Docs Reference
Works with
This page says what Geometer can check, framework by framework, protocol by protocol. It does not say "any agent", because that is not true.
On this page
How to read this page#
Each row has one of three statuses.
- Supported: Geometer found, ran and checked an agent of this kind, and the limits are listed in the row.
- Partial: some of it works. The row says what is missing.
- Not yet: nothing is built for it. If your agent runs in its own Docker image and its tools are web requests, Geometer can still check it at those boundaries. See the last section.
Supported does not mean every agent of that kind works. It means a real agent of that kind was checked from start to finish. Tell us when one does not, and this page changes.
Agent frameworks#
| Framework | Status | What it means |
|---|---|---|
| OpenAI Agents SDK (Python) | Supported | Geometer builds the image, runs the agent and watches each tool call through the framework's own hooks, so a call can be held or have its reply lost. Tested with one SDK version. Other versions run, and the report says they were not tested. Effects of tools that only change local state stay "unconfirmed". |
| Mastra (TypeScript) | Supported | The same, for Mastra agents and their tools, tested with one version. Tools built for each request, provider tools and workflow steps are listed as not observed. |
| Temporal (Python) | Supported | Geometer starts a Temporal development server next to your agent, runs a worker for your workflows and activities, and runs the entry workflow once. A workflow is the agent and an activity is a tool, so every attempt of an activity is seen, held, failed before it runs, or run with its reply lost (Temporal then retries it). Tested on one public refund agent (temporal-ai-agent-memory-state-stripe): the refund activity was reached and an at-most-once rule over its ledger held in every trial. Limits: a workflow that waits for a signal is driven with one signal Geometer chooses and labels; only top-level activity functions are tools (activities that are methods of a class built with arguments are not); a worker killed mid-activity is not tried; the Temporal server's own history is not reset between faults; a project that ships its own Dockerfile and Compose file for the worker is run as it is, with no hook in the worker; child workflows and local activities are not seen. |
| Tau2-bench style agents | Supported, with limits | Geometer finds tool kits written the tau-bench way (@is_tool methods) and the harness that runs an agent and a simulated user against them, and runs the harness once per domain. Tested on tau2-bench (airline domain, seven write tools): every tool call is seen and can be held or have its reply lost, and the tool kit's own database is read after each call. Limits: what a tool changed is known only from the kit's own state inside the process, so a repeat is reported as "held (inferred)" and never as confirmed; some write tools get no valid arguments from the stand-in model, so their repeat stays undecided; the model and the simulated user are the stand-in, so a run shows how the tools behave under faults, not whether a model solves the task. |
| Vercel AI SDK (TypeScript) | Supported | Geometer starts your project's own command with a hook in front of the ai package, so every function tool handed to generateText, streamText or an agent class is seen, held and faulted like a Mastra tool. Tested with ai 5 on a small refund agent written for the purpose (its refund tool calls Stripe, answered by the simulated Stripe); the one public example tried (ai-sdk-agents' contexts, built on ai 4) does not start under Node 22 on its own, so it was not checked. Limits: the project's own prompt is used; tools without an execute function (provider tools, tools the application answers) and tools a prepareStep callback adds are listed as not observed; a tool that streams partial results is run to its last result; the loader hook needs Node 18.19 or newer. A web server (Next.js, Express) is not run as a task. |
| OpenAI Agents SDK (TypeScript) | Supported | The same hook for @openai/agents: every function tool made with tool({ execute }) is seen, held and faulted. Tested with 0.21.0 on a small refund agent written for the purpose (the model is the stand-in; the refund tool calls the simulated Stripe). Limits: your own prompt and start command are used; hosted tools and tools the application answers are listed as not observed; what a tool changes inside the process is not confirmed. |
| LangChain.js and LangGraph.js | Supported | The same hook for tools made with tool(fn, { name }), DynamicTool and DynamicStructuredTool, so a LangGraph agent (createReactAgent) and a plain LangChain chain are checked at the tool. Tested with @langchain/core 1.3.1 and @langchain/langgraph 1.5.3 on a small refund agent written for the purpose. Limits: a class that extends StructuredTool is not seen; a graph served by langgraphjs dev is not run; the same prompt, start-command and effect limits as the other TypeScript frameworks. |
| Google ADK (TypeScript) | Supported | The same hook for new FunctionTool({ execute }) under LlmAgent. The agent's Gemini calls reach the stand-in through the SDK's base-address setting. Tested with @google/adk 2.3.0 on a small refund agent written for the purpose. Limits: the same as the other TypeScript frameworks; agent servers (adk web) are not run. |
| Claude Agent SDK (TypeScript) | Supported, with limits | The same hook for the custom tools you define with tool(name, description, schema, handler) and serve from createSdkMcpServer: each call is seen, held and faulted. Tested with 0.3.312 on a small refund agent written for the purpose, with the built-in tools switched off (tools: []). Limits: the built-in tools of the Claude Code runtime (Bash, Read, ...) run in a separate process and are not seen; with the built-in tools left on, the stand-in's tool tour spent the run's turns on them and the model traffic of one run can grow past what the check can keep (512 MB), and then the check says it could not run, with that reason; the runtime needs the SDK's platform binary to be installable in the image. |
| LangGraph and LangChain (Python) | Supported | Geometer builds the image from your own pinned dependencies, runs the agent and watches each tool call through LangChain's tool classes, so a call can be held or have its reply lost. create_react_agent, create_agent and a ToolNode all run those classes, so plain LangChain agents are covered by the same hook. Tested with langgraph 1.2.18; other versions run and the report says they were not tested. A tool that raises follows your own handle_tool_errors setting. LangGraph JS is a different framework and is not covered. An agent built inside a function is not found. |
| Pydantic AI | Supported | The same, through Pydantic AI's own wrap_tool_execute hook, for function tools and toolsets. Tested with version 2.63.0, with openai:, anthropic: and google: models. Versions before that hook existed run and say their tools are not observed. An exception from a tool ends the run unless your code handles it, as it would in production. Output tools are not tools Geometer holds. |
| CrewAI | Supported | The same, for BaseTool classes and @tool functions, including a crew written with @CrewBase. Tested with version 1.15.32. CrewAI retries a failing tool itself and Geometer keeps that. Tools from crewai_tools are subclasses of the same BaseTool, so they go through the same hook, but no run reached one. A task's text goes to the crew's input named task, request, query, prompt, input, question, message, instruction or topic (the first of these that the crew's placeholders or your inputs have), else to the only placeholder the crew has; with no such name the crew gets your own inputs and the run notes that the task text was not passed. Crew flows and several crews in one project run through the first crew only. |
| Strands Agents | Supported | The same, through Strands' own tool hooks. Strands uses Amazon Bedrock by default, so Geometer answers the Bedrock Runtime Converse and ConverseStream APIs with its stand-in. Tested with version 1.62.0. Only the default Bedrock model was run; Strands' OpenAI, Anthropic and LiteLLM providers read the same base address settings as the other frameworks but were not run. Bidirectional streaming agents and multi-agent swarms run through the entry agent only. |
| Google ADK (Python) | Supported | The same, through ADK's own plugin callbacks, for function tools. Tested with version 2.13.0. The model is answered as Gemini. Agents that ADK builds from YAML files, live (audio and video) agents and tools run by Google's servers are not covered. |
| LlamaIndex agents | Supported | The same, for FunctionAgent, ReActAgent and AgentWorkflow, through the tool classes' call and acall. Tested with llama-index-core 0.14.25. Query engine tools are watched the same way. An agent built inside a function is not found. |
| Hugging Face smolagents | Supported | The same, for ToolCallingAgent and CodeAgent tools, through Tool.__call__, which also runs when the code an agent writes calls a tool. Tested with version 1.26.0. A ToolCallingAgent runs its tool calls one after the other while checked. OpenAIServerModel and LiteLLMModel read the usual base address. Hugging Face's own inference models are not pointed at the stand-in by Geometer. |
| Agno | Supported | The same, for functions and toolkits an agent lists, through FunctionCall.execute and aexecute. Tested with version 3.1.11. A failed call is Agno's own failed result, as it would be for a tool that raised. Teams and workflows run through their entry agent only. |
| Microsoft Agent Framework | Supported | The same, for FunctionTool and @tool functions, through FunctionTool.invoke. Tested with version 1.23.0. The stand-in follows the framework's default OpenAI Responses calls, which keep the conversation on the server. Foundry, Copilot Studio and hosted agents are not covered, and neither are Semantic Kernel plugins. |
| Microsoft AutoGen | Supported | The same, for AssistantAgent tools, through run_json. Tested with autogen-agentchat 0.7.5. The agent only reaches a second tool if the project allows more than one round of tool use (max_tool_iterations), as in a real run. AG2 (ConversableAgent) has its own tool path and is not covered. |
| Haystack agents | Supported | The same, for Tool objects and @tool functions an Agent or a ToolInvoker runs, through Tool.invoke. Tested with haystack-ai 3.4.1. Pipelines made of components that are not tools are not tools to Geometer. |
| Claude Agent SDK (Python) | Supported | The same, for the custom tools of the SDK's in-process servers (@tool and create_sdk_mcp_server), through their handlers. Tested with version 0.2.174. The SDK starts Claude Code, so Geometer turns Claude Code's own tools (files, shell, sub-agents) and its settings files off for the check, unless your options choose a set. Those tools run outside your process and are not observed. The program's model calls are answered as Anthropic Messages. |
| Frameworks in other languages: Java (Spring AI), .NET (Semantic Kernel), Go, Google ADK for Java and Go | Not yet | The in-process hooks exist for Python and Node only, so function tools in these agents are not seen. An agent that runs in its own Docker image is still checked at its model, web and MCP boundaries: its model calls are answered, and the requests its tools send over HTTP or MCP are seen, held and faulted. |
Tool protocols#
| Protocol | Status | What it means |
|---|---|---|
| Web requests (HTTP) | Supported | Your agent's calls are answered by simulated tools. Geometer can hold a call, lose a reply or a request, and keep a record of every effect. Services whose address is fixed inside the client are reached through a certificate made on your own computer. HTTP/2 and mail protocols are not simulated. Stripe (api.stripe.com) and Adyen's Checkout API (checkout-test.adyen.com) are answered by the payments pack: a simulated Stripe and a simulated Adyen, each modelled on its public docs, with charges or payments, refunds, the provider's idempotency rules and (Adyen) refunds that settle later. The commerce pack answers Shopify's Admin REST API (*.myshopify.com), Zendesk (*.zendesk.com) and Intercom (api.intercom.io), and the CRM and ERP pack Salesforce (*.my.salesforce.com), HubSpot (api.hubapi.com) and NetSuite (*.suitetalk.api.netsuite.com): each a simulation modelled on its public docs, with the provider's own idempotency behaviour (often none) and, for Salesforce, the conditional update header If-Unmodified-Since. |
| MCP over HTTP | Supported | The simulated tools answer tools/list and tools/call. Servers that need credentials or live outside the simulated tools are not started. Stripe's hosted server (mcp.stripe.com) is answered by a simulated Stripe (modelled on Stripe's public docs) that lists the hosted tools and keeps refunds in the same records as the REST routes. Adyen's tools are a small catalogue written from its public docs, on a stand-in address (mcp.adyen.com); that is how the simulation names it, not a statement that Adyen offers such a server. Intercom (mcp.intercom.com), HubSpot (mcp.hubspot.com) and Salesforce (api.salesforce.com) host MCP servers; the simulated ones list tool names written for the simulation, not copied from those servers. No official MCP server was found for Shopify's Admin API, Zendesk (a client of MCP servers) or NetSuite, so those are reached over their REST routes only. |
| MCP over standard input (stdio) | Supported | A server your project holds runs inside the sandbox behind Geometer's bridge, so each tool call is seen, held and faulted like an HTTP call. This works for the images Geometer builds for OpenAI Agents SDK and Mastra projects, for configuration kept in a JSON file, and for an image you wrote yourself: Geometer builds a thin extra image over yours (pinned to the image it just built, removed after the check) that holds the bridge and a rewritten copy of the MCP configuration, and your files are not changed. Tested on one small own-Dockerfile Python agent whose stdio server refunds through a simulated Stripe: the refund was held, replies lost on the server's call and on the MCP call were enacted, an at-most-once rule over the refund held for the careful agent in every trial, and the same rule caught the agent that refunds again when no reply came. Limits: the image needs python3 or node (an image with neither keeps its server listed but not observed, and the run says so); the MCP configuration must be a JSON file inside the build context, found in the image by name and identical content; a server declared in source code (a Python dictionary or keyword arguments, a TypeScript object) is listed and run once for its tool list, and its calls are not observed; what the server does to files or databases is seen as a call, not as an effect. |
MCP started with npx or uvx | Partial | The package is installed once while the image is built, its version and hash are recorded, and it then runs without network. Covers registry packages, and git or https URL packages (installed from the source at build time, the tarball hash or git commit recorded, then run directly offline), in the images Geometer builds. A server that needs a credential and calls a host a simulated provider answers (Stripe, HubSpot) is given a placeholder credential, labelled as one in the plan and the report. Tested on @stripe/mcp 0.3.3 with a placeholder key against the simulated Stripe, and on a registry tarball URL of server-memory: both started offline, their tool calls were seen and a reply lost on them was enacted. Limits: no at-most-once verdict was reached on either (the Stripe refund attempt committed no effect in the simulation, and the memory server's effect is not visible to Geometer); a lost reply makes the MCP client wait longer than Geometer's hold limit, so that trial is reported as stalled; the git (not URL) path was not run live; Docker commands (docker run) are refused with the reason, and so is a server that needs a credential for a host Geometer does not simulate. |
| Postgres | Supported | Queries are decoded and the database is reset between runs. |
Model APIs#
Your agent's model is answered by a stand-in, so a check shows how your application behaves, not how a real model behaves. Nothing is sent to a model provider.
| API | Status | What it means |
|---|---|---|
| OpenAI chat completions and Responses | Supported | Whole answers and streams, tool calls, and answers in the shape the request asks for. A client that keeps its conversation on the server and sends only the new tool outputs with previous_response_id is followed. |
| Anthropic Messages | Supported | The stand-in answers in Claude's own format, whole or streamed, with tool_use paired to tool_result. A Mastra project whose agents use anthropic/ models ran to the end against it. The report says no Claude model was asked. A client that ignores the base address setting is reached only through the certificate path, and only if it speaks HTTP/1.1. |
| Amazon Bedrock Converse | Supported | Converse and ConverseStream are answered, whole or as Amazon's binary event stream, with tool use paired to tool results. A Strands agent on its default Bedrock model ran to the end against it, using a real boto3 client that checks the stream's checksums. The stand-in accepts any request signature and the dummy keys Geometer gives the agent. InvokeModel, the agent and knowledge base runtimes and speech streaming are not answered. |
Google Gemini generateContent | Partial | Answered whole or streamed, with function calls. The @google/genai (JavaScript) and google-genai (Python) clients read GOOGLE_GEMINI_BASE_URL, and the generated images set it, so a client that reads it is pointed at the stand-in; Vertex AI paths are answered too when the client reads GOOGLE_VERTEX_BASE_URL, but the SDK still asks for Google credentials unless the project passes an API key. A Google ADK agent (which uses @google/genai) ran to the end against the stand-in. A plain Python google-genai agent in an image you wrote gets the same settings (an agent in your own image is pointed at the stand-in through the SDK variables, not only through a setting your code names); it has not yet been run end to end here. The Live API (websockets) is not answered. |
Check a model change and Check constraint stability can call a real model you choose. Only OpenAI-compatible chat is offered for that today.
An agent with its own Docker image#
An agent that ships its own Dockerfile or Compose file is run as it is. Geometer sees its model calls and its web and MCP requests, whatever framework wrote it. It does not see tools that are plain functions inside the agent's process, because the hook that sees them is installed only in images Geometer builds.
See also#
- Alpha limitsWhat the alpha does not do yet.
- Check your agentThe steps for a first check.
Something unclear, wrong or missing on this page? Report a problem with these docs.