Check your agent

Before you check#

1. Make sure Docker is running. If not, the Docker card in the workspace shows the steps. From a terminal, geo doctor reports what is wrong.
2. Open the project. See Quickstart.
3. Read See what Geometer found on the project page. It lists agents, tasks, outside services, tests and the rules that apply. If it lists no agent task, or cannot start your app, it says what was missing under What Geometer could not do here. A check can still run your own tests, but it then covers only what those tests already cover.
4. Look at the Rules tab. Suggested rules count only once you choose Use this rule.

To see the same information without a window, run geo inspect for a read-only report of modules, toolchains, targets, ambiguities and gaps, or geo onboard for what Geometer works out about a project by itself and the one question it still has. Both read files and start nothing. Both name agents written with a framework (for example "OpenAI Agents SDK: Triage Agent and 7 others, in python-backend"), and so do geo env plan --agents, the workspace's "See what Geometer found" and geo check: they all use the same detection. A Python or JavaScript backend in a subfolder (python-backend/, server/) is a part of the project of its own, unless the folder is tests, docs or examples. When a folder has lock files for two package managers, Geometer picks one by itself and says why: the packageManager field, then the folder's own scripts, then the README's install line, then a lock file that is clearly newer. It asks only when none of these decides, and then suggests one. Change a choice with geo decide.

Run the check#

Press Check this project on the first-check page, or Check again on the board. Under What a check does you can choose the time limit for agent runs. From a terminal, geo ci runs the same automatic plan without a window, and geo explore runs only the agent exploration, with a budget in seconds (600 by default).

geo explore --budget 600

A check runs up to three kinds of work, and the check page marks each as run, not run or could not run:

• Explore. Your agent's tasks run in a container, first normally, then again with something going wrong on purpose. The first run builds your agent's containers, which can take a few minutes. After that each run takes seconds.
• Tests. Your project's own tests run in a container with a copy of your project. Most of the time goes to downloading the toolchain image the first time and installing dependencies.
• Workloads. Your app starts in a container and receives sequences of requests built from its API, each from a clean database.

Each line of the check page is one thing Geometer did, in plain words and timed from the start: preparing a container, starting a service, a run, a problem found. Press Stop check to end a check early; its containers are removed. A check keeps running if you leave the page or close the window, and a check interrupted by a crash is marked as stopped before completion, with its containers removed.

Break it: try one fault#

On the board, each row is a tool action and each column a fault. Press a quiet Try it cell (several, if you like): Geometer tries only that fault at that tool, within the usual limits, and answers in one word in the cell, Held, Broke or Couldn't tell. It covers that fault at that tool only and says nothing about other faults. From the command line: geo check --fault lost_reply --at payments.refund (a tool with a single action that changes something can be named alone).

Services you recorded#

If your agent calls an internal REST service or an MCP server that Geometer has no description of, record a session once and let the check use a copy learned from it:

geo mcp record --http http://10.0.0.5:8080
geo mcp twin 10.0.0.5-8080 --name shop
geo check --pack twin:shop

The check says which tool a twin answered (orders was answered by a twin learned from 5 recorded calls on 2026-10-29). A twin learned from a server of this project is used without the flag where nothing else answers that server. A call the recording never showed is answered not simulated: not in the recording and the check says couldn't tell for it; it is never counted as a pass. The twin's data stays in your Geometer home. See Put Geometer in front of your MCP servers.

Read the result#

When the check ends you see one of three results:

• Problem found: a rule you rely on broke. Each problem shows what happened, step by step, and can be replayed exactly.
• Possible problem: a suggested rule broke. It counts only once you agree the rule is right, on the Rules page.
• No problems found within these limits: nothing broke in the runs that were tried. It is not a promise that nothing can.

Below the result are What was found, What was not checked (including a note if the time limit was reached, so some situations were not tried) and How to repeat it. If only your own tests ran, the page says so: a clean result then means none of your tests failed, not that nothing can go wrong. Below the steps is the raw log, with the exact commands.

The board is updated by each check and by each cell you try. Quiet Try it cells are faults that were not tried. See Checks, rules and problems.

Agents built with Mastra#

When your project is a Mastra app, Geometer finds its agents and tools from your source and runs the app the way Mastra's own README tells you to try it, in dev mode. Concretely, Geometer starts the project's own server code inside the container, sends your agent a message through Mastra's HTTP interface, and the project's own login rules treat that as the local Studio user, as they do when you run mastra dev. The check page says so: "run through Mastra's server app in dev mode, as the local Studio identity". Your tools then run with the authority your own server gives them, so a lookup that needs only the case the server sets up returns data from your project's seeded local store instead of being refused. A lookup that needs the case's own email address or order id may still be refused, because Geometer's stand-in model does not know them.

Two things to know:

• Every tool call is seen before it runs, so a call can be held, failed, or have its reply lost on purpose. What a tool changes inside your app's own process is not seen directly, because only your app sees it. If your app keeps its data in a local SQLite file, Geometer copies that file (and its log) out of the container after each trial and reads it itself: which tables changed, how many rows were added, and which unique indexes exist. Tables of the framework's own bookkeeping (Mastra's memory, traces and workflow snapshots, migration tables) are labelled as such and never counted as your tool's effects. A tool call counts as having changed the database only when a new row has a unique key holding a value the call was given and the row names the tool's action; otherwise the effect stays unconfirmed. The same is done for JSON and JSONL files your app writes while it runs (a ledger kept as refunds.json, say): Geometer lists them, reads the ones that changed, and compares records before and after. A record updated in place (a counter that goes up on a retry) is the same record, not a second effect, and a record counts as a tool's effect only when it holds a value the call was given (at any depth of the call's arguments) and the record, its fields or its collection's name names the tool's action; a result log that merely quotes the call is not an effect. If neither a database nor a ledger is found, the report says "no app database or JSON ledger found; in-memory effects stay inferred".
• If your project has an evals folder, a case that says a tool must never be called (forbiddenTool) becomes a rule, labelled "from your evals". It is checked on every tool call seen. A clean result says what the chat was offered: a rule about a tool the chat is never given holds because of how the app is built, and the page says that instead of crediting the agent.

• If your README documents a flow with a person in the middle (sign in as the customer, ask a question, an approver approves), Geometer follows it: it signs in with the README's accounts, sends the README's question, waits until the case needs approval, and approves it itself (the question "should Geometer approve?" is answered "approve" by default, and the report says so). The routes and the shape of each request come from your own routes and their zod schemas. The business choices your drafting step needs on the way (for example "recommend a refund") are made by Geometer's stand-in, not by a model, and are listed in the report as Geometer's choices. If the case ends some other way, the report says how, for example that it was escalated and never asked for approval, and which answer caused it. When the refund runs, the at-most-once rule on issue_refund is checked against the rows in your app's database and the result is held, broke, or couldn't tell.

If your project does not ship @mastra/deployer, Geometer calls the agent directly and says so; a tool that needs the server's authority may then be refused.

Agents built with the Vercel AI SDK, OpenAI Agents SDK, LangChain.js, the Claude Agent SDK or Google ADK#

These agents are calls in your own code, so Geometer does not import them. It runs the command you would run (npm run start, else a dev script that is not a server, else the one file that runs the agent) with a hook in front of the framework's package. Your files and lock file are not changed. Every function tool you define with the framework's own constructor (tool(...), new FunctionTool(...), new DynamicStructuredTool(...)) is seen before it runs, so a call can be held, failed, or have its reply lost on purpose, and the report names the framework that saw it.

• Your own prompt is used. The question in your code is the task.
• A web server (Next.js, Express, vercel dev) is not run as a task.
• Tools without a function in your process are listed as not observed: provider tools, tools your application answers, and the Claude Agent SDK's built-in tools (Bash, Read).
• What a tool changes inside your process is not confirmed. A tool that calls a service Geometer simulates, such as Stripe, is.

Agents built on Temporal#

When your project uses Temporal for Python and a model client (openai, anthropic, litellm), and defines a workflow with a run method, Geometer treats the workflow as your agent and its activities as the agent's tools, so an activity that refunds or moves money is a tool that can be failed on purpose. It builds an image from your manifest, runs the official Temporal dev server (temporalio/temporal, server start-dev) next to it, starts a worker for your workflows and activities, and runs your main workflow once. The input is the one your own code or tests build for that workflow (the page names the file and line); fields your code leaves out are filled from their types. Every activity attempt is seen before it runs. A lost reply makes Temporal retry the activity, as it would for a real timeout, so a refund that must not happen twice shows up when it does. Killing the worker mid-activity and starting a new one is not tried yet. A project with a Dockerfile or Compose file are run as they say, not this way.

Some workflows stop to wait for a person (a customer's answer, an approval). When one stands still with nothing running, Geometer sends it one of its own signals or updates, as the person, so the steps after the wait are reached. The arguments come from what the workflow's own queries show (a pending question, a suggested answer), else from the handler's types; a handler that says no or stops (reject, cancel) is never sent. Every signal sent is listed in the report as Geometer's choice, not a person's, and not a model's. Geometer's stand-in model also takes the tools one per turn for a loop that sends a new request each turn, so it can get past the first tool to a decision.

Agents built with a Python agent framework#

When your project's manifest lists LangGraph, Pydantic AI, CrewAI, Strands Agents, Google ADK, LlamaIndex, smolagents, Agno, Microsoft Agent Framework, AutoGen, the Claude Agent SDK or Haystack (or your code imports one of them), and your code builds an agent at the top of a module, Geometer treats that agent as the one to check and the functions or classes it calls as its tools. The check page names the files it read and how sure it is: certain when your manifest names the framework, likely when only your code does.

It builds an image from your own dependencies, at the versions your lock file (uv.lock or poetry.lock) or requirements.txt pins. With no pins it uses the ranges your manifest allows and the page says so. A small driver in the image imports your module, installs the hook for your framework, and runs the agent once on a prompt from your README or the tool tour. Your files are not changed. Every tool call is seen before it runs, so it can be held, failed before it runs, or have its reply lost on purpose, and the framework's own error handling decides what your agent does next. A CrewAI project written with @CrewBase is run through its crew() method with the inputs your main.py writes out, or an example value for each placeholder. When a task has text, Geometer puts it in the crew's task, request, query, prompt, input, question, message, instruction or topic input (the first of these your crew's placeholders or your inputs have, else the only placeholder); with no such name the task text is not passed and the run says so.

The model is answered by the stand-in, through the framework's own settings: the OpenAI base address, and for Strands the Amazon Bedrock Runtime address (with dummy keys; the stand-in accepts any signature). Nothing is sent to a provider.

A task that sends a call above an approval limit can include the approval's id in its input ("approval":"ap-1", the field the rule names). Before each run Geometer's test approver issues that approval for that exact call, as a person with the right to approve would, and the report names it as Geometer's test approver. Run such a task on its own, without a lost reply: a retry of an approved call reuses an approval that is already spent, which the rule rightly refuses. The practice projects use this for their large refund.

The stand-in picks numbers that reach your rules instead of always writing 1. If the task text names a number next to an argument (count 250, amount of $5,000) it uses that. Otherwise, for an argument your rules bound, it takes one of the sizes around the bound, chosen by the task's id: for a limit that needs approval, the limit itself and the next unit above it; for an argument_bound, its lowest and highest allowed value and the unit just outside each. A number the argument's own schema forbids is skipped. A task always gets the same number, in every trial and every replay, because a search replays the start of earlier trials and the agent has to answer the same way each time; different tasks take different numbers, so a project with several tasks usually visits both sides of a limit, and a project with one task sees one side. The report labels each number as chosen and says why. Arguments no rule bounds stay at 1 (or the schema's lowest allowed value). Only the sizes your rules name are tried; the stand-in does not search for other numbers.

Benchmarks with tool kits (tau-bench style)#

When your project defines tool kits (class X(ToolKitBase) with @is_tool(ToolType.WRITE) methods) and a command that runs an agent and a simulated user against them (--domain, --agent-llm), Geometer runs that command for one domain at a time. The tools you declared as WRITE are the ones that change something. The agent and the simulated user both talk to Geometer's stand-in model, so a run shows how your tools behave when they fail, not how good any model is at your tasks. What a tool call changed in the kit's in-memory database is read by Geometer from inside the process, record by record, and reported as inferred evidence of the effect (never as a confirmed one), so a write that changed it can be told from a call that did not. Geometer's stand-in model reuses identifiers before it invents any: from the results of tools it called earlier, from the conversation, and from record keys in your data files, and it calls tools that only read before tools that change something. Each identifier is labelled in the model reply as reused (and from where) or invented. A write the kit still refuses (an unknown reservation, say) changes nothing and the repeat cannot be judged; the report then says "couldn't tell" for that tool.

Agents built inside functions#

Some projects build their agent inside a function (a main() of an example, a script) rather than at the top of a module, so there is nothing to import. When Geometer finds no agent it can import but finds a script that builds an Agent(...), runs it with Runner, and has a __main__ guard, it runs that script as the task, as python script.py would, with every function tool it builds seen before it runs. The script uses its own prompt, not the task's, and the report says so. Scripts that read the terminal or serve a web page are not used, and neither is an Agent(...) that only appears in a docstring or a comment. If the script's tools come from a service the sandbox cannot reach (a hosted tool server), the agent cannot start and the check says "couldn't run" with that reason.

An agent in your own image that starts an MCP server#

If your own Dockerfile image holds a stdio MCP server that your project's MCP configuration (a JSON file in the build context) starts, Geometer builds a thin extra image over yours for the check. It adds Geometer's bridge and a rewritten copy of that configuration, so each tool call the server gets is seen, held and tried with a lost reply. Your files are not changed and the extra image is removed afterwards. The image needs python3 or node; without either, the server still runs but its calls are not observed, and the run says so.

A Compose file that needs a `.env` you do not keep#

If your Compose file names env_file: .env and the project has only .env.example, Geometer reads the Compose model with the example's keys and simulated values instead of stopping: the example's plain settings as written, every key, token or password as a made-up value of the same shape (sk_test_geometer...). Nothing comes from your computer and no real secret is used. The plan lists this as a note ("simulated settings"). Your own .env is not written.

Exit codes of geo ci#

Suggested rules that broke are reported as candidates and never fail the run.

A check in which no run reached a tool call tried nothing, so it never says "No problems found". If your agent exits when it starts, the result is Couldn't run: your agent failed at start, with the agent's own error in one line and the next step: run your agent's start command yourself. If it runs but never calls a tool, the result is Couldn't tell: the agent made no tool calls. Both make geo check and geo ci exit 3, and no certificate is offered from that check.

Your project's own test commands are reported honestly. A command that fails within a minute, writes no per-test report and prints that something is missing from the environment (a browser such as Playwright's, a program, a module, a system library, a network service) is reported as Couldn't run: <reason> (the project's own test command <name>). It is not counted as a problem, and it makes the run exit 3. The full output stays in the check's log. A test that runs and fails with a per-test report is still a problem. Test files that cannot load because the sandbox lacks an optional module (No module named 'scipy': a voice, GPU or other extra your project does not install by default) are also Couldn't run, one line per test command that lists the modules, not one problem per file; a module that is your own project's, or an import that fails for another reason (cannot import name), stays a problem.

Check a model change#

If you plan to switch the AI model your agent uses, Check a model change on the project page compares the current model with a new one on the same tasks. It runs two separate experiments. First, the recorded model's answers are replayed against your application, which is an application regression check and says nothing about the new model. Second, the new model is asked fresh about your agent's own requests within a budget of calls, tokens and time; where its answer changes the agent's tool calls, that branch runs in the isolated tool world under the same rules. The cost estimate is shown before anything starts. For each task the result is one of: same, changed with rules satisfied, changed with a rule violated, the current model itself varies, or not run or unknown. Nothing here says an upgrade passed.

From a terminal:

geo model-upgrade --estimate --to ollama:qwen3:8b
geo model-upgrade --to ollama:qwen3:8b --samples 3

It exits with 1 when a rule was violated, 2 when any sample is unknown or not run, and 0 otherwise. Exit 0 means nothing was found in the samples that ran, not that the change passed.

Check constraint stability#

Check constraint stability asks whether your agent keeps the rules you approved when conditions change. It runs your agent's tasks with a few changes, each beside an unchanged run with the same seed:

• a longer conversation, with earlier messages put in front of your agent's request;
• a tool reply that asks your agent to skip an approval (the text is added to what the model receives; your tools and your agent's code see the normal reply);
• a harmless change in the wording of your agent's instructions, where doing something different is not breaking a rule;
• another model, if you choose one.

Shortening a long conversation and handing a task to another agent run only if your agent has such a step Geometer can start. Otherwise they are listed as not run, with the reason. Geometer does not imitate them.

Before it starts, Geometer shows the rules your agent is held to, each change and whether it can run, and the most model calls, tokens and money the check can use. Nothing starts until you press Start. geo stability --plan prints the plan and stops. A broken rule shows the call your agent made, whether it took effect (an attempt that was refused is different from a refund that happened), and what came before it. Explanations are marked as not established, because something that happened just before is not necessarily the cause. Run it again replays the model's recorded answers from a clean start and can become a kept check.

A result of "no rule broken" is a sample, not a promise. "Could not be decided" means something needed was missing, for example the approvals could not be read or the budget ran out, and it is never counted as a pass. A scripted model is a stand-in that shows how your application behaves, not how a real model behaves.

Next#

• Fix a problem and keep the checkWhat to do with a problem.
• Run kept checks in CIMake the check run on every change.

If your agent talks to Stripe (api.stripe.com, or the hosted tool server mcp.stripe.com) or to Adyen's Checkout API (checkout-test.adyen.com), Geometer answers with the payments pack: a simulated Stripe, modelled on Stripe's public docs, and a simulated Adyen, modelled on Adyen's public docs (neither is the real provider). Each keeps charges or payments and refunds, with the provider's own error shapes and idempotency-key rules, and the hosted tools (for example create_refund and list_payment_intents) work on the same records as the REST routes: one ledger per provider. Adyen accepts a refund at once (received) and settles it later, by a webhook event. Three seeded payments are there to refund, and the pack has named starting points (a partly refunded charge, a disputed charge, a payment in two currencies). The report and the plan say "simulated" wherever the provider appears. A refund made through a hosted tool carries no idempotency key, so a repeat after a lost reply is a second refund; that is a result about your agent, not about the provider. When a check finds a problem, the problem page shows what changed in the simulated provider during the failing run, read from its own records (for example "2 refunds of 80.00 USD on charge ch_sim0001 (80.00 USD)").

Two more packs work the same way. The commerce pack answers a shop on Shopify's Admin REST API (*.myshopify.com), Zendesk (*.zendesk.com) and Intercom (api.intercom.io, or its tool server mcp.intercom.com); the CRM and ERP pack answers Salesforce (*.my.salesforce.com), HubSpot (api.hubapi.com, or mcp.hubspot.com) and NetSuite (*.suitetalk.api.netsuite.com). Each provider is a simulation, modelled on its public docs (none is the real provider), and the report says so. They keep the records a support or back-office agent changes: orders, refunds (with Shopify's calculate step), fulfillments and gift cards as store credit; tickets and their public replies; conversations, replies and closing; accounts, opportunities and cases; deals and notes; invoices and credit memos. Each follows the provider's own idempotency behaviour, which is often none: Shopify, Intercom, HubSpot and NetSuite (a synchronous call) have no idempotency key, so a request sent again after a lost reply is made twice, while Zendesk takes one for creating a ticket. Salesforce answers 412 to an update that sends If-Unmodified-Since when the record has changed since. A project that uses both Shopify and Stripe gets both packs, so one check can judge store credit plus a card refund for one order as double compensation. Named starting points: an order whose shipping address already changed, a partly refunded order, store credit plus a card refund, an opportunity that has moved on, a credit memo already issued.

If a service of your Compose file needs the network to start (a front end that runs apk update or npm install) and nothing depends on it, Geometer leaves it out and says so on one line: not started: <service>: it needs the network at start ....