Checks, rules and problems

How the pieces fit#

A project holds one or more agents. Each agent has tasks and calls tools. Some operations of those tools change something in the world; those are actions. Rules say what must always be true about the actions. A check runs the tasks while injecting faults into the tool calls and tests the rules. When a rule breaks, you get a problem. You replay the problem, change your code, check the fix and keep the check so your CI re-runs it.

Project#

A project is a folder on your computer, such as the root of a repository, that holds your agent and the things around it. Geometer opens it read-only, works out what it contains and keeps everything it learns in its own records folder, not in your project. Opening a project does not run, install or change anything. See Where your records live.

The project page shows what Geometer found: agents and their tasks, outside services, the project's own tests and the rules that apply. Geometer reads files such as package.json, pyproject.toml, go.mod, compose.yaml and Dockerfiles. If it cannot work something out, it asks at most one grouped question with a suggested answer. Your answers are remembered for the project.

If you want Geometer to use hand-written declarations instead of inferring them, you can declare tools, tasks, rules, workloads and conversations in a .geometer/ folder, or keep them in a folder outside the project and pass it with --declarations. Declared files win over inferred ones.

Agent and task#

An agent is a program that uses an AI model to decide what to do and then calls tools: for example, a support agent that issues refunds. A task is one thing Geometer asks the agent to do, such as answering a customer who wants a refund. Geometer takes tasks from your declarations or, when you have none, from what it can read in the project, such as examples in your README and your evaluation cases. It also runs the same request twice to see what a repeated request does.

During a check, Geometer answers your agent's model calls with a stand-in, so a check does not need a model and repeats the same answers. The stand-in shows how your application and the check behave, not how a real model behaves. To try a real model, see Choose a model.

Tool and action#

A tool is something your agent calls to do work: a payment service, an email service, a database or a tool server. Each tool has operations, such as payments.refund. An operation that changes something is an action, and actions are what you grant authority for. The overview lists one row per action.

During a check, outside services are simulated. Nothing real is charged, sent or changed. Geometer records what the agent asked for and what each simulated tool did, and it keeps the tool's own record of effects apart from the agent's account of what happened.

Rules#

A rule is something that must always be true, for example "a refund is paid at most once per request". Geometer tests rules during every run.

Rules come from two places. Rules you declare are yours. Suggested rules are read from your code: your database schema, your API description, the way your agent calls outside services and your tests. A model can suggest more, if you choose one; see Choose a model.

On the Rules page, you decide what to do with each suggestion:

• Use this rule. It is checked from now on, and a run that breaks it is shown as a confirmed problem. It fails your CI only if you also choose Also check it in CI. Until then its problems stay suggestions in CI.
• Not relevant. The rule is set aside and not suggested again.

You can decide a whole group at once, and each rule is still recorded separately. Rules your database already enforces are kept apart in a collapsed section, because they rarely find anything new.

Keep a number inside a range#

A rule can keep a number that your agent sends to a tool inside a range, for example "a shop price is between 1.00 and 1000.00" or "a spawn is at most 500 units". Declare it in .geometer/rules.toml:

[[rule]]
id = "price-range"
family = "argument_bound"
tool = "shop"
operation = "set_price"
field = "price"
min = "1.00"
max = "1000.00"

field is the name of the call's argument; for a number inside another object, join the names with a dot, such as amount.value. Give min, max or both. Both are allowed values themselves, so a price of exactly 1000.00 passes. Write money as text such as "80.00" or as a whole number: Geometer compares them exactly, with no rounding. The agent may send the number as text or as a number.

Geometer checks every call the agent makes to that operation. A call outside the range is a problem, and Guard refuses it before it is sent, saying which side of the range it broke. A call with no such argument, or with a value that is not a number, is never counted as fine: the result is "cannot tell", and Guard refuses the call unless you chose otherwise. If your tool's own description already gives a minimum or maximum, Geometer suggests this rule from it, and it is checked only after you choose Use this rule.

Checks#

A check is one run of the whole routine against a project. The Check this project button starts one. There are three ways Geometer can check a project, and the project page says for each whether it will run here and why not when it will not:

• Test the AI agent with things going wrong. Geometer runs your agent's tasks and makes things fail on purpose. This finds mistakes like a customer charged twice or an email sent five times.
• Send requests to the app's API. Geometer starts your app and sends it sequences of requests, such as create, read back, change and delete, each time from a clean database. This finds requests that crash the app or leave data in a wrong state.
• Run the project's own tests. Geometer runs the tests you already have inside a container. This shows whether they pass in a clean environment, and it only finds what your tests already cover.

Two further checks start from the project page and use a model: Check a model change and Check constraint stability. See Choose a model and Check your agent.

Every check you start keeps its record (open it from Under the hood), including ones you stopped or that were cut short. A check keeps running in the background if you leave the page.

Situations and fault kinds#

A fault is a kind of failure at one tool action. The board has one row per action and one column per kind of fault Geometer can create:

Each cell of the map says Tried, the rule held, A rule broke or Not tried yet. A tick means the situation was tried within the check's budget and the rule held. An exclamation mark means a rule broke there. Untried ground is drawn as fog, so what was not checked reads as territory to explore. The map is a coarse summary of what was stored and never a percentage. A list called Beyond the map names what Geometer does not model, and the map says nothing about those. Duplicate delivery of a request, for example, is not a column.

Pick a fogged cell and press Try this situation: Geometer explores that kind of fault at that tool and says that is what it did. A crash is offered at most once per run and only where the agent is waiting at a call. A partial effect applies only to a call that declares its parts.

Problems#

A problem is a run in which a rule broke, written as what happened to a customer or record. Problems come in two kinds:

• Confirmed: a rule you rely on broke. Fix these first.
• Possible: a suggested rule broke. Decide on the Rules page whether the rule is right.

The problem page reads consequence first. It shows the rule with what was expected and observed, each step your agent or app took, the effects recorded by the simulated tools and which of them the tool itself confirmed. It also lists what the run did not cover, so you know the limits of what it shows. Under What the broken runs had in common, Geometer compares the runs where the rule broke with the runs where it held, for example that every broken run had a lost reply on the refund. With fewer than three runs on either side it says it cannot tell which differences matter. It lists what it did not compare, such as the values inside calls and the timing.

Replay#

Replay repeats a problem from a clean start, with the same answers as the first time. Because the model's answers come from a recorded tape, a replay checks your application, not the model. A problem you can repeat exactly is a problem you can fix and check again.

Replay comes in forms, and Geometer says which one it used:

• Exact. The recorded run, including the faults, is repeated from a clean reset. This is what Replay on a problem page and geo replay <finding> do. With --clean-worker, the replay runs on a second clean worker, with a fresh records folder and fresh containers from a copy of the project at the recorded revision. It reports exact, property reproduced, diverged or incompatible.
• Exact, one change. In the grid of faults tried you can replay a recorded problem with one fault changed at one call. Because a changed call changes what the agent asks its model, this form uses the project's stand-in model rather than the recorded tape, and says so.
• Bounded. A search limited to one kind of fault at one tool, under the usual budgets. It is an ordinary exploration, so it can find new problems.
• Check my fix. The recorded run on your code as it is now, plus a bounded set of nearby cases. See the next section.

A problem whose run cannot be repeated is detected before anything starts, and the replay buttons are disabled with the reason in words.

Checking a fix#

Check my fix (geo verify <finding>) replays the problem on your changed code, confirms that the run really reached the point where the rule broke, and tries a few nearby cases. It answers Still happens, Fixed within what was checked, Could not tell or Cannot check on this version. The explanation says what was checked: the recorded run and a number of nearby cases, not every situation. Geometer never calls a problem fixed without that qualification.

Kept checks#

Keep this check records a problem as a check that can be re-run on every future version of your code. Geometer stores it in its records and writes a reviewable file under .geometer/regressions/ in your project. Commit the file with your fix. geo ci and geo regression run re-execute your kept checks on the current checkout. A kept check that reproduces the problem is a confirmed failure, one that cannot be decided is missing evidence, and one that cannot run is an infrastructure failure.

When a kept check expects a different outcome than the original run, for example a fix that refuses a duplicate request with a 409, you allow that outcome explicitly with geo regression keep --allow node=outcome. That is a decision you make, and Geometer records it.

Next#

• GuardHow Geometer watches and stops calls as they happen.
• Autonomy levelsHow an action earns more authority.
• Fix a problem and keep the checkThe step-by-step guide.