educational
Code scanning, evals and action assurance check different things.
A refund agent can pass every code scan and still refund twice. What each kind of check covers, and where they overlap.
What code scanning checks
Static application security testing reads source code without running it and looks for known patterns of unsafe code: injection, hardcoded secrets, weak cryptography, unsafe deserialization and similar weaknesses catalogued in references such as CWE. It is fast, it runs in CI and it catches whole classes of bugs before code ships. Keep it.
Evaluations check what a model says or does on selected cases. Observability records what instrumented systems saw. Each answers a real question, and none of them is replaced by action assurance.
Where the failure lives
Take a refund agent. It requests an $80 refund. The provider completes it, but the reply is lost. The agent sees no confirmation and retries as a new request. The customer receives $160. Every line of code involved can be free of known weaknesses. The defect is in behavior under a fault, across the agent, the tool, the provider and the network.
A code scanner has no way to see that a reply can be lost, that the retry created a new identity instead of reusing the original, or that the business rule of one refund per request was broken. With agents there is a further layer: the model decides at runtime which tool to call and with what arguments, so the decision is not a code path a scanner can trace.
What action assurance borrows
Action assurance runs the workflow in an isolated environment, injects faults such as lost replies and concurrent requests, and checks every execution against the team's own business rules. The idea of fault injection comes from distributed-systems testing such as Jepsen; searching for any execution that breaks a declared property comes from fuzzing and property-based testing.
It then checks connected actions before they take effect, allowing, holding or denying them as a policy engine would, and keeps a record of authority, decisions, outcomes and gaps, as financial controls do. A short way to say it: fault-injection testing, a policy check and an audit trail, built for agents that take actions.
Where they overlap
Both can run in CI as a release gate, and security tools are increasingly scanning agent code, tool definitions and prompt-injection patterns. Excessive agency, where an agent has more tools, access or autonomy than it needs, is now a recognized risk category in LLM security guidance.
The distinction is the question being asked. Security tooling focuses on attackers and vulnerabilities. Action assurance focuses on whether consequential actions follow the agreed rules, including failures with no attacker at all: a lost reply, a stale approval or a well-meaning agent with too much authority.
Using them together
A practical release check for an agent that moves money uses all three: code scanning for how the code is written, evaluations for what the model tends to do, and action assurance for what happens when the agent acts under faults and handoffs. Each covers stated cases and leaves the rest explicitly unknown.
If your existing tests already exercise lost replies, races and changed approvals against your business rules, you may not need another tool. If they do not, those are the cases to add first.
Sources & corrections
- MITRE Common Weakness Enumeration (CWE)
- Jepsen: distributed systems safety research
- OWASP Top 10 for LLM Applications: LLM06:2025 Excessive Agency
Corrections: none recorded.
Related issued certificates: none. Examples are educational.