Pre-launch: continuous agent regression testing
Did your last deploy make your agent easier to break?
Aetius reconstructs a statistical twin of your production agent from your own traces, runs it against real adversarial attacks, and answers with a credible interval and an explicit decision rule, not a flaky pass/fail that flips on every run.
Built for teams shipping agent changes weekly, not one-time red-team audits.
Illustrative output. Every verdict is backed by an explicit power calculation, not a hardcoded sample-size cutoff.
The problem
Confidence and control have decoupled.
Reported vs. verified
of executives are confident their existing policies protect against unauthorized agent actions.
of organizations actually send agents to production with full security and IT approval.
AGAT Software, 2026 survey of 900+ executives and practitioners
of organizations reported confirmed or suspected AI agent security incidents in the past year
Gravitee, State of AI Agent Security 2026 · n=919
of organizations have runtime visibility into what their agents are actually doing
Gravitee, 2026
in total costs from AI-related security incidents, reported by 43% of enterprises in the past year
WitnessAI, The Hidden Cost of Enterprise AI, July 2026
Enterprises already run agents in production. They've already had incidents. And 97% expect a material one within the next year. The gap isn't awareness. It's that testing happens once, if at all, and nothing tells a team whether last week's prompt edit, tool addition, or model upgrade quietly made their agent easier to compromise.
Arkose Labs, February 2026 · n=300 enterprise leaders
How it works
A statistical twin of the agent you actually run.
Connect your traces
Point Aetius at your Langfuse project. No execution access to production is required; we work from the trace data you're already collecting.
We reconstruct your agent
From real traces, we rebuild the agent's actual structure: which roles exist, what tools each one has, how they hand off. Every tool is classified by what it can actually do, whether that's taking a sensitive action, looking up data, or ingesting untrusted content, so only the attack families that structurally apply get run.
We test the change, with statistics
Adversarial attacks run against the reconstruction before and after your change. We calibrate our own false-positive rate first, then report whether behavior moved by an amount that matters: FLAGGED, no difference detected, or INCONCLUSIVE when there isn't enough evidence to tell. Never a guess.
Why it's different
Not a once-a-year audit. Not a generic benchmark.
What it tests against
One-time audit
A snapshot of your agent, reviewed once by hand.
Generic benchmark
A generic simulated agent, unrelated to your actual tools or prompts.
Aetius
A reconstruction of your real production agent, built from your own traces.
When it runs
One-time audit
Once, usually before launch, and rarely again.
Generic benchmark
Whenever someone remembers to kick it off.
Aetius
Every time a prompt, tool, or model changes.
"Did this change make it worse?"
One-time audit
Not designed to answer that; there's nothing to compare against.
Generic benchmark
A single score that moves around on its own between runs.
Aetius
A credible interval and an explicit decision rule, validated against a measured false-positive rate.
Access required
One-time audit
Often full access to a staging or production environment.
Generic benchmark
None, but also no relationship to what you actually ship.
Aetius
Read access to trace data you already collect. No execution access to production.
For technical teams
We give you a credible interval, not a score.
Most agent security tools report a single attack-success number and let you eyeball whether it moved. That number is noisy by construction. Here's what runs underneath every verdict Aetius reports.
Self-calibration first
Before any comparison is trusted, we measure our own false-positive rate: on identical configurations run twice, and in simulation against the trace shapes real runs actually produce.
A/A calibration
A model that matches the data's structure
Attack runs are clustered within attack cases, and cases within a family share strength through partial pooling, so sparse samples get honest uncertainty instead of wild point estimates.
hierarchical Beta-Binomial model, per-arm partial pooling
Flags only for differences that matter
A change is flagged only when the entire 95% credible interval clears a practical-equivalence band; a statistically nonzero wobble smaller than that still isn't a finding. Family-level flags are corrected so testing dozens of attack families at once doesn't inflate false alarms.
ROPE decision rule, Benjamini–Hochberg FDR control
Honest early stopping
A run can stop as soon as the evidence is decisive, or as soon as the effect is credibly too small for more spend to change the verdict. The stopping procedure's false-signal rate is measured by simulation, not assumed.
sequential credible-interval checks under the ROPE rule
Fewer runs, same power
Shared randomness and covariate adjustment across paired runs cut the number of attacks needed to detect a real regression, which lowers your testing cost directly.
common random numbers, CUPED-style adjustment
Verdicts backed by power, not hunches
Every verdict is graded against what the run could actually have detected. “No difference detected” states the smallest effect the run could reliably catch; INCONCLUSIVE means not enough evidence yet, and comes with how many more runs it would take.
power analysis, minimum detectable effect
Every method is validated by simulation against real measured trace shapes, both null runs with no true difference and deliberately injected known-bad changes, before we trust a single result: the detector has to flag real regressions and stay quiet on null ones, at a measured rate, not an assumed one. The same power model that grades a finished run also prices it before it starts, so the cost of an answer is on screen before anything is spent.
FAQ
An audit tells you where you're vulnerable today, once. Aetius tells you, every time something changes, whether that change made things measurably worse. It runs continuously, not once a year.
Most tools measure how many attacks succeed. Aetius measures whether that number changed in a way that's statistically real versus normal run-to-run noise, and it tests against a reconstruction of your actual production agent's structure rather than a generic simulated one.
No execution access is required. Aetius reconstructs your agent's structure and behavior from trace data you already have, in Langfuse for example, so there's no need to grant live production access to get started.
This is an open question we're still refining onboarding for. If your trace volume is low, talk to us directly; we'd rather figure out the right path with you than force a one-size-fits-all minimum.
Contribute
The repository is open.
Aetius is developed in the open on GitHub. The reconstruction engine, the attack families, and the statistical machinery behind every verdict are all there to read, and issues and pull requests are welcome. If something in the methodology looks wrong to you, that is exactly the kind of scrutiny it is built to invite.
View on GitHubTry it
Ship agent changes without the guesswork.
The public build installs in two commands and runs against your own traces. An Anthropic key plus a Langfuse or Braintrust key is all it needs.
Try It$ curl -fsSL https://aetius-webpage.vercel.app/downloads/install_latest.sh | bash -s -- https://aetius-webpage.vercel.app/downloads/aetius-latest.whl$ aetiusRequires Python 3.11+ and pipx. Full setup notes on the Try It page.