What it costs to stop reviewing at random.
Take 10,000 agent traces, 10% of them failures, and a team that reviews 500. Random review costs nothing to choose them and finds about 50 failures. Tailwise pays Jev, a small model, to score every trace, and its 500 find 2 to 5 times as many. The scoring costs a few dollars.
| Method | Failures found | 95% interval | Model cost |
|---|---|---|---|
| Random review | 50 | none | $0.00 |
| τ-bench | 110 | 62 to 158 | $1.38 |
| τ-bench, action checks | 237 | 128 to 332 | $3.32 |
| SWE-agent | 258 | 120 to 416 | $2.50 |
| TRAIL | 156 | 0 to 500 | $1.54 |
Bars are the mean over 100 resamples; lines are 95% intervals. The full width is every review a failure. † Rough: few successful runs.
Reviews per failure found
Random review reads 10 traces for every failure it finds. With Tailwise ranking the queue:
- τ-benchCustomer-service agents, built-in checks4.6
- τ-bench, action checksRuns whose outcome the trace shows2.1
- SWE-agent†Coding agents, built-in checks1.9
- TRAIL†Research and coding agents, built-in checks3.2
Per 10,000 traces, 10% failing, 500 reviewed
“Random reviews to match” is how many traces random review would read to find as many failures as Tailwise's 500. “Per extra failure” is the model cost divided by the failures found beyond random review's 50. Jev bills $0.042 per million input tokens; output is free.
| Dataset | Model cost | Failures found (95%) | Reviews per failure | Random reviews to match | Per extra failure |
|---|---|---|---|---|---|
| Random reviewAny dataset | $0.00 | 50 | 10 | 500 | – |
| τ-benchBuilt-in checks | $1.38 | 110 (62–158) | 4.6 | 1,099 | $0.023 |
| τ-benchBuilt-in checks, trace-detectable runs | $1.31 | 152 (81–241) | 3.3 | 1,522 | $0.013 |
| τ-benchAction checks | $3.58 | 115 (72–172) | 4.3 | 1,150 | $0.055 |
| τ-benchAction checks, trace-detectable runs | $3.32 | 237 (128–332) | 2.1 | 2,366 | $0.018 |
| SWE-agent†Built-in checks | $2.50 | 258 (120–416) | 1.9 | 2,579 | $0.012 |
| TRAIL†Built-in checks | $1.54 | 156 (0–500) | 3.2 | 1,558 | $0.015 |
The direction is clear on every dataset: a ranked queue finds more failures than a random one, for little money. How many more is less certain than a single number suggests.
- The intervals are wide
Every figure is an average over 100 draws that resample whole tasks. The best one, 237 failures in 500 reviews, has a 95% interval of 128–332. Even its lower end is above random review's 50.
- Some τ-bench labels can't be judged from a trace
Some τ-bench runs fail on things no trace shows, such as an item the simulated user wanted but never named, and some tasks' recorded answers disagree with runs scored as successes. Leaving those out leaves 1,304 of 1,980 runs. With the built-in checks, Tailwise finds 110 failures on all runs and 152 on those.
- The action checks were built on τ-bench
On all τ-bench runs they find 115 failures against 110 with the built-in checks, at 2.6× the cost. Their gain shows on trace-detectable runs, and they still need confirming on agents they were not designed on.
- SWE-agent and TRAIL are rough
SWE-agent: 77% of 1,000 runs failed; TRAIL: 74% of 148 runs failed. Resampling them to 10% failing leaves few successful runs to rank against, so those figures are marked † and their intervals run wide.
- Cost grows with trace length
These traces average 3,125 to 8,515 input tokens. Longer traces cost proportionally more. Reviewer time is still the larger cost: every review Tailwise saves is minutes of an engineer.
- No LLM judge in the comparison yet
The baseline is random review, the free option most teams use. A chat-model judge on every run would cost far more per trace; how much accuracy it buys on these runs is not measured yet.
Rules are free, certain checks, such as “the agent never submitted a patch”, written for each agent before looking at any result. A rule that keeps a trace skips the model call. On these datasets they did not improve the ranking on the whole, but SWE-agent's no_submit rule decided a third of runs without the model, cutting its cost from $2.50 to $1.43.
| Dataset | Failures found: model → + rules | Model cost: model → + rules |
|---|---|---|
| SWE-agent†Built-in checks | 258 → 280 | $2.50 → $1.43 |
| τ-benchBuilt-in checks | 110 → 89 | $1.38 → $1.38 |
| τ-benchBuilt-in checks, trace-detectable runs | 152 → 151 | $1.31 → $1.31 |
| TRAIL†Built-in checks | 156 → 121 | $1.54 → $1.43 |
| τ-benchAction checks | 115 → 97 | $3.58 → $3.58 |
| τ-benchAction checks, trace-detectable runs | 237 → 210 | $3.32 → $3.32 |
Every change in failures found has a 95% interval that includes no change. Rules are most useful where a failure has a plain signature, and as free filters before the model.
- τ-bench
- Customer-service agents (GPT-4o, Claude 3.5 Sonnet) in airline and retail simulations, acting through tools for a simulated user. 1,980 runs, 40% failed.
- SWE-agent
- SWE-agent on Llama models (8B, 70B, 405B) fixing GitHub issues; a run fails when its patch does not pass the tests. 1,000 runs, 77% failed.
- TRAIL
- smolagents runs on GAIA and SWE-bench, with failures annotated by hand. 148 runs, 74% failed.
- Method
- jev-1.14.0 on Tailwise's built-in checks (and τ-bench's action checks), traces ranked by severity. Failed runs are dropped at random until 10% fail, and whole tasks are resampled, 100 times.
- An LLM judge
The same runs scored by a chat model, to put a price on the accuracy a slower, costlier judge adds.
- Agents the checks were not built on
The action checks were designed on τ-bench. They need confirming on other agents.
- Production traces
Real failures at real failure rates, rather than benchmark runs resampled to 10% failing.
- More successful runs
SWE-agent and TRAIL have few, so their figures are rough.
The harness is in the repo. It downloads the datasets, scores them through Tailwise and writes every table on this page, with the plots, to a results folder.
git clone https://github.com/tecnosam/tailwise.git && cd tailwise/bench
uv run bench run --dry-run # projected tokens and cost, no calls
uv run bench run --datasets tau,swe # τ-bench and SWE-agent, under $1