Check every agent run for cents per thousand.

Random review reads 1 run in 20 and never sees the rest. Tailwise scores every run, about 14¢ per 1,000 with the built-in checks, and puts the likely failures at the top of your queue. In our benchmark, your team finds a failure every 2.1 to 4.6 reviews instead of every 10.

Try it in 60 seconds Read the benchmark

Failures found in 500 reviewsModel cost

Random review50$0.00

τ-bench110$1.38

τ-bench, action checks237$3.32

SWE-agent†258$2.50

TRAIL†156$1.54

Failures found in 500 reviews of 10,000 traces, and what the model calls cost
Method	Failures found	95% interval	Model cost
Random review	50	none	$0.00
τ-bench	110	62 to 158	$1.38
τ-bench, action checks	237	128 to 332	$3.32
SWE-agent	258	120 to 416	$2.50
TRAIL	156	0 to 500	$1.54

Per 10,000 traces · 10% failing · lines are 95% intervals · † rough, few successful runsHow we measured

The problem

Your dashboards say healthy. The answer is wrong.

A wrong answer throws no error and runs at normal speed, so your monitoring never sees it. Your users do.

of wrong answers go unseen when your team reviews a random 5% of runs.

one agent run · trace 7f3a91 healthy

question Which magazine started first, Arthur's Magazine or First for Women?

tool search("First for Women")

tool (stopped, never searched the other)

answer First for Women

status 200 OK · no errors · normal speed

truth Arthur's Magazine (1844)

keepflagsanswer_wrongmissing_tool_calls

Every other option misses them, or costs too much.

Approach	Scores every run	Catches wrong answers	Cheap at volume
Random samplingfree, but misses 95% at a 5% rate	No	Partly	Yes
Tail sampling ruleswrong answers raise no errors	Yes	No	Yes
Evals on the sampleonly see what sampling kept	No	Partly	Partly
LLM judge on every runslow and costly at volume	Yes	Yes	No
Tailwiseone short call per run, cents per 1,000	Yes	Yes	Yes

Random sampling
- NoScores every run
- PartlyCatches wrong answers
- YesCheap at volume
Tail sampling rules
- YesScores every run
- NoCatches wrong answers
- YesCheap at volume
Evals on the sample
- NoScores every run
- PartlyCatches wrong answers
- PartlyCheap at volume
LLM judge on every run
- YesScores every run
- YesCatches wrong answers
- NoCheap at volume
Tailwise
- YesScores every run
- YesCatches wrong answers
- YesCheap at volume

Quickstart

Three commands to your first decisions.

You need Docker and a Jev API key. Compose starts the engine, NATS and Phoenix, then scores a dozen sample runs and prints each decision with its reason.

Quickstart in the docs

❯

Already instrumented? Change one endpoint.

Tailwise reads standard OpenTelemetry spans and forwards kept traces to the Phoenix or Langfuse you already run. Your agent code stays the same, and backing out is the same one-line change in reverse.

shell

# Before: your exporter points at Phoenix or Langfuse.
# After: it points at Tailwise, which forwards the traces worth keeping.
export OTEL_EXPORTER_OTLP_ENDPOINT=http://tailwise:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

How it works

Spans in. Six checks. The failures out.

Your agent

OpenTelemetry spans, over gRPC or HTTP, or JSON

Tailwise

Queue

NATS JetStream stores the trace before Tailwise replies

Score · one model call

answer_wrong
unsupported_answer
missing_tool_calls
answered_on_failure
ignored_instructions
contradicts_knowledge

Decide

Any check over its threshold keeps the trace

Model down? 5 deliveries over about 8 minutes, then the dead-letter stream. Nothing dropped silently.

The model: Jev by default, or any /v1/systemone endpoint, chosen per agent.

Kept · your back end

Phoenix, Langfuse or any OTLP back end, witheval.flags answer_wrong
eval.severity 0.57
eval.reason …

Dropped

Never sent. 2% kept at random as a check on the model.

Failure modes

What happens when it breaks.

Stored before acknowledged
The API answers 202 only after NATS JetStream has stored the trace. Delivery is at least once, from a durable stream, not an in-memory queue.
Model down: retried, then parked
A trace the model fails to score goes back on the queue after 5s, 30s, 2m and 5m. After 5 deliveries it moves to a dead-letter stream, with headers saying why, ready to replay.
Model rejects a trace: a random fallback
A trace the model refuses (400, 413, 422) is kept at the random-floor rate with rule fallback, so you still get a review sample.
Tested against a real NATS
The test suite starts a NATS server inside the test process and runs go test -race. No fakes for the queue.
Golden files pin what the model reads
The text the model reads for each test trace is checked in. The tests fail if that text changes, until someone rewrites the files and reviews the diff.
Configs checked before rollout
tailwise -check validates the config, its manifests and the secrets it names, then exits 1 on any problem.

Where it breaks

The README lists every open problem in this version. A few you should know before a pilot:

Dead letters wait for a manual replay, and nothing alerts on their count yet.
Nothing redacts fields or caps spend before the call to Jev.
Delivery is at least once, so a trace can reach your back end twice.
Embedded NATS without queue.store_dir loses queued work on a crash.

Read every known limitation

Data and security

Where your traces go.

Tailwise runs inside your network. Three things leave it, and you configure each one. With the defaults, every scored trace goes to TypeSafe's hosted Jev, a third party. A dropped trace goes nowhere after scoring.

Leaves → 1Every trace
The model endpoint
Jev at api.typesafe.ai, unless you set scorer.endpoint
The full text of every scored trace: the user's question, each tool call, the evidence and the answer.
Leaves → 2Kept traces only
Your exporters
Phoenix, Langfuse or the OTLP back ends you list
Kept traces, with the decision as eval.* attributes. Each back end gets only its own headers.
Leaves → 3Off by default
Your metrics endpoint
Only if you set observability.metrics.endpoint
The engine's own counters and timings.

Keep traces in your network

Tailwise talks to models over the System One API. Set scorer.endpoint to a server you run, for every agent or one at a time, and trace text never reaches TypeSafe. Before you send production traces to Jev, review TypeSafe's terms for retention, training use and hosting region.

Deploying, in the docs

Not built yet

Redaction before the model call
No hook strips personal data from a trace before Tailwise sends it to the model.
Authentication on intake
The API and OTLP receivers have no auth or TLS. Run Tailwise inside your VPC and put a mesh or load balancer in front.

Cost against accuracy

Pay cents per 1,000 traces. Get back hours of review.

Random review is free to choose, but it finds failures only as often as they happen. Tailwise is free and open source; you pay for one short model call per trace, about 14¢ per 1,000 traces of 3,300 tokens. In exchange, the traces your team reads are the likely failures.

The numbers on the right come from the benchmark: the share of reviewed traces that failed when 10% of traces fail and a team reviews 5%. Your agents will differ.

No spend cap yet. Every scored trace is a billed call, and only server.workers limits the rate. Heuristic rules that keep a trace skip the call.

Agents like

Traces per day10k

Tokens per trace3,300

Minutes per review5

Per month, reviewing 5% of traces
Per month	Random	Tailwise
Model calls	$0	$41.6
Failures found	1,500	3,296
Hours to find 3,296	2,747	1,250

To find as many failures at random, your team would review for about 1,497 more hours a month. Tailwise costs $41.6 of model calls instead.

15,000 reviews (5% of traces), with 10% of traces failing; Tailwise's share of failures among them is the benchmark's (22%, 95% interval 12–32%). Jev at $0.042 per million input tokens; output is not billed. Check TypeSafe for current pricing. The benchmark

Adopting it

Start with a shadow pilot.

Tailwise is at v0.2.0 with one maintainer. Run it beside what you have before you let it decide what your team never sees.

1
Add a second exporter
Send a copy of your spans to Tailwise from your OTel SDK or Collector. Your current pipeline keeps running.
2
Keep into a scratch project
Point every exporter at a separate Phoenix or Langfuse project.
3
Compare for a week
Check what Tailwise kept against what your team flagged, and watch which checks fire.

tailwise.yaml

# tailwise.yaml: a shadow pilot. Kept traces go to a scratch
# project, so nothing changes for your team until you decide.
exporters:
  - name: shadow
    endpoint: http://phoenix:6006/v1/traces
    resource_attributes:
      openinference.project.name: tailwise-shadow