The Tailwise manual

Decides which LLM agent traces are worth keeping for review. For each trace it asks a System One decision model, TypeSafe's hosted Jev by default, whether the final answer is likely wrong, applies a keep/drop policy, and exports the kept traces over OTLP to any number of trace back ends: Arize Phoenix, Langfuse, an OpenTelemetry Collector, or anything else that accepts OTLP/HTTP traces.

Traces arrive as OpenTelemetry spans (OTLP over gRPC or HTTP) or over an HTTP JSON API, and wait in a NATS JetStream queue: on the server queue.url names, or, when it is empty, on a NATS server the engine runs inside its own process. A worker pool in the same binary pulls them off the queue and does the rest.

OTLP gRPC :4317 ─┐
OTLP HTTP :4318 ─┴─> spans stored by trace ID ──> assembled once the trace goes quiet ─┐
                     (stream TAILWISE_SPANS)                                            │
                                                                                        v
POST /v1/traces ──> validate ───────────────────────────────> JetStream stream TAILWISE_TRACES
                                                                                        │
                                                                                        v
                                             worker pool ──> slice ──> Jev ──> policy ──> exporters (kept only)
                                                  │
                               system error, after max_deliver attempts
                                                  v
                                    JetStream stream TAILWISE_TRACES_DLQ
 
POST /v1/traces:score ──> validate ──> slice ──> Jev ──> policy ──> exporters   (no queue)

Quickstart

You need Docker and a Jev API key. The engine needs no config file.

export JEV_API_KEY=...        # your TypeSafe Jev API key
make up                       # the engine, NATS JetStream and Phoenix, in Docker
make samples                  # send the bundled sample traces and print each decision

make samples sends scripts/samples.jsonl, a dozen agent runs whose answers are wrong, partly correct or correct, and prints each decision next to its label, with Jev's reason in plain English:

a57d9106a210b9f56fba917038ea3be2  label=wrong    keep p_keep=0.980  above_threshold
    The final answer contradicts the evidence the agent gathered.
d5ca6bb1e89aa96b4290758d031c7716  label=correct  drop p_keep=0.040  below_threshold
    The final answer is correct, complete and supported by the evidence.

The kept ones appear in Phoenix at http://localhost:6006, in the project tailwise, with the decision as eval.* attributes on their root span. make logs follows the engine's log; make down stops everything.

At start-up the engine warns that the model has no manifest: it asks the built-in questions, which are written for any agent rather than yours. See Manifests.

Setup

Requirements:

• A Jev API key, in JEV_API_KEY
• Docker, to run the engine, NATS and Phoenix with Compose
• Go 1.26 or newer, to build or test outside Docker (not for a release binary)
• Python 3, only for make samples

From a release binary

Every release on GitHub Releases has a tailwise binary for Linux, macOS and Windows, each on amd64 and arm64, and a checksums.txt. The binary needs nothing else: the queues run inside it unless queue.url names a NATS server.

VERSION=0.1.0 OS=linux ARCH=amd64   # OS: linux, darwin; ARCH: amd64, arm64
BASE=https://github.com/tecnosam/tailwise/releases/download/v$VERSION
curl -fsSLO "$BASE/tailwise_${VERSION}_${OS}_${ARCH}.tar.gz"
curl -fsSLO "$BASE/checksums.txt"
sha256sum --ignore-missing -c checksums.txt   # on macOS: shasum -a 256 --ignore-missing -c checksums.txt
tar -xzf "tailwise_${VERSION}_${OS}_${ARCH}.tar.gz" tailwise
 
export JEV_API_KEY=...
./tailwise -version
./tailwise            # the built-in defaults: Jev, embedded NATS, Phoenix at localhost:6006

Windows archives are .zip files holding tailwise.exe. On macOS, a binary downloaded in a browser is quarantined; clear it with xattr -d com.apple.quarantine tailwise. To see kept traces, start Phoenix too (make phoenix, or phoenix serve); it takes port 4317, so run the engine with TAILWISE_OTLP_GRPC_ADDR=:14317.

With Docker Compose

make up       # docker compose up --build -d: the engine, NATS JetStream and Phoenix
make logs     # follow the engine's log
make down     # stop all three; queued traces, dead letters and Phoenix's data stay in volumes

Compose starts three services:

Inside the container the engine listens for OTLP gRPC on 4317. Compose publishes it on host port 14317 because Phoenix takes 4317 for its own OTLP gRPC receiver.

If a Phoenix is already running on the host (phoenix serve), it holds 6006 and 4317 and make up fails with "port is already allocated". Either stop it, or move the compose one with PHOENIX_PORT and PHOENIX_GRPC_PORT, or keep using the host one:

TAILWISE_EXPORT_ENDPOINT=http://host.docker.internal:6006/v1/traces \
  docker compose up --build -d --no-deps nats observer

Compose reads these variables:

To run the engine with your own config instead, mount it and pass -config; examples/README.md shows how.

On the host

make phoenix  # start only Phoenix in Docker
make run      # build and start the engine on its built-in defaults (Jev, embedded NATS)
make run CONFIG=examples/jev.yaml   # or with a config file
make test     # go vet + go test -race, fully offline
 
# Or with the queues in an external NATS JetStream, as in a cluster:
make nats
TAILWISE_NATS_URL=nats://localhost:4222 make run

make run binds 8080, 4317 and 4318. Phoenix, from make phoenix or on the host, already holds 4317, and the engine exits with an "address already in use" error; start it with TAILWISE_OTLP_GRPC_ADDR=:14317 make run instead.

Sending traces

curl -X POST localhost:8080/v1/traces:score -d @testdata/traces/wrong_river.json

{"trace_id":"wrong-river-001","decision":"keep","p_keep":0.98,"rule":"above_threshold","reason":"The final answer contradicts the evidence the agent gathered.","cause":"contradicts_evidence","model":"jev-1.14.0","latency_ms":357.8,"exported":true}

scripts/send_samples.py sends a batch:

python3 scripts/send_samples.py --sync          # print each decision next to its label
python3 scripts/send_samples.py                 # through the queue
python3 scripts/send_samples.py --otlp          # as OTLP spans, to :4318
python3 scripts/send_samples.py --dataset my-traces.jsonl -n 50   # your own traces

--dataset takes any JSONL file of traces in the JSON API's shape; an optional label on each row is printed next to its decision, and -n samples that many, equally across labels. With --otlp each trace is sent as an AGENT root span and one TOOL span per tool call.

To point an instrumented agent at the engine, give its OTLP exporter the engine's endpoint instead of Phoenix's; see Connecting an agent or service.

Flags: -config PATH (optional: without it the engine runs on its built-in defaults and the environment), -check (validate and exit), -version and -log-level debug|info|warn|error.

Configuration

The engine needs no config file. Without -config it runs on its built-in defaults: Jev as the scorer, the built-in questions, an embedded NATS server and one Phoenix exporter at http://localhost:6006/v1/traces. A config file sets only what differs; every key it leaves out keeps its built-in default. examples/ has ready-to-run variations, and tailwise.schema.yaml is the full reference: a JSON Schema with each key's type, default, limits and the rules that tie keys together. architecture.md explains how a file is loaded.

Environment overrides, which apply with or without a config file:

The standard OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_TRACES_HEADERS are not read: each exporter sends only its own headers, so one back end's credentials never reach another. Use headers_env instead.

The engine validates its configuration at start-up and exits with a clear error if:

• JEV_API_KEY, or a variable the config names for a secret, is not set
• a threshold is outside [0, 1], a duration does not parse, or a key is unknown
• the decision question is missing from the manifest (or the built-in questions) or is not noul
• queue.ack_wait is not longer than the slowest model's timeout (a queued trace gets one scoring call per delivery), since a shorter value would hand a trace to a second worker while the first is still on it

tailwise -check runs the same checks and exits, printing any warnings.

Manifests

A manifest holds the questions the model is asked:

{"model": "...", "questions": {"answer_wrong": {"type": "noul", "instructions": "..."}, "why_wrong": {"type": "choice", "criteria": {"...": "..."}}}}

• questions is sent to the model unchanged, all questions, in the manifest's order, in one call. The reason question's chosen option is the reason.
• decision, optional, lists more yes/no (noul) questions that can keep a trace (see Any question can keep a trace). Without it, the decision question decides alone. It stays in the engine and is not sent to the model.
• A relative scorer.manifest path resolves against the config file's directory.

A manifest is optional, for every System One model. Without one the model is asked the built-in questions (builtin_manifest.json), and the engine logs a warning at start-up. They are:

Each yes/no answer is the probability that the agent failed in that way. answer_wrong alone decides keep or drop; the other answers are recorded with every decision (in the log and as eval.answers.<question> on the export) so a reviewer sees which problems the model found. In the benchmark, letting any check keep a trace kept most traces without ranking failures any better, so the built-in questions have no decision block; a manifest can add one. To word the questions for your agent, start from examples/manifests/my-agent.json and set scorer.manifest (or TAILWISE_MANIFEST).

How well each check tells failed runs from successful ones depends on the agent. In the benchmark, unsupported_answer was informative on coding agents (ROC-AUC 0.78 on SWE-agent) but no better than chance on a customer-service agent whose answers cite its own lookups; missing_tool_calls was informative on both. Treat a check as a hint until you have seen it on your own agent's traces.

Any question can keep a trace

A manifest's decision block lists the yes/no questions that can keep a trace besides the decision question, in priority order, and can give any of them its own threshold:

"decision": {
  "any_of": ["unsupported_answer", "missing_tool_calls", "answered_on_failure",
             "ignored_instructions", "contradicts_knowledge"],
  "thresholds": {"contradicts_knowledge": 0.7}
}

• Any question keeps. A trace is kept when the decision question, or any question in any_of, reaches its threshold. Each question checks a different way an agent fails, and a wrong answer usually fails in one way only: a check that says "no" is not evidence the answer is right, so it never counts against keeping.
• Thresholds. thresholds sets a question's own threshold, in (0, 1]; a question without one uses policy.keep_threshold. Raise it for a check that flags too often. The decision question can have one too.
• Flags. The questions that reached their threshold are the trace's flags: the problems the model found, in priority order. They are in the decision log line (flags) and on the exported trace (eval.flags), next to every answer (eval.answers.<question>). A reviewer can filter on them, for example every kept trace flagged answered_on_failure.
• Priority. The decision question always comes first, then any_of in the order written, then the action checks. Flags are listed in that order.
• Severity: the review order. A trace's severity is the average answer of its deciding questions, flagged or not. action_wrong on a trace without actions is 0; a check that failed is left out of the average rather than counted as 0, so an outage does not bury failed traces. Review traces highest severity first. It does not change whether a trace is kept (severity in the response and the log, eval.severity on the export).
  • Why an average and not the highest answer: a trace several checks doubt is more often a real failure than one a single check is sure of. On τ-bench with the action checks, when 10% of runs fail and a team reviews the top 5%, ranking by the highest answer finds 23% failures among the traces reviewed; ranking by the average finds 47% (benchmark), on the runs whose label the trace can show.
  • Every deciding question counts the same, so list in any_of only the checks you trust on your agent: a check no better than chance pulls good traces up.
• p_keep is the highest answer among the deciding questions.
• Every listed question must be a noul question of the same manifest, listed once, and the model must answer all of them: a missing answer is a scoring error, like a missing decision answer. Questions not listed are still asked, logged and exported, but do not keep a trace.
• Each listed question is one more way to be kept, so listing more keeps more traces. Recheck the share kept after changing the list, a threshold or a question's wording.

Without a decision block, the decision question decides alone, against policy.keep_threshold.

Checking an agent's actions

An agent that acts for its users (cancels orders, books flights, sends money) fails in ways its final answer hides: it exchanges the wrong item, or never does one of the things the user asked, and then reports success. Name the tools that act in the manifest's actions block, each with what it does, worded as the user would ask:

"actions": {
  "cancel_order": "cancel an order",
  "issue_refund": "get a refund for an order",
  "update_shipping_address": "change the shipping address of an order"
}

Each trace then gets two more checks, each its own call to the model:

• action_wrong, once per action call (the last 10): does the call use an argument that is not what the user asked for? The model sees the user's request, the earlier tool results that hold the call's argument values (where the right item or order is identified, trimmed to the parts holding them), the call, and the actions after it. The trace's answer is the highest over its calls.
• unhandled_request, once per trace: for each action, did the user ask for it, and was every such request carried out or explicitly declined? The model sees the conversation (or, without one, the user's request), the actions taken and the final answer. The answer is the highest min(requested, 1 − handled) over the actions.

Both are deciding checks after the decision question and any any_of questions, with policy.keep_threshold unless decision.thresholds sets their own. They are recorded in answers and flags like any question. When one of them, and not the decision question, keeps a trace, the reason says what it found: cause wrong_action ("The agent's action cancel_order({"order_id":"#W2"}) does not match what the user asked for.") or unhandled_request ("The user asked the agent to get a refund for an order, and it neither did it nor declined."). A check that fails is left out of the decision, and reported; on the queued path a system error redelivers the trace instead.

The conversation. The completeness check reads the agent's turns: they show a request it declined, or one the user withdrew. Tailwise takes them from the agent's latest LLM span (the one with the longest llm.input_messages, plus its llm.output_messages), or from the JSON API's optional conversation, a list of {"role": "user" | "assistant", "content": ...}. Without them, the check sees only the user's request and the final answer, and flags far more successful runs (26% against 11% on τ-bench).

What it costs and buys. On τ-bench's trace-detectable runs, with grounded evidence, the checks raise the failures kept from 54% to 70%, and the successful runs kept from 26% to 40%; the share of kept runs that failed barely moves (60% to 57%). They catch the failures the final answer hides: wrong-variant exchanges the user had described go from 33% to 90%, runs that left out an action from 50% to 68%. Each trace costs one call per action call plus one, about 2,000 input tokens each.

Heuristic rules

Some failures need no model to spot: a final answer that should be JSON and is not, a tool call that errored, an agent calling the same tool with the same arguments again and again. Write them as rules. They cost nothing, their verdict is certain, and each fired rule is listed with a plain reason.

Rules in the config file apply to every agent. An agent's manifest adds its own under "rules", in the same form: a rule with the same name replaces the default, and "enabled": false turns it off for that agent.

rules:
  - name: answer_json
    on_fail: keep                # certain: keep without asking the model
    final_answer: {json_schema: {type: object, required: [order_id, status]}}
  - name: tool_failed
    on_fail: signal              # raises severity, decides nothing
    tool_output: {ok_status: true}
  - name: looping
    run: {max_repeated_calls: 3}

Each rule sets one of:

A rule fires when the trace breaks any of its expectations. on_fail says what happens then:

• keep: the trace is kept without asking the model, so it costs no tokens. Rule keep_rule, cause rule, the rule's reason ("The final answer does not match the schema: at '': missing property 'status'."), severity 1, so it is reviewed first, and p_keep null. Use it for failures that are certain.
• signal (the default): the rule counts in severity, 1 when it fires and 0 when it does not, next to the model's answers, and decides nothing. Every signal rule weighs as much as a check, so a rule that fires on many good traces pulls them up the review order: keep signal rules few and specific.

Fired rules are in the response's rules (name, on_fail, reason), the log line's rules, and the export's eval.rules and eval.rules.<name>. A rule that does not compile (a bad regular expression or schema, two rules with one name) stops Tailwise at start-up, naming the model and the rule.

One model per agent

Each agent can have its own model: its own questions, and if needed its own endpoint or key. The scorer section is the default model; agents.models adds one per agent, each complete on its own:

agents:
  attributes: [agent.name, service.name]
  models:
    support-bot:
      endpoint: https://api.typesafe.ai/v1/systemone
      model: jev-latest
      manifest: manifests/support-bot.json
    research-agent:
      endpoint: https://api.typesafe.ai/v1/systemone
      model: jev-latest
      manifest: manifests/research-agent.json
      timeout: 45s

• Which agent. For OTLP traces, the first of agents.attributes set on the root span, else on its resource: OpenInference's agent.name, then the resource's service.name by default. For the JSON API, the trace's agent field.
• Which model. The agent's entry under agents.models, else the scorer section. A trace that names no agent, or an agent not listed, is scored by the default.
• No inheritance. Each entry takes the scorer keys (kind, endpoint, model, api_key_env, manifest, decision_question, timeout, max_retries, backoff) and is complete on its own: endpoint and model are required, and nothing is taken from the scorer section. Keys it leaves out get the engine's built-in defaults, the same ones scorer gets; without a manifest it gets the built-in questions and its own warning. TAILWISE_SCORER_ENDPOINT and TAILWISE_MANIFEST change only the scorer section.
• Everywhere. The decision log, the /v1/traces:score response, the eval.agent span attribute and the metrics all carry the agent, and eval.model the model. /readyz probes the default model only, so one agent's model being down does not make the whole engine unready.

examples/agents.yaml is a complete example.

OTLP ingestion

The engine runs the standard OTLP trace receivers, so an agent's exporter can point at it as if it were a collector:

An export is answered with success as soon as the spans are stored (in NATS, once it has confirmed them, typically within a few milliseconds). If they cannot be stored the answer is gRPC UNAVAILABLE or HTTP 503, which OTLP exporters retry.

Connecting an agent or service

Point the OTLP exporter at the engine instead of at Phoenix. Nothing else in the agent changes, and no API key or header is needed.

For anything using an OpenTelemetry SDK, the standard variables are enough:

# HTTP (the SDK appends /v1/traces)
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
 
# or gRPC (host port 14317 under Docker Compose, 4317 inside the network)
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:14317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Or set it in code. In Python, with an OpenInference instrumentor:

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
 
provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
)
trace.set_tracer_provider(provider)
# then instrument as usual, e.g. GoogleADKInstrumentor().instrument(tracer_provider=provider)

The engine needs one root span carrying the question in input.value and the answer in output.value; tool calls are spans with openinference.span.kind = TOOL. The instrumentors emit these already. To send a trace by hand:

curl -i http://localhost:4318/v1/traces -H 'Content-Type: application/json' -d '{
  "resourceSpans": [{"scopeSpans": [{"spans": [{
    "traceId": "5b8efff798038103d269b633813fc60c",
    "spanId": "eee19b7ec3c1b174",
    "name": "agent",
    "startTimeUnixNano": "1760000000000000000",
    "endTimeUnixNano": "1760000001000000000",
    "attributes": [
      {"key": "openinference.span.kind", "value": {"stringValue": "AGENT"}},
      {"key": "input.value", "value": {"stringValue": "Which river flows through the capital of France?"}},
      {"key": "output.value", "value": {"stringValue": "The Rhône"}}
    ]
  }]}]}]
}'

It answers 200 {}. The decision appears in the engine's log about otlp.idle_timeout (10s) later, and the trace shows up in your exporters (Phoenix by default) if it was kept.

From spans to a decision

1. Store. Each export is split by trace ID and published to tailwise.spans.<trace id> in the TAILWISE_SPANS stream.
2. Wait. A trace's spans usually arrive in several batches, with the root span last. The trace is considered complete when no span has arrived for otlp.idle_timeout, or otlp.max_wait after its first span.
3. Assemble. The spans are merged (a span sent twice counts once) and turned into the same record the JSON API takes; see the mapping below. Spans that do not describe an agent run are discarded here, before they cost a call to the model.
4. Queue, score, decide. The record goes to TAILWISE_TRACES and from there everything is as for a JSON trace, including redelivery and the dead-letter stream.
5. Forward. A kept trace is sent to every exporter as its original spans, with the original trace and span IDs, the exporter's resource_attributes, and the eval.* attributes added to the root span. A dropped trace is sent nowhere and its spans are deleted.

Because of step 2, a decision arrives at least otlp.idle_timeout after the trace's last span.

How spans are read

The engine reads the OpenInference attributes that Phoenix's instrumentors emit.

When input.value or output.value holds a JSON object, the text is taken from a question/input/query/prompt/task key (or answer/output/result for outputs) when there is one. A tool output shaped as a function response, {"response": {"result": "..."}}, is unwrapped.

A message recorded by the Google ADK instrumentor, {"new_message": {"parts": [{"text": "..."}]}} on the way in and {"content": {"parts": [{"text": "..."}]}} on the way out, gives the text of its parts, so an ADK agent's own root span works without extra attributes.

A trace is not scorable, and is discarded with a spans discarded log line, when its root span has no input.value, or has no output.value and is not an AGENT span. That covers ordinary service spans sent to the engine by mistake.

An agent run that ended without an answer is scored like any other, with an empty final answer: its root span either sets output.value to an empty string, or is an AGENT span with no output.value, as when the run crashed or ran out of steps. Those runs nearly always failed, so they reach review instead of being dropped.

Matching what the model was trained on

The model was trained on evidence as bare paragraph text and on the agent's own tool statuses (ok, not_found, bad_args, ...). Standard spans carry neither exactly: a tool span holds the tool's rendered output, and its status is only OK or ERROR. So for a trace read from plain instrumentor spans, the slice differs a little from the training data. For the test agent, evidence items keep their [Title] prefix and a search that found nothing shows as ok with its message as evidence.

An agent can remove the difference by setting two attributes itself:

With both set, a trace sent as spans yields byte for byte the slice the same trace yields through the JSON API. A test checks this for every fixture, and scripts/send_samples.py --otlp sends spans this way.

API

POST /v1/traces

Stores a trace in the queue. The 202 is sent only after JetStream has confirmed the trace, so an accepted trace survives a restart of the engine, unless the embedded server runs without queue.store_dir.

{
  "trace_id": "7f3a91c2e8",
  "agent": "support-bot",
  "system_prompt": "You answer geography questions. Cite your sources.",
  "question": "Which river flows through the capital of France?",
  "tool_calls": [{"name": "search", "args": {"query": "capital of France"}, "status": "ok"}],
  "evidence": ["Paris is the capital of France.", "The Seine flows through Paris."],
  "final_answer": "The Seine"
}

• question and final_answer are required. final_answer may be an empty string for a run that ended without an answer.
• trace_id is optional; the engine generates 32 hex characters if it is missing.
• agent is optional and picks the model (see One model per agent); at most 128 bytes.
• conversation is optional: the user's and agent's turns, each {"role": "user" | "assistant", "content": "..."}, for the action checks.
• system_prompt is optional: the instructions the agent ran under. A model with scorer.system_prompt: true sees it, cut to 8,000 characters, above the question; otherwise it is ignored and the model sees exactly what it would without one.
• args may be any JSON value. status defaults to ok. A tool call's optional output is what it returned; models with scorer.evidence: grounded, the default, see it.
• This is the engine's own JSON API on port 8080. The OTLP receiver also serves a /v1/traces path, on port 4318, with a different body.

POST /v1/traces:score

Same input, scored synchronously without the trace queue. Meant for testing and demos. It returns as soon as the decision is made; a kept trace is queued for export and reaches every exporter's back end moments later, so a slow back end never holds up the response.

{
  "trace_id": "wrong-river-001",
  "agent": "",
  "decision": "keep",
  "p_keep": 0.98,
  "rule": "above_threshold",
  "reason": "The final answer contradicts the evidence the agent gathered.",
  "cause": "contradicts_evidence",
  "model": "jev-1.14.0",
  "latency_ms": 357.8,
  "exported": true,
  "answers": {"answer_wrong": 0.98, "unsupported_answer": 0.12},
  "flags": ["answer_wrong"],
  "severity": 0.98,
  "input_tokens": 412
}

POST /v1/otlp/traces:score

Scores one trace sent as OTLP spans, synchronously, and returns the same response as POST /v1/traces:score. The body is an OTLP/HTTP export request (application/x-protobuf or application/json, optionally gzipped) holding the spans of exactly one trace. The trace is built from the spans exactly as the OTLP receivers build it (see From spans to a decision), but without the queue or the wait for late spans, so send every span in one request. It answers 400 for spans of more than one trace, and 422 when no root span has a question and an answer. Meant for testing and benchmarks: bench/ scores datasets through it.

API docs: GET /docs and GET /openapi.yaml

/openapi.yaml serves the OpenAPI 3.0 description of this API. /docs renders it with Swagger UI, where you can also send requests. The UI's scripts load from a CDN, so /docs needs internet access in the browser; the spec does not.

The spec lives in internal/server/openapi.yaml and is embedded in the binary. Update it when you change a handler; a test fails if a route, a documented status code or a field of the decision response goes missing from it.

GET /healthz and GET /readyz

/healthz returns 200 whenever the process is up. /readyz returns 200 only if the queue is up (the NATS connection, when NATS is used) and the default model (the scorer section) answers GET /v1/models on its host within 2 seconds, and 503 otherwise. An agent's own model is not checked: when it is down, only that agent's traces are affected (retried, then dead-lettered), so it does not take the engine out of a load balancer; watch tailwise.scorer.duration with error.type per tailwise.model for it.

How a decision is made

1. Route. The trace's agent picks its model (see One model per agent).
2. Rules. The heuristic rules run. When a keep rule fires, the trace is kept (rule keep_rule) and the model is not asked.
3. Slice. trace.BuildState renders the trace into the text the model reads (see The slice).
4. Score. The slice and the model's questions (its manifest's, or the built-in ones) go to the model, in one call. A queued trace that hits a system error goes back to the queue for redelivery. On the synchronous endpoint, which has no queue, the engine retries it with scorer.max_retries and jittered scorer.backoff. The two kinds of retry never multiply.
5. Decide. The trace is kept when any question that decides (the decision question, and those in the manifest's decision.any_of) reaches its threshold; those questions are its flags (see Any question can keep a trace).
   • A question reached its threshold: keep, rule above_threshold.
   • Otherwise keep with probability random_floor, rule random_floor.
   • Otherwise drop, rule below_threshold.
6. Export. Kept traces are queued to every exporter. Each decision produces one JSON log line with trace_id, agent, decision, rule, reason, cause, p_keep, severity, model, latency_ms, answers, every yes/no answer by question, flags, the deciding questions that reached their threshold, rules, the heuristic rules that fired, as in "answers": {"answer_wrong": 0.91, "unsupported_answer": 0.84, ...}, "flags": ["answer_wrong", "unsupported_answer"].

Reasons

Every decision says in plain English why the model thinks the final answer is wrong. Jev answers with probabilities and choices, not free text, so the reason comes from a choice question, why_wrong in the built-in questions (scorer.reason_question names it). Each option is a sentence:

The model picks the option that fits, and its sentence becomes the reason. A manifest can define its own options in the same shape, with sentences that suit your agent. The reason question rides along in the same request as the decision question, so it costs no extra call. If a manifest has no reason question, the reason says only how likely the answer is wrong, and the engine warns at start-up. On fallback it says the model could not score the trace.

When the model cannot score a trace

What happens depends on whose fault it is and which path the trace took.

The random fallback keeps the trace with probability random_floor and drops it otherwise, with rule fallback either way.

The queue

Where the queues live

The engine has three kinds of queue: the trace work queue with its dead-letter stream, the store where OTLP spans wait to be assembled, and one export queue per exporter. queue.url decides where all of them live:

Both run the same JetStream code, so the rules are the same: a bounded queue (queue.max_messages, exporters[].queue_size), redelivery with backoff up to a delivery limit, then a dead letter stored in a DLQ stream; spans assembled after otlp.idle_timeout of quiet or otlp.max_wait. The embedded server suits trying the engine and single-instance setups; use an external server to share the work between replicas. With the embedded server queue.replicas must be 1. Docker Compose sets TAILWISE_NATS_URL, so the compose stack uses its own NATS container.

The rest of this section applies to both.

• Stream TAILWISE_TRACES uses work-queue retention on file storage: a trace is removed once a worker acknowledges it. A full stream refuses new traces rather than evicting unscored ones.
• Workers pull one trace at a time through the durable consumer tailwise-workers. Run more engine replicas with the same consumer name to share the work.
• Delivery is at least once. If a worker dies after exporting a trace but before acknowledging it, the trace is scored and exported again.
• Payload is the trace JSON, the same shape the API accepts, so another producer can publish straight to tailwise.traces. A trace published that way without a trace_id gets one derived from its stream sequence.
• A trace assembled from OTLP spans also carries those spans in an otlp_spans field (base64 protobuf), so a kept trace can be forwarded as the originals. nats.conf raises the NATS message limit to 8 MB for this. A trace still too big is queued without its spans and, if kept, exported as rebuilt spans instead.

NATS names

Every stream, subject and consumer is named from queue.prefix. With the default, tailwise:

The workers pull from the consumer tailwise-workers, and each exporter's workers from tailwise-workers-export. Engines with the same prefix share the work, so give each deployment on one NATS server its own prefix, for example staging.

NATS in production

The engine is a NATS client; it does not run or configure the server (nats.conf and the compose service are only for trying it). Against a secured or clustered NATS, set:

queue:
  url: tls://nats.internal:4222
  replicas: 3                                  # every stream the engine creates, kept on 3 servers
  auth:
    credentials_file: /var/run/secrets/nats/tailwise.creds   # or nkey_seed_file, user + password_env, token_env
  tls:
    ca_file: /var/run/secrets/nats/ca.pem      # a private CA
    # cert_file / key_file                     # mutual TLS

replicas is applied to every stream the engine creates (traces, dead letters, spans, pending markers, each exporter's queue); more than the cluster has servers fails at start-up. An existing stream is updated to the new count when the server allows it. Passwords and tokens come from environment variables (password_env, token_env); credential, seed and certificate files from paths, relative to the config file, typically mounted from a Kubernetes secret. The account needs JetStream access to the streams under queue.* and otlp.* and the right to create durable consumers.

Streams

Redelivery and the dead-letter stream

When a worker hits a system error (one call per delivery), it returns the trace to the queue with a delay from queue.redelivery_backoff. After queue.max_deliver deliveries it copies the trace to TAILWISE_TRACES_DLQ and removes it from the work queue. With the defaults that is 5 deliveries over about 8 minutes.

A message that is not a valid trace is dead-lettered on first sight.

A dead letter's body is the original trace. Its headers say what happened:

Each dead-lettered trace also produces an ERROR log line, trace dead-lettered.

Inspecting and replaying, with the compose stack up:

make dlq      # how many dead letters there are
 
# The NATS CLI, run from a container on the compose network:
alias nats='docker run --rm -i --network tailwise_default natsio/nats-box:latest nats -s nats://nats:4222'
 
nats stream get TAILWISE_TRACES_DLQ 1          # one dead letter, with headers
 
# Replay it once Jev is back: post its body to the API again.
nats stream get TAILWISE_TRACES_DLQ 1 --json | jq -r .data | base64 -d \
  | curl -X POST localhost:8080/v1/traces -d @-
 
nats stream purge TAILWISE_TRACES_DLQ          # clear the dead letters

NATS monitoring is at http://localhost:8222 (/jsz?streams=true for stream and consumer counts).

Exporters

Kept traces go to every back end in exporters, over OTLP/HTTP. The default is the local Phoenix:

exporters:
  - name: phoenix
    endpoint: http://localhost:6006/v1/traces
    resource_attributes:
      openinference.project.name: tailwise   # the Phoenix project

Add more entries to send to more back ends. Langfuse, for example, takes Basic auth built from its public and secret keys (examples/langfuse.yaml):

  - name: langfuse
    endpoint: https://cloud.langfuse.com/api/public/otel/v1/traces
    headers_env: LANGFUSE_OTLP_HEADERS

export LANGFUSE_OTLP_HEADERS="Authorization=Basic%20$(printf '%s' "pk-lf-...:sk-lf-..." | base64)"

Headers. An exporter's headers are its headers, plus those parsed from the variable headers_env names, which win on a name clash. The variable uses the OpenTelemetry format: key=value pairs separated by commas, values percent-encoded where needed (%20 for a space). The engine logs exporter names and endpoints, never header values, and a header that fails to parse is reported without its value.

Asynchronous. Exporting never holds up scoring. Each exporter has its own bounded queue (queue_size) and its own workers (concurrency); a kept trace is queued to every exporter and the worker moves on. The queue is in NATS JetStream when queue.url is set (stream TAILWISE_EXPORTS_<NAME>, so queued exports survive a restart, unless the embedded server runs without queue.store_dir); see Where the queues live. A slow or unreachable back end only fills its own queue: past queue_size, traces are skipped for that back end alone and each skip is reported to the error sinks (kind=full, exporter=<name>). The decision log's exported field says whether at least one exporter queued the trace.

Retries. A send returns only once the back end has answered, so a failure is never silent. A failed send is retried after backoff (default 1s, 5s, 30s), up to max_deliver tries (default 3); each retry is reported. Then the exporter gives up: the trace is reported as a severe error and moved to TAILWISE_EXPORTS_DLQ with the exporter's name, from where it can be replayed. An error a retry cannot fix (corrupt stored spans) is given up at once. Export failures never send the trace to the trace dead-letter stream: it was scored, only its delivery failed.

Shutdown. With the embedded server and no queue.store_dir, the engine first sends what is queued, within 10 seconds, since the queue goes away with the process. Otherwise it stops after the sends in flight and the rest stays queued for the next start.

What the back ends receive

The engine writes OpenInference attributes, which Phoenix reads natively and Langfuse and other OTLP back ends show as span attributes, plus its own eval.* and tailwise.* attributes.

A kept trace that arrived over OTLP is forwarded as its own spans: same trace ID, span IDs, names, timings and attributes. Two things change: the exporter's resource_attributes are set on every resource (replacing attributes of the same name), and the root span gains the decision attributes below. The agent's own service.name is kept.

A kept trace from the JSON API has no spans of its own, so the engine builds them, as a new trace with a generated trace ID. Resource: service.name=tailwise plus the exporter's resource_attributes. A root span agent_run, and one child span per tool call, named after the tool.

Root span attributes:

Tool span attributes (rebuilt traces only; forwarded tool spans are not changed):

Resource attributes worth setting per back end:

Rebuilt and forwarded spans alike are sent in one request per trace, with a 10-second timeout. A failed send is retried by the export queue only (exporters[].max_deliver and backoff), not inside the request.

Metrics

With observability.metrics.endpoint set, the engine sends its own OpenTelemetry metrics over OTLP/HTTP every interval, to a Collector or any back end that takes OTLP metrics:

observability:
  metrics:
    endpoint: http://otel-collector:4318/v1/metrics
    headers_env: METRICS_OTLP_HEADERS     # optional, "key=value,..."
    resource_attributes:
      deployment.environment: production

tailwise.agent is bounded: an agent with its own model under agents.models appears by name, a trace naming any other agent as other, and one naming none as none. Agent names come from the traces, so any client could otherwise create unlimited metric series. The decision log uses the same label; the trace's own agent name stays on its exported spans (eval.agent) and in its /v1/traces:score response, cut to 128 bytes.

A failed request or call also carries error.type: the status code (5xx for HTTP, any non-OK gRPC code), or the error kind (unavailable, timeout, rejected, ...) for model calls and sends. An error rate is the count with error.type over the total count; a latency percentile comes from the histogram. The resource has service.name=tailwise plus resource_attributes. As with exporters, the global OTEL_EXPORTER_OTLP_HEADERS is not read. The last metrics are sent on shutdown.

Deploying

Run the engine inside your private network (a VPC), next to the agents that send it traces, and control access there: security groups or firewall rules, a Kubernetes NetworkPolicy, and IAM roles or service accounts for whatever the engine itself calls. The engine does not authenticate clients on its JSON API or OTLP receivers, and serves them without TLS; put a load balancer or service mesh in front of it if traffic must be encrypted or authenticated in transit.

What leaves the network is what you configure: exporters (with their own headers), the metrics endpoint, and a hosted scorer such as Jev, which receives every scored trace's text.

On Kubernetes, install with the Helm chart in deploy/helm/tailwise: the engine, an optional NATS JetStream cluster, and an optional trace-routing gateway for several replicas with queues on embedded NATS. Kustomize can install the same chart.

Before a rollout, tailwise -check -config my-config.yaml (or plain tailwise -check for the built-in defaults) loads and validates the configuration and its manifests, checks the secrets it names are set, prints any warnings, and exits: 0 if valid, 1 with the problems if not.

Known limitations

These are open problems in the current version, not design choices to rely on.

• Dead letters are not replayed automatically. A Jev outage longer than the redelivery window (about 8 minutes with the defaults) sends every queued trace to TAILWISE_TRACES_DLQ, and they stay there until someone replays them by hand (see Redelivery and the dead-letter stream). Nothing alerts on the dead-letter count either; watch for trace dead-lettered log lines or run make dlq.
• A model that rejects everything causes silent random decisions. A 400, 413 or 422 is treated as that trace's problem and gets the random fallback. If the model starts answering that way for every request (for example after a change to its request schema), every trace is decided at random, about 98% dropped with the default random_floor, and only WARN log lines show it.
• Decisions use the model's probabilities as they come. Nothing tunes them to your agent, and any of six questions can keep a trace, so the defaults may keep too much at first. Watch the share of traces kept, and which flags keep them, and adjust decision.thresholds or the any_of list.
• Severity ranks traces but nothing acts on it yet. Exporters send kept traces in the order they are decided, not by severity, and nothing keeps a fixed share such as the top 5%: the keep decision is still each check's threshold. Sort by eval.severity in your back end to review in order.
• Every scored trace is a billed call to Jev, and leaves your network. Nothing caps the spend or redacts fields before sending; only server.workers limits the rate.
• An exporter can lose kept traces. A trace skipped because the exporter's queue was full is not sent to that back end. One it gave up on after max_deliver tries waits in TAILWISE_EXPORTS_DLQ.
• The embedded server without queue.store_dir loses queued work on a crash. Traces waiting to be scored, spans waiting to be assembled and traces waiting to be exported are in a temporary directory. Set queue.store_dir, or use an external NATS (queue.url), where that matters.
• Plain instrumentor spans do not reproduce the training slice exactly. See Matching what the model was trained on. The test agent does not set the tailwise.* attributes yet, so its traces are scored on a slightly different slice than the model saw in training. On the one run compared, the raw score moved from 0.259 to 0.266.
• Trace completeness is a guess. A trace is assembled after otlp.idle_timeout of silence. Spans that arrive later are assembled as a separate, partial trace, which is normally unscorable and discarded; the first decision is not revisited. An agent run that pauses for longer than the idle timeout between spans is split this way.
• The API and the OTLP receivers have no authentication or TLS. Anyone who can reach 8080, 4317 or 4318 can submit traces. This is by design; see Deploying.
• A trace can be scored and exported twice. Delivery is at least once and nothing deduplicates: a worker that dies between exporting and acknowledging leaves a duplicate in the back ends. The same happens if a client retries a POST /v1/traces that had in fact succeeded.
• The compose file's NATS has no authentication. It is there to try the engine, bound to loopback. In production, point queue.url at a NATS you run or buy, with queue.auth, queue.tls and queue.replicas (see NATS in production).

Shutdown

On SIGINT or SIGTERM the engine stops accepting requests and OTLP exports, stops pulling from the queue, and lets in-flight traces finish for up to server.shutdown_timeout. Spans not yet assembled and traces still queued stay in JetStream for the next start. If the timeout runs out, in-flight model calls are cancelled and those traces go back to the queue; they are not dead-lettered and get no fallback decision. Last, the exporters send the traces still queued for them and shut down (up to 10 seconds in all).

Contributing

See CONTRIBUTING.md for the code layout, tests, the golden slice files, commit conventions, releases and every make target.