ReqTrace · routing dashboard
Which model should do which step? Quality is measured first; cost only counts for work the eval says is safe to move. Fictional Novapump NP-200 data, all documents synthetic.
Scorecard: three ways to run the same pipeline
Same documents, same prompts, same answer key. Only the routing changes. Measured from each run's log.
Planted defects caught
One column per defect in the answer key.
Where the work happened
Output tokens by step (output is the expensive side of an API bill).
What a bigger document would cost est.
Takes the measured token profile of the demo SRS and scales its content to your page count. Fixed prompt overhead is charged per call, and long inputs are split into more calls. Prices are illustrative: change them.
Out-of-process edit: caught by two independent layers
Someone edited the legacy test protocol after sign-off. A hash comparison flags it; separately, the AI draft step refuses to reuse the edited step.
Routing policy, decided by evidence
A step runs locally only if the ledger shows it is safe.
How to read this
- Measured numbers come from
llm_log.jsonl. The "list cost" is the per-call figure the Claude Code CLI reports itself. - Projected numbers use
pricing.json(illustrative rates) and a straight-line scale-up. They ignore prompt caching, batch discounts and negotiated rates, and they assume the local model keeps its quality on a longer document, which the eval above only demonstrates at demo size. - Local tokens are free per token. The cost shown for them is electricity at the machine wattage in
pricing.json. The real price of local is time: see the "local model time" cell above. - Local token counts use a different tokenizer from the hosted models, so token counts differ a little between columns.