Accuracy you can check, not accuracy we assert.
Coverage at or above 90%, accuracy on answered questions at or above 99%, answered error at or below 1% — enforced as code constants.
These are gates, not averages.
They are held as code constants and enforced by the release gate. A build that misses one does not ship — the gate CLI exits non-zero. None of them is a measured average from a fleet, and this page never presents them as one.
- Release gate
- at or above90%Coverageanswered correct ÷ total
- Release gate
- at or above99%Accuracy on answeredanswered correct ÷ answered
- Release gate
- at or below1%Answered error rateanswered wrong ÷ answered
The classifier reads the live certainty gate's own output, so the measurement is of what production actually does.
A case can assert that the system must refuse.
Abstention cases are first-class. A golden case may state that the correct behaviour is to abstain — and then answering it is a failure, exactly as returning the wrong figure would be. An evaluation suite without that concept can only ever reward a system for saying more.
Every golden case is classified as answered-correct, answered-wrong, escalated or clarified — and an answer that shipped wrong is a P0.
The shape the release gate permits at its exact boundary, derived from three code constants. It is not a measured result, and TaxOrch does not publish one. Answered correct: 90%. Answered wrong: 0.91%. Escalated or clarified: 9.1%.
- 90%Answered correctCoverage is answered-correct over total, and the bar is 90%.
- 0.91%Answered wrongAnswered error is capped at 1% of answered cases. It is a P0 and the metric driven toward zero.
- 9.1%Escalated or clarifiedWhat the system declined to answer. The gate constrains the size of this band, not how it divides — so neither does this drawing.
The shape the release gate permits at its exact boundary, derived from three code constants. It is not a measured result, and TaxOrch does not publish one.
Every golden case lands in exactly one of four outcomes
- Answered correct
- The system produced a figure and the figure was right. The only outcome that counts toward coverage.
- Answered wrong
- The system produced a figure and the figure was wrong. A P0 — the one outcome the architecture exists to prevent.
- Escalated
- The certainty gate sent it to a human rather than shipping it. Not a failure; the design working.
- Clarified
- The system returned a specific question instead of a figure, because a fact it needed was missing.
Ten categories, four verdicts, and no heuristic pass.
With no active corpus release — or a required-but-unavailable vector backend — the verdict is blocked. Not a warning, not a degraded pass. A system that guesses when its evidence is missing is the failure mode this product is built against, and the gate is held to the same rule as the assistant.
Demonstration run "A passing run": overall verdict Pass. Every category cleared against pinned models, corpus and rule packs.
- CalculationDeterministic engines produce the expected figures.Pass
- ExtractionDocuments yield the fields the pipeline depends on.Pass
- RetrievalThe right in-force passages come back for the question.Pass
- End to endThe whole pipeline, question to contract.Pass
- AbstentionThe system declines where it is supposed to decline.Pass
- Golden accuracyThe golden set's answered cases return the right numbers.Pass
- Golden abstentionThe golden set's must-abstain cases actually abstain.Pass
- Source freshnessNo registered source has gone stale past its update cadence.Pass
- Privacy isolationTenant data stays inside its tenant.Pass
- Latency and costThe run stays inside its performance and spend envelope.Pass
These three runs demonstrate documented gate behaviour. They are not records of real runs — per-category results are not published, and a screenshot of a green board would prove nothing about the next build.
- Pass
- Every category cleared. The build proceeds.
- Fail
- A category did not clear. The gate CLI exits non-zero and the build stops.
- Blocked
- The gate could not evaluate — no active corpus release, or a required vector backend unavailable. There is no heuristic pass.
- Waived
- A recorded, audited exception. Never available for figure fabrication or a live-country bar.
Two things a waiver can never cover.
A waiver requires a recorded reason and is audited. It is a deliberate, attributable decision — not a flag someone can set to make a red board go green.
Gate waivers are one of the event types written to the audit log, alongside model route changes, case finalisation and deletion.
- Never waivable
Figure fabrication
A number that does not trace to a calculator output or cited text. There is no reason good enough, and no signature that unlocks it.
- Never waivable
A live-country bar
The threshold a jurisdiction must clear to be exposed to users. A country cannot be waived into coverage it has not earned.
10 gates behind one live verdict.
Ready, not-ready or degraded, with per-gate remediation text. A fresh environment correctly returns not-ready: the system is fail-closed by construction, so nothing works until it has been made to work.
- 01Active corpusA validated corpus release is promoted and active.
- 02Retrieval backendThe vector service is reachable, or explicitly not required here.
- 03Calculators and coverageCalculators load and coverage profiles are seeded.
- 04Release gate passedThe most recent release gate returned pass or waived.
- 05Source healthRegistered sources are healthy and not stale.
- 06Model healthRouted models report healthy.
- 07Claim verifier enabledClaim-level verification is switched on.
- 08General-knowledge fallback disabledThe unsafe fallback path is off.
- 09Backup and restoreBackup is provisioned and verified.
- 10Monitoring and alertsMetrics and alerting are provisioned.
An answer from six months ago can be put back on trial.
Answers are pinned to a corpus release, model version, and calculator version, so any answer can be reproduced later.
Most systems answer “the product gave a wrong answer” with an apology and a prompt change. This is the runbook instead, and it only works because every answer pins its own provenance.
- 1
Capture the pinned versions
From the answer contract itself: model, corpus release, rule pack, calculator version, prompt version, commit.
- 2
Reproduce it
Replay the answer through the eval runner using those exact versions. Not a similar question — that question, on that build.
- 3
Roll back
Restore the prior corpus release or prompt version. Promotion is immutable and rollback restores the previous release and its index, so a bad release cannot stay active.
- 4
Add a golden case
Encode the correct expected behaviour. If the right behaviour was to abstain, the case asserts abstention — and answering it becomes a failure.
- 5
Re-run the gate
Confirm it passes with the new case in the set. The defect is now a permanent regression guard rather than a fixed ticket.
The defect leaves the process as a golden case rather than as a closed ticket. It cannot come back without failing the gate first — which is the difference between fixing a bug and fixing a class of bug.
Everything here is meant to be verifiable.
The coverage matrix, the evaluation results and the answer contract are all published, including what they do not cover. Read them before you talk to anybody.
Bring the question you expect it to get wrong.
TaxOrch provides decision support, not professional tax advice. Exact results apply only within declared coverage. TaxOrch does not file returns or execute payments. Review all outputs before filing.