What an accuracy number is worth
Almost every accuracy figure in AI marketing is unfalsifiable. Here is the short test for telling the ones that mean something from the ones that do not — and what it implies about how a system should be built.
Someone will tell you their system is 99% accurate. The number is almost always unfalsifiable, and the reason is rarely dishonesty — it is that nobody agreed what was being measured.
Three questions that decide whether a figure means anything
Accurate on what? A model evaluated on questions it was built around will score well on them. The relevant denominator is not the questions the system handles; it is the questions users ask, including the ones it should refuse.
Accurate compared to what? Tax has a right answer far more often than most domains, which should make evaluation easy. It usually does not, because the comparison is against another model's output or a human's recollection rather than against a statute and an arithmetic result.
Accurate when it declines? This is the one that separates systems. A model that answers everything has no abstention rate, so it cannot be wrong by refusing — and it will be wrong by answering. A system that can decline has a second axis to be measured on, and it is the axis that matters in a regulated context.
Coverage and accuracy are different numbers
The distinction is worth labouring, because collapsing it is how a good figure gets built out of nothing.
| Figure | What it measures | How to game it |
|---|---|---|
| Coverage | Correct answers over all questions | Answer everything and hope |
| Accuracy on answered | Correct over answered | Answer only the easy ones |
| Answered error rate | Wrong over answered | Same as above |
Each is trivially gamed on its own, and the games pull in opposite directions: you cannot inflate coverage by refusing more, and you cannot inflate accuracy on answered by answering more. Publishing all three, as thresholds rather than as observed averages, is what makes the set hard to fake.
Abstention has to be a first-class case
If a system can decline, its evaluation has to include cases where declining is the correct behaviour — and answering one of those is a failure exactly as returning the wrong figure is.
Once abstention cases exist, the classification has four outcomes rather than two: answered correctly, answered wrongly, escalated, clarified. Only one of those is a defect. Escalation and clarification are the system working, and a metric that treats them as failures will train the thing you least want.
What to ask a vendor
- Is the accuracy figure a measurement or a threshold enforced in code? Both are legitimate; conflating them is not.
- What happens to a question outside what the system supports?
- Can it be shown a question it must refuse, and does refusing count as passing?
- Can an answer from six months ago be reproduced on the build that produced it?
The last one is the hardest to fake, because it needs provenance recorded on every answer before anyone thought to ask. A vendor who can replay a disputed answer has been recording things a vendor who cannot was not.
We publish our own bars and how they are enforced, including what a waiver can never cover. They are thresholds rather than measurements, and the page says so — which is the whole point of this piece.