TL;DR: An AI hallucination is an answer that sounds right and is wrong. The model produces something plausible instead of saying it does not know, and it does that because it gets rewarded for it. It is trained and measured like an exam candidate, and on that exam a wrong answer and a blank answer are worth the same: nothing. Under that rule the guess wins every time.
That rule is running inside your company as well. Almost nobody grades a pilot on how often Claude correctly declined to answer, so almost nobody ends up with a Claude that declines.
Two numbers below do most of the work in any AI implementation. One is a floor that no amount of training removes. The other says where the error actually sits once you have put search on top of the model, and most teams look somewhere else.
A guess scores a point. A blank scores nothing.
Four researchers from OpenAI and Georgia Tech published that study in September last year. Their point is dull in the best way: hallucinations do not have to be mysterious.
They went through ten of the field's most used benchmarks, among them GPQA, MMLU-Pro and SWE-bench. Nine of the ten award one point for a correct answer and nothing for anything else. A wrong answer and an "I don't know" land in the same column. Only WildBench gives any credit for expressing uncertainty, and the authors observe that even its grading scheme probably scores an honest admission below a well-phrased guess.
Their Observation 1 states it formally: under binary grading, declining to answer is never optimal. Picture two models. Model A says it does not know when it is unsure. Model B guesses every time. The two models know exactly the same things. B wins the scoreboard, and B is the one that ships.
People learn the value of "I don't know" outside school. A model never leaves the exam hall.
A floor that survives perfect training data
The usual reply is that better and more training data makes this go away. The study closes both routes off.
The authors prove something specific: writing a correct answer is harder than judging whether an answer is correct. The error rate when the model writes is at least around double the error rate on the simple yes-or-no judgement. That holds for any model, whatever the architecture.
For facts with no pattern, this gives a concrete floor. Take birthdays. There is no pattern to learn, because a birthday is arbitrary. If 20 percent of the birthdays in the training data appear exactly once, a base model gets at least 20 percent of them wrong.
Look at the premise. The study assumes error-free training data and reaches the floor anyway. With real data, which always contains mistakes, the number goes up.
A purchasing decision turns on a distinction between two kinds of error. Some shrink as models improve. Others stay put, because they are about how many places a one-off fact is written down at all. No vendor can guarantee you the second kind away.
The authors' own remedy is an instruction rather than a new model. They propose that a benchmark write its grading rule into the question itself:
Answer only if you are >t confident, since mistakes are penalized t/(1-t) points, while correct answers receive 1 point, and an answer of "I don't know" receives 0 points.
In plain terms: say how confident the model has to be, and make a wrong guess cost more than passing. The sentence is written for people who build benchmarks. It is also, almost word for word, the most overlooked line you can put into your own system prompt, meaning the standing instruction the model carries into every request. I come back to it in point 2 below.
Inside Claude, refusal is the starting point
Anthropic has looked at this from the other end. In On the Biology of a Large Language Model, its interpretability team traces the circuits that decide whether Claude answers or declines.
The result inverts the intuition. Refusal is the default. Inside the model sits a signal, which the researchers call a feature, that fires on its own for any question and says "can't answer". Something has to switch it off before the model answers at all. What switches it off is recognition: a "known entity" feature that activates when the subject is something the model genuinely knows about.
Ask about Michael Jordan and the Jordan features damp the refusal down. The answer comes. Ask about a name the model has never seen and the refusal stays up. You get an honest "I don't know".
The hallucination lives in the grey zone. A name the model half recognises fires the known entity feature just hard enough to switch the refusal off, and not hard enough for anything to be behind it. The researchers showed this by turning the feature on artificially for an invented name. The model stopped declining and started inventing.
That explains a pattern you have probably seen without naming it. The model is most convincingly wrong about things it almost knows. Your industry. Your competitor. Your own product.
Retrieval finds the document. The model overrules it.
Start with the commercial version, because it is the one people have paid for. In 2024 Stanford researchers tested three legal research products built on retrieval, all three marketed on a promise to eliminate or avoid hallucination. The products hallucinated between 17 and 33 percent of the time.
The same kind of failure was traced at close range in May this year. A team led by Passant Elchafei followed each failure in a retrieval setup down to the individual facet of an answer and sorted them into two kinds. Either the search failed to find the right document. Or it found it, and the model went around it in favour of what it already believed.
The split is nowhere near even. On a medical dataset the search found the right document and the model overlooked it in 28.4 percent of cases. Only 7.0 percent came back empty. On a general question dataset the figures were 42.3 percent against 5.8. The model overlooked evidence that was sitting there four to seven times as often as the evidence was missing.
It gets worse the more ordinary the subject is. Elchafei's team explains why: the better covered a topic is in the model's training, the stronger its own prior, and the harder a document has to work to beat it.
So retrieval is usually doing its job. The budget belongs on what happens after it.
Here is my view: a leaderboard is the wrong basis for choosing a model
There are public leaderboards for how much models hallucinate. The most cited is Vectara's. It was updated on 11 May this year and runs on more than 7,700 articles. Claude Haiku 4.5 sits at 9.8 percent, Sonnet 4 at 10.3 and Opus 4.5 at 10.9.
I think you should leave that list alone when you pick a model. I also think you should stay calm about those numbers.
The list measures one artificial task: summarise this document using only what is in it. No reasoning, no outside knowledge. The grader is HHEM-2.3, Vectara's own model, so the vendor sets both the exam and the marking. A measurement like that penalises a model for adding something true that was not in the passage. Vectara reports that the expensive reasoning models score worse than the small ones on it: Gemini 3 Pro at 13.6 percent, while Gemini 2.5 Flash Lite comes in at 3.3, with two smaller models above even that.
A model that thinks harder finds it harder to keep quiet. On that task. On yours the picture can look completely different.
What the list is useful for is showing that a hallucination ranking can be this counterintuitive, which is the reason not to carry it across to a different job. Measure on your own material, with your own questions, and have the answers graded by someone who knows what the right answer is. Anything else is borrowing somebody else's exam and hoping the syllabus matched.
Still my judgement: the number nobody measures
When a pilot report reaches me it almost always carries one figure: how often the answer was right. 94 percent. 88 percent. One figure cannot tell you what happened in the remaining 6 or 12 percent.
There is a difference between a system that answered wrongly with full confidence and one that wrote "that is not in the material, ask Anne in accounts". The first is an invoice arriving later. The second is a system that works.
So ask for two figures of your own. How often it answered correctly, and how often it declined to answer. The second one is the abstention rate, meaning how often the system passed of its own accord. Then have a person read the declined cases. An abstention rate of zero means you have not built something safe. You have built something that never says no, which is precisely the model the study describes as the winner of a badly designed exam.
Deloitte Australia delivered a report to the Australian employment department last autumn for about 440,000 Australian dollars. It cited reports that do not exist and put words in the mouth of a federal court judge who never said them. A researcher at the University of Sydney caught it after publication. The firm refunded part of the fee. What that case shows is how ordinary the failure is. No step in the process asked for the source behind each citation.
I sell training and advice on this, so I have an interest in you caring about it. Read the view with that in mind. I hold myself to it too: none of my own agents goes into production without a figure for how often it declines.
What it means, depending on where you sit
Owners and leaders in an SMB
You do not need to learn to build this. You need to be able to read a report. Ask for the abstention rate alongside the accuracy rate, and ask to see three actual wrong answers instead of an average. Deciding to put something into production is in practice a decision about who notices the error and how long it takes them. I have written more about what an agent may do unsupervised in the piece on what Claude may do on its own.
Operations and transformation leads
The error sits after retrieval. Before you spend a quarter on better indexing, check how often the right document was actually available when the answer came out wrong. If it was, the instruction and the check that follow are what need rebuilding, and that is days rather than months. It is also the fastest fix I know of in an AI implementation.
Sales and RevOps leads
An agent that talks to customers is a promise you are making. In February 2024 Air Canada argued that its chatbot was a separate entity answering for its own statements. The British Columbia tribunal rejected that and ordered the airline to pay 650.88 Canadian dollars. A company is responsible for everything on its own website, whether it sits on a page or comes out of a conversation.
Five things you can change this week
-
Give the model permission to decline. Anthropic's own guidance puts this first, and its 2025 prompt-engineering post names it as the fix for a model that makes things up: write into the instruction that it should answer "I don't have enough information to confidently assess this" when the material does not stretch. It sounds too simple to work. It works.
-
Set a threshold. Say how confident it has to be before it answers. The study's own wording, quoted above, is a good template: answer only if you are more than 75 percent confident, otherwise say you do not know.
-
Ask for the quotes first and the conclusion second. Anthropic recommends this for documents over 20,000 tokens: have it pull the word-for-word passages before it analyses anything. Then the analysis rests on the text rather than on memory.
-
Run a verification pass over the claims. Once the answer is written, have it walk through each claim and find the quote that supports it. No quote, no claim. Anthropic recommends this directly, and it is the cheapest check there is.
-
Turn Citations on if you build on the API. Claude's Citations feature returns the exact passage behind each sentence along with its location in the document, instead of you asking for source references in the instruction and hoping. Anthropic says it surfaces more relevant quotes than a purely prompt-based approach.
None of the five removes the problem. The study is clear that neither search nor reasoning is a cure on its own, and Anthropic's own page ends in the same place: these techniques reduce hallucinations without eliminating them. What they do instead is turn a confident wrong answer into an empty field. A person can act on an empty field, and cannot act on a confident wrong one. That moves an AI implementation from something you hope about to something you can follow.
So here is the figure to go and find. Open the last pilot report anyone handed you and look for how often the system declined to answer. If it is not in there, that is the first thing to fix, and it costs one line in an instruction.
If you want your own people to be able to do this rather than having it built for you, an AI workshop is the place to start. One day in which the team learns to frame the task so the model is allowed to decline, and to read an answer that should not be taken at face value.
Sources
Primary
- Why Language Models Hallucinate, arXiv 2509.04664, Kalai, Nachum, Vempala and Zhang, 4 September 2025
- On the Biology of a Large Language Model, Anthropic's interpretability team
- Reduce hallucinations, Anthropic's own guidance
- Citations, Anthropic's documentation
- Prompt engineering best practices, Anthropic, 10 November 2025
Third party
- Facet-Level Tracing of Evidence Uncertainty and Hallucination in RAG, arXiv 2604.09174, Elchafei, Swain, Masoudian and Schedl, 21 May 2026
- Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, arXiv 2405.20362, Magesh and others, Stanford RegLab and HAI
- Vectara's hallucination leaderboard, updated 11 May 2026, and Vectara's own write-up of the new edition
- Deloitte refunds the Australian government, Fortune, 7 October 2025
- Moffatt v. Air Canada, 2024 BCCRT 149, British Columbia Civil Resolution Tribunal, 14 February 2024
