Executive summary
Large language models generate text by predicting likely next tokens from patterns in their training and in the material they are given. That process can produce a real reference or a plausible invented one in equally fluent language, and research finds the confidence in a model's wording is a poor guide to whether it is right. This piece explains where that confidence comes from, a practical way to judge which tasks need the most checking, and three habits that help a team decide what to check before work leaves the firm.
A summary of a long document comes back from an AI tool fluent and well organised. Six of its points are right. The seventh is wrong, and it reads in the same steady voice as the other six.
People who use these tools every day tend to know that errors happen, and many have built their own ways of catching them. This piece looks at why a wrong answer can arrive in the same tone as a right one, and what that means for deciding what to check.
1How a model writes
A large language model writes by prediction. The Office of the Australian Information Commissioner (OAIC) describes generative AI as working in a way "in which the next word, sub-word, pixel or other medium is predicted based on likelihood". The units a model predicts are called tokens, and a token can be a whole word, part of a word or a punctuation mark. Given the text so far, the model predicts a likely next token, then the next, drawing on patterns in its training and on any material it has been given. Every word it hands you is produced that way, including the ones that happen to be facts.
Illustration of next-token probabilities
Possible ways to finish "The contract was signed on 14 ___". Hover a bar to try another word.
"The contract was signed on 14 March"
The US National Institute of Standards and Technology (NIST) uses the word confabulation in its Generative AI Profile, defining it as "a phenomenon in which GAI systems generate and confidently present erroneous or false content in response to prompts". NIST notes that some commenters see "hallucination" as giving AI human characteristics it doesn't have. Our AI glossary uses the more common word, and both describe the same thing: plausible content that turns out to be false.
2Confidence in the wording
The confidence in a model's wording and the chance that its answer is correct are separate measures. Researchers also treat a third one separately: the model's own internal uncertainty about an answer.
In a 2025 study presented at EMNLP, a major conference on language processing, researchers found that language models "often use assertive language when conveying false claims". They tested whether the uncertainty models express in words matches their internal uncertainty, across a range of models and datasets, and found that the models "largely fail at this task". Standard prompting improved the match only marginally.
For the reader, how sure an answer sounds is a poor guide to whether it is correct.
3Two references
The Law Society of NSW describes one form of hallucination as a tool producing "a reference that appears legitimate, but is in fact fictitious". Its guide adds that "the AI models currently in use cannot properly identify when the output they are returning is factual."
One of the two references below exists, and we made the other up for this article.
Which one exists?
Pick the reference you think is real.
The invented one has the right shape: plausible author names, a plausible year, a journal title that sounds like a journal, and a subject close enough to the topic that a reader looking for support might accept it on a quick read.
4How the scoring works
Models are compared on scoreboards as they are developed. OpenAI's researchers are careful about what that does: "While evaluations themselves do not directly cause hallucinations, most evaluations measure model performance in a way that encourages guessing rather than honesty about uncertainty."
In their example, a model is asked for the birthday of a person it has no information about. If it guesses a date, it has a one in 365 chance of being right. A model that says "I don't know" scores zero every time. In OpenAI's words, "over thousands of test questions, the guessing model ended up looking better on scoreboards than a careful model that admits uncertainty."
Expected score over one million questions
Expected points for a model that guesses compared with one that says "I don't know".
Scored on accuracy only.
Right = 1 point. Wrong or "I don't know" = 0.
Confident errors penalised.
Right = 1. Wrong = minus 1. "I don't know" = 0.
The full paper and OpenAI's summary of it argue for a change to the scoring. OpenAI's advice is to "penalize confident errors more than you penalize uncertainty, and give partial credit for appropriate expressions of uncertainty."
Some benchmarks already score that way. Stanford's 2026 AI Index reports results from AA-Omniscience, a benchmark by Artificial Analysis that asks 6,000 open-ended knowledge questions across six domains, including law and health. It penalises wrong answers and applies no penalty for refusing to answer. On that benchmark, hallucination rates across 26 models ranged from 22% to 94%. The figures describe performance on that benchmark only, and they are separate from how often each model is wrong in everyday work.
OpenAI also reports that its GPT-5 model "has significantly fewer hallucinations especially when reasoning, but they still occur."
5Who supplies the substance
One practical way to decide how much checking a task needs is to ask who supplied the substance. The table below is our own framework for that. The further down it a task sits, the more of the substance tends to come from the model, and the harder it is to judge while you read.
Who supplies the substance (a Konvey framework)
From tasks where you supply the substance to tasks where the model does. Filter by who supplies it, or hover a row.
| What you're asking for | Who supplies the substance | Exposure | What needs checking |
|---|---|---|---|
| Tightening something you drafted | You | Low | That the meaning hasn't changed |
| Changing the register of an email | You | Low | That the tone suits the reader |
| Summarising a document you've read | Shared | Medium | Any point the final work relies on |
| Summarising a document you haven't read | The model | High | Everything the decision rests on |
| The position under a standard or in a jurisdiction | The model, or shared if you supplied the source material | High to very high | All of it, at source |
| Supporting references, figures, authorities | The model | Very high | Every one, individually |
| Counts, totals, date arithmetic | The model | Very high | Every number, by a person or a suitable tool |
6Counts and dates
The OAIC uses counting to show how this goes wrong. Without protective measures, it says, a model asked how many b's are in banana "will generally state there are two or three 'b's in banana", partly because of the way it splits text into tokens. Tokenisation helps explain some letter-counting failures. More broadly, language models can also make mistakes with counting and arithmetic unless the task is handed to a suitable tool, such as a spreadsheet, a calculator or a short piece of code.
Nearly right
Watch the model count, then check it yourself. Illustrative figures.
Items on a schedule.
Days between two dates.
Mon 2 MarchWed 18 March
The count arrives in the same even tone as the rest of the answer, and a figure that is nearly right can pass a quick read and go out with the work.
AI Uplift Program
See how your firm checks AI output
The investigation looks at how work drafted with AI is checked before it leaves the firm, and measures how long your everyday work takes today. You keep the findings either way.
7Three habits
Three habits help a person decide what to check.
Checking at source what will be relied on
Names, figures, dates, references, quotations, and any content a client or third party will act on. In the 2025 Victorian Lawyer Census, AI users who answered the quality assurance questions reported their checks: 76% compared the legal accuracy of output with other sources, and 50% checked specifically for hallucinations. For tax practitioners, the Tax Practitioners Board says they "should verify and review AI generated content for accuracy throughout each step of the workflow". Who does that checking in a firm is the subject of Are you the human in the loop?
Asking it to point to the source
If the model is working from a document, ask it to identify the passage that supports each important claim, then open that passage yourself. A passage or citation the model gives you is something to verify, and the claim stands only if the passage says what the claim says. Some lawyers in the Victorian census described prompting AI for sources or citations, "asking AI to show sources", to better assess reliability.
Knowing which model is in use and what it can reach
A model given your document, or one that can run a live search, has material it can point to, which makes its claims easier to check, though it can still be wrong. People who are unsure whether a model has that material may trust its answer as though it does.
8Where we come in
A firm's AI policy sets which tools, which accounts and what information goes where. Whether a particular figure or reference can be relied on is decided by the person checking the work, and that judgement is something the whole team can learn.
What we often see is that people understand this to different degrees, because each person picked it up in their own way, from what they read and the tools they use. A person in accounts payable may have as sharp a sense of it as a senior associate.
One way to see where your firm sits is to take a small sample of recent work that was drafted partly with AI and check the specifics: the names, figures, dates and references.
Our investigation looks at how AI-assisted work is checked across your firm today and measures how long your everyday work takes, so you start from a baseline of your own. Then comes AI Foundations, the short training everyone in the firm completes, and cohort workshops for each team or skill level. The findings belong to your firm whether or not we go any further together.
For a clear picture of what gets checked, and by whom, across your firm, reach out to our team. The first conversation works out whether your firm is the right fit for the AI Uplift Program, and whether we are the right people to help.
Before AI-assisted work goes out
Frequently asked questions
What is an AI hallucination?
Plausible content from an AI tool that turns out to be false, such as an invented case, figure or quotation. The Law Society of NSW notes that the models in use "cannot properly identify when the output they are returning is factual."
Does a confident answer mean the model is sure?
The confidence in the wording is a poor guide. Researchers presenting at EMNLP in 2025 found that models "often use assertive language when conveying false claims", and that the uncertainty they express in words matches their internal uncertainty poorly.
Is hallucination the right word for it?
NIST uses confabulation, which it defines as a phenomenon in which AI systems "generate and confidently present erroneous or false content". It notes that some commenters see "hallucination" as giving AI human characteristics. Both words are in common use.
Does this happen less with a better model?
OpenAI reports that GPT-5 "has significantly fewer hallucinations especially when reasoning, but they still occur." On the AA-Omniscience benchmark reported in Stanford's 2026 AI Index, hallucination rates across 26 models ranged from 22% to 94%, figures that apply to that benchmark only.
Which tasks carry the most risk?
In our framework, the ones where the model supplies the substance: summarising a document you haven't read, stating a position under a standard, producing supporting references, and counting or date arithmetic.
About the author
Julia works with professional services firms on how AI actually gets used across the firm, from what people are already doing with it through to the position the firm takes on it.
Primary sources
Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang, Why Language Models Hallucinate, arXiv:2509.04664, 4 September 2025arxiv.orgOpenAI, Why language models hallucinate, 5 September 2025openai.comGabrielle Kaili-May Liu, Gal Yona, Avi Caciularu, Idan Szpektor, Tim G. J. Rudner and Arman Cohan, MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs, Proceedings of EMNLP 2025, November 2025aclanthology.orgStanford Institute for Human-Centered Artificial Intelligence, The 2026 AI Index Report, chapter 3, Responsible AI (AA-Omniscience, 26 models, 6,000 knowledge questions), April 2026hai.stanford.eduLegal Services Research Centre, Victorian Legal Services Board and Commissioner, Generative AI Use in the Legal Profession: Findings from the 2025 Victorian Lawyer Census, 1,887 Victorian lawyers responding, quality assurance figures among AI users, 20 April 2026lsbc.vic.gov.auTax Practitioners Board, TPB(GS) 55/2026 The use of Artificial Intelligence and the Code of Professional Conduct, 22 July 2026tpb.gov.auThe Law Society of NSW, A Solicitor's Guide to Responsible Use of Artificial Intelligence, January 2026lawsociety.com.auOffice of the Australian Information Commissioner, Guidance on privacy and the use of commercially available AI products, updated 17 January 2025 (foundational)oaic.gov.auNational Institute of Standards and Technology, NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024 (foundational)nvlpubs.nist.govGeneral information about how firms organise their AI use. It is not legal, tax or professional advice.