Generative AI tools can produce fluent, confident answers that are simply wrong. This article explains what that failure is, how the US National Institute of Standards and Technology (NIST) treats it, and what a team can do about it in daily work.
What does it mean when an AI system “hallucinates”?
A hallucination is output that sounds authoritative but is invented or false: a court case that never existed, a statistic nobody published, a feature a product does not have. The system is not lying in the human sense. It has no intent. It is producing text that fits the pattern of a plausible answer.
The term is everywhere in tech coverage, but it is a loose metaphor. A person who hallucinates perceives something that is not there. A language model does something different: it generates the most likely continuation of a prompt, and sometimes the most likely continuation is false. For a primer on how these systems work, see our explainer on what generative AI is.
Why NIST prefers the word confabulation
NIST, the US standards agency, published a Generative Artificial Intelligence Profile (document number NIST AI 600-1) on July 26, 2024. According to NIST’s publication page, it was last updated on April 8, 2026. The profile is a companion resource to the AI Risk Management Framework, known as AI RMF 1.0, which NIST released in January 2023 and which is voluntary.
In its announcement of the profile, NIST describes generative AI systems “confabulating or ‘hallucinating’ output” and treats the two words as equivalent. Confabulation is a term borrowed from psychology, where it describes filling a gap in memory with a made-up but confident account. Many researchers consider it a better fit than hallucination, because the model is not perceiving anything. It is filling a gap with a plausible story.
For readers, the practical point is simple: both words describe the same risk, and standards documents use the more precise one.
Where this sits among the profile’s 12 risks
NIST says the profile “centers on a list of 12 risks and just over 200 actions that developers can take to manage them.” Its announcement names three of those risks:
- a lowered barrier to entry for cybersecurity attacks;
- the production of misleading or hateful content;
- generative AI systems producing made-up output, the confabulation discussed here.
The fact that confabulation is on a list of twelve matters. It tells organizations that invented output is not a rare glitch to be handled case by case. It is a recognized category of risk, to be assessed, measured and managed like any other. The profile is voluntary guidance, not law, so no one is required to follow it, but it gives teams a shared vocabulary and a checklist-style starting point.
Why models make things up
Without going into engineering detail, a few causes are widely understood:
- Prediction, not lookup. A language model predicts likely words. It does not consult a database of verified facts unless the system around it is built to.
- Gaps in training data. Where the model saw little or conflicting information, it can still produce a confident-sounding answer.
- Pressure to answer. Systems tuned to be helpful may attempt an answer rather than say “I don’t know.”
- Vague prompts. Ambiguous questions invite the model to guess what the user meant.
These explanations are general rather than quotations from NIST, and the exact mix differs by model and task.
Why a rare error still matters
Confabulation is hard to manage because it hides well. A wrong answer written in polished prose looks the same as a right one. The risk is not only that errors occur but that people stop checking.
A short worked example shows how this scales. The numbers below are illustrative, not measured rates.
| Scenario | Value |
|---|---|
| AI-drafted customer answers per month | 2,000 |
| Share containing a material error (assumed) | 2% |
| Wrong answers per month | 2,000 × 0.02 = 40 |
| Share a reviewer spot-checks | 10% |
| Answers checked | 2,000 × 0.10 = 200 |
| Wrong answers expected among those checked | 200 × 0.02 = 4 |
| Wrong answers that reach customers | 40 − 4 = 36 |
An error rate that sounds small still means dozens of wrong answers every month. And a spot-check catches only the proportion it samples. The lesson is that the review design, not just the model, decides how many errors reach the public.
Which tasks are safest, and which are risky?
Not every use of generative AI carries the same exposure. A rough sorting helps.
Lower risk: brainstorming headlines, rewording your own text, summarizing a document you have supplied and can compare against, drafting an outline.
Higher risk: citing laws, cases or studies from memory; giving medical, legal or financial guidance; stating product specifications; producing numbers without a source; and any output that goes to customers with no human review.
A useful test is whether a wrong answer would be obvious to the person reading it. If the reader cannot easily tell, the task needs stronger controls.
How can a team reduce the risk?
There is no setting that removes confabulation, but several practices lower it and limit the damage.
Ground the system in your own documents
Instead of asking a model to answer from general memory, give it approved source material and instruct it to answer only from that material. This approach, often called retrieval-augmented generation (the system fetches relevant documents first and answers from them), reduces invented answers but does not eliminate them, because the model can still misread what it retrieved.
Require sources and check them
Ask for the passage or document behind each claim, then verify the source exists and says what the answer claims. Fabricated citations are a classic confabulation pattern, so a reference that cannot be opened should be treated as unverified.
Keep a person accountable for outputs that matter
Assign named reviewers for anything customer-facing, legal, financial or safety-related. Review should be a defined step with a record, not an informal habit.
Test before launch and monitor afterward
Build a set of realistic questions with known answers, run them before deployment, and rerun them when the model or the prompts change. After launch, sample live outputs and log complaints. Our guide to the NIST AI Risk Management Framework explains how its Govern, Map, Measure and Manage functions turn this into an ongoing routine.
Design for honest uncertainty
Tell the system, and the users, that “I could not find that” is an acceptable answer. Interfaces that show where an answer came from make errors easier to spot.
Do AI agents make the problem bigger?
Possibly. When a chatbot gives a wrong answer, a person can read it and decide what to do. When an AI agent, software that plans and takes actions on a user’s behalf, acts on a made-up fact, the error can turn into a wrong email, a wrong record or a wrong purchase before anyone sees it. Our overview of how AI agents work covers the basics. The sensible response is to limit what an agent can do without approval, and to put checks at the points where an action cannot easily be undone.
Where to start
Organizations adopting generative AI can start small. List where the tools are already used. Sort those uses by the harm a wrong answer could cause. For the high-harm group, add grounding, source checks and a named reviewer. Write the rules down, and revisit them when tools change. Broader principles are in our piece on using AI responsibly in your business.
Conclusion: treat confabulation as a managed risk
NIST’s decision to list confabulation among 12 named generative AI risks reflects a shift in thinking: made-up output is a normal property of current systems that has to be designed around, not a surprising bug. Teams that match controls to the stakes of each task, verify sources and keep humans accountable can use these tools productively while keeping wrong answers away from the people who rely on them.
This article offers general information only and is not legal, compliance or professional advice.
Sources: National Institute of Standards and Technology
