What is an AI hallucination?
An AI hallucination is an answer from an AI model that sounds authoritative but is wrong or made up: a statistic nobody measured, a court case that never happened, a command option that does not exist, or a vulnerability ID attached to the wrong product. The term is used mostly for large language models, and the danger is in the delivery. A hallucinated answer arrives in the same fluent, confident style as a correct one, so nothing in the text signals that it needs checking.
Why models hallucinate
A language model generates text by repeatedly choosing a likely next token based on patterns in its training data. It does not look facts up or test its claims unless the application gives it a tool to do so. When its training data was thin, outdated or contradictory on a topic, or the question assumes something false, the most plausible continuation can still be wrong. Models are also tuned to be helpful, which favors producing an answer over admitting uncertainty. Grounding a model in retrieved documents reduces hallucination, but the model can still misread or embellish what it was given.
Example: a confident wrong command
An administrator asks an assistant how to find Active Directory users who have not signed in for 90 days. The assistant replies with Get-ADUser -LastLogonDays 90 -Disable and explains each switch. Neither parameter exists. Run as written, the command simply fails, but a hallucinated answer to a firewall, cloud permission or hardening question can be valid syntax that does the wrong thing, such as opening a port the administrator meant to close. A working approach uses Search-ADAccount -AccountInactive -TimeSpan 90.00:00:00 -UsersOnly, checked against Microsoft's documentation and tried on a few accounts before anything is disabled.
Package hallucination and slopsquatting
Coding assistants sometimes recommend libraries that do not exist. Because many invented names come back when the same prompt is asked again, an attacker can collect them, publish packages under those names on public registries such as npm or PyPI with a malicious install script, and wait for developers to install what the assistant suggested. The tactic is called slopsquatting. A 2024 study of code from 16 models found that about one in five package names suggested by open-source models, and about one in twenty suggested by commercial models, did not exist. Unlike typosquatting, the name does not imitate a real package; the AI made it up.
Reducing the risk
- Treat AI output as a draft or a claim, not as evidence, and require human review before it drives a decision.
- Ask for sources and confirm that each one exists and says what is claimed.
- Test commands, scripts and configuration changes in a lab before production.
- Ground assistants in authoritative internal documents, and allow them to say they do not know.
- Check every suggested package's publisher, age, download history and repository; install through an internal registry with an allowlist; pin versions with hashes.
- Do not let coding agents add new dependencies without approval.
Hallucination vs. poisoning and bias
A hallucination needs no attacker; the model invents the error on its own. If someone planted the false statement in the model's training data or in a document it retrieved, the issue is poisoning. If the model's errors fall consistently on one group, region or language, the issue is bias. Slopsquatting is the case where an attacker profits from a hallucination after the fact.
AI hallucination on the CompTIA Security+ exam
Hallucinations are among the AI risks in SY0-801 objective 2.6, and they return with AI-assisted security operations under objective 4.6, where a confident but wrong summary or alert explanation can mislead an analyst. Scenario wording about an AI tool producing a fake citation, a nonexistent command or event ID, or advice that sounded right and was false points here; an attacker registering a package name that an AI invented points to package hallucination. The best answers add verification against an authoritative source, testing or human review, rather than trusting the output or simply switching models. Distractors are poisoning, when someone planted the falsehood, and bias, when the errors are skewed. Hallucinations also appear in A+ Core 2 (220-1202) objective 4.10, which covers the limitations of AI.