Where Do LLM Hallucinations Actually Come From? An Engineer's Firsthand Observations

A Problem That's Haunted Me for a Long Time

After running AI Agents at SFD Lab for nearly two years, I've been burned by LLM hallucinations more times than I can count. The most absurd case: I asked GPT to look up the version number of an npm package, and it confidently told me "the latest version is 3.2.1." The package didn't exist. Not a wrong version number — the entire package name was made up.

This isn't a bug. It's a design characteristic of large language models — we just call it a "hallucination" when that characteristic manifests as incorrect information.

The Nature of Hallucinations: A Probability Game

To understand hallucinations, you first need to understand what LLMs are actually doing. They're not "looking up" information — they're predicting "the most likely next token" based on context. It's a purely statistical process.

Here's an example: when you ask "Who were the founders of quantum mechanics?", the model's internal process goes roughly like this:

Scan the frequency of names appearing near "quantum mechanics" in training data
Calculate probability weights for "Planck," "Bohr," "Heisenberg," and "Schrödinger"
Sample from those probabilities and output

This works remarkably well most of the time, because there's plenty of training data. But when a question touches on niche domains, recent events, or genuinely fuzzy territory, the model still follows the same logic — only this time, what gets sampled is content that "sounds plausible but doesn't actually exist."

The model doesn't know what it doesn't know. That's the core problem.

Four Types of Hallucinations, Four Different Root Causes

After two years of running experiments, I've broken hallucinations down into roughly four categories:

1. Knowledge Cutoff Hallucinations

The most common kind. Training data has a cutoff date, and the model has no knowledge of anything after that point — but instead of saying "I don't know," it will infer an answer that "sounds reasonable" from what it does know.

Our approach at SFD Lab: for any task involving real-time information, we force-connect the web_search tool. We don't give the model the option of answering from memory.

2. Detail Distortion Hallucinations

These are the most insidious, and the most dangerous. The big picture is right, but the details are wrong. Think: an API documentation page that's correct overall, but one parameter's type is flipped; a paper citation with a real title, but a fabricated DOI.

Engineers dread this type the most, because a quick read-through feels fine — you only discover the problem when you actually run the code.

3. Logical Consistency Hallucinations

Give a model a long conversation, and it might assert A early on, then reach a conclusion of not-A later — both stated with equal confidence. This is a limitation of the attention mechanism: beyond the effective context window, the weight of early information decays.

4. Overconfidence Hallucinations

This isn't about incorrect content per se — it's about how the model expresses itself. When delivering an uncertain answer, the model's tone is often just as definitive as when it's certain. That makes it very hard for users to know when they should go verify something.

Why Does GPT-4 Hallucinate Less Than GPT-3.5?

Your instinct might be "because it's bigger and better," but the real reason is more nuanced.

Beyond scale, the more critical change was the introduction of RLHF (Reinforcement Learning from Human Feedback), along with later fine-tuning specifically targeting "uncertainty expression." The model was trained to say "I'm not sure" or "I'd recommend verifying this" when uncertain, rather than forcing out a plausible-sounding answer.

But RLHF introduced a new problem: models became more "people-pleasing." They'll shift their position based on user reactions, even when the original answer was correct. That's another form of hallucination — not factual error, but positional instability.

Engineering-Side Mitigation Strategies

In SFD Lab's actual workflow, we rely on a few standard techniques to combat hallucinations:

Tools before memory. For any scenario requiring precise information, ask the tool first (database, API, search) — not the model's "memory." The model handles reasoning and comprehension; it doesn't handle fact storage.

Structured output. We require the model to output JSON with a confidence field. That confidence score isn't fully reliable, but it filters out a chunk of the most egregious results.

RAG instead of memory. For scenarios requiring precise citations, we use RAG (Retrieval-Augmented Generation) to inject document chunks into context, rather than letting the model answer from impression. The document chunks are ground truth; the model just summarizes and formats.

Isolated verification. In multi-agent systems, we separate the agent responsible for generating content from the agent responsible for verifying it, with no shared conversation history. That way, the verification agent can't be led astray by the generation agent's errors.

A Counterintuitive Conclusion

Hallucinations cannot be fully eliminated — this isn't a technical problem, it's a fundamental property of probabilistic sampling.

The more pragmatic mindset is: accept that hallucinations exist, and design your system around them. Airplanes don't stop flying because failures are possible — they manage risk through dual engines, redundant backups, and pre-flight checks.

In agent engineering, this means: don't treat LLMs as omniscient databases. Treat them as smart-but-fallible assistants. Give them tools, give them constraints, give them verification mechanisms — then they can work reliably.

One direction we're currently testing at the lab is "cross-model validation": send the same question to two different models, compare the outputs, and route significant divergences to mandatory human review. Early results suggest this can reduce high-risk hallucinations by around 60%, at the cost of doubling API call expenses. Whether it's worth it depends on how much precision your application demands.

The one-line summary: LLM hallucinations are fundamentally a probability game. The fix isn't making the model "smarter" — it's building guardrails into the system around it.