Table of Contents
My first AI hallucination
First of all welcome to my first after summer #noHypeAI taking AI easy post! Nice to see you here.
When generative AI was still at the beginning of its hype cycle, I decided to use it to help prepare one of my lightning talks.
You can watch the talk here.
I asked the AI to help me collect materials, research the topic, find useful reports, and organise everything into something I could present.
And it did.
At first, the results looked impressive. The explanations were clear, the research appeared relevant, and the whole process felt almost suspiciously easy.
Then I asked it to provide links to the reports and sources it had used.
That was when the trouble started.
Around half of the links were broken, irrelevan or competely fabricated. Some pointed to pages that no longer existed. Others led to completely unrelated content. A few of the reports appeared not to exist at all.
The AI had not simply misunderstood my request.
It had confidently invented sources that looked exactly like the kind of sources that should exist.
That was my first practical lesson in AI hallucinations: an answer can look polished, detailed, and completely believable while still being wrong.
And no, asking the AI to “only provide real links” did not magically fix the problem.
At that time it obviously felt like an obvious failure (it was also a few years ago so the models had slightly more room for improvement). Why would a system provide a source that did not exist?
The answer becomes clearer once we stop thinking of an AI model as a search engine or database.
The confident intern problem
You ask an AI assistant a perfectly reasonable (and simple one my think) question.
First of all, it gives you a clear answer. Secondly, the explanation sounds logical. Thirdly, the formatting is excellent. And lastly, there may even be bullet points. Perfect surface.
There is only one minor problem – it made the whole thing up.
Perhaps it invented a NuGet package (as in my case), referenced a method that does not exist, confidently described a court case with the wrong verdict, or provided a link that leads directly to the internet’s version of an empty parking lot.
This behaviour is usually called an AI hallucination.
The name makes it sound mysterious, as if the model briefly saw something strange while crossing the digital desert. In reality, hallucination is not a weird side effect added to an otherwise perfect factual machine.
It comes directly from how language models work.
And unfortunately, typing:
Do not hallucinate.
does not install a truth module.
AI is generating, not looking up
A large language model is not primarily a database.
It does not normally receive your question, locate a verified record containing the answer, and return that record to you. No magic:
SELECT TOP (1) Answer
FROM VerifiedFacts
WHERE Question = @question
AND IsAccurate = 1
ORDER BY ConfidenceScore DESC;A language model estimates which token is likely to come next, based on the tokens already present in the conversation. A token might represent a whole word, part of a word, punctuation, or another encoded piece of text.
It generates an answer one token at a time.
That sounds simple, but after training on huge amounts of text, the model learns extremely complex patterns. It learns how documentation usually looks, how developers discuss errors, how APIs are named, how explanations are structured, and which concepts commonly appear together.
This allows it to produce useful answers about an enormous range of topics.
It also allows it to produce extremely convincing nonsense.
Imagine asking:
How do I configure the EnableSuperReliableMode option in ASP.NET Core?
There is no standard ASP.NET Core option with that name.
However, the prompt looks similar to thousands of legitimate framework questions. The model knows what a technical answer should look like. It may therefore generate a plausible namespace, configuration example, version requirement, and explanation.
The model is not necessarily checking whether EnableSuperReliableMode exists.
It is continuing the pattern.
Fluency and factuality are not the same feature.
Why does it not simply say “I don’t know”?
Truth being told.. sometimes it does.
Modern models are increasingly trained to express uncertainty, refuse unsupported requests, and admit when information is missing.
But there is a difficult balance between being cautious and being useful.
A model that answers every uncertain question with “I don’t know” would probably hallucinate less. It would also be deeply annoying.
Imagine a coding assistant behaving like this:
“Please consult a qualified stack-trace professional.” – not exactly the productivity revolution we were promised.
Language models are trained and evaluated to provide helpful answers. In many evaluation settings, guessing can be rewarded when the guess happens to be correct, while refusing to answer receives no credit.
Research published by OpenAI argues that this creates an incentive problem: standard evaluations often reward successful guesses more than honest uncertainty. As a result, models may learn that attempting an answer is generally more useful than declining. [1]
In human terms, we have created a very knowledgeable intern who believes leaving a field empty is worse than entering something that looks reasonable.
What could possibly go wrong?
Missing information creates empty spaces
Hallucinations become more likely when the model lacks the information required to answer reliably.
This may happen because:
- the information was not present in the training data,
- different training sources contradicted each other,
- something has changed after model’s school (training) time,
- the question is vague,
- it skipped the important context,
- or the requested detail never existed in the first place.
The model still needs to generate the next token.
It cannot pause the universe, conduct an investigation, and return next Thursday with an audited report. Unless tools such as search, retrieval, or database access are available, it can only work with patterns learned during training and information currently included in its context.
When the evidence is incomplete, plausible text may fill the gaps.
This is why hallucinations often appear in very specific details:
- dates,
- names,
- package versions,
- citations,
- URLs,
- configuration properties,
- statistics,
- and obscure API methods.
The general explanation may be correct while one tiny, production-breaking detail is invented.
My favourite type of correctness.
Why “AI, please, do not hallucinate” is not enough
Adding “do not hallucinate” to a prompt can help a little.
It signals that accuracy matters and may encourage the model to be more cautious. More useful instructions include:
- say when information is uncertain,
- do not invent missing details,
- distinguish facts from assumptions,
- provide sources,
- and ask for missing context.
But instructions do not change the model’s fundamental architecture.
The model is still generating a likely response. It still may not know which part of its internal representation is correct, outdated, incomplete, or fictional.
Consider telling a developer:
Do not write any bugs.
A lovely requirement.
We should add it to Jira immediately.
The instruction communicates the goal, but it does not provide the developer with complete requirements, reliable data, tests, documentation, or supernatural powers.
Similarly, “do not hallucinate” does not give the model access to facts it does not have. It also does not guarantee that the model can recognise every incorrect statement before producing it.
Confidence in the writing is not proof of confidence in the underlying fact
What’s next?
Hallucinations are not just a prompting problem. They are a system-design problem.
In the next article, we will look at practical ways to reduce them using reliable context, retrieval, tools, validation, and human review.
Coming soon: You Cannot Eliminate AI Hallucinations, but You Can Design for Them. (or something similar, I haven’t decided yet)
In a meantime make sure to check out some of developer’s tips on how to effectively work with AI part 1 and part 2, both equallyhelpful!
Sources
OpenAI, “Why Language Models Hallucinate” 2025.
Google, “Introduction to Large Language Models” Machine Learning Crash Course.