One of the most appealing uses of AI in training is instant feedback (or at least, it seems like it should be).
Instead of waiting for an instructor to review someone’s performance, an AI system can respond immediately and give them something to think about before their next time working with the instructor.
That can be incredibly useful, but it often creates a subtle problem.
Large language models are very good at spitting out a confident and eloquent justification of just about anything… even when that thing doesn’t make sense.
Confidence Can Fool an AI Too
We normally talk about AI hallucinations as the model inventing information.
It might fabricate a source, describe a feature that doesn’t exist, or confidently provide the wrong answer to a factual question.
Feedback introduces another version of the same problem.
Instead of inventing the answer, the model can invent a convincing explanation for why the learner’s answer was good (or bad) even if it wasn’t.
Imagine a learner gives a confident but flawed explanation during an exercise.
If you ask an LLM to evaluate that response in isolation, the model has a lot of linguistic signals to work with. The learner may sound knowledgeable. Their answer may use appropriate vocabulary. The reasoning may resemble other strong answers the model has encountered. Maybe your prompts are strong enough to counter all that. Maybe not.
From the model’s perspective, the response has all the hallmarks of competence.
But that doesn’t always mean it’s right.
Language Models Are Built to Find Patterns
This problem makes more sense when we remember what a language model actually does.
LLMs operate on learned relationships within language. This makes them good at recognizing patterns and generating text that fits those patterns.
Which makes them excellent tools for assisting in parsing and interpreting open-ended learner responses.
But evaluating a response requires something more. The system needs to know what success actually looks like, what we’re measuring against, how we reliably quantify the results.
If the evaluation itself is being improvised by the same type of model that is interpreting the learner’s answer, the distinction can become blurry fast, and in a way that’s impossible to sort out after the fact.
A polished answer may receive too much credit because it has the hallmarks of confidence.
A confident learner with a long answer may overwhelm the weight of the system prompts, causing the system to reinterpret the exercise outside what you intended.
The AI may even generate a compelling explanation for why an unexpected answer was actually a clever alternative.
Sometimes it may be, sure. But sometimes the model is simply being taken along for the ride.
Feedback Needs an Anchor
Effective feedback should be grounded in something outside the language model.
That can mean defined learning objectives, expected behaviors, or another explicit framework for evaluating performance.
The AI can still help interpret what the learner said. It can also help turn the evaluation into natural, useful feedback.
But it should never be allowed to invent the standard and judge the learner against that invented standard.
At SkillSaige, this distinction is important to how our roleplaying scenarios work (and why they’re not just another chatbot).
The underlying simulation tracks what the learner is trying to accomplish and how their actions affect the interaction. The language model helps us interpret natural language, but the model is not given authority to decide whether everything the learner says was secretly correct.
That makes feedback more consistent, and it makes the system more predictable, reliable, and diagnosable without losing the benefits of flexible AI in language processing.
Good Feedback Should Sometimes Disagree With You
There is also an important educational consequence here. Practice loses much of its value if the system is too eager to validate the learner.
The purpose of a simulation is not to produce a pleasant conversation with an AI. It is to give someone a safe place to try something, see what happens, and improve when they make mistakes.
Sometimes that means the feedback needs to say that an approach did not work. Sometimes the simulated person should become less cooperative, or even frustrated. Change the topic. Cut things off.
A language model can make those moments sound natural, but it should not be deciding what happens in the training itself.
Instant AI feedback has enormous potential, assuming it is implemented correctly.
But that’s a pretty big assumption. It’s a fine line between using an AI to translate abstract but deterministic results into understandable language, and letting the model decide the results wholesale.
The important question is not, in the end, whether AI can provide feedback quickly.
It’s whether the system has something reliable to base that feedback on.
If you want to learn more about what I think about the broader implications of AI hallucinations, I have a longer article on my blog about this topic. And for a saner, more powerful way to leverage AI in your business or institution and invest in your people, check out a demo of SkillSaige.
