# What to do when your AI assistant gives a wrong answer

Source: https://cplt.tech/blog/what-to-do-when-ai-gives-a-wrong-answer
Author: Stéphane Lepain
Published: 2026-11-16T09:07:35.112816+00:00
Tags: wrong-answers, small-business, human-review

---

When your AI assistant gives a wrong answer, find out what it was working from before you blame the model. Open the source it used, fix the document or the instruction if the fault is there, keep the question on a list you re-test, and decide which kinds of answer a person approves before anyone relies on them.

That order is my own suggestion for a small team. Think of it as a routine to adapt, offered as my judgement rather than a validated method.

## Expect it to happen again

OpenAI's research note [Why language models hallucinate](https://openai.com/index/why-language-models-hallucinate/), published in September 2025, defines hallucinations as "plausible but false statements generated by language models." It also says accuracy will never reach 100% because "some real-world questions are inherently unanswerable."

Giving the assistant your own documents helps and still leaves wrong answers. In a [preregistered 2024 study of AI legal research tools](https://arxiv.org/abs/2405.20362), researchers found that Lexis+ AI and two Thomson Reuters tools each hallucinated between 17% and 33% of the time, even though they use retrieval-augmented generation. That was lower than a general-purpose chatbot, and the authors concluded that the providers' claims of eliminating hallucinations were overstated. Your documents and questions will differ, so I read the figure as a reason to build a routine, not as a forecast for your assistant.

## Start with what it was shown

I would take each wrong answer through four questions:

1. Which document did it use? Open it. A superseded price list or an old policy can explain a wrong answer, and the fix is to update or retire the document.
2. Was the answer in your documents at all? If not, the assistant should have said so and passed the question to a person. OpenAI's note says models "can abstain when uncertain." Ask your supplier how yours behaves when the answer is missing.
3. Did it have the right document and misread it? Keep that case. It tells you which kind of question needs a person's approval.
4. Was the question ambiguous? "What is the return window?" has two answers if you sell two product lines. Ask staff to name the product and the date in the question.

## Keep the question

Write the question, the correct answer and the source on a shared list. When documents, settings or the model change, ask the whole list again and compare. A wrong answer then becomes a test your assistant has to pass next time.

The [acceptance checks for a knowledge assistant](https://cplt.tech/blog/internal-knowledge-assistant-acceptance-checks) show how to write the expected behaviour before you read an answer. Start your list with the questions your team asks most.

## Decide who approves what

A customer reads what your assistant says as what your business says. So I keep a person on any answer that promises something on your behalf. Refunds, prices and deadlines are the first ones I would list.

Who is responsible for a wrong answer depends on your country and your terms, so ask a lawyer before you put an assistant in front of customers. The [customer email article](https://cplt.tech/blog/can-ai-answer-customer-emails) shows how I sort routine questions from the ones that need approval.

## An assistant on your own server needs the same routine

A model you run yourself also gives wrong answers. Where it runs changes who controls the documents and the settings. The routine above stays the same: source, fix, keep the question, decide who approves.

If you want an [assistant that answers from your company documents](https://cplt.tech/services/ai-knowledge-assistant) and shows the page it used, bring the five questions your team asks most to a [free 45-minute call](https://cplt.tech/contact). I will help you turn them into the first test list.

## Frequently asked questions

### Why does my AI keep getting answers wrong?

Common causes are an old document, an answer missing from the documents, a question that fits two products or years, and a model that guesses. OpenAI's research says some real-world questions are inherently unanswerable, so a good assistant says when it is unsure.

### Who should approve an AI assistant's answers?

Anyone who can check the answer against a document, and a person with authority for anything that commits the business. Responsibility for a wrong answer depends on your country and your terms, so ask a lawyer before customers see the assistant.

### Can I stop an AI assistant from ever giving a wrong answer?

I would be wary of a supplier who promises that. Build a routine instead: check the source, correct the cause, re-test the question and keep a person on answers that commit your business.
