Why does your chatbot answer confidently and wrongly?
Key Takeaways
For busy support leads: the failure that costs you money is not the assistant saying it does not know. That outcome is fine, and customers forgive it. The failure that costs you money is a confident wrong answer about a refund window, a cancellation policy or a data-deletion request, and that almost always comes from retrieval finding an article that is nearly right. Nearly right is worse than missing, because missing triggers a handoff and nearly right does not.
- 1Write the eval set before the content. Sixty real questions, frozen, graded weekly. Without it you are guessing about progress.
- 2Separate wrong from abstained. Averaging both into one accuracy number hides the only failure mode that actually hurts you.
- 3Document the negative space. Articles stating what your product does not do prevent more hallucinations than any prompt instruction.
- 4One claim, one place. Two articles half-answering the same question get retrieved together and blended into something neither of them says.
- 5Every article carries a running cost. If you will not reread it next quarter, do not publish it this quarter.
Retrieval systems do not understand your product. They find passages that look semantically close to the question and hand them to a model that is very good at writing fluent prose from partial information. When the closest passage is about a similar feature rather than the right one, the model fills the gap. That gap-filling reads as confidence.
This is why the standard advice, write more articles, sometimes makes things worse. Adding a fifth partially-relevant article about billing does not improve retrieval; it gives the model five near-misses to blend instead of two. Precision beats volume, and it is not close.
The size of the residual risk is documented. A preregistered study of commercial legal research tools found that Lexis+ AI, Westlaw AI-Assisted Research and Ask Practical Law AI "each hallucinate between 17% and 33% of the time" (Stanford RegLab and HAI, 2024), and those are purpose-built retrieval products from vendors that had claimed otherwise. Even in the easiest possible setup, summarising a document handed straight to the model, Vectara's leaderboard shows hallucination rates ranging from 1.8% to 23.5% depending on the model, on a leaderboard maintained by a company that sells retrieval software. Your job is not to reach zero. It is to make sure the residue lands on cheap questions.
The practical consequence: your first pass should be subtractive as often as additive. Before writing anything new, search your own knowledge base for the twenty questions you most fear getting wrong and read what comes back. If two articles disagree, that is your first ticket.
What should you build before you write a single article?
An evaluation set. Take sixty real questions from your ticket history, verbatim, including the badly spelled and rambling ones. Write the correct answer next to each. Freeze the list. This is now the only honest measure of whether your knowledge work is helping.
Sixty is not arbitrary. It is small enough that one person can grade it in an hour and large enough that a five-point move is not noise. Weight it deliberately: roughly half from your highest-volume topics, roughly half from your highest-risk ones, since those are different lists and the risky half never has the volume to show up naturally.
Rerun the same sixty every time you make a meaningful change to the content. The value is in the frozen list. The moment you start adding questions you just fixed, the score goes up and tells you nothing.
Which content earns its keep first?
Not your top ticket topics. The right first pass ranks by cost of being wrong, not by volume, because volume tells you where the work is and risk tells you where the damage is.
| Topic | Volume | Cost if wrong | Write it |
|---|---|---|---|
| Refunds and cancellation | High | High | First |
| Billing, invoices, proration | High | High | First |
| Security, privacy, data deletion | Low | Very high | First |
| Getting started and setup | High | Low | Second |
| Feature how-to | High | Low | Second |
| Integrations and configuration | Medium | Medium | Second |
| Contract and legal terms | Low | Very high | Do not automate, hand off |
That bottom row is a deliberate stance. Some questions should never be answered by generation at all, no matter how well documented. Contract interpretation, legal commitments and anything a regulator would read belong in a clean human handoff, configured explicitly rather than left to the model's judgement about its own confidence.
Why is documenting what you do not do the highest-leverage content?
Because "no" is the answer your knowledge base is structurally worst at giving. Ask any assistant whether your product supports a feature you have never built, and retrieval returns the nearest adjacent feature. The model reads a page about something similar and produces an encouraging answer about a capability that does not exist. Customers then buy, or stay, on that basis.
The fix takes an afternoon. Write one page listing the things you are most often asked for and do not offer, using the exact phrasing customers use rather than your internal naming. Add a short "not supported" line to each integration page for the specific configurations that fail. Where a workaround exists, say so; where none exists, say that too.
In our experience this single page moves the wrong-answer rate more than any other content, and it is close to the least popular thing to write, because nobody enjoys publishing a list of their own gaps. Do it anyway. A customer who gets a straight no in ten seconds is far less expensive than one who discovers it during onboarding.
How does chunking change the way you write?
Retrieval does not fetch whole articles. It splits documents into passages and fetches the passages that match, which means a section can arrive at the model with no title and no surrounding context. Write for that.
Three habits follow directly. Repeat the subject inside each section rather than relying on the page title: write "To cancel a Pro subscription, open Billing" instead of "To cancel, open Billing". Put the answer in the first two sentences under each heading, because a passage that opens with three sentences of preamble scores worse against the question. And use headings that restate the question a customer would type, since heading text usually travels with the chunk.
The corollary is the familiar advice about one topic per article, but for a mechanical reason rather than an aesthetic one. An article covering billing, pricing and account management gets retrieved for all three and answers each one thinly, because the matched passage is competing with the rest of the document for relevance.
What does a real grading rubric look like?
One accuracy percentage is the most common measurement mistake in this whole exercise. It merges two outcomes with completely different costs. Grade into four buckets instead.
| Grade | What it means | Acceptable? | The fix |
|---|---|---|---|
| Correct | Fully and accurately answered | Yes | None |
| Incomplete | True, but missing a step or a condition | Tolerable | Expand the source article |
| Abstained | Said it did not know, or handed off | Tolerable, but track the trend | Write the missing article |
| Wrong | Confident and inaccurate | Never | Find and fix the near-miss article |
The two-sided design is not ours. Google DeepMind's FACTS Grounding benchmark judges 1,719 examples as accurate only if "fully grounded in information contained in the provided document, with no hallucinations", and separately disqualifies responses that "don't sufficiently address the user's request", which stops a model scoring well by being uselessly evasive. It is vendor-funded work judged by frontier models including Google's own, so treat the leaderboard sceptically, but the design point stands: grade abstention and inaccuracy separately or you will optimise one into the other.
Then set a launch gate that reflects the asymmetry: zero wrong answers in the billing, cancellation, security and legal subset, regardless of what the overall numbers look like. A high abstention rate is an inconvenience you fix with content. A single confident wrong answer about a refund policy is a customer complaint and, in some industries, a compliance conversation.
How do you kill contradictions before they reach the model?
Retrieval has no idea which of two documents is newer or more authoritative. It sees two plausible passages and passes both along. If your old pricing page and your new pricing page both survive in the corpus, the assistant will eventually quote the old one, and you will find out from a customer.
Run a deduplication pass by question rather than by document. Take your twenty most important questions, retrieve for each, and list every article that comes back. Where two articles both make a claim, one of them must become the canonical source and the other must either link to it or lose the claim entirely. Partial overlap is fine. Competing statements of fact are not.
Then designate a single canonical location for each of your volatile facts: pricing, limits, retention periods, supported regions, SLA numbers. Every other article references it rather than restating it. This is the discipline that keeps a knowledge base honest as it grows, and it is the one that decays first when nobody owns it.
There is a decided case on exactly this. Air Canada's chatbot described a bereavement-fare policy that contradicted a page elsewhere on the airline's own site, and the tribunal rejected both the argument that the chatbot was "a separate legal entity that is responsible for its own actions" and the argument that the customer should have checked one part of the site against another (Moffatt v. Air Canada, 2024 BCCRT 149). Reconciling your own contradictions is your job, not the customer's.
What can a knowledge base never answer?
Anything account-specific. "Why was I charged forty-seven dollars last Tuesday" is not a document question, it is a data question, and no amount of writing will produce the answer. The same goes for order status, current usage against a limit, seat counts and whether a specific invoice was paid.
This distinction should shape your entire plan. Split your top thirty questions into document questions and data questions before you start. The document half is a content project. The data half needs system integrations so the assistant can perform a live lookup, and a defined handoff for when the lookup is unavailable.
If a vendor demo answered an account-specific question convincingly, they had an API connection behind it, not a better model. Ask which one you are being shown, because the implementation effort differs by an order of magnitude.
How do you keep it current without a full-time owner?
By capping the corpus. Every article you publish is a recurring obligation, and the arithmetic is unforgiving. Forty articles at ten minutes each per quarterly review is under seven hours, one person, one afternoon. Four hundred articles is over sixty-six hours a quarter, which nobody has, which is precisely why large knowledge bases rot in place.
So prefer sixty correct articles to four hundred mixed ones, and treat publishing as a decision with an ongoing cost rather than a free win. Set a hard trigger for immediate updates, not a schedule: any pricing change, policy change, or feature removal updates the canonical article the same day it ships.
Two feedback sources cost nothing and outperform any review calendar. First, the list of questions where the assistant abstained, which is your content gap list, generated for free. Second, the conversations a human took over after an AI answer, which is where wrong answers hide, since the customer rarely bothers to tell you the bot was wrong.
Where this breaks
If your product ships significant changes weekly, your knowledge base will be wrong faster than you can repair it. Document only the stable layer, billing, account management, policy, and route product mechanics to humans until the surface settles.
If you support multiple languages with translated articles, you now maintain several corpora that drift apart independently. Budget for that or serve one canonical language and translate at answer time.
If fewer than a handful of repeat questions arrive per day, skip the whole project. A good FAQ page and fast human replies will beat a mediocre assistant and cost you nothing to maintain.
And the honest limit on our side: Corebee will not repair a contradictory knowledge base for you, and neither will anyone else's model. The retrieval quality is the vendor's problem. The consistency of what gets retrieved is yours.
What should you do in the first two weeks?
Week one: pull sixty real questions from ticket history, write the correct answers, and freeze the list. Run it against whatever you have today, even if that is a bare help centre, and record the four-bucket baseline. Read every wrong answer and trace it to the article that caused it.
Week two: write the negative-space page, fix the contradictions you found, and split your top thirty questions into document questions and data questions. Rerun the same sixty. If the wrong count in your high-risk subset is not zero, do not launch publicly yet; run internally with agents reviewing answers until it is.
If you want to run that loop on real traffic without a procurement conversation first, you can start a free trial and grade your own sixty questions in an afternoon.