How to Reduce Chatbase Wrong Answers and Escalations

November 12, 2026

Chatbase answers wrongly for three reasons: the source doc was ambiguous, the chunk retrieval grabbed the wrong section, or the system prompt let it improvise. All three are fixable without changing models, and most teams can cut bad answers in half within a week of auditing real conversations.

The method is unglamorous. Read what users actually asked, classify the failures, fix in order of frequency. This guide explains the classification and the fixes.

Why does the same question get different answers?

Retrieval is probabilistic. Ask “how do I cancel” and the retriever may pick your cancellation help article on one phrasing and a billing FAQ on the next.

The symptom is two contradictory replies in the same day. The cause is usually near-duplicate content across pages: one page says cancellations take effect immediately, another says at period end. Both are indexed. The bot cites whichever scores higher.

Fix: find the duplicates and delete or consolidate them. Chatbase’s source view shows which document a reply cited, so you can trace every wrong answer back to its page. Do that for twenty conversations and the pattern appears fast.

How do you audit Chatbase conversations efficiently?

Don’t read everything. Sample by failure, not by volume.

  1. Open the Conversations view and filter to threads where the user asked the same thing twice. Repetition means the first answer failed.
  2. Tag each thread: prompt problem, chunk problem, missing content, or needs live data.
  3. Count the tags for one week. Whichever bucket is largest gets fixed first.
  4. Ignore one-off phrasing oddities; they do not repeat.

A typical week on a docs-heavy SaaS site: 40% missing content, 25% chunk problems, 20% prompt problems, 15% needs live data. That distribution tells you where to spend the hour.

What are escalation rules worth setting?

Escalation rules catch the cases the bot should not handle. Without them, a bot trained on docs will attempt to answer legal, refund, and security questions from marketing pages.

Rules that pay off:

  • Route questions containing “refund”, “chargeback”, “legal”, or “GDPR” straight to a human.
  • Escalate after two failed attempts in one conversation.
  • Offer a human handoff when the confidence score is below your threshold.
  • Collect an email before handoff so the transcript survives the session.

Yes, this lowers your deflection rate. It also lowers the number of furious customers who were told the wrong refund policy. Deflection is not the goal; resolved conversations are.

How should you measure Chatbase performance honestly?

Deflection rate alone is misleading. A bot that says “I don’t know” to everything has a great deflection score and no value.

Track a small set instead:

Metric What it catches
Resolved-without-escalation rate Real deflection, not stonewalling
Repeat-question rate Answers that missed
Thumbs-down per 100 conversations Quality drift after doc changes
Handoff accuracy Wrong escalations, wrong routing

Review these weekly for the first month. After that, monthly is enough unless you ship doc changes, which always move the numbers.

What does the paid tier add that matters?

The free plan is enough to test whether the approach works at all. Paid tiers raise the message and character allowances, add integrations (Slack, Zendesk, Notion), and remove the Chatbase badge from the widget.

None of that improves accuracy. Accuracy comes from content hygiene and prompt discipline, both of which you can do on the cheapest plan. Upgrade when you hit the message cap or need a helpdesk integration, not before.

Where the tool genuinely falls short

Chatbase cannot reason over structured data. Questions like “how many seats does my plan include” get the generic plan description, not the user’s actual entitlements. Teams that need this end up with a custom bot on their own backend plus Chatbase for the prose side.

It also does not handle multilingual content gracefully when one language is thin. A bot trained mostly on English docs gives short, evasive answers in other languages rather than admitting the gap.

Quick Summary

  • Trace wrong answers to cited documents; duplicate contradictions are the most common root cause.
  • Audit by failure type weekly: missing content, chunk problems, prompt problems, live-data needs.
  • Escalate refund, legal and security questions instead of letting the bot improvise.
  • Measure resolved-without-escalation rate, not raw deflection rate.
  • Paid tiers raise limits and add integrations but do not improve answer accuracy.