How to Stop AnswerRocket From Returning Wrong Numbers

November 4, 2026

AnswerRocket gets the join right and the metric wrong. That is the failure mode to design against: SQL that runs, a chart that looks reasonable, and a number that quietly disagrees with finance.

Every fix below is a definition, a role, or a review habit — not a prompt trick. The assistant reads your schema and your semantic layer. It does not read your intent.

Which definitions should you write first?

Start with the metrics that appear in board decks. For most companies that is five to eight numbers: revenue, ARR, active users, churn, conversion, ticket volume.

For each one, write down:

  • The exact table and column it comes from
  • The filter clause (status, date range, excluded test accounts)
  • The grain (daily, monthly, per account)
  • The owner who signs off on changes

A one-page doc beats a 60-field glossary nobody maintains. Six definitions used consistently will fix most wrong numbers.

How do you test whether the assistant is right?

Build a regression set. Ten questions with answers you already know from a finance report or a dbt test.

Ask them after every schema change. If revenue moves 2% without a business reason, something shifted.

Run it monthly as well. Warehouses drift quietly — a backfill, a renamed column, a new currency.

This takes 20 minutes a month and catches the failures that damage trust fastest. Nothing else in the setup has that return.

Where do wrong numbers come from that definitions do not fix?

Duplicated rows from a join. A one-to-many join multiplies revenue. Look for a row count in the SQL that should equal your order count.

Timezone boundaries. “Last 30 days” in UTC versus local cuts differently depending on the hour. Force one timezone in the connection settings.

Currency mixing. Multi-currency accounts summed without conversion produce meaningless totals. Either convert in the warehouse or restrict the project to one currency.

Test accounts and internal users. Exclude them at the role or view level, not in every question.

Soft deletes. The row exists with a deleted_at timestamp. Filter it.

What access controls should you set up before rollout?

Run the assistant on a read-only service account with role-scoped views, one per project or business unit. If the account can see all tenants, so can the assistant, and so can anyone with a login.

Three rules that hold up in audits:

  • No raw PII columns in any view the assistant can reach.
  • Row-level filters enforced in the view, not in the assistant.
  • Query logs retained and reviewed; the SQL it wrote is your audit trail.

Pair it with a proper warehouse engine if speed is the problem. ClickHouse handles billion-row scans where a general warehouse stalls, which changes what questions are realistic.

How should a team actually use it day to day?

Analysts use it to start a question, then verify the SQL. Ops and support staff use saved, pinned answers with the SQL hidden — those have been reviewed once and do not need re-checking.

Do not open free-form questioning to the whole company in week one. Open three pinned answers to the sales team, collect questions, then formalise the next five.

That rollout pattern keeps wrong-number complaints at zero for the first month, which is roughly what it takes for a team to trust a new analytics tool.

Quick Summary

  • Wrong answers come from undefined metrics, not from the model. Write 5-8 core definitions with owner and filter.
  • Keep a regression set of 10 known-answer questions and re-run it monthly and after every schema change.
  • Watch for join fan-out, timezone boundaries, mixed currencies, test accounts and soft deletes.
  • Use read-only, view-scoped warehouse roles; enforce row filters in the view, not the assistant.
  • Roll out with pinned reviewed answers first, free-form chat later.