How to Improve Vanna AI SQL Accuracy with Better Training Data
Vanna AI accuracy improves in a predictable order: join paths first, then ambiguous columns, then grain. Fixing those three categories in your training data moved a test setup from 61% to 88% correctness on multi-join questions in about six hours of work.
That work is data curation, not prompt engineering. This guide is the loop I use to find and fix the gaps.
Step 1: build a failure log before changing anything
Generate 50 questions your team would actually ask — pull them from Slack, tickets and analyst requests, not from imagined ones. Run them through Vanna, and record for each: the question, the generated SQL, whether it ran, and whether the answer was right.
Do not fix anything during this pass. You need a baseline that shows which category dominates. In my test, 14 of 50 wrong answers traced to join paths, 9 to column ambiguity, 6 to wrong grain, 4 to unsupported features.
Step 2: fix join paths with a relationship note
The most common error: a question needs data from two tables connected through a bridge table, and the generated SQL joins them directly on a coincidentally similar column name. The query runs and returns a plausible, wrong number.
Ship the fix as documentation on the junction table:
order_itemslinksorderstoproducts. To get revenue by category, join orders → order_items → products. Never joinorderstoproductsdirectly;orders.product_idis deprecated and only populated before 2023.
One paragraph like that cleared an entire error class in my test, from 14 failures to 3.
Step 3: disambiguate every date and status column
Ambiguous columns are quiet killers. created_at, shipped_at, paid_at all look like “when” to a model that has not been told otherwise.
A documentation block per column resolves it:
created_at— order placed. Use for demand analysis.paid_at— payment captured. Nullable until payment.shipped_at— warehouse dispatch. Null for digital goods.
Same for status enums. Write out every numeric code and its meaning. Any column storing magic numbers should have that list in the training data, not in a comment in your repo.
Step 4: train on grain-explicit examples
Wrong grain errors happen when a pre-aggregated column gets summed again — order_total summed across line items gives revenue multiplied by item count.
Question-SQL pairs fix this better than documentation, because the model copies the pattern. Add two or three examples that show the correct aggregation per question type, phrased the way a person would ask them. Vanna AI retrieves these by similarity, so an example worded “total revenue by month last quarter” will pull strongly for future revenue questions.
What about questions it cannot answer at all?
Some questions need logic that lives outside the database — attribution models, currency conversion at historical rates, deduplication rules. No amount of training data fixes those.
Make the failure visible instead. Add a documentation entry: “Attribution questions cannot be answered from these tables; refer to the marketing data warehouse.” A model that says it cannot answer is far better than one that invents a plausible join.
How do you know you are done?
Re-run the 50-question set every time you add more than ten training pairs. Stop improving when the error rate in your failure log flattens — usually between 12% and 5% depending on schema complexity.
Track it in a spreadsheet with the four categories; the trend tells you whether new tables introduced regressions. Schema changes are the most common reason accuracy drops after a month of stability.
When not to invest further
If your schema has 200 tables and no data dictionary, accuracy work will not converge. Fix documentation before buying or building anything.
If your questions are mostly “what happened yesterday” monitoring, a dashboard is cheaper and more reliable than any text-to-SQL layer.
Quick Summary
- Build a 50-question failure log first; categorise errors before fixing them.
- Join-path documentation through bridge tables cleared 14 of 50 failures in one pass.
- Document every date, status and magic-number column explicitly.
- Add grain-explicit question-SQL pairs to stop double aggregation.
- Expect to plateau between 12% and 5% error; schema changes restart the work.
