How to Evaluate AI Legal Tools Before Buying Them
Legal AI has a trust problem, and it is earned. Models have cited cases that do not exist. Firm-wide rollouts stall when partners find an error in a memo that a junior associate never checked. Buying well means testing Harvey or any competitor on your own documents, not a vendor’s demo set.
Quick answer: Build a test set of 30-50 documents with known correct answers, run each candidate tool against it, and measure citation accuracy, recall of material clauses, and false-confidence rate. Demo accuracy rarely survives contact with real files.
This is the framework I would use before signing an annual contract. It takes two weeks and it is the cheapest part of the decision.
What should your test set contain?
Pull from matters you already closed, where you know the outcome. Include the messy ones: scans of scans, documents with unusual formatting, contracts in a second language if you handle them. A test set of only clean PDFs will flatter every tool equally and tell you nothing.
Which metrics actually matter?
| Metric | What it measures |
|---|---|
| Citation accuracy | Do cited passages exist and say what is claimed? |
| Clause recall | Fraction of material clauses a reviewer would flag |
| False confidence | How often it states a wrong answer plainly |
| Time to first draft | Minutes saved versus manual review |
Citation accuracy is non-negotiable. A tool that cites real, correct passages but misses some clauses is usable with review. A tool that invents authorities is not usable at all, at any speed.
How do you design the test?
- Pick 10 contracts with a partner’s annotated review as ground truth.
- Run each tool blind — no vendor engineer sitting in.
- Have a mid-level associate score outputs against the annotations.
- Repeat on 5 documents outside your usual practice area.
- Compare not just accuracy but how long verification took.
That last step is where honest answers live. If verifying the AI’s output takes nearly as long as reading the contract, the tool has not saved you anything.
What about data handling and confidentiality?
Ask for the deployment model in writing: where does data live, is it used for training, what happens on deletion, and can it run in your environment. Tools like Harvey offer enterprise controls, and general assistants usually do not. If a vendor is vague here, treat it as a red flag rather than a gap to be filled later.
How do you compare against doing nothing?
Run the manual baseline. Take the same 10 contracts, have an associate review them unaided, and record the time and the clauses found. Without this number you cannot tell whether the tool helped or just moved work around. It is often the most illuminating part of the whole exercise.
What are the warning signs?
- Demos run only on the vendor’s cherry-picked documents.
- No citation, or citations you cannot click through to a source.
- Accuracy claims without a defined test set behind them.
- Vague answers on training data usage.
- A pilot that quietly excludes your hardest documents.
How should you roll out?
Even after a good pilot, start narrow: one workflow, one team, mandatory human review on every output for the first month. Track error types, not just error counts. If the same class of mistake keeps appearing — dates, defined terms, cross-references — that tells you what to constrain in the workflow rather than abandoning the tool.
Who this framework is for: legal teams evaluating any AI vendor, whether Harvey or a general LLM with careful prompting. Who can skip it: nobody buying an enterprise legal tool should skip it, but individual practitioners doing occasional drafting have less to lose.
Quick Summary
- Test on 30-50 of your own documents, including messy ones.
- Citation accuracy and false confidence matter more than raw speed.
- Measure verification time, not just drafting time.
- Establish a manual baseline to know if the tool helps at all.
- Roll out one workflow at a time with mandatory human review.
