How to Use Ashby Scorecards to Cut Interviewer Bias
Ashby scorecards are per-interview evaluation forms where an interviewer scores defined attributes on a scale and writes supporting evidence. They appear in the candidate’s profile the moment an interview is scheduled, and the whole point is that they get submitted before the debrief conversation.
That ordering is the mechanism. Once a group discusses a candidate, individual judgments collapse toward whoever spoke first and most confidently. Ashby’s design assumes independent scoring first, discussion second. Teams that skip the first half get the paperwork without the benefit.
What is inside an Ashby scorecard?
Each scorecard pulls its attributes from the interview plan. Typically you will see:
- Attribute scores on a 1–4 scale (strong no through strong yes).
- An overall recommendation — strong no, no, yes, strong yes.
- Evidence fields per attribute, where the interviewer is asked to quote or describe what they observed.
- Optional tags or notes depending on your configuration.
Ashby pushes evidence fields deliberately. A score without evidence is an opinion; a score with a quote is a data point you can bring into debrief.
How do you run the scorecard workflow?
- Right after the interview, before the next meeting. Same day is the workable standard; same hour is better.
- Open the interview from the interviewer’s calendar invite or from My Tasks in Ashby.
- Score every attribute. Do not leave any blank — a blank reads as neutral and skews averages.
- Write evidence for the attributes scored low or high. Middle scores need less text.
- Submit. It locks.
Ashby has a setting that blocks debrief scheduling until scorecards are in. Turn it on. A gentle reminder gets ignored; a blocked calendar slot does not.
What to do when an interviewer wants to change a score
They can add a note after debrief rather than editing the score. If your team genuinely needs to revise, an admin can reopen it, but every reopen should have a reason logged. Frequent reopens mean the rubric is wrong, not the interviewer.
How do you read the scorecard data without fooling yourself?
Ashby’s candidate view shows attribute averages across interviewers. Two patterns to watch:
| Pattern | What it looks like | What it usually means |
|---|---|---|
| Bimodal | Scores split 1 and 4 | Genuine disagreement, or two interviewers assessed different things |
| Flat high | Everyone scores 4 on everything | Rubric too easy, or interviewer training gap |
| One outlier | One 1 among four 4s | Worth reading that person’s evidence in full before dismissing |
Bimodal is the interesting one. It usually signals a rubric problem — attribute X means something different to two interviewers. Fix the guideline text in the interview plan, not the candidate outcome.
Does structured scoring actually reduce bias?
Partially. Structured interviews with defined attributes and anchored scales measurably outperform unstructured conversation on predictive validity — that is the well-replicated finding from the industrial-organisational psychology literature. What does not work is having the form and ignoring it.
Specific biases scorecards help with: halo effect (one strong answer colouring everything else), similarity bias (the interviewer who “just clicked” with the candidate), and recency (the last 10 minutes dominating memory). What they do less well against is affinity bias in the debrief, where people rationalise a gut call with the evidence they remember.
A countermeasure that costs nothing: have the debrief facilitator read all scorecards aloud before anyone comments. It forces the group to hear low scores before a consensus forms.
What are the setup mistakes?
- Too many attributes. Six attributes in a 45-minute interview means the interviewer scores from impression, not observation. Cap at three.
- Vague attribute names.
Culture Fitis unfalsifiable.Collaborates on ambiguous problemsis scorable. - No evidence requirement. Enable required evidence on attributes marked as key for the role.
- Scorecards not attached to the interview plan. If the interview has no plan attributes, Ashby generates an empty form and interviewers write free text — the thing you were trying to avoid.
Where this approach struggles
Very senior or highly specialised roles resist attribute scoring. A principal engineer’s strengths are often not decomposable into three scored dimensions, and forcing it can produce false precision. For those, Ashby allows free-form strong-signal captures that feed debrief instead of scores, which is the better tool.
High-volume hourly hiring is the other mismatch. Scorecard rigor per candidate at 200 candidates a week becomes a data-entry job. Use it for the final stage only.
Key Takeaways
- Ashby scorecards must be submitted before debrief or the anti-bias mechanism fails.
- Three attributes per interview is the practical ceiling.
- Require written evidence on high and low scores; middles need less.
- Attribute averages that split bimodally usually indicate a rubric problem, not a candidate problem.
- Enable the setting that blocks debrief scheduling until scorecards are submitted.
