How to Build a Labeling Queue in Humanloop That Actually Gets Done
Human review is the part of the eval stack that quietly dies. The queue fills, nobody opens it, and three weeks later the dataset is 40 items behind production. Humanloop gives you the queue, the rubric notes and the sync back to datasets — the rest is process design.
What follows is the process that keeps a labeling effort alive past week two, plus the parts of Humanloop that make it possible.
Why do labeling queues stall?
Three reasons, in order:
-
The queue is unbounded. Reviewers see “2,847 pending” and do nothing. A 20-item batch with a visible end gets finished.
-
The rubric is an adjective. “Mark as good if the answer is helpful” produces agreement around chance level and reviewers stop trusting their own clicks.
-
There is no consumer. Labels go into a dataset nobody runs an eval against, so the work has no visible effect. Reviewers notice that.
Fix all three and review happens without being chased.
How do you sample from production instead of labeling everything?
Log every call, then sample. Humanloop’s logging gives you every call with its prompt version and inputs; the sampling decision is yours.
The sample that works: 50 random calls a week for the baseline, plus every call that got a thumbs-down plus every call that hit an error path. The random 50 tells you the steady state. The negative cases tell you where quality actually breaks.
Labeling 100% of traffic is wasted effort. Labeling only failures gives you a dataset that is 90% bad examples, and a model trained or evaluated on that does not represent production.
How do you write a rubric people can apply consistently?
As a short ordered checklist, not a paragraph. For an answer-quality task:
-
Does the answer only use facts present in the provided context?
-
Does it answer the question that was asked, not an adjacent one?
-
Is every number in the answer traceable to the context?
-
If the answer says it does not know, is that correct given the context?
Four binary questions. A reviewer marks the first one that fails. That produces a label with a reason attached, which is far more useful than a 1–5 score because it points at the fix.
Score scales are a trap. Reviewers calibrate differently on a 1–5 scale and drift over a long session. Binary plus reason code is stable.
How do you measure whether reviewers agree?
Have two people label the same 30 items and compute agreement. Cohen’s kappa above 0.6 means the rubric is workable; below 0.4 means the rubric is unclear and you should not trust the labels yet.
Run this once per rubric change, not per week. If agreement drops after you edit the rubric, the edit made it worse.
Do not chase perfect agreement. Some items are genuinely ambiguous — an answer that is technically correct but unhelpfully terse. Agreeing that a case is ambiguous is itself a useful result; move it to a separate bucket.
How do labels get back into evals?
Humanloop annotations sync into datasets. The pattern:
- Reviewer labels a batch. 2. Confirmed-correct and confirmed-wrong items get promoted into the eval set, capped at a few hundred items. 3. The eval runs on every prompt or model change. 4. Items the model now gets wrong become new labeling candidates.
That last step keeps the loop alive: the eval set is not static, it drifts toward the cases that are currently hard.
What does this cost in people-hours?
A rough plan for a team with one product: 50 random plus maybe 30 negative cases a week, at roughly 20 seconds per item with a binary checklist, is about 25 minutes of review per week. That is affordable. The version where you label 500 items per week without automation is 3 hours and will not survive a busy sprint.
Bulk labeling is also where quality drops. Sessions longer than 45 minutes degrade agreement noticeably. Split the work across two days instead of one.
When is human labeling the wrong tool?
If your scorer can be written deterministically — valid JSON, contains a required field, under a length cap — label nothing. Automation is free and consistent.
If you have a domain expert but only 30 examples total, you may get further by reading those 30 yourself and fixing the prompt than by building a queue. Labeling becomes worthwhile when the volume of cases exceeds what you can hold in your head.
And if nobody owns the eval set, the queue will die. This is an ownership problem, not a tooling problem.
Quick Summary
-
Queues stall from unbounded batches, vague rubrics and labels nobody uses; fix all three before adding reviewers.
-
Sample 50 random production calls per week plus every thumbs-down and error case; do not label 100% of traffic.
-
Write rubrics as 3–5 binary checklist questions and record the first failure reason, not a 1–5 score.
-
Check inter-annotator agreement with Cohen’s kappa per rubric change; above 0.6 means the rubric works.
-
Promote confirmed labels into the eval set and let items the model now fails seed the next batch.
