How to Roll Out Dust Assistants to a Support Team
Getting one assistant working is a weekend project. Getting twenty support agents to trust it is an operations problem, and most rollouts fail there — not because the answers are bad, but because nobody owns quality after launch.
This guide is for the person who already built a working assistant in Dust and now has to put it in front of a team. It covers structure, review loops, and the numbers to watch.
How should you organize assistants across a support team?
By task family, not by seniority or team name. Three assistants usually cover an eight-person support team:
- Product answers — reads the docs database, answers how-it-works questions.
- Policy and billing — reads the policy space, answers refund and plan questions.
- Escalation drafting — takes a ticket thread and drafts the customer reply plus an internal summary.
Each one has a narrow source scope. A single general assistant that reads everything produces answers that blend refund policy with product docs, which is exactly the kind of confidently wrong output that destroys trust in week one.
Who owns assistant quality?
Name one person. They review a sample weekly, edit the instructions when answers drift, and are the only person who can edit published assistants.
If everyone can edit, instructions accumulate contradictory rules within a month — usually appended by whoever was annoyed by a specific answer that week. Set edit permissions on published assistants to admin-only and let everyone else submit corrections through a form or a Slack thread.
How do you review answers without reading everything?
Sample 20 interactions a week. Stratify: five from the busiest channel, five from new agents (they phrase questions worst, which exposes gaps), and ten random.
Score each on three things only:
- Correct — matches the source document.
- Grounded — cites a real page, not a plausible one.
- Useful — the agent did not have to rewrite it.
Track grounded rate weekly. It is the leading indicator; correctness follows it. If grounding falls below about 90%, something changed upstream — a renamed Notion page, a revoked connector token — and no prompt edit will fix it.
What Slack structure works?
One channel per assistant, plus one private channel for corrections. Do not put an assistant in #general.
A dedicated channel makes it obvious which assistant answered which question, gives you a clean interaction log for sampling, and lets you mute or unpublish one assistant without killing the rest.
The corrections channel matters more than it sounds. Agents paste bad answers there, and the owner batches instruction fixes. Without it, corrections happen in DMs and disappear.
Which questions should be routed away from the assistant?
- Anything about a specific customer’s account state. The assistant reads documents, not your billing database.
- Legal and compliance questions where a wrong answer has a cost beyond a bad ticket.
- Angry customers. A drafted reply from an AI reads flat when the customer is furious, and agents know it.
Document these exclusions in the channel topic. Agents who hit one bad answer in a sensitive case stop using the tool entirely, and winning them back takes months.
What metrics tell you it is working?
Three, and none of them are message count.
- Draft adoption rate — share of assistant replies sent with only minor edits. Above 40% is a real signal; below 20% means the output format is wrong for your workflow.
- Time to first response — should drop measurably in the first month. If it does not, agents are reading the assistant output and then writing from scratch.
- Escalation accuracy — whether the assistant correctly flagged cases it could not answer. This is the trust metric.
A team that hits 40% adoption, keeps grounding above 90%, and drops first response time is getting real value. A team with high message count and low adoption is paying for a novelty.
What breaks at month three
Source drift, mostly. Documents get restructured, pages get archived, someone renames the “Refund policy v3” page to “Refunds (current)” and half the citations break.
Mitigation: pin the document IDs in the assistant config where the connector allows it, and run a monthly test suite of 15 questions with known answers. Fifteen questions takes twenty minutes and catches most drift.
The second break is instruction bloat. Every complaint adds a rule. Once instructions pass roughly two pages, the model starts ignoring the middle. Consolidate quarterly, and delete rules that describe cases nobody has hit in a month.
Quick Summary
- Organize assistants by task family with narrow source scopes; one general assistant reading everything blends policies and produces wrong answers.
- Name a single owner with edit rights on published assistants; let everyone else submit corrections through one channel.
- Sample 20 interactions weekly and track grounded rate — below 90% signals a broken connector or renamed page.
- One channel per assistant; never publish to a general channel.
- Target above 40% draft adoption and monthly test runs of 15 known-answer questions to catch source drift.
