How to Build a Support Triage Assistant With Cohere
Support triage is a classification problem with high stakes and a small budget, which suits the Cohere API: classify a ticket by intent, extract the order reference, and route with a confidence threshold. Cohere‘s structured output makes the routing decision machine-readable instead of prose you have to parse.
A working version handles four intents and one fallback, and takes an afternoon to build.
What should the triage output look like
A fixed JSON shape:
intent: billing, shipping, product defect, account accessurgency: high if the message contains a dispute, legal or safety signalreference: an order or account number if present, otherwise nullconfidence: a number you threshold on
A schema in the prompt plus a validation step on the API response gets you there. Reject responses that do not match the schema and retry once.
How do you build a test set before writing prompts
Pull 200 real tickets, hand-label them, and record the labels. Without this you cannot tell whether a prompt change helped. Teams that skip it spend weeks tweaking prompts by feel.
Split 150 for prompt development and 50 as a held-out check. Report accuracy per intent, not overall, because billing tickets are easy and defect tickets are not.
What are the routing rules
| Condition | Route |
|---|---|
| confidence > 0.8 | Auto-route by intent |
| confidence 0.5 to 0.8 | Route with a review flag |
| confidence < 0.5 | Human triage queue |
| urgency = high | Always human, regardless |
That last row protects you. An automated reply to a legal threat is worse than a slow one.
Never auto-answer a ticket above the urgency threshold. Route it, do not reply.
How do you keep costs sane
Classification calls are small: a ticket plus a short instruction. The expensive part is the follow-up if you let it draft replies. Use the smaller Command variant for classification and reserve the larger one for drafting, and cache common intents.
Watch the monthly request rate limits on your tier. A spike from a product launch can throttle routing at the worst moment, so set a queue that degrades to human triage rather than failing.
Where does it fit compared with a full support bot
Triage only decides where a ticket goes; it does not answer. If you want a customer-facing bot that answers from documentation, a tool like Chatsimple covers that layer, and the two can run together: bot for self-serve, Cohere for routing what escalates.
What are the honest limits
Sarcasm and multi-issue tickets confuse intent labels; a ticket mentioning both a refund and a shipping delay gets one label. Handle those with a rule that flags mentions of two intents. Non-English tickets need the multilingual variant, and accuracy drops on short messages with no reference number.
Not worth building for a team under 20 tickets a day, where the routing is faster by hand.
Quick Summary
- Classify intent, extract reference, threshold on confidence, route.
- Label 200 real tickets first; prompt tuning without that is guesswork.
- High-urgency signals always go to a human, never to an auto-reply.
- Use the small model for classification and the large one only for drafting.
- Degrade to human triage when rate limits are hit.
