AI Ticket Triage: How to Set It Up the Right Way
AI ticket triage: what to classify (urgency, category, sentiment, escalation) and how to combine the signals into actions, with thresholds per question.
Triage is not routing with extra steps. Routing asks "whose is this?"; triage asks "what is this, how bad is it, and who needs to look?" Skip the design step and you risk ending up with a single "AI priority" field nobody trusts. This guide covers AI ticket triage: deciding what to classify and how the signals combine.
AI ticket triage: what to classify
Four dimensions earn their keep:
- Category / team. Single choice: the routing decision. Descriptions must be mutually exclusive (see the automated ticket routing guide for option design).
- Urgency. A rating on your scale: does this block work, is a customer losing money, is it cosmetic? Urgency is about impact and time-sensitivity, not politeness.
- Sentiment. Cheap to score, useful as a modifier (an angry customer on a routine issue still deserves a glance), misleading as a priority driver. A frustrated but trivial ticket is still trivial.
- Escalation need. Yes/no: does this need a human now, beyond the normal queue? This is your safety question, and it deserves its own threshold.
One thing not to ask the model: customer tier, contract value, VIP status. That's a CRM lookup keyed on the customer ID, joined after the model runs. Models guess those badly and you already know them exactly.
Designing the triage logic
Write the policy before wiring anything:
- Which dimension decides what? Typically: category sets the queue; urgency sets the position; sentiment modifies within a band (never overrides a level); escalation overrides everything into a human's hands.
- Combine explicitly. "Priority = urgency, plus one level if sentiment is negative AND customer tier is enterprise" is a rule you can audit. "Ask the AI how important this is" is not.
- Use bands rather than points. Triage outputs feed your service-level agreements (SLAs) and queues; broad bands (low/normal/high) survive model wobble better than fine-grained 1–100 scores.
- Every auto-decision has an exit. For each action the triage triggers, define what happens when confidence is low.
Asking for all of it in one call
With a decision model, the dimensions are typed questions in one request, each with its own threshold, because a wrong urgency call costs more than a wrong sentiment call:
from saina import Saina
client = Saina('https://helm.internal.example', api_key='YOUR_KEY')
result = client.ask(
model='saina-helm-0.8b',
state=ticket_text,
mode='decision', # required for per-question thresholds
questions={
'category': {
'type': 'single_choice',
'question': 'Which team should handle this?',
'options': category_options, # your routes, as ID -> description
},
'urgency': {
'type': 'rating',
'question': 'How urgent is this, considering impact and time sensitivity?',
'levels': ['Routine; no one is blocked', # level 0
'Should be handled today', # level 1
'Someone is blocked or losing money'], # level 2
'threshold': 0.85,
},
'sentiment': {
'type': 'single_choice',
'question': 'How does the customer sound?',
'options': {'calm': 'Neutral or positive',
'frustrated': 'Annoyed but functional',
'angry': 'Hostile, threatening to leave'},
'threshold': 0.6, # low stakes: loose bar
},
'escalate': {
'type': 'yes_no',
'question': 'Does this need a human now, ahead of the queue?',
'threshold': 0.9, # strictest bar: if neither yes nor no reaches it, a human looks
},
},
)
answers = result['answers']
urgency = answers['urgency']['level'] # 0, 1, 2, or None if rejected
escalate = answers['escalate']['selected'] # True, False, or None if rejected
if escalate is not False: # yes, or unsure: a human looks now
send_to_on_call(ticket_id, answers)
elif urgency is None or answers['category']['selection'] is None:
send_to_review(ticket_id, answers)
else:
enqueue(answers['category']['selection'], band=urgency,
bump=answers['sentiment']['selection'] == 'angry')
Answers come back as a plain dict with per-option probabilities and, in decision mode, a reason for anything rejected. Then your rules do the combining; the model doesn't "decide priority" in some unspecifiable way. Note the escalation rule: a rejected escalation answer is treated like a "yes", so a strict threshold there sends more tickets to a person, never fewer.
Hosted decision APIs, such as OpenAI's GPT-6 Luna Decisions, also take many questions per call, so the same triage design works there; check your provider's docs for how thresholds are set, or apply them to the returned probabilities in your own code. Helm is the self-hosted option: the tickets stay on your own servers and the model version stays pinned.
Making it safe to trust
- Per-question thresholds. Strict where mistakes are expensive (escalation at 0.9, urgency at 0.85), loose where they're cheap (sentiment at 0.6). These are starting values to tune on your own tickets, not validated settings.
- Watch the escalation question's false-negative rate hardest. It's the one whose miss means an angry customer waited.
- Feedback loop. Reviewers' corrections go back into the labeled set (a small sample of your own tickets for a first check, a larger one before you tune thresholds); re-run monthly, and after any change to questions, options or model version.
- Report per dimension. A single overall score hides where it fails; "category X% correct, urgency Y%, escalation misses Z%" (illustrative placeholders, measure your own) is a system you can improve.