Support Ticket Classification: Methods Compared
Support ticket classification starts with the label set: single vs. multi-label, how many categories, handling overlap, and keeping labels stable over time.
Most support ticket classification problems are taxonomy problems. Every method you might pick inherits the flaws of the label set, so this guide covers taxonomy design first and compares the methods after.
Designing the label set for support ticket classification
Single-label vs. multi-label
Ask: can a ticket genuinely be two things at once? "Refund request for the item that never arrived" is billing and delivery/shipping; forcing one label loses information.
- Single-label fits routing (one team owns the ticket).
- Multi-label fits tagging (several topics can apply).
- They can coexist: one
single_choicerouting question plus onemulti_choicetagging question. If you're using a decision model, that's two questions in one request, each tag scored independently (set a lower per-tag threshold than for routing, e.g. 0.5 as a starting point to tune, or few tags will clear). On a chat model it's one structured-output prompt with a schema to validate.
How many categories
- 5–12 for a first pass (a rule of thumb). Beyond that, add hierarchy: first department, then team. Flat sets of 40 invite overlap.
- Merge anything that shares a queue and an SLA. Categories that route identically are noise for the model and for you.
- Split anything reviewers argue about often. Recurring arguments mean the boundary is real and undescribed.
Handling overlap
Overlap is a definitions defect. Fixes, in order: sharpen descriptions with concrete examples and non-examples ("billing: charges and invoices; not password resets even if the user mentions billing"); merge; or make it multi-label and let both apply.
Keeping labels stable over time
New products create new categories. Plan for it:
- Stable IDs, evolving descriptions. Renames shouldn't break your joins.
- A change ritual: new label → update descriptions → re-run the labeled test set → check drift on neighbors (adding "refunds" will steal from "billing"; that's correct, measure it).
- A junk drawer you monitor. A growing share of
otherusually means a missing category; read a sample monthly.
The methods compared
| Method | Effort to start | Ongoing effort | Cost model | Consistency | Multi-label | Notes |
|---|---|---|---|---|---|---|
| Manual tagging | None | Scales with volume, humans drift | Per seat (agent time) | Person-dependent | Yes | Ground truth for everything else |
| Keyword rules | Low | Rule sprawl; phrasing defeats it | Free | Deterministic | Clunky | Keep as hard overrides |
| Chat LLM prompts | Low | Prompt/model drift; schema validation | Per token | Variable | Via structured output | No reliable built-in confidence |
| Fine-tuned classifier | High (data + training) | Retrain per label change | Training compute, then cheap inference | High | Per-task training | Fixed label set is the constraint |
| Hosted decision model (e.g. GPT-6 Luna Decisions) | Low | Provider versions | Per token | High per version | Yes/no per tag (check your provider) | Real probabilities |
| Self-hosted decision model (e.g. Helm) | Medium (a server) | You maintain | Flat infrastructure | High per pinned version | Native multi-label | Data stays in; 1–256 questions/call |
Fair reading of that table: chat LLMs classify well in many cases; their weakness is the interface (drift, no reliable confidence), not comprehension, and structured outputs already take care of most parsing. Fine-tuned classifiers are excellent and cheap per call once built; their tax is retraining whenever labels change. Decision models keep the label set in the request, so a label change is a data change. Hosted ones such as GPT-6 Luna Decisions need no server; you pay per token and follow the provider's version schedule. A self-hosted one like Helm keeps data in your infrastructure at a flat cost, and you run and update the server yourself. Whether either does as well as a tuned classifier on your tickets is something to test with both on your labeled set.
Choosing, given a taxonomy you trust
- Labels change monthly → decision model (options are request data) or chat prompt.
- Labels frozen, huge volume, ML capacity → fine-tuned classifier.
- Data can't leave → self-hosted: classifier or open-weights decision model.
- Low volume, option set in flux, no need for a confidence signal → chat prompt with structured outputs.
- Whatever you choose: same labeled set (50–100 examples for a first check, 200+ for tuning thresholds), same test ritual (see the routing guide), same review branch. The taxonomy work is shared; only the engine differs.