How to Choose a Classification Probability Threshold

A classification probability threshold is one number doing an economic job: every point you raise it buys fewer mistakes and more human reviews. This guide is about choosing that number deliberately. For where the scores come from and what they mean, start with the AI confidence scores guide. for building the branch once you've chosen the number, see Build a human handoff trigger.

Coverage vs. precision: the trade-off, concretely

Two quantities move against each other as you raise the threshold:

• Coverage: the share of items whose top probability clears the threshold and get handled automatically.
• Precision on the auto-handled set (also called selective accuracy): of the items handled automatically, the share that are correct. This is accuracy measured only on the items above the bar.

At a low threshold such as 0.5 you auto-handle most of what the model prefers (with three or more options the top probability can sit below 0.5, so even then some items fall through): high coverage, and the most exposure to the model's errors. At 0.95 most items go to review: errors among auto-handled items get rare, but not zero, and the automation is doing little. The right number is where the costs balance:

(mistakes that slip through) × (cost of a wrong action) vs. (items deferred) × (cost of a review)

A wrong action that emails a customer a refund is expensive; a wrong tag costs a glance. Same model, different thresholds. That's why thresholds are per question, not per system.

The procedure

1. Build a labeled set. 50–100 real items with agreed answers are enough for a first check; for tuning thresholds, use 200+. Include the hard cases; a threshold tuned on easy data is optimistic.

2. Score them. Run the scorer (decision model or otherwise) and record, per item, the full distribution, or at least the top probability and the gap between the top two.

3. Sweep the threshold. For each candidate value (0.5, 0.6, …, 0.95), compute on your set:

• Coverage (share above the bar) and the count of items above it
• Precision on the auto-handled set
• Implied reviews and mistakes at your traffic volume

Mind the sample size. At high thresholds the auto-handled subset gets small. With ~200 labels and 40% coverage you're measuring precision on ~80 items, where 99% means about one error, and one more or one fewer error swings it by more than a point. Require a minimum count per row (say 50+ auto-handled items) before trusting it, or show a confidence interval next to each precision figure.

4. Find your knee. Precision-vs-coverage typically has a knee: a region where giving up a little coverage buys a lot of precision. Near the knee, price the two error currencies and pick.

5. Sweep threshold and margin together. A second bar on the top-two gap (min_margin) can reject "torn" answers, but only in part of the grid. For single-choice and yes/no questions the probabilities sum to 1, so any answer with top probability ≥ t already has a gap ≥ 2t − 1. At t = 0.8 that's 0.6, so a margin of 0.05 or 0.2 never rejects anything. A margin bar only binds when min_margin > 2t − 1, i.e. at lower thresholds with several options (for example threshold 0.5 with min_margin 0.2). So sweep the pair as a grid, not one after the other. With Helm's decision mode, the threshold defaults to 0.8 and can be overridden per question, min_margin defaults to 0, and every answer comes back with a reason (accepted, below_threshold, below_margin or tie); logging those reasons is your workflow's job, and it is what makes the step auditable.

6. Run it in shadow mode. Run the chosen pair logging-only for a week before enforcing it. The numbers from your labeled set meet reality here.

Worked example (illustrative numbers)

Suppose 1,000 items/day, review cost ~$3 each, and a wrong auto-action costing ~$50 on average. Sweeping a threshold on a labeled 500-item sample might show:

(Illustrative numbers, not data. Mistakes/day = auto-handled items × (1 − precision); cost = mistakes × $50 + reviews × $3.)

Here 0.8 is the cheapest row. Going down to 0.7 roughly triples the mistakes and adds ~$850/day; going up to 0.9 cuts mistakes by about 8/day but adds 210 reviews, ~$215/day more. Below the knee mistakes dominate the bill; above it reviews do. Your curve, and your two prices, are the ones that matter.

Per-class thresholds

Not all errors are equal within one question: misrouting to "other" is cheap, mislabeling something "legal-hold" is not. A decision-mode threshold applies to the question as a whole, not per option, so per-class strictness has to come from somewhere else. Two options:

• Lower question threshold + per-class cut-offs in the workflow: set a permissive threshold on the question, then check the returned probabilities in your workflow and require a higher value before acting on the expensive class (e.g. act on "legal-hold" only at ≥ 0.95).
• A separate yes/no gate for the expensive class: ask "Is this a legal hold?" as its own yes/no question with its own, stricter threshold.

Either way, measure per-class error rates at your chosen operating point and confirm each is priced.

Re-tune when the world changes

New categories, new phrasings, model updates, seasonal traffic: re-run the sweep on fresh labeled data monthly, and after any change to the options or the model version. Also re-run it any time the fallback rate or the mistake rate moves.

Checklist

• Labeled set: 50–100 items for a first check, 200+ for tuning, hard cases included.
• Sweep table computed: coverage, count, precision (with interval or minimum count), reviews, mistakes.
• Both error currencies priced (review cost, wrong-action cost).
• Threshold and margin swept together as a grid.
• Per-class error rates reviewed at the chosen operating point.
• Shadow week before enforcing; re-tune monthly and after option or model changes.

Related posts

• AI confidence scores
• Model uncertainty scores
• Build a human handoff trigger