Blog · October 24, 2026

AI Confidence Scores Explained: How to Use Them in Workflows

AI confidence scores explained: where they come from, ranking quality vs. calibration, and how workflows use them for thresholds, review queues and escalation.

Saina · October 24, 2026 · 5 min read

Automation needs a number to branch on: handle automatically or send to review, act or escalate. That number is a confidence score, and where it comes from decides whether it means anything. This guide covers what these scores are, what they can and can't tell you, and the workflow patterns that use them well.

Where a confidence score comes from

Pattern 1: the model writes it out. You ask "how confident are you?" and the model replies "90% sure". This is generated text. It is a stylistic claim, not a measurement: the same model will happily say 90% to questions it gets wrong, and the number moves with phrasing more than with evidence. You cannot put a threshold on this and call it a control.

Pattern 2: token log-probabilities (chat models, where exposed). The model's actual probability for the tokens it emitted. A real signal, closer to a measurement, but its shape depends on tokenization and phrasing ("billing" vs " Billing"), and provider support varies. Useful with care; check current provider docs rather than assuming.

Pattern 3: per-option probabilities from a scoring model. A decision model scores every option and returns the probability distribution directly: {"billing": 0.94, "technical": 0.04, "other": 0.02}. The number exists because the architecture produced it, not because someone asked in prose. This is the cleanest interface for workflow branching. Saina Helm returns it on every answer. Hosted decision models, GPT-6 Luna Decisions among them, return per-option probabilities as well; the field names and response shape differ by provider, so check the docs for the one you use. The workflow patterns below apply to any of them. (Background on decision models vs. chat LLMs: LLM decision making.)

Ranking quality vs. calibration

Two different properties hide behind "is this score any good?", and they support different uses.

Ranking quality: do higher scores go with more correct answers? If items scored 0.9 are right more often than items scored 0.7, the score ranks well. That is all an empirically tuned threshold needs: you sweep candidate cut-offs on labeled data, measure accuracy above each one, and pick. The score doesn't have to mean anything in absolute terms.
Calibration: does 0.9 mean "right about 90% of the time"? You need this only when you want to read the number directly as a probability, for example to estimate expected errors without a labeled sweep, or to compare scores across questions or models.

A well-ranking but miscalibrated score is still useful for thresholds; it just can't be quoted as a percentage. Calibration is measured, not assumed: bucket predictions by stated score and compare each bucket's stated vs. actual accuracy on a labeled set (a reliability curve). No model's probabilities should be called calibrated until that measurement exists for the relevant distribution, including ours. Treat every "confidence" as unproven until you've checked it on your traffic, which is also why thresholds get tuned per deployment rather than copied between them.

Helm's numbers: `probabilities`, `threshold` and `confidence`

Helm returns three related things, and they are easy to mix up:

probabilities: the per-option distribution. In decision mode, threshold compares the raw top probability from this distribution (default 0.8).
confidence: a separate normalized field, not the top probability. For single_choice it is (p_max − 1/n) / (1 − 1/n), so 0 means "uniform over n options" and 1 means "all mass on one option". For yes_no it is 2·p_max − 1. A three-option answer with top probability 0.8 has confidence 0.7.
reason: why the answer was accepted or rejected (below).

Per question type: single_choice and yes_no use threshold and min_margin; rating questions use the threshold but ignore min_margin; multi_choice applies the threshold per label (each label is accepted independently), so at the 0.8 default often nothing clears and tag questions usually need a lower per-question threshold.

How workflows use confidence

Thresholds. Act automatically only when the top probability clears a bar; below it, review. The threshold is a cost dial (review handling vs. mistake handling), tuned on your data, per question. Choosing a probability threshold covers the sweep.

Margins. The gap between the top two options flags a model torn between two answers. Because probabilities sum to 1, a top probability of at least t already guarantees a gap of at least 2t − 1, so a margin bar only adds anything when it is larger than that, i.e. with lower thresholds and many options. Example (illustrative): threshold 0.45, scores billing 0.48 / refunds 0.46 / technical 0.04 / other 0.02. The top clears 0.45, but a min_margin of 0.1 rejects it. Helm checks, in order: tie (top two exactly equal) → below_threshold → below_margin → accepted. Model uncertainty scores covers margin, entropy and what each signal catches.

Review queues with context. Every deferred item should arrive with the full distribution and the reason, so the human sees the model's dilemma rather than a bare ticket. Reviewed items are labeled data, and the loop compounds. Build a human handoff trigger builds this.

Monitoring. The share of rejected answers per question, trended over time, is an early warning for drift. Catch LLM classification errors covers dashboards and alerts.

Escalation chains. Confidence decides where a case goes: clear → auto; torn → human; important-and-torn → the big model or the senior person.

In n8n with Helm

The Saina community node (self-hosted n8n) has two outputs: Selected receives accepted decisions, Fallback receives rejected ones (below_threshold, below_margin, tie). Each node asks one question; set a per-question Confidence Threshold (default 0.8) and Minimum Margin (default 0) under Advanced. Wire Selected to the action and Fallback to your review queue. A full routing build is in n8n support ticket routing.

What confidence is not

Not correctness. A well-calibrated 0.7 is still wrong about three times in ten. Confidence tells you how much to trust the process, not whether this answer is right.
Not a substitute for an eval set. Thresholds tuned without labeled data are vibes with decimal points.
Not comparable across models. Your 0.8 and another model's 0.8 are different units until you've measured both on the same data.

Checklist

Confidence source identified per node (written-out / logprobs / scoring model).
Written-out confidence is treated as decoration, not a control.
Threshold tuned on labeled data (as a rough starting point, 50–100 examples for a first check, 200+ for tuning); calibration checked before reading scores as percentages.
Threshold (and margin, where it can bind) set per question; reasons logged.
Review queue items carry distributions and reasons.