Blog · October 27, 2026

Build a Human Handoff Trigger Using AI Confidence Thresholds

Build a human handoff trigger with AI confidence thresholds: step-by-step in n8n and in code, reading rejection reasons, logging handoffs, and retuning.

Saina · October 27, 2026 · 5 min read

You have picked a confidence threshold to flag uncertain AI decisions (Choosing a probability threshold covers the threshold-and-margin sweep). This guide builds the mechanism: the branch that sends flagged decisions to a person, in n8n and in code, with the logging that makes it improve over time.

The pattern

Decision step ── accepted ──► act automatically
            └── uncertain ──► review queue ──► human decides ──► outcome logged
                                                    └─ feeds the next tuning round

Three requirements make it work: the uncertain branch is derived from explicit numbers (threshold, margin), the review queue is somewhere humans actually look, and outcomes loop back. The pattern needs per-option probabilities from the decision step. Saina Helm returns them on every answer, and hosted decision models such as GPT-6 Luna Decisions return them as well, with their own field names; the n8n and code sections below use Helm, and the logging and retuning sections apply to any of them.

Using confidence thresholds to flag uncertain AI decisions in n8n

The Saina community node (@run-saina/n8n-nodes-saina) runs on self-hosted n8n only; it is a community node, not verified by n8n, so it can't be installed on n8n Cloud. On n8n Cloud, use the HTTP Request template described in step 2. The node source reviewed on 2026-10-23 hard-codes the saina-helm-0.8b model ID, so against a Helm 2 server (the release-pinned walkthrough setup) use the HTTP Request path there as well, with model set to saina-helm-2-0.8b.

1. The decision node. Fill in Context (the ticket text), Question ("Which team should handle this?") and Answers (one label per line). One question per node. Under Advanced: Mode Decision (the default), Confidence Threshold (default 0.8) and Minimum Margin (default 0, the required gap between the top two probabilities).

A margin only adds anything when it exceeds 2 × threshold − 1: probabilities sum to 1, so at threshold 0.8 every accepted answer already has a gap of at least 0.6, and a margin of 0.2 never fires. To make the margin do work, lower the threshold. An example to tune, not a recommendation: threshold 0.5 with margin 0.2 (2 × 0.5 − 1 = 0, so the margin binds) on a question with several teams.

2. The two outputs are your branch. Selected receives accepted decisions; Fallback receives rejected ones (below_threshold, below_margin, tie). Every item on Selected is accepted, so no IF node is needed; read the answer at $json.saina.answers.<resultKey>.selection (the Result Key defaults to answer). Inference errors stop the workflow by default. They go to Fallback only if you set the node's error handling to use the Fallback output; those items have saina.reason set to inference_error and no probabilities, so your review step should handle them separately. Invalid form inputs always stop the workflow.

With the HTTP Request template (http-request-portable.json, no custom package, works on n8n Cloud against a public HTTPS endpoint), add the IF yourself and branch on the reason, not on whether selection exists (a rejected answer still has a selection field; it's just null):

IF {{ $json.answers.team.reason === "accepted" }}  →  auto-branch (switch on selection)
ELSE                                               →  review-branch (carry $json.answers.team.probabilities + reason)

HTTP errors stop the workflow unless you configure n8n's error output. If your API doesn't return reasons, compute them: reject when top < threshold or top − second < margin.

3. The review queue. Whatever your team lives in: a Slack channel message ("route decision needed: ticket 4512, billing 0.55 vs refunds 0.41, reason below_margin"; with threshold 0.5 and margin 0.2, the 0.14 gap is under the margin), a Google Sheet row, an email digest. Include the distribution and the reason: a reviewer who sees the model's dilemma resolves it in seconds; one who sees a bare ticket re-triages from scratch.

4. The closing loop. After the human decides, write the outcome back next to the logged decision: ticket ID, probabilities, reason, human's choice, timestamp. This table is your tuning dataset and your audit trail.

In code

import os
from saina import Saina, SainaError

client = Saina('https://helm.internal.example', api_key=os.environ['SAINA_API_KEY'])

try:
    result = client.ask(
        model='saina-helm-2-0.8b',
        state=ticket_text,
        mode='decision',
        threshold=0.5,      # example to tune
        min_margin=0.2,     # binds because 0.2 > 2 * 0.5 - 1
        questions={
            'team': {
                'type': 'single_choice',
                'question': 'Which team should handle this?',
                'options': route_options,
            },
        },
    )
except SainaError as err:
    # request or inference failed: no probabilities, hand off
    handoff(ticket_id=..., probabilities=None, reason=f'error: {err}')
else:
    answer = result['answers']['team']
    if answer['reason'] == 'accepted':
        route(ticket_id=..., team=answer['selection'])
    else:
        # below_threshold / below_margin / tie
        handoff(ticket_id=..., probabilities=answer['probabilities'],
                reason=answer['reason'])

The decision model returns the reason as data, so the branch is an if on a field, not a regex on generated text. In distribution mode (the default when mode is omitted) you'd do the arithmetic yourself on probabilities: same policy, explicit numbers. With a hosted decision API that returns probabilities but no reason codes, that arithmetic is the whole branch: reject when the top probability is under your threshold or the top-two gap is under your margin, and log the reason you computed.

Logging handoffs

Helm returns the reason on every answer; it doesn't log anything, so storing it is your job. Log every deferral with: timestamp, item ID, the full probability distribution, the reason, the model revision, and (later) the human's decision and the time it took. Three uses:

Using reviewed items to retune

  1. Monthly, and after any change to the options or the model version (or when the handoff rate drifts), rerun the threshold sweep. Reviewed items are all low-confidence cases, so on their own they bias the labeled set; combine them with a random sample of auto-handled items, checked by hand, so the set covers both sides of the threshold. Aim for 200+ labeled items for tuning.
  2. Check the human decisions against the model's probabilities: if reviewers often pick the model's second choice, the handoff is catching real races; if they pick options the model scored near zero, something about the input or options is off.
  3. Feed wording fixes and new options back, re-run the test set, adjust thresholds, and ship the update like any other change.

Checklist

Sign up to try a decision All posts