Chapter 8: Confidence and Confidence-Gated Routing

What Confidence Measures

TypeSafe's confidence works like a five-minute utilization average: it collapses many possible values into one summary figure, and is therefore useful but lossy. Every Choice and Score answer already carries a full probability distribution, and confidence is a statistic computed from it — a property of numbers the model already produced, not a heuristic bolted on afterward.

Peaked Versus Flat Distributions

The shape that matters is concentration. A flat shape, with probability spread across several options, means low confidence. A single peak on one option means high confidence. Two answers to the same NOC triage question illustrate the point:

Both answers return the same choice, because choice is simply the highest-probability option. If your code reads only choice, the two incidents are indistinguishable — but Incident A is a fabric event that could hardly be anything else, while Incident B is a 34% guess in a four-way race where random guessing scores 25%.

Figure 8.1: The same choice, two distributions

Incident A — peaked                    Incident B — flat
confidence ~0.93                       confidence ~0.38

dc_fabric     ####################     dc_fabric     #######
              0.91                                   0.34
network_core  #                        network_core  ######
              0.05                                   0.29
wan_transport |                        wan_transport ####
              0.03                                   0.22
wireless      |                        wireless      ###
              0.01                                   0.15

choice = dc_fabric                     choice = dc_fabric

Same selected label, same field value, entirely different events. The left one is a fabric alert that could hardly be anything else. The right one is a 34% guess in a four-way race where blind guessing scores 25%.

Confidence as a Summary; Probabilities as the Full Picture

Confidence is the five-minute average; probabilities is the per-sample detail. Consider two low-confidence answers that both report roughly 0.4:

{
  "owner": {
    "choice": "dc_fabric",
    "probabilities": {
      "dc_fabric": 0.46,
      "network_core": 0.44,
      "wan_transport": 0.06,
      "wireless": 0.04
    },
    "confidence": 0.41
  }
}

...versus a genuinely four-way split at 0.30 / 0.26 / 0.23 / 0.21. The first is a two-horse race: the model has firmly ruled out wireless and transport and cannot separate fabric from core — actionable, route to a combined queue. The second is noise, and the right move is a human.

This is the hierarchical fallback idea: across 60 SEC filings with a threshold at 0.9, confident predictions were 90% accurate, uncertain predictions forced to be specific were 40% accurate, and those same uncertain predictions reported one level up as divisions were 70% accurate. Degrading gracefully nearly doubled the accuracy of exactly the cases the system would otherwise get wrong.

Low Confidence as a Signal the Question or State May Be Wrong

Persistent low confidence deserves the same suspicion network engineers apply to an impossible interface counter — check whether you polled the right OID before blaming the device. Three causes account for most of it:

• Overlapping criteria. If two options' descriptions both say "switching infrastructure," the model has no basis for separating them. Use structured criteria (what, not_for, examples) to draw the boundary explicitly.
• Thin state. A syslog line with no peer address, device role, or site tag genuinely does not identify an owning team — the low confidence is correct, and the fix is enriching state before you ask.
• A question that is not atomic. "Which team should own this and is it urgent?" is two questions; the symptom is broad, persistently mediocre confidence across many incidents rather than a hard split between two options.

One clarification worth holding onto for the rest of the chapter and the next: this chapter's confidence ladder is not the same thing as Chapter 7's Noul-probability ladder. Noul probability scores the likelihood that a statement is true; confidence scores how concentrated a Choice or Score answer's distribution is. They are different quantities that happen to share the same 0-to-1 range, so a given Noul probability value and a confidence value of the same magnitude are not interchangeable — the two ladders are tuned independently. Where the action class is the same, the two tables agree by design: read-only enrichment clears at 0.50 on both, routing and assignment at 0.60, and anything destructive at 0.95 plus corroboration.

The Three-Tier Threshold Pattern

The pattern fits in one sentence: "The answer tells you what; confidence tells you whether to act." Confidence-gated routing uses the confidence score as a decision filter alongside the answer, so the same choice value leads to different behavior depending on how strongly the model held it — the QoS-classification analogy: a marking doesn't change what a packet contains, only what the network is willing to do with it.

These are starting points, not settings — "the correct threshold values depend on your domain and the performance of the model for your use case." The two pages disagreeing slightly is itself the lesson: illustrative bands you replace with measured ones.

Figure 8.2: Three-tier confidence-gated routing

High Confidence: Act Without Human Involvement

The high tier is for answers where the probability mass sits on one option and the action is worth automating. In the NOC service, an access point that deauthenticates and re-associates inside a known maintenance window — a pattern closed as noise hundreds of times — should not consume a human minute at 0.97 confidence. Two rules keep this tier safe: write the confidence value and question name onto every automated action for auditability, and sample it — a random 2% of auto-actioned tickets into a weekly human review queue is how you learn a 0.97 answer stopped being 97% accurate.

Medium Confidence: Proceed with Caution or Flag

The middle tier is where most real traffic lands. Between 0.6 and 0.85, lower-risk actions such as checking an account balance proceed automatically, while higher-stakes actions request confirmation rather than acting independently. In incident terms, assigning to a team is the balance check — cheap to get wrong, trivially reversible — while auto-resolving is the transfer approval, where a wrong call closes a real outage silently. So the medium tier performs the reversible action and attaches a review flag: a work note carrying the confidence value, the runner-up option, and a marker asking a human to confirm.

Low Confidence: Route to a Human or a Fallback System

Below the floor, the model is telling you it does not know. "Route to a human" has a formal name — selective prediction, where a system may abstain rather than predict, with explicit risk control so the review queue is not overwhelmed. The binding constraint is capacity: if 30% of alerts fall below the floor and the night shift is two people, you have built a queue nobody drains, not a safety net. As the human queue lengthens, the confidence threshold for automation must decrease, meaning the bar for escalation must rise — a counter-cyclical rule, best expressed as a runtime value rather than a constant compiled into the triage service.

Scaling Thresholds to Risk

One threshold for a whole service is the most common mistake in this pattern. Different actions within the same system should be gated at different levels depending on the consequences of getting it wrong — matching thresholds to consequences prevents both over-caution (rejecting valid high-confidence requests) and dangerous under-caution (acting on uncertain high-risk decisions). The mental model is a change management process: a show command needs no approval, a VLAN description edit needs a peer, and a BGP policy change needs a CAB review.

The last row surprises people: auto-resolving a ticket feels gentler than pushing a config change, so teams gate it loosely — but it deserves the strictest gate because it is the only action with no natural detection path. A bad config change announces itself through alarms; a wrongly closed incident announces itself when a customer calls three hours later.

Figure 8.3: Risk-scaled threshold ladder

Start Conservative, Measure, Then Loosen

1. Shadow mode, two weeks. Ask the questions on every incident and log answer and confidence, but take no action.
2. Read-only tier only. Turn on tagging and enrichment at 0.50. Nothing can break.
3. Assignment at a deliberately high bar. Enable auto-assignment at 0.90, and measure the reassignment rate.
4. Loosen in steps, measuring each. Drop to 0.85, then 0.80, then 0.75, and stop when the reassignment rate crosses what your team tolerates.
5. Auto-resolution last, on one narrow class. Enable it for a single well-understood noise pattern at 0.95, with 100% audit sampling for the first month.

Plotting Confidence Against Accuracy on Historical Tickets

Everything above depends on one table you can build from tickets you already closed — a reliability diagram: partition predictions into roughly ten equal-frequency bins, plot each bin's average predicted probability against the accuracy actually observed, and compare against the ideal diagonal where confidence equals accuracy.

Read three things off it. The model is roughly calibrated — accuracy tracks confidence bucket by bucket. Pick each threshold by naming the accuracy you require: if auto-resolution must be 98% correct, the table says 0.95 and you automate 30% of volume. And find the cliff — the steepest drop in the table (13 points, between the 0.75–0.84 and 0.60–0.74 buckets) is a natural threshold location, since just above it you capture most available coverage and just below it accuracy degrades fast. Rebuild the table monthly — it is your drift detector.

Figure 8.4: The threshold-tuning loop

Worked Example: Auto-Close, Assign, or Page

One client.system_one(...) call asks two questions about an incoming ServiceNow incident: a Choice for the owning team — with a fifth option for known transient noise — and a Score for severity. The confidence values then select one of three actions against the ServiceNow Table API. Incidents are updated with PATCH /api/now/table/incident/[sys_id]; PATCH updates only the fields you send, unlike PUT which replaces the whole record. The state field is numeric — 1 = New, 2 = In Progress, 3 = On Hold, 6 = Resolved, 7 = Closed.

Every tunable value lives in one dictionary at the top, so raising the escalation bar during a major incident is a config edit rather than a code change:

CONFIG = {
    # TypeSafe
    "model": "jev-latest",

    # ServiceNow
    "sn_instance": "https://acme.service-now.com",
    "sn_incident_table": "/api/now/table/incident",
    "sn_state_in_progress": "2",
    "sn_state_resolved": "6",
    "sn_close_code": "Solved (Permanently)",

    # Confidence thresholds -- STARTING POINTS from the TypeSafe docs,
    # replaced by values read off our own confidence-vs-accuracy table.
    "auto_resolve_min_confidence": 0.95,   # terminal action: strictest gate
    "assign_min_confidence": 0.60,         # reversible action: middle tier
    "review_flag_max_confidence": 0.85,    # below this, flag for human review

    # Severity gate for paging on low-confidence answers
    "page_min_severity": 3.0,              # index into the severity legend

    # Routing targets. Values are sys_ids from sys_user_group in YOUR
    # instance -- a group's display name also resolves, but sys_id is
    # the reference that survives someone renaming the group.
    "noise_option": "known_noise",
    "fallback_group": "<sys_id of Network Operations>",
    "team_groups": {
        "dc_fabric":     "<sys_id of DC-Fabric>",
        "network_core":  "<sys_id of Network-Core>",
        "wan_transport": "<sys_id of WAN-Transport>",
        "wireless":      "<sys_id of Wireless-NOC>",
    },
}

def triage(incident: dict) -> dict:
    response = client.system_one(state=state, questions=QUESTIONS, model=CONFIG["model"])
    owner = response.answers["owner"]
    severity = response.answers["severity"]
    sys_id = incident["sys_id"]

    # Tier 1: high confidence on known noise -> auto-resolve
    if owner.choice == CONFIG["noise_option"] and owner.confidence >= CONFIG["auto_resolve_min_confidence"]:
        return auto_resolve(sys_id, owner)

    # Tier 2: medium confidence on a real team -> assign, flag if borderline
    if owner.choice != CONFIG["noise_option"] and owner.confidence >= CONFIG["assign_min_confidence"]:
        return assign(sys_id, owner)

    # Tier 3: below the floor -> page or park for a human
    return escalate(sys_id, owner, severity)

ServiceNow Incidents: Auto-Resolve Known Noise at High Confidence

This is the strictest gate in the service, and it fires only when the model both picks known_noise and is peaked on it. The close_notes record the confidence and the runner-up, so a reviewer auditing a wrongly closed ticket sees exactly how close the call was. A 403 on the PATCH is not a transient failure to retry — it will fail forever until someone grants the integration user the role.

Assign to a Team at Medium Confidence with a Review Flag

The assignment is the reversible action, so it proceeds automatically across the whole medium band. What changes inside the band is the review flag: between 0.60 and 0.85 the work note asks a human to confirm; above 0.85 it does not. Writing the top three probabilities into the note makes graceful degradation visible — on a near-tie between two teams, the receiving engineer sees it immediately and knows where to bounce it. work_notes is the internal activity log, and u_ai_confidence is a custom field so the monthly bucket table is a database query rather than a log-parsing exercise.

Page On-Call at Low Confidence for High-Severity Guesses

Notice what the low tier does not do: throw the answer away. It assigns to Network Operations, the parent group above the four specific teams — a hierarchical fallback, reporting a coarser level rather than forcing a narrow category — and the best guess plus the full distribution travel with the ticket. The severity gate keeps paging humane: low confidence on a low-severity event is a queue item, while low confidence on something scored major or critical is a page, because an unclassified potential outage is exactly when a human must look now.

Figure 8.5: Auto-close, assign, or page worked example