TypeSafe AI for Network and DevOps Engineers

A practical guide to TypeSafe AI’s System One model (Jev) built on the official documentation, teaching network engineers and beginner DevOps engineers to make fast, typed, confidence-aware decisions over Cisco, Arista, Juniper, and Aruba data and integrate them with ServiceNow, Splunk, and Salesforce.

Table of Contents

• Chapter 1: Introduction: Fast Decisions, Not Generated Text
• Chapter 2: AI Fundamentals for Network Engineers
• Chapter 3: Getting Started: Your First System One Request
• Chapter 4: State: Feeding the Model the Right Context
• Chapter 5: The Choice Primitive: Routing and Classification
• Chapter 6: The Score Primitive: Severity, Risk, and Spectrums
• Chapter 7: The Noul Primitive: Yes/No with Real Probabilities
• Chapter 8: Confidence and Confidence-Gated Routing
• Chapter 9: Designing Decisions: Atomic Questions, Fan-Out, and Composite Scoring
• Chapter 10: Advanced Structure and Model Limitations
• Chapter 11: Integrating with the Tools You Already Run
• Chapter 12: Capstone and Next Steps: Building a NOC Triage Pipeline

Chapter 1: Introduction: Fast Decisions, Not Generated Text

Learning Objectives

• Explain what TypeSafe AI and the Jev model are, and how a System One model differs from a generative LLM
• Describe the three primitives (Choice, Score, and Noul) and the typed answer each one returns
• Identify network-operations tasks that fit fast, bounded judgments versus tasks that genuinely need text generation
• Trace how a NOC triage flow would use typed answers instead of parsing LLM prose

What TypeSafe AI Is

The Jev System One Model and the POST /v1/systemone Endpoint

Every network engineer has written the same script at least once. Syslog arrives, a regular expression fires, and a chain of if statements decides whether the event is worth a page at 3 a.m. The regex works until a vendor changes a message string, or until the judgment you need is not “does this string match” but “is this a customer-impacting outage or a maintenance window nobody told us about.” That second question is not pattern matching. It is judgment, and judgment is the part your script has never been able to do.

TypeSafe AI is a platform built for exactly that gap. Instead of treating AI models as text generators whose output a human reads, TypeSafe provides what its documentation calls AI primitives: modular, composable building blocks designed for software integration [Source: https://docs.typesafe.ai/introduction.md]. A primitive is a single, narrowly scoped question with a declared answer shape. You do not ask a primitive to explain itself. You ask it to decide, and it hands the decision back in a form your code can use without translation.

Jev is TypeSafe’s flagship model and the first System One model. The documentation describes it plainly: Jev evaluates “typed questions against a state and returns structured results directly” [Source: https://docs.typesafe.ai/introduction.md]. Three words in that sentence carry the whole architecture, and they will recur in every chapter of this book:

• State is the thing being judged. A syslog line, a ServiceNow ticket body, a JSON blob of BGP neighbor data, a Splunk alert payload. State is your evidence.
• Question is the judgment you want made about that state, expressed as a typed primitive with written instructions and criteria.
• Typed answer is what comes back: a selected label, a numeric score, or a probability, along with a distribution over the possibilities.

Every one of those evaluations goes to a single API endpoint, POST /v1/systemone, and all TypeSafe models are served from it [Source: https://docs.typesafe.ai/models.md]. There is no separate endpoint for classification, no different endpoint for scoring, no chat endpoint versus a completion endpoint. One endpoint, one request shape, many questions per request. If you have ever appreciated a vendor API that exposes one well-designed call instead of fourteen overlapping ones, you will recognize the design instinct here.

Figure 1.1: System One Request and Response Shape

sequenceDiagram
    participant Code as Your Code
    participant Endpoint as POST /v1/systemone
    participant Jev as Jev Model

    Code->>Endpoint: state plus questions dictionary
    Endpoint->>Jev: evaluate all questions in parallel
    Jev-->>Endpoint: typed answers with probabilities
    Endpoint-->>Code: response.answers keyed by question name

The practical consequence is speed. Most System One queries complete in roughly 100 milliseconds, which puts them in the latency budget of real-time interfaces and inline event pipelines rather than batch jobs [Source: https://docs.typesafe.ai/concepts/how-to-build-with-system-one.md]. That is fast enough to sit inside a syslog ingestion path without becoming the bottleneck.

Typed Answers and Probability Distributions Instead of Free Text

Here is the friction that TypeSafe was built to remove. Traditional large language models are excellent at producing human-readable text, but that strength becomes a liability the moment you need machine-readable output. The typical workflow is to coerce a text generator into producing something structured, then parse the result back into a code-consumable format, adding complexity and new failure points at every step [Source: https://docs.typesafe.ai/introduction.md].

System One models skip that round trip. Rather than generating freeform prose, they return typed, structured answers directly: a specific choice selection, a numerical score, or a probability distribution. The documentation’s summary of the approach is blunt — “text generation, no parsing” [Source: https://docs.typesafe.ai/introduction.md].

Typed answers conform to the JSON schemas your code already expects, which eliminates the need to “recover a value from generated prose” [Source: https://docs.typesafe.ai/concepts/how-to-build-with-system-one.md]. Each answer drops straight into a conditional, a threshold comparison, or a field on a ServiceNow record. Think of it the way you think about SNMP versus screen-scraping show output. Both can tell you an interface is down. Only one of them gives you a typed value at a known OID that will not break when the vendor reformats the CLI banner in the next maintenance release.

A second property matters more than it first appears: answers come back as probability distributions, not just point values. When Jev routes a ticket to the “routing” team, it also reports how much probability mass landed on “wireless” and “security.” That distribution is the raw material for every escalation rule you will build in Chapter 8.

The Three Primitives at a Glance: Choice, Score, and Noul

TypeSafe offers exactly three question types, and each returns a different answer shape [Source: https://docs.typesafe.ai/primitives.md]. Three is a small number on purpose. Most of the judgment work in a NOC decomposes into these shapes.

Three points of precision, because they drive real design decisions later. Choice selects from a known set with no inherent order [Source: https://docs.typesafe.ai/primitives.md] — routing team versus wireless team versus security team, which do not sit on a scale. Score positions a judgment along a spectrum and can return a value between your defined levels [Source: https://docs.typesafe.ai/primitives.md], which maps naturally onto things network engineers already scale: syslog severities, QoS classes, change-risk tiers. A score of 1.7 on a three-level rubric tells you something a hard bucket would have thrown away. Noul returns the probability itself as the signal, with no separate confidence calculation [Source: https://docs.typesafe.ai/primitives.md]: a Noul of 0.97 for “this event indicates a hardware failure” is answer and certainty in one number, while 0.51 is the model saying it cannot separate the two cases.

All three can be combined in a single request and evaluated in parallel against the same state, which makes them efficient building blocks for composite judgments [Source: https://docs.typesafe.ai/primitives.md]. This is worth pausing on. You are not making three API calls; you are making one call that asks three questions about the same syslog line.

Here is the shape, so you can see it before Chapter 3 walks through installation and authentication. The state is a single BGP adjacency message off a Cisco IOS-XE core router, and we ask three questions about it at once:

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

client = TypeSafeClient()

syslog_line = (
    "Mar 14 02:17:44 core-rtr-01 %BGP-5-ADJCHANGE: "
    "neighbor 192.0.2.14 Down BGP Notification sent"
)

response = client.system_one(
    state=syslog_line,
    questions={
        "customer_impacting": Noul(
            instructions="This event indicates loss of customer-facing connectivity",
        ),
        "owning_team": Choice(
            instructions="Which NOC team should own this event",
            criteria={
                "routing": "BGP, OSPF, IS-IS, and static routing problems",
                "wireless": "Access point, WLC, and RF issues",
                "security": "Firewall, ACL, and policy enforcement issues",
            },
        ),
        "severity": Score(
            instructions="Operational severity of this event",
            criteria=[
                "Informational, no action needed",
                "Degraded, investigate during business hours",
                "Outage, page the on-call engineer now",
            ],
        ),
    },
)

print(response.answers["customer_impacting"].noul)  # 0.94
print(response.answers["owning_team"].choice)       # "routing"
print(response.answers["severity"].score)           # 2.1

Read that as an engineer, not as an AI practitioner. The state is your evidence. The questions dictionary holds independent judgments, each with instructions in plain English and, for Choice and Score, criteria defining the options or rubric levels. Nothing in the response needs parsing: response.answers["owning_team"].choice is a string you can hand to a ServiceNow assignment-group lookup, and response.answers["severity"].score is a float you can threshold. Chapter 3 covers client setup, the TYPESAFE_API_KEY environment variable, and the jev-latest default; Chapters 5 through 7 take each primitive apart.

Key Takeaway: TypeSafe AI supplies typed AI primitives rather than a text generator, and Jev is its first System One model, evaluating typed questions against a state through a single POST /v1/systemone endpoint. The three primitives — Choice, Score, and Noul — return a selected label, a position on a scale, and a yes-probability respectively, and all three can be asked about the same state in one parallel request.

System One Versus Generative LLMs

Kahneman’s Fast and Slow Thinking as the Naming Origin

The name comes from cognitive science. In the framework popularized by psychologist Daniel Kahneman, human cognition runs in two modes: System 1 is fast, automatic, and intuitive, while System 2 is slower and more deliberate. TypeSafe adopts the distinction directly — “System 1 thinking is fast and intuitive. System 2 is slower and more deliberate” — and builds Jev to emphasize speed and focus [Source: https://docs.typesafe.ai/concepts/system-one.md].

The analogy holds up in a NOC. When a senior engineer glances at a syslog line and says “that’s wireless, not routing,” they are not walking a decision tree; they are recognizing a pattern in under a second on a decade of exposure. That is System 1. When the same engineer sits down to design a new OSPF area layout, sketching options and tracing failure modes, that is System 2.

Confusing the two modes is the root of most disappointing AI deployments. System One models are purpose-built for rapid, constrained judgments suitable for software automation, not for open-ended reasoning [Source: https://docs.typesafe.ai/concepts/system-one.md]. If the task is “look at this and decide,” you want System One. If the task is “write the post-incident narrative for the customer,” you want a generative model. Telling the two apart is a recurring theme of this book.

Why Parsing LLM Prose in Code Is Fragile

Suppose you skip TypeSafe and do this with a generative model. You write a careful prompt asking for JSON, call the API, and json.loads() the response. It works in testing. What goes wrong in production explains why typed answers are an architectural choice rather than a convenience.

When developers manually parse LLM-generated JSON, they implicitly make six assumptions, each of which fails silently at some rate [Source: https://dev.to/pockit_tools/llm-structured-output-in-2026-stop-parsing-json-with-regex-and-do-it-right-34pk]:

1. Validity — that the output is syntactically valid JSON. In practice models wrap output in markdown code fences, add a conversational preamble (“Sure! Here’s the analysis:”), or stop generation mid-token.
2. Schema compliance — that every required field is present and no extra ones appear. Field names drift subtly between requests, optional fields come and go, and models invent fields nobody asked for.
3. Type correctness — that strings are strings and numbers are numbers. Severity scores arrive as "high" instead of 0.85; counts arrive as "five" instead of 5.
4. Value constraints — that values stay in range. Confidence scores exceed 1.0, timestamps fail to parse, enumerated values outside your defined set appear.
5. Consistent formatting — that structure is identical across inputs. Capitalization shifts, field order changes, optional elements appear and vanish.
6. No extraneous content — that the JSON is the only thing returned. Explanatory text before and after the payload is common, requiring pre- and post-processing to isolate the data.

Industry analysis puts unconstrained prompt engineering at roughly 80 to 95 percent valid output, with no guarantee for the remainder [Source: https://dev.to/pockit_tools/llm-structured-output-in-2026-stop-parsing-json-with-regex-and-do-it-right-34pk]. Network engineers know what to do with a number like that: a link dropping 5 percent of frames is not a working link, it is an incident. A pipeline processing fifty thousand syslog events a day at 95 percent parse success throws twenty-five hundred exceptions a day, each needing handling, logging, and retry logic.

The industry’s answer to this is constrained decoding, which enforces validity during generation instead of after it. The mechanism is straightforward once described: the system compiles your schema or grammar into a finite state machine, tracks that state as tokens are produced, and at each step masks the model’s next-token probabilities so only tokens that keep the output on a valid path remain selectable — invalid tokens get their logits set to negative infinity before sampling [Source: https://zeroentropy.dev/concepts/constrained-decoding/]. The result is what one analysis calls “mathematical guarantees instead of statistical ones”: the output is guaranteed valid, so you never get a parse error, never retry, and never need a fallback parser [Source: https://tianpan.co/blog/2026-03-03-structured-generation-reliable-llm-output]. Major providers including OpenAI, Google, and Anthropic now ship structured output modes built on this technique as standard API features [Source: https://dev.to/lycore/structured-outputs-how-we-stopped-parsing-llm-responses-by-hand-3lgb].

TypeSafe goes a step further than bolting a schema onto a text generator. Because Jev is built to emit typed answers as its native output rather than as a constrained special case, there is no prose layer to constrain in the first place [Source: https://docs.typesafe.ai/introduction.md].

Here is the before-and-after, side by side, for the single task of routing one syslog event:

The parallelization row is a genuine architectural difference rather than a performance tweak. Questions run simultaneously because “one primitive’s result does not become hidden context that changes another primitive’s result” [Source: https://docs.typesafe.ai/concepts/how-to-build-with-system-one.md]. In an agent loop built on a generative model, asking “which team owns this?” before “how severe is it?” can change the severity answer, because the first answer became context for the second. With System One, your severity judgment is independent of your routing judgment — the same property that makes a stateless load balancer easier to reason about than a stateful one.

Figure 1.2: Generative LLM Prose Parsing Loop

flowchart TD
    A[Write prompt asking for JSON] --> B[Call generative LLM]
    B --> C[Receive text blob]
    C --> D[Strip code fences]
    D --> E[Parse JSON]
    E --> F{Parse succeeded}
    F -- No --> G[Retry or raise exception]
    G --> B
    F -- Yes --> H[Validate keys and types]
    H --> I[Use value in code]

Figure 1.3: One Parallel Typed Call

flowchart TD
    A[Build questions dictionary] --> B[Call system one with state and questions]
    B --> C[All questions evaluated in parallel]
    C --> D[Typed answers returned]
    D --> E[Use value in code]

One honest caveat, because it will save you an incident review later: structured outputs guarantee shape, not truth. The model can still misjudge, misattribute, or produce a plausible but wrong answer. You receive reliably shaped data whose accuracy still depends on the model’s judgment and the quality of the state you fed it [Source: https://zeroentropy.dev/concepts/constrained-decoding/]. Typed answers eliminate parsing failures. They do not eliminate the need for confidence thresholds, human review paths, and the evaluation discipline covered in Chapters 8 through 10.

Calibrated Probabilities as a First-Class Output

The third differentiator is the one that changes how you design pipelines. System One models return typed answers with calibrated probabilities [Source: https://docs.typesafe.ai/concepts/system-one.md]. A calibrated probability is one where the number means what it says: across a large set of cases where the model reports 0.80, roughly 80 percent should turn out to be correct. This is the difference between a confidence number you can set a threshold against and a number that merely sorts results.

Why this matters operationally: structured outputs enable threshold-based routing, escalating uncertain cases to humans or to more expensive models based on calibrated confidence scores rather than guessing from ambiguous text [Source: https://docs.typesafe.ai/concepts/how-to-build-with-system-one.md]. That is a design pattern you already run. It is the same shape as a routing protocol’s administrative distance, or a QoS policy that puts unclassified traffic in a scavenger class rather than dropping it. High-confidence traffic takes the fast path; low-confidence traffic takes the path that costs more and involves a human.

Compare that to what you get from a generative model. Asked how confident it is, a text model will say something like “I’m fairly confident this is a routing issue, though it could also relate to the wireless infrastructure.” You cannot threshold on “fairly.” With a Choice answer, you get a confidence value and a full probabilities distribution across every option you defined, and you can write:

answer = response.answers["owning_team"]
if answer.confidence >= 0.85:
    assign_to_queue(answer.choice)
else:
    assign_to_queue("noc_triage_review")

That four-line pattern is the backbone of Chapter 8’s confidence-gated routing, and it is only possible because the confidence signal is a number in a field rather than an adverb in a sentence.

Key Takeaway: The System One name comes from Kahneman’s fast, intuitive mode of thinking, and Jev is built for that mode rather than for deliberation or composition. Parsing prose fails at rates that are unacceptable in a production pipeline, while typed answers with calibrated probabilities let you route on explicit thresholds — but structural guarantees are not accuracy guarantees, and the model can still be confidently wrong.

Why This Matters for Network Operations

Analogy: Longest-Prefix-Match Lookup Versus Asking a Consultant for an Essay

Consider two ways of answering “where should this packet go?”

The first is a forwarding table lookup. The router takes a destination address, performs a longest-prefix match, and returns a next hop. The answer is typed — an interface and a next-hop address, not a paragraph. It arrives in microseconds, and no part of the forwarding path parses prose.

The second is hiring a consultant to write a design document on optimal traffic engineering for that flow. The output is genuinely valuable, far more nuanced than a table entry, and takes three weeks. You would never put it in the data plane.

Both artifacts are legitimate; you just would not swap them. A System One call is the forwarding-table lookup of AI judgment: bounded question, typed answer, roughly 100 milliseconds, safe inline in a pipeline [Source: https://docs.typesafe.ai/concepts/how-to-build-with-system-one.md]. A generative LLM is the consultant’s essay, belonging in the control plane of your process where a human reads it.

Most disappointing AI-in-NOC projects are essays deployed into data planes: someone wires a chat model into an alert pipeline, asks it to “analyze this alert and recommend an action,” then spends three months writing regexes to extract the recommendation. The problem was never the model’s intelligence. It was the mismatch between output shape and job.

Where Judgment Calls Hide in NOC Workflows

Once you start looking for bounded judgments, you find them everywhere in operations. They are the steps in your runbooks that say “determine whether,” “assess the,” or “assign to the appropriate.” Those phrases are where an experienced human currently supplies a judgment your automation could not.

Syslog triage. Cisco NX-OS, Arista EOS, Juniper Junos, and Aruba AOS-CX all report interface and adjacency events, and all phrase them differently. The judgment “is this event customer-impacting?” is identical across vendors even though the string patterns are not. That is a Noul over a raw log line. Your regex library must be rewritten per vendor and per software release; a well-written Noul instruction does not.

Ticket routing. A ServiceNow incident arrives with a free-text description written by a field technician at 2 a.m. Choosing the assignment group is a Choice over a fixed set of teams — exactly the “routing decisions” use case the primitive was designed for [Source: https://docs.typesafe.ai/primitives.md]. Today a triage engineer reads it, or a keyword rule sends anything containing “slow” to the wrong queue.

Change review. A proposed Junos or IOS-XE change reaches CAB approval. “How risky is this?” is a Score against a rubric you write once: routine, elevated, high-risk. The score can fall between levels, which is closer to how a change advisory board actually thinks than a three-way radio button [Source: https://docs.typesafe.ai/primitives.md].

Alert deduplication. Is this new Splunk alert the same problem as the incident opened eleven minutes ago? A Noul over a state containing both payloads.

Customer impact classification. Does this incident need a Salesforce case because a named customer is affected? Another Noul, with a confidence threshold deciding whether a human confirms before anything customer-visible is created.

Each of the five is a question a competent engineer answers in about two seconds using judgment rather than calculation, and each currently either consumes human attention or is approximated badly by brittle rules.

Code Owns Control Flow; the Model Supplies Programmable Common Sense

This is the architectural principle that should govern everything you build in this book. System One positions AI as a structured decision-making component inside deterministic software workflows, not as an autonomous agent that runs your network. The documentation states the rule directly: “Keep control flow, deterministic rules, and side effects in code. Break broad judgments into narrow, typed questions with explicit instructions and criteria” [Source: https://docs.typesafe.ai/concepts/how-to-build-with-system-one.md].

Read that as a separation-of-duties statement, because it is one. Your Python code decides what happens: which API to call, which ticket to open, whether to page someone, what to do when a call times out. The model contributes one thing — a judgment your code could not compute — and contributes it as a value, not an action. The model never touches a device, closes a ticket, or pages anyone. Your code does those things, having consulted the model the way it consults any other data source.

Figure 1.4: Code Owns Control Flow; the Model Supplies Judgment

flowchart TD
    A[Event arrives] --> B[Code decides which typed questions to ask]
    B --> C[Jev supplies typed judgment]
    C --> D{Code evaluates the judgment}
    D -- Confidence high --> E[Code takes automatic action]
    D -- Confidence low --> F[Code routes to human review]
    E --> G[Code performs side effects such as ticket update or page]
    F --> G

The other half of the rule is “break broad judgments into narrow, typed questions.” The temptation with any capable model is to ask one big question: “analyze this alert and tell me what to do.” The System One philosophy pushes the opposite way, emphasizing atomic, well-scoped questions composed together in code rather than complex reasoning inside a single prompt, prioritizing reliability and developer control over prompt engineering [Source: https://docs.typesafe.ai/primitives.md]. Three narrow questions combined with an if statement you wrote are auditable. One broad question whose internal reasoning you cannot inspect is not.

The useful framing is that the model supplies programmable common sense. It knows “BGP Notification sent” is more serious than “interface counters cleared” without you encoding that, and that a ticket mentioning a conference room and a projector is probably wireless. The intelligence about your network — escalation matrix, maintenance windows, customer tiers, on-call rotation — stays in your code, in version control, where you can review and test it.

Key Takeaway: A System One call belongs in the fast path of an operations pipeline the way a forwarding-table lookup belongs in a data plane, while generative text belongs where a human reads it. Bounded judgments hide throughout NOC workflows — syslog triage, ticket routing, change review, deduplication, customer-impact calls — and the durable design rule is that code keeps control flow and side effects while the model supplies narrow, typed judgments.

How This Guide Is Organized

The Running Example: A Multi-Vendor NOC Triage Service

Every chapter in this book builds toward one system, so that concepts arrive attached to something you could actually deploy. That system is a multi-vendor NOC triage service.

It takes in syslog and alert data from Cisco IOS-XE and NX-OS, Arista EOS, Juniper Junos, and Aruba AOS-CX devices, along with alerts forwarded from Splunk and incidents already open in ServiceNow. For each event, it asks Jev a small set of typed questions: which team owns this, how severe is it, is it customer-impacting, is it a duplicate of something already open. It then writes team, severity, and confidence back into ServiceNow, and for customer-facing situations, creates or updates a case in Salesforce. Where confidence is low, it routes to a human review queue instead of acting.

Figure 1.5: NOC Triage Flow

flowchart LR
    A[Syslog] --> D[State]
    B[Splunk alerts] --> D
    C[ServiceNow incidents] --> D
    D --> E[Jev typed questions]
    E --> F["Typed answers: team, severity, impact, duplicate"]
    F --> G[ServiceNow fields updated]
    F --> H[Salesforce case for customer impact]
    F --> I[Human review queue if low confidence]

You will meet pieces of that service constantly. Chapter 5 builds its routing Choice, Chapter 6 builds its severity Score, Chapter 7 builds its impact Nouls, Chapter 8 builds the confidence gate that decides between automatic action and human review, Chapter 11 wires it to the real ServiceNow, Splunk, and Salesforce integrations, and Chapter 12 assembles the whole pipeline end to end.

Here is the roadmap:

Chapter 10 deserves a word of warning in advance. Jev is a System One model, which means it is fast and intuitive rather than deliberative, and it has corresponding weaknesses — notably with arithmetic and with instructions it interprets more literally than you intended. This book covers those limitations honestly rather than pretending they do not exist, because designing around a known limitation is ordinary engineering and discovering one in production is not.

Vendors and Tools Used in Examples

The examples in this book use equipment and platforms that show up in real multi-vendor enterprise networks:

You do not need all of these to follow along. The point of the multi-vendor framing is that the judgment layer stays the same while the syslog dialects differ, which is precisely the argument for putting judgment in a typed question instead of a per-vendor regex library. If you run only Cisco, the Arista examples still teach you something: they show how little the question changes when the evidence format does.

Prerequisites: Basic Python, JSON, and REST Familiarity

Three things are assumed, and only three.

Basic Python — functions, dictionaries, if statements, and installing a package with pip. No decorators, async programming, or type-checker fluency is required to follow the reasoning in any chapter, and every core example through Chapter 10 is a plain synchronous function. Two later chapters do reach for more: Chapter 3 mentions the asynchronous client in passing, and the integration examples in Chapters 11 and 12 are written as a FastAPI web service, which means @app.post decorators and a Pydantic model for the incoming payload. Those are framework mechanics rather than TypeSafe concepts — Chapter 11 flags exactly which lines are FastAPI scaffolding, and you can read past them without losing the argument.

JSON — enough to read a nested object and see that {"choice": "routing", "confidence": 0.91} has a string field and a float field. Every request and response in this book is JSON underneath, and Chapter 4 leans on it when we send structured state rather than a raw log line.

REST familiarity — what an HTTP POST is, what an API key does, and why a request can fail with a connection error versus an authentication error versus a validation error. If you have ever called a ServiceNow or Meraki REST API from a script, you are past the bar.

What is explicitly not assumed is machine learning. Chapter 2 supplies exactly the background you need — tokens, probabilities, calibration, and why a model’s confidence is a number worth trusting — and no more. Every AI concept in this book is introduced in networking terms first and generalized second, because that is the order in which it sticks.

Key Takeaway: This guide is organized around one continuously developed system, a multi-vendor NOC triage service that reads Cisco, Arista, Juniper, Aruba, and Splunk events and writes judgments back into ServiceNow and Salesforce. Twelve chapters move from primitives to confidence gating to full integration, and the only prerequisites are basic Python, JSON literacy, and REST familiarity — no machine-learning background required.

Chapter Summary

The central claim of this book is that a large share of the judgment work in network operations does not need generated text. It needs fast, bounded decisions that software can act on. TypeSafe AI is built for that category, providing typed AI primitives rather than a prose generator, and Jev is its flagship System One model — evaluating typed questions against a state and returning structured results directly through a single POST /v1/systemone endpoint, typically in around 100 milliseconds. Three primitives cover most of the ground: Choice for selecting from an unordered set, returning choice, probabilities, and confidence; Score for positioning a judgment on a defined spectrum, returning score, legend, probabilities, and confidence; and Noul for clean yes/no questions, returning a single noul probability that is itself the signal. All three can be asked about the same state in one parallel request.

The contrast with generative LLMs is architectural rather than stylistic. Parsing prose in production code rests on six assumptions that fail silently — validity, schema compliance, type correctness, value constraints, consistent formatting, and no extraneous content — and unconstrained prompting delivers valid output only 80 to 95 percent of the time, a rate no network engineer would accept from a link. Typed answers conform to the schema your code expects, so there is no value to recover from prose, no retry logic, and no fallback parser. Just as important, calibrated probabilities arrive as first-class fields, which turns escalation from a guess into a threshold comparison. What typed answers do not do is guarantee accuracy: the model can still be wrong, and confidence gating plus human review paths remain part of responsible design.

For network operations specifically, the governing discipline is a division of labor. Code keeps control flow, deterministic rules, and side effects; the model supplies programmable common sense as narrow, typed judgments that your code combines. A System One call belongs inline in a pipeline the way a longest-prefix-match lookup belongs in a forwarding path, while generative text belongs wherever a human is going to read it. The chapters ahead build a multi-vendor NOC triage service on that principle, moving from state design through each primitive, into confidence-gated routing and decision decomposition, through Jev’s real limitations, and finally into ServiceNow, Splunk, and Salesforce integration. Chapter 2 supplies the small amount of AI background that makes the rest land, and Chapter 3 gets you making live calls.

Key Terms

Chapter 2: AI Fundamentals for Network Engineers

Learning Objectives

• Define tokens, inference, and classification in plain language, using networking terms you already know
• Distinguish classification and scoring from open-ended text generation, and explain why the difference matters for automation
• Explain what a calibrated probability distribution is and why calibration is the property that makes confidence thresholds safe
• Decide which parts of a NOC workflow should stay deterministic code and which parts can safely be handed to a model

Models, Tokens, and Inference

What a Token Is, and Why Request Size Is Measured in Tokens

A token is the basic unit a language model processes. Roughly, a token is a word or a fragment of a word: “inference” might be a single token, while “engineering” could split into two [Source: https://www.f5.com/company/blog/ai-inference-and-tokens-oh-my]. A model does not read characters and it does not read lines. It reads a sequence of tokens, and everything about the economics of a model request — size limits, price, speed — is expressed in tokens.

This is less exotic than it sounds. A user thinks in “a file transfer”; your switch thinks in frames and your billing thinks in bytes. An MTU limits a transport-layer unit, not sentences, and sentences get chopped to fit. Tokens are the same kind of abstraction — the model’s MTU and its byte counter rolled into one.

Two practical consequences follow. First, a token is not a networking object. Tokens are produced by a tokenizer on the inference server, inside the AI stack, and never travel across your network. Tap the link between your triage service and the model API and you will see JSON text on the wire, not tokens. That matters the moment someone asks you to rate-limit, budget, or set an SLO on token consumption: no packet capture or NetFlow record will give you a token count. You either run a tokenizer at a gateway or consume the counts the model server reports back [Source: https://www.f5.com/company/blog/ai-inference-and-tokens-oh-my]. TypeSafe takes the second route — every response carries a usage object with input_tokens and output_tokens, so your service logs consumption per request the way it logs interface counters [Source: https://docs.typesafe.ai/sdk/python/api/types/responses.md].

Second, different models tokenize differently. GPT-4, LLaMA, and other architectures split the same string into different token sequences, so an accurate count requires the specific model’s tokenizer [Source: https://www.f5.com/company/blog/ai-inference-and-tokens-oh-my]. If you run more than one model, do not assume one token budget transfers to the other — the same discipline you apply before trusting that two vendors’ “input errors” counters mean the same thing.

For the NOC triage service we build across this book, the token budget is dominated by what you put into state. A single Cisco IOS-XE syslog line is a few dozen tokens; a ServiceNow ticket with a long description and three customer replies is several hundred; a full show tech-support dump is enormous and almost never what you want to send. Request size is a design choice you make, not a property of the incident.

Inference as a Stateless Request/Response Call

Inference is the act of running a trained model over an input to produce an output. Operationally, it is a stateless HTTP request/response call, and you should treat it exactly like any other REST API dependency in your stack: you send a payload, you get a payload back, and the server remembers nothing about you afterward. TypeSafe exposes this as a single endpoint, POST /v1/systemone, which the Python SDK wraps as client.system_one(...) [Source: https://docs.typesafe.ai/introduction/quickstart.md].

“Stateless” deserves emphasis because chatbot experience teaches the opposite intuition. A chat assistant appears to remember what you said three messages ago; it does not, the client is resending the whole conversation each time. For a typed decision API there is no conversation at all. Each call carries its own state (the text, JSON object, or array being evaluated) and its own questions, and is answered on its own merits [Source: https://docs.typesafe.ai/sdk/python/api/clients/sync/client.md]. If your triage service needs the model to know this is the fourth flap on the same interface today, that fact has to be in the state you send.

Figure 2.1: Inference as a stateless request/response call

sequenceDiagram
    participant Triage as Triage Service
    participant API as TypeSafe API
    Triage->>API: POST /v1/systemone with state and questions
    API->>API: Process request with no memory of prior calls
    API-->>Triage: Response with answers and request_id
    Note over Triage,API: Each call is independent and idempotent

If you have built anything against a stateless API, the operational playbook transfers directly:

• Timeouts and retries are yours to own. The SDK exposes retry and timeout overrides per call and raises distinct exceptions for validation problems (TypeSafeError), unsuccessful server responses (TypeSafeAPIError), and connection or timeout failures (TypeSafeAPIConnectionError) [Source: https://docs.typesafe.ai/sdk/python/api/clients/sync/client.md]. Because each call is independent, retrying is safe and re-running a failed triage is idempotent.
• Every response is traceable. The response carries a request_id from the x-typesafe-request-id header plus the model that answered [Source: https://docs.typesafe.ai/sdk/python/api/types/responses.md]. Log both alongside your ServiceNow ticket number, as you would a correlation ID through a load balancer. When someone asks six weeks later why a ticket went to the wrong team, that pair is your evidence.

One piece of internal machinery explains performance behavior you will observe. Inference runs in two phases with different bottlenecks. The prefill phase pushes the entire input through the model’s layers in parallel and produces the first output token; it is compute-bound and measured by time to first token (TTFT). The decode phase then generates each subsequent token one at a time, each depending on all previous ones; it is memory-bandwidth-bound and measured in tokens per second (TPS). A technique that speeds up one phase often does nothing for the other, so TTFT and TPS must be measured and optimized separately [Source: https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering].

Prefill is serialization delay — the time to clock a large frame onto the wire, scaling with size. Decode is per-unit processing delay, paid once per output token. A model that emits a long essay pays decode cost on every token of it; a model that emits a label and a probability vector barely enters the decode phase at all. That is the structural reason the typed approach in this book is fast.

Latency, Throughput, and Cost per Million Tokens

Three numbers describe a model dependency in production, and all three have direct counterparts in link engineering.

Latency is the round-trip time for one inference call. As with network latency, the number that matters for a triage service is the tail, not the average: if your ServiceNow integration has a webhook timeout, a p99 above it produces silently untriaged tickets — the same failure signature as a BFD timer set tighter than your worst-case path latency.

Throughput is how much work the endpoint will accept per unit time, expressed in tokens per second and requests per minute. Think of it as the committed information rate on a circuit: you are buying a rate, and traffic above that rate is shaped or dropped. A Monday-morning maintenance window that dumps four hundred Arista EOS interface-flap alerts into Splunk in ninety seconds is a microburst against your model endpoint, and the fix is the one you would apply to a congested uplink: queue, pace, and batch rather than blast. Published rate limits for jev-latest are covered later alongside model selection; the point here is architectural, not numeric.

Cost for token-based APIs is quoted per million or per billion tokens and is usually dominated by input rather than output. Because you control input size, cost control is a data-shaping problem: sending an entire show tech-support when three syslog lines and two ticket fields would do is the AI equivalent of leaving a packet capture running on a metered circuit.

The vocabulary in this section maps onto networking concepts closely enough to be worth keeping as a reference:

Key Takeaway: A token is the model’s unit of accounting, produced inside the AI stack and never visible on your wire, so token counts come from the API’s usage field rather than from your monitoring. Inference is a stateless request/response call, with the timeouts, retries, and request IDs that implies. Latency, throughput, and per-token cost define the dependency, and all three are shaped mainly by how much you choose to send.

Classification Versus Generation

Picking from a Bounded Set Versus Producing Open-Ended Text

Classification is choosing one option from a defined set. Generation is producing open-ended text. These are different products of the same underlying technology, and confusing them is the single most common reason AI projects in network operations go sideways.

Figure 2.2: Classification versus generation

flowchart TD
    A[Model Input] --> B{Bounded Answer Set Defined}
    B -->|Yes| C[Classification]
    C --> D[Choice, Score, or Noul]
    D --> E[Testable and Comparable Output]
    B -->|No| F[Generation]
    F --> G[Open Ended Text]
    G --> H[Requires Human Judgment]

Generation is what most people have seen: you ask a question, you get paragraphs. The output space is effectively infinite — nothing constrains the answer to a known set, nothing guarantees a comparable answer tomorrow, and nothing tells you how sure the model was. It is the right tool when a human will read the result and apply judgment: drafting a post-incident summary, explaining an unfamiliar protocol behavior, sketching a design document.

Classification constrains the output before the model answers. You declare the allowed answers, and the model’s job is to distribute confidence across exactly those options. In the TypeSafe SDK this takes three concrete forms, each declared as a question you pass in the questions mapping [Source: https://docs.typesafe.ai/introduction/quickstart.md]:

• Choice(instructions=..., criteria={...}) — pick one label from a dictionary of labels and their meanings. This is classification in its purest form.
• Score(instructions=..., criteria=[...]) — place the input on an ordered rubric you define as a list, low to high.
• Noul(instructions=...) — a yes/no judgment, returned as a probability of “yes.”

Here is what that looks like for a NOC triage decision. A ServiceNow ticket arrives, created from a Splunk alert that correlated syslog from a Cisco IOS-XE core router and an Arista EOS leaf:

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

client = TypeSafeClient()

ticket = """INC0104882 - Branch 118 users report intermittent loss of the
payroll app since 08:40. Core1 (IOS-XE) logged
%BGP-5-ADJCHANGE: neighbor 10.10.0.9 Down BGP Notification sent
twice this morning. Leaf-2 (Arista EOS) shows Ethernet4 flapping.
Caller says two other branches are fine."""

response = client.system_one(
    state=ticket,
    questions={
        "team": Choice(
            instructions="Which NOC team should own this incident",
            criteria={
                "routing": "BGP, OSPF, route policy, and WAN reachability",
                "switching": "Campus and data-center L2, VLANs, port flaps, STP",
                "wireless": "APs, WLAN controllers, RF and client association",
                "transport": "Circuits, optics, carrier faults, and physical plant",
            },
        ),
        "severity": Score(
            instructions="How severe is the business impact described",
            criteria=[
                "Single user or cosmetic",
                "One site degraded, workaround exists",
                "Multiple sites or a business-critical app down",
            ],
        ),
        "customer_facing": Noul(
            instructions="The incident affects an externally visible service",
        ),
    },
)

print(response.answers["team"].choice)      # "routing"
print(response.answers["severity"].score)   # e.g. 1.8
print(response.answers["customer_facing"].noul)  # e.g. 0.21

Three things are worth noticing. The set of teams is closed — the model cannot invent a fifth team, because criteria defines the universe of valid answers. The severity rubric is yours, in your operational language, not a generic 1–10 scale the model imagined. And the yes/no question returns a number between 0 and 1 rather than the word “yes,” because the useful output is a degree of belief, not a verdict [Source: https://docs.typesafe.ai/sdk/python/api/types/responses.md].

Why Bounded Outputs Are Testable and Comparable

A bounded output is testable in the ordinary software sense. If the answer must be one of routing, switching, wireless, or transport, you can run three hundred historical ServiceNow tickets whose true owning team you already know, compute an accuracy number, diff last month against this month, and fail a CI build when accuracy drops. None of that works on a paragraph of prose — there is no assertion you can write against “the answer sounded reasonable.” Bounded outputs are also comparable: two tickets that both return switching at 0.9 confidence are records you can count, group, alert on, and trend, while free-text explanations cannot be aggregated without a second parsing layer that adds its own errors.

This is why TypeSafe frames its approach as AI with “software-like properties such as structure, reliability, observability, testability, speed, consistency, and low cost,” aimed at machine-to-machine interaction rather than human-facing chat [Source: https://docs.typesafe.ai/introduction/machine-learning-primer.md]. The target is not a system that impresses a reader but one that behaves like a function you can write tests for.

There is a performance dividend too. A classification answer is a handful of output tokens; a generated explanation is hundreds or thousands, each paid for sequentially in the memory-bandwidth-bound decode phase [Source: https://blog.bytebytego.com/p/a-guide-to-ai-inference-engineering]. Choosing classification is a latency and cost decision as well as a correctness one.

The ACL Analogy: A Match Decision Versus a Design Document

The cleanest way to hold this distinction is to compare two things you already do.

An ACL entry makes a bounded decision. A packet arrives, the rule set matches it against a finite ordered list of conditions, and the outcome is permit or deny. You can test it with known packets, diff it between two devices, prove what it will do without running it, and log every hit. It is auditable because the answer space was defined before the packet arrived.

A network design document is open-ended. It is prose for humans, it varies by author, and no automated test says it is correct. It is valuable — you cannot build a network from ACL entries alone — but it is reviewed by judgment, not by assertion.

A Choice question is an ACL match; a chatbot prompt is a design document. Both are legitimate, but only one belongs in a path where the output feeds an automated action. If you are about to put generated text into a field another system will act on, you have written a design document where an ACL belonged.

Key Takeaway: Classification picks one option from a set you defined in advance; generation produces open-ended text with no bounded answer space. Bounded outputs can be unit-tested against labeled history, compared across time, aggregated into metrics, and returned in a few tokens instead of thousands. The test is simple: if another system will act on the answer, it needs to be an ACL match, not a design document.

Probabilities and Calibration

Reading a Probability Distribution Across Options

A classification model does not really pick one answer. It produces a probability distribution — a confidence value for every allowed option, summing to 1.0 — and the “answer” is just the option that came out on top. A ChoiceAnswer in the TypeSafe SDK carries exactly this: the selected label, a confidence score, and probabilities per label [Source: https://docs.typesafe.ai/sdk/python/api/types/responses.md].

Take the team question from the ticket above. Conceptually the answer looks like this:

{
  "team": {
    "choice": "routing",
    "confidence": 0.71,
    "probabilities": {
      "routing": 0.71,
      "switching": 0.19,
      "transport": 0.07,
      "wireless": 0.03
    }
  }
}

Read it like a routing table, not a verdict. Four candidates were considered. Routing won at 0.71. Switching is a genuine runner-up at 0.19 — unsurprising, since the ticket mentions an Arista leaf port flapping alongside the BGP adjacency change. Transport sits at 0.07, consistent with “flapping could be an optic.” Wireless is effectively ruled out at 0.03. The distribution is not noise around the answer; it is a summary of the real ambiguity in the ticket, and a good triage engineer would rank the candidates the same way.

Figure 2.3: Probability distribution across four NOC teams

flowchart LR
    T[Incident Ticket] -->|0.71| Routing[Routing Team]
    T -->|0.19| Switching[Switching Team]
    T -->|0.07| Transport[Transport Team]
    T -->|0.03| Wireless[Wireless Team]

With only a label, your code has one behavior. With a distribution, you get graduated behavior that mirrors how a NOC already runs:

team_answer = response.answers["team"]
team = team_answer.choice            # "routing"
confidence = team_answer.confidence  # 0.71

if confidence >= 0.85:
    assign_to_queue(team)                        # auto-assign
elif confidence >= 0.60:
    assign_to_queue(team, flag="review")         # assign, mark for review
else:
    assign_to_queue("NOC-Triage")                # a human decides

That structure — act automatically when confident, flag when moderately confident, escalate when not — is the same tiering you apply to syslog severity, where a level 3 error pages someone and a level 6 informational message goes to a log bucket.

A note on exactness: the quickstart documents attribute access for the primary values (response.answers["team"].choice, .score, .noul) [Source: https://docs.typesafe.ai/introduction/quickstart.md], and the response reference states that each ChoiceAnswer includes a selected label, confidence, and per-label probabilities, while each ScoreAnswer includes an expected score, confidence, a rubric legend, and probabilities per integer score [Source: https://docs.typesafe.ai/sdk/python/api/types/responses.md]. Chapter 3 works through the complete response object field by field; the JSON above shows the shape, which is what matters here.

What “Calibrated” Means: 80% Should Be Right About 80% of the Time

A distribution is only useful if the numbers mean something. Calibration is the alignment between predicted probability and observed outcome frequency. A well-calibrated model assigns 80% confidence to decisions that turn out correct about 80% of the time, and what it calls 20% likely happens about 20% of the time [Source: https://www.giskard.ai/glossary/model-calibration]. When confidence systematically diverges from accuracy, the model is miscalibrated [Source: https://www.giskard.ai/glossary/model-calibration].

You already depend on calibration in instrumentation, you just call it something else. An interface utilization gauge reading 80% is trustworthy because 80% on the gauge corresponds to 80% of line rate in reality. If the gauge read 80% whenever real utilization was anywhere between 40% and 95%, you would not build a capacity-planning threshold on it — you would replace the gauge. Model confidence is that gauge, and calibration is whether it reads true.

To check calibration, bin decisions by stated confidence and compare each bin against how often those decisions were actually right. Here is a well-calibrated triage model after 3,000 tickets whose true owning team was later confirmed:

Every row’s observed accuracy lands close to its bucket. That is the whole test. Its formal version is Expected Calibration Error (ECE), which bins predictions and reports the weighted average deviation between confidence and accuracy — a single scalar you can track over time [Source: https://www.emergentmind.com/topics/expected-calibration-error-ece].

Now imagine that row had come back at 71% instead of 95.7%. The model would still pick the right team most of the time, so a casual review would find nothing wrong — but your auto-assign rule at 0.85 would be silently wrong nearly three times in ten. That is not hypothetical: modern deep neural networks are systematically overconfident, producing confidence estimates that do not reflect their true correctness likelihood, driven by network depth, width, weight decay regularization, and batch normalization, to the point where a network may express 90% confidence while being right 70% of the time [Source: https://arxiv.org/abs/1706.04599]. For network automation the consequence is direct: a model that says “this configuration change is 95% safe” while its real accuracy at that level is 70% will approve unsafe changes at a catastrophic rate [Source: https://www.giskard.ai/glossary/model-calibration].

This is precisely the problem TypeSafe designs against. RLCD (Reinforcement Learning for Calibrated Decisions) trains models to output decisions and probabilities aligned with true outcome frequencies rather than generated text carrying arbitrary confidence, in deliberate contrast to RLHF (Reinforcement Learning from Human Feedback), which optimizes for human-preferred responses and can reinforce “sycophancy and confident-sounding hallucinations,” and RLVR (Reinforcement Learning with Verifiable Rewards), which produces strong reasoning models at the cost of speed and expense [Source: https://docs.typesafe.ai/introduction/machine-learning-primer.md]. RLHF carries a subtler problem called mode dropping: the model narrows its output distribution toward preferred styles while suppressing alternatives [Source: https://docs.typesafe.ai/introduction/machine-learning-primer.md]. For a triage decision, a suppressed alternative is exactly the 0.19 on switching that told you the ticket was ambiguous. Human preference and machine trustworthiness are different optimization targets [Source: https://docs.typesafe.ai/introduction/machine-learning-primer.md].

When calibration drifts it can often be repaired without retraining. Temperature scaling — a single-parameter variant of Platt scaling that rescales logits before the softmax — is remarkably effective at very low cost [Source: https://arxiv.org/abs/1706.04599], as are isotonic regression and beta calibration [Source: https://www.giskard.ai/glossary/model-calibration]. Miscalibration is a fixable measurement problem, not a permanent defect.

Aggregate Calibration Versus Individual Correctness

Calibration is a property of a population of decisions, not of any single decision. When the model says 0.90 and is wrong, that is not evidence of miscalibration — a perfectly calibrated model at 0.90 is supposed to be wrong about one time in ten. If it were never wrong at 0.90 it would be under-confident, and you would be routing to humans decisions the model actually knows. Three practical implications follow.

First, you cannot audit calibration by looking at incidents. Pulling up the one ticket that went wrong tells you almost nothing about whether your thresholds are sound. You need the bucket table: hundreds or thousands of decisions with confirmed outcomes, grouped by confidence.

Second, a threshold is a statement about acceptable error rates, not a promise of correctness. Auto-assigning at 0.85 against a calibrated model accepts roughly one mis-assignment in seven at the boundary in exchange for the volume you no longer route through a human. That is a business tradeoff to make explicitly with the NOC manager and revisit. Mis-assigned tickets are recoverable; a mis-executed change may not be, which is why change automation deserves a far higher threshold than routing automation.

Third, calibration must be monitored continuously, because distributions shift: new vendors get onboarded, a Junos fleet replaces an aging platform, a Splunk correlation rule changes the shape of the alerts that reach you. Teams should gate releases on confidence or risk scores, use calibration as the evidence that those thresholds mean what operators expect, and verify calibration in production by monitoring ECE or a similar metric so drift is detected rather than discovered [Source: https://www.giskard.ai/glossary/model-calibration] [Source: https://www.emergentmind.com/topics/expected-calibration-error-ece]. Treat it like any drifting baseline — your interface error-rate baseline is not a number you set once and never re-examine.

Key Takeaway: A classification answer is a probability distribution across every allowed option, and the runner-up values tell you how ambiguous the input was. Calibration means those numbers correspond to reality — 80% confidence is right about 80% of the time — and it is verified across buckets of many decisions, never on a single case. Thresholds are only safe on a calibrated model, so measure calibration before you automate and monitor it after.

Where AI Belongs in a Network Workflow

Deterministic Logic Stays in Code

Deterministic logic is any computation whose answer follows from the input by fixed rules — same input, same output, provably. Interface status checks, threshold comparisons, date arithmetic, regex matching, field lookups, unit conversions. These belong in code permanently, and no amount of model quality changes that.

The reason is not that a model would necessarily get them wrong; it is that a model might, and you would have no way to prove otherwise. if utilization > 80 is verifiable by inspection and a two-line unit test. “Ask the model whether utilization is high” can only be validated statistically — you have traded a proof for a probability and gained nothing.

Arithmetic and date math are the same trap. Language models are pattern-matching engines that generate output statistically likely to follow the input [Source: https://www.rconfig.com/blog/from-copilot-to-chaos-the-real-pitfalls-of-ai-driven-network-automation]; they are not calculators or clocks. “How many minutes between the first %LINK-3-UPDOWN and ticket creation?” is a datetime subtraction. “Is this maintenance window still open?” is a comparison. Compute both in Python, then put the computed value into state as a fact if the model needs it.

This is the division of labor you already practice between BGP and a route-map: the protocol computes best paths by deterministic rules, the route-map encodes policy judgment. Keep the same boundary here.

Judgment Calls for the Model

What benefits from a model is the class of questions whose answer depends on interpreting unstructured human or device language, where a reasonable engineer would give a judgment rather than a computation:

• Interpreting a ticket. “Branch users say things are slow sometimes, mostly in the afternoon, the new VoIP handsets seem fine though” has no parseable fields. Which subsystem does that description implicate? A regex cannot answer; a Choice can.
• Ranking severity from described impact. A caller’s words carry impact information no counter does. “Nobody in the warehouse can scan anything” is operationally severe even if every interface is up. A Score against your rubric turns that language into a comparable number.
• Spotting a risky change. Does a proposed change touch core routing policy, a shared-services VLAN, or a device with no redundant peer? That is a Noul — a risk judgment returned as a probability you can threshold.
• Reconciling multi-vendor descriptions of the same event. A Cisco IOS-XE %LINEPROTO-5-UPDOWN, an Arista EOS log line, a Junos message with its own TAG field, and an Aruba AOS-CX entry may all describe one physical fault in four dialects. Deciding they are the same story is interpretation.

The pattern across all four: the input is language, the output is a bounded label or score, and the alternative is a human reading the text.

Figure 2.4: Deciding what stays in code versus what goes to the model

flowchart TD
    A[NOC Task] --> B{Can the Answer Follow Fixed Rules}
    B -->|Yes| C[Keep in Code]
    B -->|No| D[Ask the Model]
    D --> E[Model Returns Bounded Label or Score]
    E --> F[Code Decides the Action and Executes It]
    C --> F

Here is the rule of thumb in table form, with tasks from the NOC triage service:

Note the last three rows: the model supplies a judgment, your code decides what to do with it and performs every action. The model never touches the device.

Common Failure Modes of Putting an LLM in the Control Path

The documented consensus across industry and research is direct: LLMs should not operate as autonomous pilots in network automation [Source: https://www.rconfig.com/blog/from-copilot-to-chaos-the-real-pitfalls-of-ai-driven-network-automation]. Four failure modes explain why.

Semantic blindness. An LLM is a pattern-matching engine producing text statistically likely to follow a prompt, with no awareness of your network’s history, unstated business rules, inter-device dependencies, or organizational constraints. The result is syntactically correct but semantically unsafe output: “Optimize BGP for the new Dallas link” can be read as license to alter long-standing peering policy and disrupt critical flows, every generated line being valid BGP syntax. The IETF has formally identified this gap between linguistic intent and physical control authority as unresolved in draft guidance on LLM-assisted network management [Source: https://www.rconfig.com/blog/from-copilot-to-chaos-the-real-pitfalls-of-ai-driven-network-automation].

Amplified blast radius. Traditional automation errors — a typo, an incomplete script — are usually localized. AI-driven mistakes trade “a high frequency of small, localized mistakes for a low frequency of massive, catastrophic failures”: an AI updating ACLs across hundreds of firewalls on flawed security logic can open an enterprise-wide vulnerability in a single push, and a documented case study of a single incorrect AS number in an automated routing script caused major service disruption — a class of flaw an AI can replicate across thousands of configuration lines at once [Source: https://www.rconfig.com/blog/from-copilot-to-chaos-the-real-pitfalls-of-ai-driven-network-automation].

Hallucination and undetectable error. A hallucination is confident output not grounded in reality — invented commands, incorrect syntax, or a feature that does not exist in the target device OS version. Such errors hide at scale: a single misconfiguration inside a thousand-line generated script escapes review because nobody can audit generated configuration line by line, and semantic violations raise no syntax error — they manifest only during failure [Source: https://www.rconfig.com/blog/from-copilot-to-chaos-the-real-pitfalls-of-ai-driven-network-automation].

Context drift and structural violations. LLMs struggle with consistent state across sequential actions, so a multi-step change can pass through intermediate states that violate constraints, and a decision made at step 1 can be contradicted at step 10 [Source: https://www.rconfig.com/blog/from-copilot-to-chaos-the-real-pitfalls-of-ai-driven-network-automation]. Research on LLM-generated topologies found systematic violations of the constraints that matter — unidirectional links where full duplex was required, loops in ring topologies, interface oversubscription, single points of failure — degrading sharply as networks grew from 12–32 nodes to 173 or more. That scaling curve is the dangerous part: a lab test on a small topology can look fine while masking failures that appear only at production scale [Source: https://arxiv.org/html/2607.00292v1].

Figure 2.5: Failure path of an LLM holding control authority

flowchart TD
    A[LLM Given Direct Control Authority] --> B[Generates Syntactically Valid Output]
    B --> C{Semantically Safe}
    C -->|Unknown to the Model| D[Semantic Blindness]
    D --> E[Change Pushed at Scale]
    E --> F[Amplified Blast Radius]
    E --> G[Hallucinated Command or Syntax]
    E --> H[Context Drift Across Steps]
    F --> I[Catastrophic Failure]
    G --> I
    H --> I

The responsible pattern is human-in-the-loop with automated safeguards: models generate candidates for expert review, AI-influenced configuration requires validation, and the pipeline provides configuration diffing, test coverage validation, rollback, staged deployment with canary validation, and audit trails showing which components were AI-generated versus human-authored [Source: https://www.rconfig.com/blog/from-copilot-to-chaos-the-real-pitfalls-of-ai-driven-network-automation].

Notice how much of that risk evaporates under the typed approach. A model that can only return one of four team labels with a probability distribution cannot hallucinate a nonexistent Junos command, cannot violate a topology constraint, and cannot push anything to a device. It has no control authority — it produces a judgment, and deterministic code decides what happens next. That containment is the architecture this book is built on.

Key Takeaway: Anything provable — status, thresholds, arithmetic, dates, string matching, and every write action — stays in deterministic code, because a proof beats a probability. The model handles interpretation of language: which team owns this, how bad the described impact is, whether a change looks risky. Keeping the model out of the control path defuses the documented failure modes of generative automation, because a bounded label with a probability cannot execute anything.

Chapter Summary

A token is the model’s unit of accounting, invisible to your network monitoring and reported back in the API’s usage field. Inference is a stateless request/response call with all the ordinary operational habits that implies: timeouts, retries, request IDs, and capacity planning against a published rate. Because inference splits into a compute-bound prefill phase and a memory-bandwidth-bound decode phase, the length of the answer directly drives latency and cost — the first of several reasons short, bounded answers beat long, generated ones.

Classification and scoring bound the answer before the model speaks. A Choice over four NOC teams, a Score against your own severity rubric, and a Noul about customer impact are testable against labeled history, comparable across tickets, aggregable into metrics, and cheap to produce; generation is none of those things. Each bounded answer also carries a probability distribution, and the runner-up values are real information — 0.71 on routing with 0.19 on switching says the ticket is genuinely ambiguous, exactly as a human triage engineer would judge it. Those numbers are trustworthy only if the model is calibrated, which is verified across buckets of many decisions and monitored as your traffic and vendor mix change. Modern neural networks are systematically overconfident by default, which is why TypeSafe’s RLCD training targets calibrated decisions and probabilities rather than persuasive text.

The payoff is a division of labor that carries through the rest of this book. Deterministic logic — interface state, threshold comparisons, date math, string matching, and every write to a device or ticketing system — stays in code, where it is provable. The model handles judgment calls that require reading language. The documented catastrophes of AI-driven network automation share one root cause: a generative model holding control authority. Take that authority away, constrain the output to a bounded set, attach a calibrated probability, and gate the action on a threshold your team chose deliberately. The next chapter puts this into practice with your first real client.system_one call.

Key Terms

Chapter 3: Getting Started: Your First System One Request

Learning Objectives

• Obtain a TypeSafe API key from the console and store it in the TYPESAFE_API_KEY environment variable
• Install the typesafe-sdk package with pip or uv on Python 3.10+ and instantiate a TypeSafeClient
• Construct a system_one request with a state argument and a questions dictionary containing named Choice, Score, and Noul questions
• Read answers, probabilities, confidence, and usage from a SystemOneResponse and translate them into NOC triage decisions

Environment Setup

Everything in this book runs through one Python package and one environment variable. If you have ever configured a RADIUS shared secret on a switch or dropped an API token into an Ansible vault, the pattern will feel familiar: the credential lives outside the code, the code picks it up at runtime, and nothing sensitive ends up in your Git repository.

Creating an API Key and Storing It in TYPESAFE_API_KEY

A TypeSafe API key is issued from the TypeSafe console, and the SDK expects to find it in an environment variable named TYPESAFE_API_KEY. The official SDK documentation is explicit that after installation you “set the TYPESAFE_API_KEY environment variable using credentials from the TypeSafe console” [Source: https://docs.typesafe.ai/sdk/python.md]. You do not pass the key into the constructor in the normal case — the client reads it for you. The quickstart states that the client “automatically reads your TYPESAFE_API_KEY environment variable and uses the jev-latest model by default” [Source: https://docs.typesafe.ai/introduction/quickstart.md].

Think of the key the way you think of an SNMPv3 credential or a TACACS+ key: it identifies you, it authorizes you, and it is the thing an auditor will ask about. Treat it accordingly. Export it in your shell for interactive work, put it in a .env file that is listed in .gitignore for local development, and store it in your platform’s secret store — Ansible Vault, HashiCorp Vault, a Kubernetes Secret, or your CI system’s encrypted variables — for anything that runs unattended.

# Interactive shell (bash/zsh) — good for experimenting
export TYPESAFE_API_KEY="ts_live_your_key_here"

# Confirm the variable is actually visible to the process that will use it
echo "${TYPESAFE_API_KEY:0:8}..."   # print only a prefix, never the whole key

# Persist it for your user (Linux/macOS). Restart the shell afterward.
echo 'export TYPESAFE_API_KEY="ts_live_your_key_here"' >> ~/.bashrc

A detail that bites automation engineers constantly: an environment variable exported in your interactive shell is not automatically visible to a systemd service, a cron job, a Jenkins agent, or a Docker container. When your script works by hand and fails under the scheduler, the environment is the first thing to check — the same way you would check whether a route exists in the VRF the traffic actually lands in, not the global table you were looking at.

Installing typesafe-sdk with pip or uv on Python 3.10+

The SDK requires Python 3.10 or higher and installs with either of the two common Python package managers [Source: https://docs.typesafe.ai/introduction/quickstart.md]. Both pip install typesafe-sdk and uv add typesafe-sdk are documented as supported installation paths [Source: https://docs.typesafe.ai/sdk/python.md].

# Check your interpreter first — 3.10 or higher is required
python3 --version

# Create an isolated environment so the SDK does not collide with
# netmiko, pyATS, or whatever else lives in your system Python
python3 -m venv ~/venvs/noc-triage
source ~/venvs/noc-triage/bin/activate

# Option A: pip
pip install typesafe-sdk

# Option B: uv (faster resolver, manages the project's dependency file)
uv add typesafe-sdk

# Store the key for this project and verify the interpreter can see it
export TYPESAFE_API_KEY="ts_live_your_key_here"
python3 -c "import os; print('key present:', bool(os.environ.get('TYPESAFE_API_KEY')))"

Use a virtual environment. Network automation boxes accumulate a dense pile of libraries — netmiko, napalm, pyATS, ncclient, three versions of requests — and a per-project environment is the equivalent of putting each service in its own VRF instead of letting everything leak into the global routing table.

Verifying Access Before You Write Real Code

Before writing triage logic, prove three things independently: the package imports, the key is readable, and the API answers. The documentation for the model catalog notes that “All models use the same API endpoint: POST /v1/systemone” [Source: https://docs.typesafe.ai/models.md], and that endpoint is what every example in this book exercises. The model names themselves — jev-1.13.0, jev-latest, and jev-preview — are published in the models reference [Source: https://docs.typesafe.ai/models.md], so you can confirm which alias you intend to pin without making a call at all.

The cheapest live check is the model catalog itself. The models reference states that you can query the available models with GET /v1/models using your API key [Source: https://docs.typesafe.ai/models.md]. It consumes no input tokens, so it is the network engineer’s equivalent of a ping before a traceroute: it proves the key is accepted and the API is reachable before you spend anything on a real question.

curl -s https://api.typesafe.ai/v1/models \
  -H "Authorization: Bearer $TYPESAFE_API_KEY"

A successful reply lists the model identifiers and aliases documented in the models reference. A 401 here means the key is missing or wrong; fix that before touching Python. (Confirm the exact base URL and header format against the API reference for your account, since the models page documents the path but not every transport detail.)

The most reliable end-to-end smoke test is the smallest possible real request: one short state, one Noul question. If it returns, your key is valid, your network path to the API is open, your Python version is acceptable, and your billing account is live — four checks in one round trip.

import os

from typesafe_sdk import Noul, TypeSafeClient

assert os.environ.get("TYPESAFE_API_KEY"), "TYPESAFE_API_KEY is not set"

client = TypeSafeClient()

response = client.system_one(
    state="Interface GigabitEthernet1/0/24 changed state to down.",
    questions={
        "mentions_an_interface": Noul(
            instructions="The text refers to a specific network interface",
        ),
    },
)

print("model:", response.model)
print("request_id:", response.request_id)
print("answer:", response.answers["mentions_an_interface"].noul)

The model and request_id fields are part of every response’s metadata — request_id carries “the x-typesafe-request-id response header” [Source: https://docs.typesafe.ai/sdk/python/api/types/responses.md]. Log that request ID everywhere. When you open a support case or correlate a weird classification against your own logs, it is the only identifier that ties your side of the conversation to theirs, and it plays exactly the role a transaction ID plays in a ServiceNow integration log.

When that smoke test fails, the failure almost always falls into one of a handful of buckets. Three exception types cover nearly everything you will hit while getting started: TypeSafeError for validation issues such as empty questions or empty criteria, TypeSafeAPIError for unsuccessful server responses, and TypeSafeAPIConnectionError for connection or timeout failures [Source: https://docs.typesafe.ai/sdk/python/api/clients/sync/client.md]. Knowing which of the three you got tells you whether the problem is in your request, on the server, or in the path between them. The SDK actually exposes a finer hierarchy beneath these — a dedicated class per HTTP status, including TypeSafeRateLimitError with a retry_after_ms property and a separate TypeSafeAPITimeoutError — and Chapter 11 works through the full set when we build the production receiver. For a first smoke test, the three above are enough.

Key Takeaway: The entire setup is one package and one environment variable: install typesafe-sdk on Python 3.10 or higher, export TYPESAFE_API_KEY, and let TypeSafeClient() pick the key up on its own. Prove access with the smallest possible real request rather than assuming it works, and log the returned request_id from day one so that every future classification can be traced back to a specific call.

Figure 3.1: Environment setup sequence, from interpreter check to a verified live request

flowchart TD
    A["Check Python version is 3.10 or higher"] --> B["Create a virtual environment"]
    B --> C["Install typesafe-sdk with pip or uv"]
    C --> D["Set TYPESAFE_API_KEY environment variable"]
    D --> E["Verify with GET /v1/models"]
    E --> F["Run smallest real request as smoke test"]

Anatomy of a Request

A System One request has exactly two mandatory parts. You give Jev something to look at, and you give it a set of named, typed questions about that thing. Everything else — model selection, timeouts, retries, extra headers — is optional tuning.

Figure 3.2: Anatomy of a System One request and response

flowchart LR
    State["state"] --> Request["POST /v1/systemone"]
    Questions["questions dictionary"] --> Request
    Model["model (optional)"] --> Request
    Request --> Answers["answers"]
    Request --> Usage["usage"]

The state Argument: What the Model Evaluates

state is the content under evaluation. The client reference defines it as “Text, a JSON object, or an array to evaluate” [Source: https://docs.typesafe.ai/sdk/python/api/clients/sync/client.md]. That flexibility matters more in network operations than it might first appear. A raw syslog line is text. A parsed Splunk alert is a JSON object. A list of the last ten interface transitions on a port is an array. All three are legal state values, so you can feed Jev the data in whatever shape your pipeline already produces instead of flattening everything into a string first.

The mental model that works best for network engineers is a packet-classification one. state is the packet header plus payload — the thing being inspected. The questions are the ACL entries and QoS classifiers you are matching against it. Just as a classifier can only act on fields that are actually present in the packet, Jev can only answer from what you put in state. If your syslog line does not contain the hostname, no question about which device produced it can be answered reliably, no matter how the question is worded.

That has a direct design consequence for the multi-vendor NOC triage service running through this book. Before you call system_one, decide what belongs in the state. For a single syslog event, the raw line is usually sufficient and cheap. When triage depends on context — “is this the fourth flap on this port in ten minutes?” — that context has to be in the state as a structured object, because Jev evaluates what you give it and nothing else.

Keep the state tight. You are billed on input tokens, at $42 per billion input tokens, with output tokens free [Source: https://docs.typesafe.ai/models.md]. A state is not a log archive; it is the evidence for a specific decision.

The questions Dictionary: Named Choice, Score, and Noul Questions

questions is “a non-empty mapping of question names to question objects” [Source: https://docs.typesafe.ai/sdk/python/api/clients/sync/client.md]. The keys are names you invent; the values are one of the three question types introduced in the previous chapter. Those three types are documented as Noul for yes/no questions, Choice for multiple options, and Score for rating on a scale [Source: https://docs.typesafe.ai/sdk/python/usage.md].

Two details in that table are worth pausing on. First, Choice criteria are a dictionary — label to description — while Score criteria are an ordered list. The quickstart shows both shapes side by side: criteria={"billing": "Payment or subscription issues", ...} for a Choice, and criteria=["Calm, just stating facts", "Frustrated but civil", "Very angry, strong language"] for a Score [Source: https://docs.typesafe.ai/introduction/quickstart.md]. The list ordering is the rubric: position zero is the bottom of the scale and the last entry is the top.

Second, Choice descriptions are optional in the sense that the SDK usage documentation shows a form where each label maps to None, as in Choice(instructions="What is the tone?", criteria={"calm": None, "angry": None}) [Source: https://docs.typesafe.ai/sdk/python/usage.md]. For production triage, write the descriptions anyway. A label named routing means one thing to you and another to a model that has to infer your intent from six characters. Descriptions are the equivalent of the remark line on an ACL: technically optional, operationally essential.

The names you choose for the keys become the names you read back out of the response, since answers are “keyed by question name” [Source: https://docs.typesafe.ai/sdk/python/api/types/responses.md]. Pick names that will survive contact with your downstream systems — if the answer ends up in a ServiceNow field called u_assignment_team, naming the question subsystem and mapping it once in code is cleaner than naming it q1.

The model Field and the jev-latest Alias

model is an optional parameter on system_one that overrides the client default, and when it is None the call inherits the client’s configured model [Source: https://docs.typesafe.ai/sdk/python/api/clients/sync/client.md]. That default is jev-latest [Source: https://docs.typesafe.ai/introduction/quickstart.md].

TypeSafe publishes three ways to reference the model, and the distinction is the same one you already make between a symlink and a versioned filename when staging IOS-XE images on a TFTP server.

Jev is TypeSafe’s flagship model and “the first System One model,” currently at version 1.13 [Source: https://docs.typesafe.ai/models.md]. For a NOC triage service that writes severity and team assignments into ServiceNow, there is a real argument for pinning to jev-1.13.0 in production while running jev-latest in a staging path — the same change-control discipline that stops you from upgrading code on sixty access switches because the vendor published a new release. Alias in the lab, pinned version in production, and a deliberate promotion in between.

# Pin explicitly for a production triage path
response = client.system_one(
    state=syslog_line,
    questions=triage_questions,
    model="jev-1.13.0",
)

The remaining optional parameters — retry, timeout, extra_headers, and extra_body — are also accepted per call, where retry sets a retry policy for that specific call, timeout overrides the timeout in seconds, extra_headers adds request headers as key-value pairs, and extra_body supplies extra top-level request-body fields that are shallow-merged into the request [Source: https://docs.typesafe.ai/sdk/python/api/clients/sync/client.md]. In a syslog pipeline, timeout and retry are the two you will reach for first, because a triage service that blocks indefinitely on one call is worse than one that drops the event and logs it.

Key Takeaway: A request is state plus questions: the evidence and the typed questions asked about it. state accepts text, a JSON object, or an array, so structured alerts do not need flattening, and questions is a non-empty mapping whose keys become the keys of the response. Everything else, including model, is optional — but pinning jev-1.13.0 in production instead of riding jev-latest is the same change-control instinct you already apply to device firmware.

Worked Example: Classifying a Cisco Syslog Line

Now build a real triage call. The scenario is the running example for this book: a multi-vendor NOC ingests syslog from Cisco, Arista, Juniper, and Aruba devices and needs to route each event to a team, assign a severity, and decide whether a human needs to act tonight.

The State: A Raw %LINEPROTO-5-UPDOWN Message

Cisco IOS-XE syslog messages follow the pattern %FACILITY-SEVERITY-MNEMONIC: Message-text, optionally preceded by a timestamp [Source: https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/17_xe/syslogs/17-14-x/b-system-message-guide-17-14-x.html]. The FACILITY is a code identifying the subsystem that generated the message — LINEPROTO for line protocol status changes, LINK for physical interface status, SYS for system events, OSPF and BGP for routing protocol events [Source: https://networklessons.com/system-management/cisco-ios-syslog-messages]. The severity digit runs 0 through 7, where lower numbers mean higher severity, and the MNEMONIC is a short uppercase code such as UPDOWN or CONFIG_I that uniquely identifies the message type [Source: https://networklessons.com/system-management/cisco-ios-syslog-messages].

Here is the event, exactly as it would arrive at a collector from an IOS-XE access switch:

*Feb 14 09:40:12.418: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/24, changed state to down

Read it the way an experienced engineer does. LINEPROTO means this is a Layer 2 line protocol event, not a Layer 1 one — LINK-3-UPDOWN reports physical changes while LINEPROTO-5-UPDOWN reports Layer 2 protocol status, and the two typically occur together during interface transitions [Source: https://networklessons.com/system-management/cisco-ios-syslog-messages]. The 5 is the device’s own severity, Notice, defined as “normal but significant conditions” and typically used for interface state changes and configuration modifications [Source: https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/17_xe/syslogs/17-14-x/b-system-message-guide-17-14-x.html].

This is precisely where the device’s built-in severity stops being enough. Cisco stamps every LINEPROTO-5-UPDOWN as a 5 whether the port is an unused desk drop or the uplink carrying your voice VLAN. GigabitEthernet1/0/24 is a 1/0/x access-stack port, which is a strong hint that this is an edge port — but the severity digit cannot express that. Asking Jev a Score question about operational severity produces a judgment the facility code structurally cannot.

That same reasoning ports across vendors, which is why this pattern scales to the whole NOC. Arista EOS implements the same eight severity levels as Cisco and supports standard RFC 3164 format [Source: https://www.arista.com/en/um-eos/eos-system-event-logging]; Junos OS follows UNIX syslog conventions with the same eight levels, using a TAG field that plays the role of Cisco’s mnemonic [Source: https://www.juniper.net/documentation/en_US/junos13.2/topics/reference/general/syslog-facilities-severity-levels.html]; Aruba AOS-CX supports per-syslog-server severity configuration [Source: https://help.central.arubanetworks.com/latest/documentation/online_help/content/aos-cx/cfg/conf-cx-logging.htm]. Because the severity scale is shared across all four vendors, one set of typed questions can cover the entire estate.

The Questions: Which Subsystem, How Severe, Is It Actionable

Three questions, one per type, answered in a single call:

• subsystem — a Choice, because the answer must be one of a fixed set of teams your NOC actually staffs.
• operational_severity — a Score, because severity is a graded judgment along an ordered scale, not a category.
• actionable_tonight — a Noul, because paging someone is a yes/no decision.

import os

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

client = TypeSafeClient()

syslog_line = (
    "*Feb 14 09:40:12.418: %LINEPROTO-5-UPDOWN: Line protocol on "
    "Interface GigabitEthernet1/0/24, changed state to down"
)

triage_questions = {
    "subsystem": Choice(
        instructions="Which NOC team owns the subsystem this message came from",
        criteria={
            "interface": "Physical port, line protocol, or link state events on a switch or router interface",
            "routing": "OSPF, BGP, EIGRP, or other routing protocol adjacency and convergence events",
            "system": "Device configuration changes, reloads, process crashes, or platform hardware faults",
            "wireless": "Access point, WLAN, or wireless controller events",
        },
    ),
    "operational_severity": Score(
        instructions=(
            "How severe this event is for network operations, considering the "
            "scope of impact rather than the severity digit in the message"
        ),
        criteria=[
            "Routine or expected state change on a single edge port with no user impact",
            "Localized issue affecting one access port or one user",
            "Service-affecting issue on an uplink, trunk, or shared segment",
            "Widespread outage affecting a site, a routing domain, or many users",
        ],
    ),
    "actionable_tonight": Noul(
        instructions=(
            "This event needs a human to take action during the current shift "
            "rather than being reviewed during business hours"
        ),
    ),
}

response = client.system_one(
    state=syslog_line,
    questions=triage_questions,
)

Note what is happening in the Choice criteria. Every label carries a description written in the vocabulary of a network engineer, not a data scientist — “OSPF, BGP, EIGRP, or other routing protocol adjacency and convergence events” gives Jev the same signal a runbook gives a new NOC hire. The Score criteria are ordered from least to most severe, and the instruction explicitly tells the model to reason about scope of impact rather than copying the 5 out of the message. That instruction matters: without it, a literal reading of “severity 5” is a reasonable answer to “how severe is this,” and the model will give you what you asked for rather than what you meant. Write the rubric so that the thing you want measured is the thing described.

One request, three answers. That is the efficiency argument for typed questions in a triage pipeline: you are billed on input tokens, so sending one state with three questions costs far less than three separate calls that each re-send the same syslog line.

Figure 3.3: The syslog triage call as a request/response sequence

sequenceDiagram
    participant Client as Python Client
    participant API as TypeSafe API
    Client->>API: state plus subsystem, operational_severity, actionable_tonight
    API-->>Client: typed answers with confidence and probabilities
    Client->>Client: Check confidence threshold
    Client->>Client: Route to team or escalate to human

Running the Request Synchronously and Asynchronously

The example above uses the synchronous client. The SDK provides both forms — the Python SDK offers “asynchronous and synchronous Python clients for the TypeSafe API” [Source: https://docs.typesafe.ai/sdk/python.md], with TypeSafeClient for synchronous operations and AsyncTypeSafeClient for asynchronous ones [Source: https://docs.typesafe.ai/sdk/python.md]. The documented guidance is straightforward: “For async work, use AsyncTypeSafeClient() with await, while the synchronous version works with standard blocking calls” [Source: https://docs.typesafe.ai/sdk/python/usage.md].

A synchronous client blocks until the answer comes back. That is exactly right for a cron job that classifies last night’s ServiceNow backlog, for a CLI tool an engineer runs by hand, or for any script where simplicity beats throughput. An asynchronous client lets many requests be in flight at once, which is what you want when a Splunk forwarder is handing you hundreds of events a minute and you do not want each one waiting on the previous one’s round trip.

import asyncio

from typesafe_sdk import AsyncTypeSafeClient

async def triage(syslog_line: str):
    client = AsyncTypeSafeClient()
    response = await client.system_one(
        state=syslog_line,
        questions=triage_questions,
    )
    return response

response = asyncio.run(triage(syslog_line))

A note on fidelity: the official documentation states that the async client is used with await and names the class AsyncTypeSafeClient [Source: https://docs.typesafe.ai/sdk/python/usage.md], but it does not publish a full async code sample with lifecycle details such as whether the client should be used as a context manager or explicitly closed. The snippet above follows the documented pattern only. Check the SDK reference for the async client before wiring it into a long-running service, and if you are unsure, start synchronous — a correct blocking pipeline in production beats an async one you are guessing at.

Throughput is worth sizing before you choose. The published rate limits are 250,000 tokens per second and 1,200 requests per minute [Source: https://docs.typesafe.ai/models.md]. At 1,200 requests per minute, a synchronous single-threaded loop will hit wall-clock latency as its ceiling long before it hits the API’s limit, which is usually the trigger for moving to the async client.

Key Takeaway: A complete triage call is one raw syslog line as state and three typed questions — a Choice for the owning team, a Score for operational severity, and a Noul for whether to page someone — sent in a single request. Write Choice descriptions and Score rubric levels in operational language, and state explicitly when you want judgment about impact rather than the severity digit already in the message. Start with the synchronous TypeSafeClient and move to AsyncTypeSafeClient only when event volume, not curiosity, demands it.

Reading the Response

The call returns a SystemOneResponse. Everything you need — the answers, the model that produced them, the token usage, and the request identifier — hangs off that one object.

The answers Object Keyed by Question Name

answers is “a dictionary of all answer objects keyed by question name” [Source: https://docs.typesafe.ai/sdk/python/api/types/responses.md]. The keys are exactly the keys you supplied in questions, which makes the round trip predictable: ask about subsystem, read back answers["subsystem"].

Alongside answers, the response exposes three type-specific collections: nouls holds “yes/no answers keyed by question name,” choices holds “choice answers keyed by question name,” and scores holds “score answers keyed by question name” [Source: https://docs.typesafe.ai/sdk/python/api/types/responses.md]. The SDK usage guide describes the same pattern from the caller’s side — access results through result.nouls, result.choices, and result.scores respectively [Source: https://docs.typesafe.ai/sdk/python/usage.md]. Use answers when you want to iterate over everything uniformly; use the typed collections when a piece of code only cares about one kind of answer, such as a paging rule that reads only nouls.

Figure 3.4: Structure of a SystemOneResponse for the syslog triage example

graph TD
    Response["SystemOneResponse"] --> Answers["answers dictionary"]
    Response --> Usage["usage"]
    Response --> RequestId["request_id"]
    Response --> Model["model"]
    Answers --> Subsystem["subsystem: choice, confidence, probabilities"]
    Answers --> Severity["operational_severity: score, confidence, legend"]
    Answers --> Actionable["actionable_tonight: noul"]
    Usage --> InputTokens["input_tokens"]
    Usage --> OutputTokens["output_tokens"]

The response also carries metadata: request_id (the x-typesafe-request-id response header), model (the model used to answer the request), and raw_http_response (the underlying HTTP response object) [Source: https://docs.typesafe.ai/sdk/python/api/types/responses.md]. That last one is an escape hatch — headers, status codes, anything the typed fields do not surface.

The shape below assembles the documented field names into the JSON structure a response represents. The field names are from the documentation; the specific values are illustrative for our %LINEPROTO-5-UPDOWN example.

{
  "request_id": "req_01hx9k3m2q8w7",
  "model": "jev-1.13.0",
  "usage": {
    "input_tokens": 214,
    "output_tokens": 0
  },
  "answers": {
    "subsystem": {
      "choice": "interface",
      "confidence": 0.94,
      "probabilities": {
        "interface": 0.94,
        "routing": 0.03,
        "system": 0.02,
        "wireless": 0.01
      }
    },
    "operational_severity": {
      "score": 1.12,
      "confidence": 0.71,
      "legend": {
        "0": "Routine or expected state change on a single edge port with no user impact",
        "1": "Localized issue affecting one access port or one user",
        "2": "Service-affecting issue on an uplink, trunk, or shared segment",
        "3": "Widespread outage affecting a site, a routing domain, or many users"
      },
      "probabilities": {
        "0": 0.19,
        "1": 0.56,
        "2": 0.21,
        "3": 0.04
      }
    },
    "actionable_tonight": {
      "noul": 0.22
    }
  }
}

choice, score, noul, probabilities, and confidence

The quickstart demonstrates the attribute access pattern directly: response.answers["is_urgent"].noul, response.answers["department"].choice, and response.answers["frustration"].score [Source: https://docs.typesafe.ai/introduction/quickstart.md]. Each answer type also carries confidence and probability data — a NoulAnswer provides the probability of a yes answer on a 0–1 scale, a ChoiceAnswer provides the selected label, a confidence score, and probabilities per label, and a ScoreAnswer provides the expected score, confidence, a rubric legend, and probabilities per integer score [Source: https://docs.typesafe.ai/sdk/python/api/types/responses.md].

A fidelity note, stated once for the whole book. The documentation confirms .noul, .choice, and .score as attribute access patterns in worked code [Source: https://docs.typesafe.ai/introduction/quickstart.md], and documents confidence, probabilities, and legend as fields the answer objects contain [Source: https://docs.typesafe.ai/sdk/python/api/types/responses.md], but it does not publish a code sample that reads those three. Every example from here to Chapter 12 therefore writes them as attributes — answer.confidence, answer.probabilities, answer.legend — because that is consistent with the accessors the docs do show. The wire-level JSON field names choice, score, noul, confidence, probabilities, and legend are the authoritative part; if your installed SDK spells an attribute differently, print one answer object and adjust. This note is not repeated in later chapters.

Two behaviors surprise people coming from traditional APIs. First, a Noul returns a probability, not a boolean. The quickstart’s example prints 0.999 for an urgency question [Source: https://docs.typesafe.ai/introduction/quickstart.md] — that is a number you threshold, not a yes you act on. Second, a Score returns an expected score that can be fractional. The quickstart’s frustration score is 1.035 [Source: https://docs.typesafe.ai/introduction/quickstart.md], which sits just above rubric level 1 in a three-level rubric. In our syslog example, 1.12 means the model is mostly on “localized issue affecting one access port or one user” with meaningful weight still on the routine level below it — which is exactly the right read for a single edge port going down.

subsystem = response.answers["subsystem"]
severity = response.answers["operational_severity"]
actionable = response.answers["actionable_tonight"]

print(f"team:       {subsystem.choice}")
print(f"severity:   {severity.score:.2f}")
print(f"page now:   {actionable.noul:.3f}")

# Route to a human when the model is not sure enough to act alone
if subsystem.confidence < 0.80:
    print(f"LOW CONFIDENCE — queue for review (request_id={response.request_id})")
else:
    print(f"Assigning to team: {subsystem.choice}")

That confidence gate is the single most important habit to form early, and it is the reason typed, confidence-aware answers beat free text for operations work. An automation that assigns tickets at 94% confidence and escalates to a human at 60% is trustworthy. One that assigns every ticket with equal conviction is not, and the first time it silently routes a core routing failure to the wireless queue, your team will stop trusting the whole pipeline. Chapter by chapter this book tightens those thresholds, but the structure never changes: read the answer, read the confidence, decide whether the machine or a person makes the call.

The usage Object and Tracking Input Tokens for Cost

usage reports token counts including input_tokens and output_tokens, each reported either as an integer or as None [Source: https://docs.typesafe.ai/sdk/python/api/types/responses.md]. Handle the None case — a usage field that is absent should not crash a syslog pipeline.

Cost tracking for Jev is unusually simple because only one side is billed. Jev costs $42 per billion tokens, input only, with output tokens free [Source: https://docs.typesafe.ai/models.md]. That means the only number that drives your bill is input_tokens, and the only lever you have is how much state and how many question descriptions you send.

usage = response.usage
input_tokens = usage.input_tokens if usage and usage.input_tokens is not None else 0

# $42 per 1,000,000,000 input tokens; output tokens are free
cost_usd = input_tokens * 42 / 1_000_000_000

print(f"input_tokens={input_tokens} cost=${cost_usd:.8f} request_id={response.request_id}")

Run the arithmetic once and the economics of the design become obvious. At $42 per billion input tokens, a request in the low hundreds of tokens costs a tiny fraction of a cent, and a NOC processing a million syslog events a month at roughly 200 input tokens each lands around 200 million input tokens — a little over eight dollars. The expensive mistake is not the per-event cost; it is re-sending the same state three times because you asked three questions in three separate calls, which triples the only number you are billed on. Bundle your questions.

Log input_tokens and request_id together on every call from the start. When someone asks in three months why the bill moved, you want per-request data, not an estimate — the same reason you keep interface counters instead of guessing at utilization.

One honest limitation to carry forward: Jev is a System One model built for fast, typed judgment, not calculation. Do not ask it to total your token spend, compute a percentage, or do arithmetic on values inside the state. Do the math in Python, as above, and let Jev do what it is good at — classifying, scoring, and telling you how sure it is.

Key Takeaway: answers is keyed by your question names and yields noul as a probability, choice as a label, and score as an expected value that may be fractional, each accompanied by confidence and probabilities that tell you how firmly the model holds its answer. Gate your automation on confidence rather than acting on every answer identically, and log usage.input_tokens with request_id on every call — input tokens are the only thing billed, at $42 per billion, with output free.

Chapter Summary

Getting from zero to a working System One request takes three steps and about five minutes. Install typesafe-sdk on Python 3.10 or higher with pip or uv, export your API key into TYPESAFE_API_KEY, and construct a TypeSafeClient() that reads the key automatically and defaults to the jev-latest model. Prove the path works with the smallest real request you can write — one short state and one Noul — before building anything on top of it, and log the request_id from the very first call so every classification is traceable.

A request is two things: state, which accepts text, a JSON object, or an array, and questions, a non-empty mapping of names you invent to Choice, Score, and Noul objects. The worked example took a single raw IOS-XE line — %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/24, changed state to down — and asked three questions about it at once: which team owns it, how severe it really is for operations, and whether it needs a human tonight. That structure is what makes the multi-vendor NOC triage service possible, because Arista EOS, Junos, and Aruba AOS-CX all share the same eight-level severity scale, so one set of typed questions covers the whole estate. It also exposes the gap that motivates the entire approach: the device stamped this event severity 5 whether the port is a desk drop or a voice uplink, and only a judgment question can tell those apart.

The response closes the loop. answers is keyed by your question names and gives you noul as a probability, choice as a selected label, and score as an expected value that may land between rubric levels — each carrying confidence and probabilities so you can decide whether to act automatically or escalate. usage.input_tokens is the only number that drives cost at $42 per billion input tokens with output free, which is exactly why bundling several questions into one call is both faster and cheaper than issuing them separately. From here, the next chapters go deeper into each question type, into writing instructions and criteria that hold up against messy real-world device output, and into wiring these answers into ServiceNow, Splunk, and Salesforce.

Key Terms

Chapter 4: State: Feeding the Model the Right Context

Learning Objectives

• Choose between string, JSON object, and JSON array state for a given input
• Build structured state from network data sources such as show commands, Arista eAPI, NETCONF, and Splunk events
• Reference specific parts of state in question instructions using dot-and-bracket paths
• Filter state to avoid context rot and stay within the token budget

The Three Shapes of State

In a System One request, state is the material the model looks at, and questions are the judgments you want made about it. TypeSafe draws that line deliberately: state holds content, questions hold the decisions, and keeping the two apart is what makes requests stay organized as they grow [Source: https://docs.typesafe.ai/concepts/state.md]. If you have ever separated a routing table from a route-map, the split will feel familiar. The table is the data. The route-map is the policy applied to it. You do not stuff policy clauses into the prefix list.

TypeSafe accepts exactly three shapes for that data: a string, a JSON object, or an array of text values [Source: https://docs.typesafe.ai/concepts/state.md]. Choosing among them is the first real design decision in a System One workflow, and it is not arbitrary — the shape you pick determines whether your question instructions can point at a specific piece of the input or have to describe it in words.

Figure 4.1: The Three Shapes of State

graph TD
    State["State"] --> A["String"]
    State --> B["JSON Object"]
    State --> C["JSON Array"]
    A --> A1["Single Cisco syslog line: LINEPROTO-5-UPDOWN"]
    B --> B1["ServiceNow incident plus interface counters"]
    C --> C1["Last twenty Aruba AOS-CX syslog lines, oldest first"]

String State for a Single Message or Log Line

The simplest state is a plain string — “a message, article, or passage” in the documentation’s phrasing [Source: https://docs.typesafe.ai/concepts/state.md]. For NOC triage, the natural candidate is a single syslog line that already carries everything the decision needs.

from typesafe_sdk import Choice, TypeSafeClient

client = TypeSafeClient()

state = (
    "<189>Sep 17 14:02:11 dc1-acc-07 %LINEPROTO-5-UPDOWN: "
    "Line protocol on Interface GigabitEthernet1/0/24, changed state to down"
)

response = client.system_one(
    state=state,
    questions={
        "subsystem": Choice(
            instructions="Which network subsystem does this message come from",
            criteria={
                "layer2_port": "Physical port or line protocol transitions",
                "routing": "BGP, OSPF, or EIGRP adjacency and route events",
                "wireless": "AP association, radio, or WLAN events",
                "platform": "Power, fan, temperature, or supervisor events",
            },
        ),
    },
)

Reserve strings for genuinely single-piece inputs [Source: https://docs.typesafe.ai/concepts/state.md]. The moment you catch yourself concatenating a ticket summary, a device hostname, and three counters into one blob separated by newlines, you have outgrown the string shape. What you have actually built is an object with the field names deleted — and you have made it impossible for a question to point at any one part of it.

Object State with Descriptive Field Names

The documentation’s own default is the object: “Use an object for most requests so each part of the state has a descriptive name and its relationships remain clear” [Source: https://docs.typesafe.ai/concepts/state.md]. Two things come with that. First, the field name is itself context — admin_down_by_change_ticket tells the model something that a bare true never could. Second, named fields are addressable, which is what makes the path references in the next section possible.

{
  "incident": {
    "number": "INC0042771",
    "short_description": "Uplink flapping on dc1-core-a since 13:50",
    "assignment_group": "Network Operations"
  },
  "device": {
    "hostname": "dc1-core-a",
    "platform": "Arista EOS",
    "site": "DC1",
    "role": "spine"
  },
  "interfaces": {
    "Ethernet1": {
      "description": "to dc1-leaf-03 Et49",
      "lineProtocolStatus": "down",
      "linkStatusChanges": 214
    }
  }
}

Grouping matters as much as naming. TypeSafe’s guidance is to keep related information together — the documentation’s example combines a refund request with the policy that governs it in a single state object rather than splitting them across requests [Source: https://docs.typesafe.ai/concepts/state.md]. The network translation is direct: an incident and the device telemetry that proves or disproves it belong in one object, because the judgment you want depends on the relationship between them.

Array State for Ordered Messages or Records

An array is the right shape for “a sequence of messages or records” [Source: https://docs.typesafe.ai/concepts/state.md] — where the items are the same kind of thing and their order carries meaning. A syslog burst is the canonical case. The order tells you whether the BGP session dropped before or after the interface did, and that ordering is frequently the whole diagnosis.

[
  "Sep 17 13:50:02 dc1-core-a Ebra: %LINEPROTO-5-UPDOWN: Interface Ethernet1, changed state to down",
  "Sep 17 13:50:03 dc1-core-a Rib: %BGP-5-ADJCHANGE: peer 10.0.2.3 Down - interface flap",
  "Sep 17 13:50:41 dc1-core-a Ebra: %LINEPROTO-5-UPDOWN: Interface Ethernet1, changed state to up",
  "Sep 17 13:51:07 dc1-core-a Ebra: %LINEPROTO-5-UPDOWN: Interface Ethernet1, changed state to down"
]

Note the constraint hiding in the specification: an array state is an array of text values [Source: https://docs.typesafe.ai/concepts/state.md]. A flat list of log lines qualifies. If your records are themselves structured — each with a timestamp, a host, and a severity you want addressable — put the array inside an object field instead, which also lets you label it (recent_syslog) and pair it with the incident it relates to.

Text-Only: No Images, Audio, or Video

The limit is stated plainly: “State must be a string, JSON object, or array of text values. Images, audio, and video are not supported (yet)” [Source: https://docs.typesafe.ai/concepts/state.md]. For network work this rules out more than it first appears. You cannot hand Jev a screenshot of a NetFlow graph, a Visio topology export, a photo of a fiber patch panel a field tech sent to the ticket, or a raw .pcap.

The workaround is always the same: render it to text before it becomes state. A packet capture becomes tshark -r capture.pcap -T fields ... output. A utilization graph becomes the numeric series or, better, the handful of summary statistics the decision actually needs. A topology becomes an adjacency list — {"dc1-core-a": ["dc1-leaf-01", "dc1-leaf-02"]} — which is more useful to the model than the picture ever was, because it is already the data the picture was drawn from.

Key Takeaway: State comes in exactly three shapes — string, JSON object, and array of text values — and objects are the recommended default because descriptive field names preserve both meaning and addressability [Source: https://docs.typesafe.ai/concepts/state.md]. Use a string only for a genuinely single passage, an array for ordered same-kind records, and convert any non-text artifact into text before it can become state.

Building State from Network Sources

Network devices already speak JSON. The work is rarely extraction — it is selection. A single show interfaces on a busy switch returns far more than any one decision needs, and in the next section you will see why shipping all of it actively hurts you.

Arista eAPI and Cisco NX-API JSON Output as State

Arista’s eAPI (Extensible API) exposes EOS commands over JSON-RPC 2.0. Clients POST to the /command-api endpoint on the switch, authenticate with HTTP basic auth, and send a list of commands — show or configuration — in a single transaction [Source: https://www.arista.com/assets/data/pdf/Whitepapers/Arista_eAPI_FINAL.pdf]. The payload is small and rigid:

{
    "jsonrpc": "2.0",
    "method": "runCmds",
    "params": {
        "version": 1,
        "cmds": ["show interfaces", "show vlan"],
        "format": "json"
    },
    "id": 1
}

Each parameter is fixed in a predictable way: jsonrpc is always "2.0", method is always "runCmds" for eAPI, version is 1, cmds is the command list, format selects "json" or "text", and id matches the response back to the request [Source: https://gist.github.com/fredhsu/8970833]. Because eAPI runs several commands in sequence inside one HTTP transaction, you can collect every snapshot a decision needs without opening a connection per command [Source: https://www.arista.com/assets/data/pdf/Whitepapers/Arista_eAPI_FINAL.pdf].

The response comes back as an array with one element per command, each a structured JSON object rather than scraped text. For show interfaces, output is organized hierarchically with interface names as keys and nested properties for bandwidth, speed, duplex, line protocol status, and error counters [Source: https://gist.github.com/fredhsu/8970833]. Trimmed to a single port, it looks like this — exact field sets vary by EOS release, so treat the shape, not the field list, as the lesson:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": [
    {
      "interfaces": {
        "Ethernet1": {
          "name": "Ethernet1",
          "description": "to dc1-leaf-03 Et49",
          "interfaceStatus": "connected",
          "lineProtocolStatus": "down",
          "bandwidth": 100000000000,
          "duplex": "duplexFull",
          "mtu": 9214,
          "interfaceCounters": {
            "inErrors": 0,
            "outErrors": 0,
            "inputErrorsDetail": { "crcErrors": 1893, "symbolErrors": 0 },
            "linkStatusChanges": 214
          }
        }
      }
    }
  ]
}

Cisco’s NX-API on Nexus platforms does the same job with a different wrapper. Requests also use JSON-RPC 2.0, with method set to cli (text output converted to JSON), cli_array (structured array output), or cli_ascii (raw ASCII), and the command supplied as params.cmd [Source: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/programmability/guide/b_Cisco_Nexus_9000_Series_NX-OS_Programmability_Guide_7x/NX_API.html]:

{
    "jsonrpc": "2.0",
    "method": "cli",
    "params": {
        "cmd": "show interfaces",
        "version": 1.0
    },
    "id": 1
}

NX-API authenticates with HTTP basic auth and then issues an nxapi_auth session cookie valid for ten minutes, which spares you re-sending credentials on every call in a polling loop [Source: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/programmability/guide/b_Cisco_Nexus_9000_Series_NX-OS_Programmability_Guide_7x/NX_API.html]. Responses arrive inside an ins_api envelope carrying type, version, sid, and an outputs array whose entries each hold input, msg, code, and body; for interface commands the body contains a TABLE_interface with ROW_interface entries listing interface name, admin and operational status, VLAN membership, duplex, speed, and error statistics [Source: https://developer.cisco.com/docs/cisco-nexus-9000-series-nx-api-cli-reference/latest/interface-commands/].

That envelope difference is the practical takeaway. Arista hands you the data directly; Cisco wraps it in ins_api and a TABLE/ROW structure [Source: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/programmability/guide/b_Cisco_Nexus_9000_Series_NX-OS_Programmability_Guide_7x/NX_API.html]. Neither wrapper belongs in your state. Both are transport artifacts — the equivalent of leaving the HTTP headers attached to a JSON body you are about to parse. Strip to the payload, then strip again to the fields the decision needs.

Junos and Aruba CLI Output: When to Keep Raw Text Versus Parse First

Not every source hands you clean JSON. Juniper devices are commonly automated through NETCONF — the Network Configuration Protocol, used with YANG data models to provide standardized, cross-platform device communication — typically via the PyEZ library [Source: https://gist.github.com/fredhsu/8970833]. NETCONF’s value in multi-vendor shops is exactly that standardization: where eAPI and NX-API each define their own request envelope and field names, NETCONF and YANG give you one model to code against when device diversity makes per-vendor parsers unsustainable.

But NETCONF and PyEZ return XML-derived structures, and Aruba AOS-CX consoles, TACACS session logs, and one-off show tech captures often give you nothing but text. So the decision becomes: parse it into fields, or paste it in raw?

That last row deserves emphasis. CLI output is written for humans, and its phrasing often carries diagnostic nuance — admin down versus notconnect versus errdisabled mean three different things to an engineer. If your parser collapses all three to "down", you have destroyed the signal before the model ever sees it. When in doubt, keep the vendor’s own string as the field’s value, and let the field name supply the structure:

state = {
    "device": {"hostname": "bld3-idf2-sw1", "platform": "Aruba AOS-CX"},
    "interface_1_1_12_raw": (
        "Interface 1/1/12 is down (Administratively down)\n"
        "  Admin state is down\n"
        "  Link state: down for 4 days (since Fri Sep 13 09:14:51 UTC 2026)\n"
        "  Link transitions: 3\n"
        "  Description: AP-3F-NORTH\n"
    ),
}

The second row of the table is the hard rule. A raw blob can be reasoned about as a whole, but it cannot be addressed. The moment a question needs to say “look at this value,” you need a field.

Combining a ServiceNow Incident with Device Telemetry in One Object

Now the running NOC triage example, end to end. An incident lands in ServiceNow; a collector pulls live interface state from the Arista spine the incident names; the two are assembled into one object state and a single System One request produces the routing, severity, and root-cause hints that get written back to the ticket.

Start with collection. This function performs the eAPI call and returns just the results array:

import os
import httpx

EAPI_URL = "https://dc1-core-a.example.net/command-api"


def run_cmds(cmds: list[str]) -> list[dict]:
    """Execute EOS commands via eAPI and return one result object per command."""
    payload = {
        "jsonrpc": "2.0",
        "method": "runCmds",
        "params": {"version": 1, "cmds": cmds, "format": "json"},
        "id": 1,
    }
    resp = httpx.post(
        EAPI_URL,
        json=payload,
        auth=(os.environ["EOS_USER"], os.environ["EOS_PASS"]),
        timeout=10.0,
    )
    resp.raise_for_status()
    return resp.json()["result"]

Next, the selection step — the one that matters most. Out of the dozens of keys eAPI returns per interface, six inform a triage decision:

def summarize_interfaces(eapi_result: dict, names: list[str]) -> dict:
    """Reduce a show interfaces result to the fields a triage decision uses."""
    summary = {}
    for name in names:
        raw = eapi_result.get("interfaces", {}).get(name)
        if raw is None:
            continue
        counters = raw.get("interfaceCounters", {})
        summary[name] = {
            "description": raw.get("description"),
            "interfaceStatus": raw.get("interfaceStatus"),
            "lineProtocolStatus": raw.get("lineProtocolStatus"),
            "linkStatusChanges": counters.get("linkStatusChanges"),
            "inErrors": counters.get("inErrors"),
            "crcErrors": counters.get("inputErrorsDetail", {}).get("crcErrors"),
        }
    return summary

Then the assembly. The ServiceNow incident contributes three fields — number for traceability, short_description because it is what the reporter actually said, and assignment_group because the current owner is context for whether a reassignment is warranted:

def build_triage_state(incident: dict, interfaces: dict, syslog: list[str]) -> dict:
    """Assemble one object state from a ServiceNow incident and device telemetry."""
    return {
        "incident": {
            "number": incident["number"],
            "short_description": incident["short_description"],
            "assignment_group": incident["assignment_group"],
        },
        "device": {
            "hostname": "dc1-core-a",
            "platform": "Arista EOS",
            "site": "DC1",
            "role": "spine",
        },
        "interfaces": interfaces,
        "recent_syslog": syslog[-8:],
    }


result = run_cmds(["show interfaces"])[0]
state = build_triage_state(
    incident=snow_incident,
    interfaces=summarize_interfaces(result, ["Ethernet1", "Ethernet2"]),
    syslog=splunk_rows,
)

The same pattern extends to every source in the running example. A Splunk saved search returns rows; you keep the timestamp, host, and message text and discard the search metadata. A NETCONF/PyEZ call on a Junos router returns an XML-derived structure; you pull the two or three leaves your question needs. In every case the collector’s job is to hand build_triage_state a small dictionary, and build_triage_state’s job is to give each one a descriptive name and put them side by side.

Figure 4.2: Data Source to State Pipeline

flowchart LR
    A["Arista eAPI"] --> N["Normalize and Filter"]
    B["Cisco NX-API"] --> N
    C["Junos NETCONF"] --> N
    D["Aruba AOS-CX CLI"] --> N
    E["Splunk Saved Search"] --> N
    F["ServiceNow Incident"] --> N
    N --> S["Object State"]

Key Takeaway: Arista eAPI and Cisco NX-API both return show output as structured JSON over JSON-RPC 2.0, differing mainly in response wrapping — Arista’s direct JSON versus Cisco’s ins_api envelope with TABLE/ROW entries [Source: https://gist.github.com/fredhsu/8970833]. Strip the transport wrapper, select only the fields a decision needs, keep vendor CLI wording raw when its exact phrasing carries meaning, and assemble everything into one named object.

Referencing State in Questions

Building good state is half the job. The other half is telling each question which part of it to look at.

Dot-and-Bracket Notation Such as ticket.messages[0].text

Once state is a JSON object, question instructions can point at specific fields using dot-and-bracket paths — dot notation for named fields, bracket indexing for array positions and keys [Source: https://docs.typesafe.ai/primitives.md]. A dot-and-bracket path such as ticket.messages[0].text or interfaces["Ethernet1"].lineProtocolStatus clarifies exactly which components should inform a given judgment [Source: https://docs.typesafe.ai/primitives.md].

If you have ever written a JSONPath expression in an Ansible filter or an xpath in a NETCONF filter, the syntax will look like something you already know. The difference is where it takes effect: you are not extracting a value in code, you are telling the model which part of the state a particular judgment hinges on. The whole state is still present; the path is emphasis, not a filter.

Here is the triage request, with each question aimed at the part of state that governs it:

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

client = TypeSafeClient()

response = client.system_one(
    state=state,
    questions={
        "owning_team": Choice(
            instructions=(
                "Using incident.short_description and the port states in "
                "interfaces, decide which team should own this incident"
            ),
            criteria={
                "data_center": "Spine, leaf, or server-facing switching inside a DC site",
                "wan": "Carrier circuits, edge routers, and site-to-site connectivity",
                "wireless": "Access points, WLAN controllers, and client association",
                "unknown": "Not enough evidence to place the incident with one team",
            },
        ),
        "impact": Score(
            instructions=(
                "Rate the operational impact using interfaces and recent_syslog, "
                "weighting device.role when judging blast radius"
            ),
            criteria=[
                "Cosmetic or already recovered; no traffic affected",
                "Redundant path degraded; traffic still flowing",
                "Active traffic loss on a redundant link",
                "Full outage on a non-redundant path",
            ],
        ),
        "physical_layer_suspected": Noul(
            instructions=(
                'interfaces["Ethernet1"].lineProtocolStatus is down with rising '
                "crcErrors, which points at a cable, optic, or patch fault rather "
                "than a configuration change"
            ),
        ),
    },
)

print(response.answers["owning_team"].choice)
print(response.answers["owning_team"].probabilities)
print(response.answers["impact"].score)
print(response.answers["physical_layer_suspected"].noul)
print(response.usage.input_tokens)

Two details are worth pausing on. The Noul instruction is written as a statement, not a question, because a Noul returns the probability that the statement is true [Source: https://docs.typesafe.ai/primitives.md]. And the paths appear inside instructions as ordinary prose references — you are naming a location in the state, in a form the model can match against the keys you supplied.

One honest caveat: Jev reads instructions literally and is not a calculator. Do not write an instruction like “flag this if linkStatusChanges exceeds 200 in the last hour.” Compute the threshold in Python, put the result in state as a named boolean or label, and let the question judge what that result means.

Placing Policy Documents Alongside the Request They Govern

The documentation’s guidance on keeping related information together uses a refund request paired with the relevant policy as its example — the request and the rules that govern it belong in one state object [Source: https://docs.typesafe.ai/concepts/state.md]. That pattern maps onto network operations more often than any other.

A change-freeze calendar, an escalation matrix, a maintenance-window policy, a customer SLA tier — these are the network equivalent of the refund policy. If you want the model to judge whether an incident warrants paging the on-call engineer, the escalation policy must be in the state, not assumed to be in the model’s weights. TypeSafe’s build guidance is explicit that you should not rely on knowledge stored in model weights when current information is available from your own database [Source: https://docs.typesafe.ai/concepts/how-to-build-with-system-one.md]. Your escalation matrix changed last quarter; the model’s training data did not.

state = {
    "incident": {
        "number": "INC0042771",
        "short_description": "Uplink flapping on dc1-core-a since 13:50",
        "assignment_group": "Network Operations",
    },
    "device": {"hostname": "dc1-core-a", "site": "DC1", "role": "spine"},
    "interfaces": interfaces,
    "escalation_policy": (
        "Page the on-call network engineer immediately for: any spine or core "
        "device with an active traffic-affecting fault; any site isolation; any "
        "WAN circuit down at a single-homed site. Queue for business hours for: "
        "single access-port faults, AP outages affecting fewer than 20 clients, "
        "and any fault on a device with a healthy redundant path."
    ),
    "change_freeze": {
        "active": True,
        "window": "2026-09-15 through 2026-09-19",
        "note": "Quarter-end freeze; only break-fix changes approved by CAB.",
    },
}

Now a Noul with the instruction “the escalation_policy requires paging the on-call engineer for this incident” is answerable from evidence the request actually contains. Without the policy field, the same question forces the model to guess at your organization’s rules — and a confident guess against the wrong policy is worse than no answer.

Every Question Sees the Same State Independently

State is sent once and every question in the questions dictionary is evaluated against the whole of it. That has three consequences worth internalizing.

Figure 4.3: Every Question Sees the Same State Independently

flowchart TD
    S["State Sent Once"] --> Q1["Question: owning_team"]
    S --> Q2["Question: impact"]
    S --> Q3["Question: physical_layer_suspected"]
    Q1 -.-> N1["No visibility into other answers"]
    Q2 -.-> N1
    Q3 -.-> N1

First, questions do not see each other’s answers. The impact score cannot be told to “use the team you picked in owning_team.” If a decision genuinely depends on a previous decision, that is two requests: run the first, put its answer into state as a new field, and run the second. Think of it as a two-stage route-map rather than one clause referencing another’s output.

Second, paths scope attention, not visibility. Writing interfaces["Ethernet1"].lineProtocolStatus does not hide recent_syslog from that question. Every field you include is visible to every question — which is precisely why the filtering discipline in the next section is not optional. A field that is noise for one question is noise in every question.

Third, state and questions share one budget. The number of questions you can ask in a single request is limited only by the request’s token budget, which state and questions share; that budget is around 32,000 tokens, roughly 150,000 characters of English text [Source: https://docs.typesafe.ai/primitives.md]. Asking twelve questions instead of three is nearly free on the state side — you pay for that large interface dump once, not twelve times — which is a strong argument for batching every judgment about one incident into a single request.

Key Takeaway: Dot-and-bracket paths such as incident.short_description and interfaces["Ethernet1"].lineProtocolStatus let a question’s instructions name exactly which part of state governs it [Source: https://docs.typesafe.ai/primitives.md]. Put the policies that govern a decision in state alongside the request rather than trusting model weights [Source: https://docs.typesafe.ai/concepts/how-to-build-with-system-one.md], and remember that every question sees the whole state independently and shares one token budget with it.

Filtering and Token Budget

You have a 32,000-token budget and a show interfaces output that could consume a good chunk of it. The temptation is to include everything and let the model sort it out. That instinct is wrong, and the research on why is unambiguous.

Context Rot: Irrelevant Fields Degrade Answers

Context rot is the phenomenon where model performance degrades progressively as input context grows — even on straightforward retrieval and reasoning tasks [Source: https://www.trychroma.com/research/context-rot]. It is not a marketing caveat. A comprehensive evaluation of 18 leading frontier models found that every single tested model exhibited degradation with longer contexts, with the severity depending on where relevant information sits, how semantically similar the query is to the surrounding content, and whether topically related distractors are present [Source: https://www.trychroma.com/research/context-rot].

The best-documented form is the “Lost in the Middle” problem. Stanford researchers placed identical factual information at different positions in a context window and measured accuracy: roughly 70-75% when the fact sat at the beginning, 55-60% when it sat in the middle, and 70-75% again at the end [Source: https://cs.stanford.edu/~nfliu/papers/lost-in-the-middle.arxiv2023.pdf]. That is a 15-20 percentage point swing driven purely by position — same fact, same relevance, different placement — and it reproduces across six major model families, indicating a property of transformer architectures rather than a quirk of any one implementation [Source: https://redis.io/blog/context-rot/].

For a network engineer, the closest mental model is a TCAM. Attention is a finite budget distributed across tokens, and as context grows it is diluted across non-essential content [Source: https://www.understandingai.org/p/context-rot-the-emerging-challenge]. When irrelevant content makes up 70-80% of the input, the model is partitioning that budget between signal and noise, and the softmax mechanism that distributes attention naturally de-emphasizes middle tokens [Source: https://www.understandingai.org/p/context-rot-the-emerging-challenge]. Every token spent on mtu, duplex, and interfaceAddress for a decision that hinges on error counters is a token not spent on the error counters.

The subtler finding is the dangerous one: topically related distractors are worse than obviously irrelevant ones [Source: https://www.trychroma.com/research/context-rot]. Including the interface counters for 47 healthy ports alongside the one flapping port is exactly that failure mode. The healthy ports are the same kind of data, in the same format, with the same field names — maximally confusable with the signal. A field of completely unrelated text would do less damage.

TypeSafe’s own build guidance draws the conclusion directly: “Include only the context relevant to the current questions. This helps the model avoid distractions and context rot” [Source: https://docs.typesafe.ai/concepts/how-to-build-with-system-one.md].

Selecting Only the Fields a Decision Needs

Decomposing the input state is the second step in TypeSafe’s System One workflow design guide, and it consists of asking, field by field, whether this data actually informs the judgment at hand [Source: https://docs.typesafe.ai/concepts/how-to-build-with-system-one.md]. Here is that audit applied to the show interfaces output from earlier, for a triage decision about team ownership, impact, and physical-layer suspicion.

Two of those rows generalize into rules. The lastStatusChangeTimestamp row is the “precompute, don’t ask” rule: derived values belong in Python, and the result goes into state as a named field. The healthy-ports row is the “same-shape noise is worst” rule: when filtering, be most aggressive about data that looks like the signal.

Positioning is the last lever. Since accuracy is measurably higher at the beginning and end of a context window than in the middle [Source: https://cs.stanford.edu/~nfliu/papers/lost-in-the-middle.arxiv2023.pdf], order your state object so the decision-critical fields come first — incident and the affected interfaces at the top, long supporting material like escalation_policy and bulk recent_syslog lower down. When you cannot avoid a long field, put the most important thing in the state before and after it rather than burying it in the middle [Source: https://redis.io/blog/context-rot/].

The Roughly 32,000-Token Request Budget and How to Stay Under It

The request’s token budget is around 32,000 tokens — roughly 150,000 characters of English text — and state and questions draw from the same pool [Source: https://docs.typesafe.ai/primitives.md]. For network data that ratio is optimistic: JSON keys, braces, quotes, and punctuation tokenize less efficiently than prose, so budget conservatively.

A quick pre-flight estimate before the call, and the authoritative count after it:

import json

def rough_tokens(state_obj) -> int:
    """Conservative pre-flight estimate: ~4 characters per token for JSON."""
    text = state_obj if isinstance(state_obj, str) else json.dumps(state_obj)
    return len(text) // 4

BUDGET = 32_000
estimate = rough_tokens(state)
if estimate > BUDGET * 0.6:
    raise ValueError(f"State is {estimate} tokens; filter before sending.")

response = client.system_one(state=state, questions=questions)
print(response.usage.input_tokens)  # actual count, from the response

The 60% threshold is a working discipline, not a documented limit: it leaves room for your questions, for criteria strings that grow as you refine them, and for the state being larger than usual on a bad day. The usage object on the response reports actual input_tokens, so log it and let real traffic tell you where your true ceiling sits.

A pre-send checklist for a triage state:

Applied to the running example, the filtered triage state — one incident, one device block, two interfaces with six fields each, an eight-line syslog slice, and an escalation policy — lands in the low hundreds of tokens. The unfiltered version, with the full show interfaces output for all 48 ports still wrapped in its JSON-RPC envelope, would run to several thousand. The filtered version is not merely cheaper and faster. Per the research, it is more accurate [Source: https://www.trychroma.com/research/context-rot].

Figure 4.4: Before and After Filtering

flowchart TD
    A["Raw show interfaces output: 48 ports, full field set"] --> B["Select Fields a Decision Needs"]
    B --> C["Compact Object State: two interfaces, six fields each"]
    C --> D["Within Token Budget: roughly 32000 tokens"]

Key Takeaway: Context rot is measurable degradation from long or noisy context, affecting every frontier model tested, with a documented 15-20 point accuracy swing based on where a fact sits in the window [Source: https://www.trychroma.com/research/context-rot] [Source: https://cs.stanford.edu/~nfliu/papers/lost-in-the-middle.arxiv2023.pdf]. Include only the fields your questions actually use, be most ruthless about same-shape distractors like healthy-peer counters, put decision-critical fields near the start, and keep the combined state and questions well under the roughly 32,000-token shared budget [Source: https://docs.typesafe.ai/primitives.md].

Chapter Summary

State is the material a System One request evaluates, and it comes in three shapes: a string for a single passage, a JSON object for named related fields, and an array of text values for an ordered sequence of records [Source: https://docs.typesafe.ai/concepts/state.md]. Objects are the recommended default because descriptive field names do double duty — they carry meaning the model can use, and they make each piece of state addressable from a question’s instructions. Images, audio, and video are not supported, so any non-text artifact in a network workflow — a packet capture, a topology diagram, a utilization graph — has to be rendered to text before it can become state [Source: https://docs.typesafe.ai/concepts/state.md].

Network sources make this straightforward on the collection side and demanding on the selection side. Arista eAPI and Cisco NX-API both return show output as structured JSON over JSON-RPC 2.0, differing mainly in whether the payload arrives directly or inside an ins_api envelope with TABLE/ROW entries [Source: https://gist.github.com/fredhsu/8970833] [Source: https://developer.cisco.com/docs/cisco-nexus-9000-series-nx-api-cli-reference/latest/interface-commands/]; NETCONF with YANG serves the same purpose across vendors, notably on Junos via PyEZ [Source: https://gist.github.com/fredhsu/8970833]. In every case the collector’s job is to strip the transport wrapper, select the handful of fields a decision actually turns on, and assemble them — alongside the ServiceNow incident and any governing policy — into one well-named object. Then dot-and-bracket paths such as incident.short_description or interfaces["Ethernet1"].lineProtocolStatus let each question name the part of that object it depends on [Source: https://docs.typesafe.ai/primitives.md].

The discipline that ties it together is filtering. Context rot is a measured property of transformer models, not a theoretical concern: every one of 18 frontier models tested degraded with longer context, and identical facts score 15-20 points lower when buried mid-window than when placed at either end [Source: https://www.trychroma.com/research/context-rot] [Source: https://cs.stanford.edu/~nfliu/papers/lost-in-the-middle.arxiv2023.pdf]. Topically related distractors — the counters for 47 healthy ports next to the one that flapped — do more damage than obviously unrelated content. With state and questions sharing a budget of roughly 32,000 tokens [Source: https://docs.typesafe.ai/primitives.md], the practice that keeps you inside the budget and the practice that makes answers more accurate turn out to be the same practice: include only what the questions need [Source: https://docs.typesafe.ai/concepts/how-to-build-with-system-one.md]. The next chapter puts that well-built state to work with the Choice primitive, where the quality of your criteria determines how cleanly a triage decision lands.

Key Terms

Chapter 5: The Choice Primitive: Routing and Classification

Learning Objectives

• Recognize when a decision has unordered options and therefore belongs to Choice rather than Score
• Write contrastive criteria using the what, not_for, and examples fields
• Interpret the choice, probabilities, and confidence fields of a Choice answer
• Avoid the common failure modes of classification: overlapping options, a missing catch-all, criteria that merely restate the label, and option sets that are too large

When to Use Choice

Every NOC runs on classification. A syslog message arrives and somebody decides it belongs to the WAN team. A ticket lands in the queue and somebody decides it is a wireless problem, not a firewall problem. A config snippet gets pasted into a chat channel and somebody decides it is Junos, not IOS. Those decisions are fast, they are made hundreds of times a day, and they are almost always made by the most experienced person on shift — which is exactly the wrong use of that person’s time.

The Choice primitive is the typed question you ask when a decision reduces to picking one label out of a fixed list. In TypeSafe AI’s System One model, a Choice is “a System One question type for selecting a single option from a predefined set,” and the model returns “the selected option, probability distribution across all options, and a confidence score” [Source: https://docs.typesafe.ai/primitives/choice.md]. You supply the list. The model never invents a new category, never returns two categories, and never returns prose. That constraint is the entire point: a classifier whose output space you control is a classifier you can wire directly into automation.

Unordered Options: Teams, Vendors, Categories

A Choice question has three required parts [Source: https://docs.typesafe.ai/primitives/choice.md]:

That criteria map is the option set. The JavaScript SDK’s type definition describes it precisely as “labels mapped to descriptions, or null for undescribed labels,” using an index signature of the form [label: string]: EntryType [Source: https://docs.typesafe.ai/sdk/javascript/api/type-aliases/ChoiceCriteria.md]. In plain terms: the keys are yours to name, the values describe what each key means, and a null value means “this label has no description, figure it out from the name.”

Notice what the type does not contain: any notion of sequence, rank, or distance. There is no first option and no last option. Swapping two entries in the map does not change the meaning of the question. That is the defining property of a Choice-shaped decision — the options are unordered. Teams are unordered. Vendors are unordered. Ticket categories are unordered. “Wireless” is not greater than “WAN,” and there is no midpoint between them.

In our multi-vendor NOC triage service, three decisions fit this shape immediately:

• Which team owns this incident? — WAN, Wireless, Security, Data Center
• Which vendor produced this configuration or log line? — Cisco IOS, Arista EOS, Junos, Aruba AOS-CX
• What kind of event is this? — link flap, routing adjacency change, authentication failure, hardware fault

Each of those is a Choice. None of them is a number in disguise.

Choice Versus Score: Does the Order of Options Matter?

The clean test is a single question: if I reorder my options, does the meaning change? If reordering is meaningless, use Choice. If the options sit on a line from less to more, use Score.

Figure 5.1: Choosing between Choice and Score

flowchart TD
    A["New classification decision"] --> B{"Does reordering the options change the meaning?"}
    B -->|"No"| C["Options are unordered"]
    B -->|"Yes"| D["Options sit on a scale"]
    C --> E["Use Choice: criteria as a map"]
    D --> F["Use Score: criteria as a list"]

The two primitives even look different in code, and the difference is a useful mnemonic. Choice takes a map — Choice(instructions=..., criteria={...}) — because a map has no meaningful order. Score takes a list — Score(instructions=..., criteria=[...]) — because a list does. When you find yourself wanting to write criteria={"low": ..., "medium": ..., "high": ...}, stop: you have an ordered scale wearing a Choice costume, and you are throwing away the model’s ability to land between two rungs. Use Score instead.

The two primitives compose well, and the TypeSafe documentation recommends exactly that composition. The intent-routing pattern uses “choice questions to identify the primary intent, then employ[s] additional scoring questions for complexity assessment,” a layered approach that “avoids unnecessary expense by determining routing criteria upfront” [Source: https://docs.typesafe.ai/patterns/intent-routing.md]. In NOC terms: one Choice decides who gets the ticket, and one Score decides how loudly to wake them up. You can ask both in the same request.

Analogy: A VLAN Assignment Versus a QoS Priority Level

If you want a one-sentence version of this chapter to keep in your head, use this one: Choice is a VLAN assignment, Score is a QoS priority level.

A VLAN ID is a label. VLAN 10 might be user data and VLAN 20 might be voice, but 20 is not “twice as much” as 10, and there is no VLAN 15 sitting halfway between them in meaning. You cannot average two VLANs. Renumbering them to 110 and 120 changes nothing about what they mean. Access ports get exactly one of them. That is a Choice: discrete, unordered, mutually exclusive, exhaustive over the set you defined.

A QoS priority level is the opposite. CoS 5 really is above CoS 3, which is really above CoS 0. The values sit on a line, “higher” is meaningful, and a value halfway between two class definitions is a coherent answer rather than a bug. That is a Score.

The mistake engineers make when they first pick up these primitives is treating severity as a Choice — “is this P1, P2, P3, or P4?” — because that is how the ticketing tool renders it. The ticketing tool is showing you buckets on a scale. Ask for the scale with Score, then bucket it yourself with deterministic Python thresholds you can explain to your change board. Ask for the team with Choice, because “the team” is genuinely a set of labels.

Key Takeaway: Use Choice when the answer is one label from a fixed, unordered set — a team, a vendor, a category — and Score when the answer sits on a line from less to more. The structural tell is the shape of criteria: a map for Choice, a list for Score. Reordering a Choice’s options must not change the question’s meaning; if it does, you have a scale and should be using Score.

Writing Effective Criteria

The criteria map is where classification accuracy is won or lost. The model sees your labels and your descriptions and nothing else about your intent. Two options that read similarly to the model will produce a split probability distribution, no matter how obvious the distinction is to you at 3 a.m. with twelve years of tribal knowledge.

Plain-String Criteria for Simple Cases

The simplest form maps each label to a one-line description string. Start here. Most classification problems in network operations are genuinely easy, and a plain-string criteria map is faster to write, faster to read in a code review, and cheaper in tokens.

from typesafe_sdk import TypeSafeClient, Choice

client = TypeSafeClient()  # reads TYPESAFE_API_KEY from the environment

event_kind = Choice(
    instructions="What kind of network event does this syslog message describe?",
    criteria={
        "link_state": "A physical or logical interface changed up/down state",
        "routing": "A routing protocol adjacency, neighbor, or session changed state",
        "auth": "An authentication, authorization, or accounting event",
        "hardware": "A power supply, fan, optic, or line card fault",
        "other": "Anything that does not clearly fit one of the categories above",
    },
)

Because the type definition permits null for undescribed labels, you may also write {"link_state": None, "routing": None} and let the label names carry the whole meaning [Source: https://docs.typesafe.ai/sdk/javascript/api/type-aliases/ChoiceCriteria.md]. That is defensible when your labels are unambiguous industry terms — bgp, ospf, isis — and a trap otherwise. A bare label like escalate tells the model nothing about when to escalate. Write the description.

Structured Criteria with what, not_for, and examples

Plain strings stop working the moment two options overlap in the model’s mind. The official guidance is explicit: “When similar options cause confusion, use structured criteria objects with custom fields instead of simple strings” [Source: https://docs.typesafe.ai/primitives/choice.md]. Instead of a string, the value becomes an object, and the documented pattern looks like this:

{
  "return_status": {
    "what": "Progress of a return already sent",
    "not_for": "Whether and how an item can be returned",
    "examples": [
      "Has my return arrived yet?",
      "When will my refund be paid?"
    ]
  }
}

That example is from the docs’ retail domain, but read it as a template and the network translation is immediate. what is the inclusion rule. not_for is the exclusion rule — the neighboring option this one keeps getting confused with. examples are concrete inputs that unambiguously belong here.

Two details matter more than they look. First, the field names are yours: “Field names are user-defined and non-reserved” [Source: https://docs.typesafe.ai/primitives/choice.md]. There is no schema forcing what/not_for/examples; you could write covers, excludes, and sample_tickets. Second, and this is the reason the convention works at all, the model sees both the names and the values, so the docs advise using “descriptive labels like what, not_for, and examples to disambiguate options” [Source: https://docs.typesafe.ai/primitives/choice.md]. The key not_for is itself a signal to the model that what follows is an exclusion. A key named field2 would carry the same text and none of the meaning. Stick to the documented convention unless you have a strong reason not to.

The technique the documentation is pointing at has a name worth learning: contrastive criteria. You are not describing each option in isolation; you are describing each option against its nearest neighbour. The guidance is to “use contrastive descriptions to prevent option overlap” and to “provide concrete examples that exemplify the option’s intent” [Source: https://docs.typesafe.ai/primitives/choice.md]. Here is the same idea applied to two NOC categories that constantly bleed into each other:

from typesafe_sdk import Choice

wifi_vs_security = Choice(
    instructions="Which team should own this incident?",
    criteria={
        "wireless": {
            "what": "Client association, roaming, RF, AP, or WLAN/SSID problems",
            "not_for": "A client that associates successfully but is then blocked by policy",
            "examples": [
                "Users on SSID CORP drop every few minutes in Building 4",
                "AP-3F-12 is stuck in a reboot loop",
            ],
        },
        "security": {
            "what": "Firewall policy, NAC/802.1X authorization, VPN, and access-control problems",
            "not_for": "RF coverage or AP hardware, even when the user is on Wi-Fi",
            "examples": [
                "802.1X authentication rejects contractors on the guest VLAN",
                "Site-to-site VPN tunnel to the Dallas branch is down",
            ],
        },
    },
)

Read the two not_for lines together. They form a boundary: the wireless team owns getting the client onto the RF medium, the security team owns what the client is allowed to do afterward. That boundary exists in your organisation already. The not_for field is where you write it down.

A practical rule for when to upgrade from strings to objects: write plain strings first, look at the probability distributions on a sample of real tickets, and add structure only to the options that keep splitting. Structured criteria cost tokens. Spend them where the confusion actually is.

Adding an other or unknown Option to Catch Edge Cases

A Choice answer is always one of your labels. The model does not have the option of declining. If a ticket about a failed Salesforce SSO integration reaches a Choice whose criteria list only WAN, Wireless, Security, and Data Center, it will not return an error — it will return one of those four, with the probability mass smeared across whichever two feel least wrong.

This is why every production Choice needs a catch-all option: an explicit unknown, other, or needs_human label whose description is “does not clearly match any other option.” A catch-all gives the model somewhere honest to put its uncertainty, which does two things. It keeps genuinely out-of-scope inputs out of your good categories, and it turns “I don’t know” into a routable signal rather than a wrong answer.

"unknown": {
    "what": "Not clearly a network incident, or missing the detail needed to route it",
    "not_for": "A clear network incident that merely spans two teams",
    "examples": [
        "Please reset my Salesforce password",
        "Something is broken, call me",
    ],
},

Note the not_for on the catch-all. Without it, unknown becomes a magnet: the model learns that anything even slightly hard goes there, and your automation rate collapses. The catch-all is for inputs that are out of scope or underspecified, not for inputs that are merely close calls. Close calls are handled by confidence, which is the subject of the next section.

The catch-all pairs with the second layer of protection the documentation recommends: confidence-based gating. The guidance is to always check confidence before routing, with “low confidence (< 0.5) on intent → escalate to humans,” summarised in the docs as “if we don’t have enough confidence to classify, route to a human agent” [Source: https://docs.typesafe.ai/patterns/intent-routing.md]. So you get two independent escape hatches — the model can say unknown, and your code can say “confidence too low, hold it.” Use both.

Common Mistakes in Criteria Design

Key Takeaway: Start with plain-string criteria and upgrade only the confusable options to structured objects with what, not_for, and examples — the model reads the field names as well as the values, so the documented naming convention is doing real work. Every production Choice needs a catch-all option, because the model must return one of your labels no matter what arrives. Write descriptions that name the evidence, not synonyms of the label.

Worked Example: Routing ServiceNow Tickets to Network Teams

Here is the first half of the NOC triage service end to end: a ServiceNow incident comes in, a single Choice decides which network team owns it, and — if the model is confident enough — the assignment_group field gets written back through the Table API.

Figure 5.2: ServiceNow incident routing and write-back sequence

sequenceDiagram
    participant Svc as NOC Triage Service
    participant TS as TypeSafe AI
    participant SNOW as ServiceNow Table API

    Svc->>TS: Send incident text as Choice question
    TS-->>Svc: Return team, probabilities, and confidence
    Svc->>Svc: Evaluate confidence gate
    Svc->>SNOW: "PATCH /api/now/table/incident/{sys_id}"
    SNOW-->>Svc: Return updated record with assignment_group

Criteria for WAN, Wireless, Security, Data Center, and Unknown

import os
from typesafe_sdk import TypeSafeClient, Choice

client = TypeSafeClient()  # reads TYPESAFE_API_KEY from the environment

TEAM_CRITERIA = {
    "wan": {
        "what": "Branch/site connectivity, MPLS and internet circuits, SD-WAN overlays, "
                "WAN edge routers, and carrier faults",
        "not_for": "Problems contained inside a single building's LAN or inside a data center fabric",
        "examples": [
            "Denver branch has been offline since 02:14; carrier ticket CX-88213 open",
            "BGP session to ISP on the Chicago WAN edge is flapping every few minutes",
        ],
    },
    "wireless": {
        "what": "Client association, roaming, RF coverage, access points, WLAN controllers, "
                "and SSID configuration",
        "not_for": "A wireless client that associates successfully but is then denied by policy, "
                   "and wired switchport problems",
        "examples": [
            "Users on SSID CORP in Building 4 drop every few minutes",
            "AP-3F-12 will not join the controller after the firmware upgrade",
        ],
    },
    "security": {
        "what": "Firewall policy, NAC and 802.1X authorization, VPN tunnels, and access-control lists",
        "not_for": "RF coverage or AP hardware, and routing problems with no policy component",
        "examples": [
            "802.1X rejects contractors and drops them onto the guest VLAN",
            "Site-to-site VPN to the Dallas branch is down after a firewall change",
        ],
    },
    "data_center": {
        "what": "Top-of-rack and spine switching, EVPN/VXLAN fabric, server-facing ports, "
                "and east-west connectivity inside a data center",
        "not_for": "Circuits that leave the data center, which belong to WAN",
        "examples": [
            "Leaf-07 lost its MLAG peer link and half the rack went dark",
            "VXLAN tunnel endpoints on spine-02 stopped learning MAC addresses",
        ],
    },
    "unknown": {
        "what": "Not clearly a network incident, or missing the detail needed to route it",
        "not_for": "A clear network incident that merely touches two network teams",
        "examples": [
            "Please reset my Salesforce password",
            "Internet is slow",
        ],
    },
}

Five options: four real teams and one catch-all. Every real team carries a not_for that names its nearest neighbour — WAN excludes the LAN, wireless excludes policy denials, security excludes RF, data center excludes anything that leaves the building. Those four exclusions draw the boundaries between the four teams, which is precisely what contrastive criteria are for [Source: https://docs.typesafe.ai/primitives/choice.md].

Now the request. The ticket text goes into state, and the question goes into questions:

incident = {
    "sys_id": "62826bf03710200044e0bfc8bcbe5df2",
    "number": "INC0042817",
    "short_description": "Unable to connect to office wifi",
    "description": (
        "Since the 07:30 maintenance window, users in Building 4 associate to SSID CORP "
        "but get no IP address. AP-4F-02 and AP-4F-03 are up on the controller. "
        "The access switch uplink was re-trunked during the window."
    ),
    "category": "inquiry",
}

state = f"""ServiceNow incident {incident['number']}
Short description: {incident['short_description']}
Description: {incident['description']}
"""

response = client.system_one(
    model="jev-latest",
    state=state,
    questions={
        "team": Choice(
            instructions="Which network team should own this incident?",
            criteria=TEAM_CRITERIA,
        ),
    },
)

One more optimisation before we read the answer. The docs note that you can “ask multiple Choice questions in a single request for parallel evaluation,” which “costs minimal additional tokens while providing comprehensive answers” [Source: https://docs.typesafe.ai/primitives/choice.md]. In the real triage service, questions therefore carries the team Choice, a vendor Choice, and a complexity Score together in one call to POST /v1/systemone — the layered pattern from the intent-routing guidance, at roughly the cost of the first question [Source: https://docs.typesafe.ai/patterns/intent-routing.md].

Reading the Probability Spread When Two Teams Are Plausible

Every Choice answer carries three fields: choice is “the highest-probability option,” probabilities is the “full probability distribution (sums to 1.0),” and confidence is “a 0-1 score reflecting how concentrated the probability is” [Source: https://docs.typesafe.ai/primitives/choice.md]. Here is a representative response body for the ticket above — illustrative numbers, but the shape and field names are the documented ones:

{
  "answers": {
    "team": {
      "choice": "wireless",
      "probabilities": {
        "wan": 0.03,
        "wireless": 0.46,
        "security": 0.07,
        "data_center": 0.41,
        "unknown": 0.03
      },
      "confidence": 0.44
    }
  },
  "usage": { }
}

Read that distribution the way you would read a traceroute with two equal-cost paths. The choice field says wireless, and if you looked only at choice you would page the wireless team and move on. But 0.46 versus 0.41 is not a decision, it is a tie — and the tie is correct, because the ticket genuinely contains both signals. Users are on an SSID (wireless) and an access switch uplink was re-trunked during the window (a wired VLAN problem, which in this org sits with the data center/campus switching team). The model is telling you the truth about an ambiguous ticket.

The confidence field is the compact version of that same story. The documentation puts it plainly: “A flat shape, with probability spread across several options, means low confidence. A single peak on one option means high confidence” [Source: https://docs.typesafe.ai/primitives/choice.md]. Confidence is not “how likely the top answer is to be right” in isolation — it is a measure of how peaked the distribution is. Two options at 0.46 and 0.41 produce a low number; one option at 0.93 produces a high one.

This is where the confidence gate goes, and the threshold comes straight from the routing pattern — low confidence, under 0.5, escalates to a human [Source: https://docs.typesafe.ai/patterns/intent-routing.md]:

CONFIDENCE_FLOOR = 0.5

answer = response.answers["team"]
team = answer.choice
confidence = answer.confidence
spread = sorted(answer.probabilities.items(), key=lambda kv: kv[1], reverse=True)

if team == "unknown" or confidence < CONFIDENCE_FLOOR:
    runner_up = spread[1]
    escalate_to_noc_lead(
        incident,
        reason=f"top={team} ({spread[0][1]:.2f}), runner_up={runner_up[0]} ({runner_up[1]:.2f}), "
               f"confidence={confidence:.2f}",
    )
else:
    assign_incident(incident["sys_id"], team, confidence)

A word about that CONFIDENCE_FLOOR. The 0.5 comes straight from the intent-routing pattern, which is the right place to start when you have no data of your own [Source: https://docs.typesafe.ai/patterns/intent-routing.md]. It is a documented default, not a measured value. Chapter 8 shows how to replace it by plotting confidence against accuracy on your own closed tickets, and the capstone service in Chapter 12 settles on a floor of 0.60 and an auto-assign line of 0.85 after doing exactly that. Treat 0.5 as the number you ship on day one and expect to move.

Two things about that gate are worth copying into your own code. First, it escalates on the catch-all and on low confidence — those are different failure modes, and both belong in the same branch. Second, when it escalates it hands the human the runner-up and its probability. A NOC lead who sees “wireless 0.46 / data center 0.41” makes the call in four seconds. A NOC lead who sees “the AI wasn’t sure” reads the whole ticket from scratch, and you have saved nobody any time.

Figure 5.3: Ticket routing across five teams with a confidence gate

flowchart TD
    A["ServiceNow incident text"] --> B["Choice: team classification"]
    B --> C["Probability distribution across wan, wireless, security, data_center, unknown"]
    C --> D{"Confidence below 0.5, or top choice is unknown?"}
    D -->|"Yes"| E["Escalate to NOC lead with runner-up probability"]
    D -->|"No"| F["Auto-assign incident to matched team"]

Writing the Assignment Group Back to ServiceNow

The last step is mechanical. The ServiceNow field you want is assignment_group, a reference to the sys_user_group table, and it accepts either a sys_id (preferred and more reliable) or the group’s display name [Source: https://www.servicenow.com/docs/r/api-reference/rest-apis/c_TableAPI.html]. Build the mapping from your Choice labels to real sys_ids once, and keep it in config rather than in code:

# sys_ids come from the sys_user_group table in YOUR instance — these are placeholders
# except the first, which is the group used in the ServiceNow documentation's example.
SNOW_GROUPS = {
    "wireless":    "287ebd7da9fe198100f92cc8d1d2154e",
    "wan":         "<sys_id of the WAN group>",
    "security":    "<sys_id of the Security group>",
    "data_center": "<sys_id of the Data Center group>",
}

The Table API exposes incidents at https://{instance}.service-now.com/api/now/table/incident, where {instance} is your instance name [Source: https://www.servicenow.com/docs/r/api-reference/rest-apis/c_TableAPI.html]. A POST to that URL creates a record. To update an incident that already exists, append its sys_id to the path and PATCH it:

import httpx

INSTANCE = os.environ["SNOW_INSTANCE"]
# Basic auth keeps this example short. Chapter 8 explains why production
# integrations should send an OAuth 2.0 bearer token instead.
AUTH = (os.environ["SNOW_USER"], os.environ["SNOW_PASSWORD"])


def assign_incident(sys_id: str, team: str, confidence: float) -> dict:
    url = f"https://{INSTANCE}.service-now.com/api/now/table/incident/{sys_id}"
    resp = httpx.patch(
        url,
        auth=AUTH,
        headers={"Accept": "application/json", "Content-Type": "application/json"},
        json={"assignment_group": SNOW_GROUPS[team]},
        timeout=10.0,
    )
    resp.raise_for_status()
    record = resp.json()["result"]
    log.info(
        "assigned %s to %s (sys_id=%s, confidence=%.2f)",
        record["number"], team, record["assignment_group"], confidence,
    )
    return record

A successful Table API write returns the record with its fields — number, sys_id, assignment_group, state, short_description, urgency, impact, priority, and timestamps such as created_on and updated_on [Source: https://docs.servicenow.com/bundle/washingtondc-api-reference/page/integrate/inbound-rest/task/explore-rest-api-for-table.html]. Read assignment_group back off the response rather than assuming your write landed; ACLs can silently decline a field.

Four operational cautions, all of which will bite you in a real instance:

• Roles. The calling account needs the snc_platform_rest_api_access role to use the REST Table API at all, and some fields — incident urgency, for example — sit behind field-level ACLs that require the ITIL role [Source: https://www.servicenow.com/docs/r/api-reference/rest-apis/c_TableAPI.html].
• ACLs are per field. “All fields within a record may not be available for update depending on access control lists” [Source: https://www.servicenow.com/docs/r/api-reference/rest-apis/c_TableAPI.html]. A 200 response is not proof that assignment_group changed.
• Patch, don’t recreate. On an insert, “unspecified fields are set to their data type’s null value” [Source: https://www.servicenow.com/docs/r/api-reference/rest-apis/c_TableAPI.html]. Updating the existing record by sys_id changes only the fields you send; POSTing a replacement blanks everything you left out.
• Mandatory fields vary. assignment_group is optional when creating incidents, but “it may be mandatory in specific organizational configurations” [Source: https://www.servicenow.com/docs/r/api-reference/rest-apis/c_TableAPI.html]. Confirm against your own instance, not the documentation defaults.

If you want the model’s confidence recorded on the ticket for later audit, put it in a journal or custom field that you have confirmed exists in your instance — field names outside the documented set above vary by deployment, so verify before you wire it up.

Key Takeaway: A production routing loop is three moves: ask one Choice over your teams plus a catch-all, gate on confidence below 0.5 and on the catch-all label, then PATCH assignment_group on /api/now/table/incident/{sys_id} with the group’s sys_id. When you escalate, hand the human the runner-up option and its probability — the probability spread is the most useful thing the model gives you on an ambiguous ticket.

Worked Example: Detecting Vendor from a Config Snippet

The second Choice in the triage service answers a different question: what is this? Snippets arrive pasted into tickets, scraped from backups, and attached to change requests, often with no device metadata at all. Before you can parse a config or match it against a golden template, you need to know whose syntax it is.

Figure 5.4: Vendor-detection flow from config snippet to handler

flowchart LR
    A["Config snippet"] --> B["Choice: vendor detection with examples"]
    B --> C["Vendor label: cisco_ios, arista_eos, junos, aruba_aoscx, or unknown"]
    C --> D["Downstream config parser or template matcher"]

Distinguishing Cisco IOS, Arista EOS, Junos, and Aruba AOS-CX Syntax

The four platforms fall into two families. Cisco IOS and Arista EOS “utilize a hierarchical CLI syntax that is structured similarly, while Juniper Junos utilizes a set-based CLI syntax with braces” [Source: https://netlab.tools/caveats/]. Aruba AOS-CX joins the first family: its configuration “follows a similar hierarchical structure to Cisco IOS” [Source: https://help.central.arubanetworks.com/latest/documentation/online_help/content/aos-cx/cfg/conf-cx-config-workflow.htm]. Junos is the outlier — it “organizes commands hierarchically within curly braces while IOS uses a flat file structure,” and it “stores changes in a candidate configuration before committing to the active configuration, unlike IOS which applies changes immediately” [Source: https://www.juniper.net/documentation/en_US/junos/topics/concept/junos-cli-configuration-mode-overview.html].

Here are the four snippets, all doing roughly the same job on an uplink:

! Cisco IOS
configure terminal
interface GigabitEthernet0/0/1
 description Link to Core Switch
 switchport mode trunk
 switchport trunk native vlan 1
 switchport trunk allowed vlan 1,10,20,30
 no shutdown

! Arista EOS
configure terminal
interface Ethernet24
   description Link to Core Switch
   switchport access vlan 33
   spanning-tree portfast
   spanning-tree bpduguard enable
   no shutdown

# Junos (set form)
set interfaces ge-0/0/0 description "Link to Core"
set interfaces ge-0/0/0 unit 0 family ethernet-switching vlan members vlan-10
set interfaces ge-0/0/0 unit 0 family ethernet-switching vlan members vlan-20
commit

# Junos (hierarchical form)
interfaces {
    ge-0/0/0 {
        description "Link to Core";
        unit 0 {
            family ethernet-switching {
                vlan {
                    members vlan-10;
                    members vlan-20;
                }
            }
        }
    }
}

# Aruba AOS-CX
configure terminal
interface 1/1/1
   description Link to Core Switch
   vlan trunk native 1
   vlan trunk allowed 1,10,20
   no shutdown

The discriminating markers, extracted:

Two vocabulary traps are worth flagging because they will confuse a careless classifier and a careless engineer equally. In Cisco and Aruba, “trunk” means an 802.1Q VLAN-tagged interface; AOS-CX calls aggregated interfaces a Link Aggregation Group (LAG), while the older ArubaOS-Switch called aggregation a trunk [Source: https://help.central.arubanetworks.com/latest/documentation/online_help/content/aos-cx/cfg/conf-cx-config-workflow.htm]. And protocol placement differs: Junos puts BFD timers inside the routing protocol configuration rather than on individual interfaces [Source: https://netlab.tools/caveats/]. Those are exactly the kinds of details that belong in not_for and examples.

Using examples in Criteria to Sharpen Boundaries

The examples field earns its keep here, because vendor detection is pattern matching on literal strings and literal strings are what examples carries. The documentation’s advice to “provide concrete examples that exemplify the option’s intent” translates directly into “paste real config lines” [Source: https://docs.typesafe.ai/primitives/choice.md].

from typesafe_sdk import Choice

vendor_question = Choice(
    instructions="Which network operating system produced this configuration snippet?",
    criteria={
        "cisco_ios": {
            "what": "Cisco IOS/IOS-XE: flat running-config, interfaces named by media type "
                    "and slot/port, VLANs configured with switchport commands",
            "not_for": "Snippets whose interfaces are bare numeric triples like 1/1/1, "
                       "or that use 'vlan trunk allowed' instead of 'switchport trunk allowed vlan'",
            "examples": [
                "interface GigabitEthernet0/0/1",
                "switchport trunk allowed vlan 1,10,20,30",
                "switchport trunk native vlan 1",
            ],
        },
        "arista_eos": {
            "what": "Arista EOS: Cisco-like hierarchical CLI, interfaces named Ethernet<n> "
                    "with no media-type prefix, three-space indentation",
            "not_for": "Interfaces with a media-type prefix such as GigabitEthernet, "
                       "which indicate Cisco IOS",
            "examples": [
                "interface Ethernet24",
                "switchport access vlan 33",
                "spanning-tree bpduguard enable",
            ],
        },
        "junos": {
            "what": "Juniper Junos: either 'set' statements or a brace-delimited hierarchy with "
                    "semicolon-terminated statements, and a candidate config committed explicitly",
            "not_for": "Any snippet that uses switchport or vlan trunk commands",
            "examples": [
                "set interfaces ge-0/0/0 unit 0 family ethernet-switching vlan members vlan-10",
                "description \"Link to Core\";",
                "commit",
            ],
        },
        "aruba_aoscx": {
            "what": "Aruba AOS-CX: Cisco-like hierarchy, interfaces named as a bare numeric "
                    "member/slot/port triple, VLANs configured with 'vlan trunk' commands",
            "not_for": "Snippets using 'switchport', which indicate Cisco IOS or Arista EOS",
            "examples": [
                "interface 1/1/1",
                "vlan trunk native 1",
                "vlan trunk allowed 1,10,20",
                "vsf member 1",
            ],
        },
        "unknown": {
            "what": "Not a switch or router configuration, or too short or generic to attribute "
                    "to a specific network operating system",
            "not_for": "A snippet that clearly belongs to one of the listed platforms",
            "examples": [
                "no shutdown",
                "hostname core-sw-01",
            ],
        },
    },
)

Look at how the not_for fields chain. cisco_ios excludes numeric-triple interfaces and vlan trunk; arista_eos excludes media-type prefixes; aruba_aoscx excludes switchport; junos excludes both VLAN dialects. Each exclusion points at a specific competitor using a specific literal token. That is contrastive criteria doing the work a regex would otherwise do, except that it degrades gracefully on snippets your regex never anticipated.

A high-confidence answer looks like this:

{
  "answers": {
    "vendor": {
      "choice": "arista_eos",
      "probabilities": {
        "cisco_ios": 0.04,
        "arista_eos": 0.91,
        "junos": 0.01,
        "aruba_aoscx": 0.02,
        "unknown": 0.02
      },
      "confidence": 0.89
    }
  }
}

One peak, everything else near zero — the single-peak shape the documentation describes as high confidence [Source: https://docs.typesafe.ai/primitives/choice.md]. Compare that to the ticket-routing answer earlier and the difference is visible at a glance, which is the practical reason to log the full probabilities map and not just choice.

Testing with Ambiguous Snippets

You cannot tune criteria you have not stress-tested. Build a small fixture set of deliberately hard snippets, run them through the Choice, and assert on the shape of the distribution rather than only on the top label. Ambiguous inputs should produce ambiguous distributions; that is the correct behaviour, not a bug to be tuned away.

The last three rows are the ones that will change your criteria. A snippet from an unlisted platform is the argument for the catch-all in concrete form: with no unknown option, that input must come back as Cisco, Arista, Junos, or Aruba, and your downstream template matcher will happily parse it against the wrong grammar. And the Junos brace-form row illustrates the tuning loop in miniature — when a correct answer arrives with low confidence, the fix is almost always another entry in examples, not a rewrite of what.

A final note on expectations. Jev is a fast System One model, and it reads literally. It is very good at “does this text contain the token switchport” and much weaker at anything requiring arithmetic or multi-step inference. Vendor detection plays to its strengths because the evidence is surface-level tokens. Do not extend the same Choice into “is this config compliant with our standard?” — that is a different question with a different failure mode, and it belongs in a different primitive and probably a different tier of your pipeline [Source: https://docs.typesafe.ai/patterns/intent-routing.md].

Key Takeaway: Vendor detection is a textbook Choice: four unordered platform labels plus unknown, with examples carrying literal config tokens and not_for naming the competing platform’s distinctive syntax. Test with deliberately ambiguous snippets and assert on the probability distribution, because a flat spread on no shutdown is the model behaving correctly — a confident answer there would be the real bug.

Chapter Summary

Choice is the primitive for decisions whose answer is one label from a fixed, unordered set. You supply the option set as a criteria map — keys are the labels the model may return, values are descriptions or null — and the model returns choice (the highest-probability option), probabilities (the full distribution, summing to 1.0), and confidence (how concentrated that distribution is) [Source: https://docs.typesafe.ai/primitives/choice.md]. The test for whether a decision belongs here is whether reordering the options changes the question. Teams, vendors, and categories are VLAN-like: discrete labels with no ranking. Severity, risk, and complexity are QoS-like: points on a line, and they belong to Score.

Criteria quality is what separates a classifier you can automate from one that spends its life in the escalation queue. Plain-string descriptions handle the easy cases. When options start competing, upgrade to structured objects using the documented what / not_for / examples convention — the model reads the field names as well as the values, so descriptive names are part of the signal, and contrastive descriptions are the recommended defence against option overlap [Source: https://docs.typesafe.ai/primitives/choice.md]. Every production Choice also needs a catch-all, because the model must answer with one of your labels no matter what arrives. The four recurring mistakes are overlapping options, a missing catch-all, criteria that restate the label, and option sets grown so large that probability mass cannot concentrate anywhere; the fix for the last one is multi-tier classification, a broad Choice followed by a narrow one [Source: https://docs.typesafe.ai/patterns/intent-routing.md].

The two worked examples are the two halves of the same production loop. Routing a ServiceNow incident means one Choice over WAN, Wireless, Security, Data Center, and Unknown; a gate that escalates on the catch-all or on confidence below 0.5 [Source: https://docs.typesafe.ai/patterns/intent-routing.md]; and a PATCH to /api/now/table/incident/{sys_id} setting assignment_group to the sys_id of the matching sys_user_group record [Source: https://www.servicenow.com/docs/r/api-reference/rest-apis/c_TableAPI.html]. Detecting a vendor from a config snippet means a second Choice over Cisco IOS, Arista EOS, Junos, and Aruba AOS-CX, with literal config tokens in examples and each competitor’s distinctive syntax in not_for. Both can ride in the same request, since additional Choice questions cost minimal extra tokens [Source: https://docs.typesafe.ai/primitives/choice.md]. The habit to carry forward is to log the whole probabilities map, not just choice — the spread is where the ambiguity lives, and it is the most useful thing you can hand a human when the automation steps back. The next chapter takes up Score, the primitive for every judgment that turned out to be a scale wearing a Choice costume, where the answer can legitimately land between two rungs of your rubric.

Key Terms

Chapter 6: The Score Primitive: Severity, Risk, and Spectrums

Learning Objectives

• Recognize when a decision lies on an ordered spectrum and belongs to Score
• Design score levels that separate cleanly, using a summary sentence and supporting signals (written as the what and examples fields)
• Interpret a score that falls between two levels and use the legend
• Choose correctly among Score, Choice, and Noul for a given judgment

When to Use Score

Ordered Levels: Severity, Urgency, Frustration, Blast Radius

A Score is a TypeSafe question type for evaluating content against ordered, descriptive levels. You hand Jev an ordered list of level descriptions running from low to high, and the response comes back with a numerical score, a probability distribution across those levels, and a confidence measurement [Source: https://docs.typesafe.ai/primitives/score.md].

The word that matters there is ordered. A Score fits only when the levels form a meaningful sequence — when level 2 is genuinely “more” of something than level 1, and level 3 is more still. In a NOC, that describes a large share of the judgments you make every shift:

• Severity of a fault: cosmetic, degraded, service-affecting, outage.
• Urgency of an alert: watch it, work it this week, work it today, work it now.
• Customer frustration in a support case: calm, impatient, angry, threatening to leave.
• Blast radius of a change: one interface, one site, one region, the core.
• Confidence in a root cause, capacity headroom pressure, staleness of a CMDB record — all spectrums.

The documentation is explicit that Score works best for spectrum-based judgments like bug severity, customer satisfaction, or experience levels, and that you should reach for Choice when your categories are unordered and Noul when the question is genuinely yes/no [Source: https://docs.typesafe.ai/primitives/score.md]. Picking the wrong one here is the most common design mistake engineers make with System One, so make the distinction concrete before writing any code.

The test to apply: if you can shuffle the options without losing information, it is a Choice. If shuffling them destroys meaning, it is a Score. “Routing / wireless / security” survives shuffling. “Single interface / single site / regional / core” does not.

Why a Noul of 0.5 Is Not Medium

Engineers new to System One often try to build a spectrum out of a Noul. The reasoning feels sound: a Noul returns a number between 0 and 1, so surely 0.2 means “a little urgent” and 0.9 means “very urgent.” The documentation rejects it directly: a value of 0.5 does not mean medium, and there is no separate confidence metric because the probability itself encodes both the direction of the judgment and the degree of certainty in it [Source: https://docs.typesafe.ai/primitives/noul.md].

A Noul of 0.5 means Jev thinks the proposition is about as likely true as false — a statement about uncertainty, not magnitude. Ask “is this alert urgent?” and get 0.5, and you have learned that Jev cannot tell, not that the alert is moderately urgent. Those are different operational situations: the first means a human should look, the second means schedule it for Thursday. Collapsing them into one number is how automated triage pipelines quietly start making bad calls.

A Score, by contrast, separates the two concerns cleanly. The score field tells you where on the spectrum the content sits. The confidence field tells you how sure Jev is about that placement. A blast radius score of 2.0 with confidence 0.91 and a blast radius score of 2.0 with confidence 0.31 are both “regional,” but only the second one should wake somebody up for a second opinion. You cannot express that with a single Noul probability, no matter how you threshold it.

An Analogy You Already Use: Syslog Severity 0 to 7

Every network engineer already thinks in ordered levels, because syslog forced them to. When you configure a logging threshold on a Cisco IOS-XE box or an Arista EOS switch, you pick a severity number from 0 to 7 — emergency, alert, critical, error, warning, notice, informational, debug — and the device sends you that level and everything more severe. Nobody has to explain why level 3 sits between level 2 and level 4. The scale is ordered by construction, each level has a concrete definition attached to it, and engineers use the definitions rather than the numbers when they argue about whether something is really a level 2.

A Score is the same idea with two differences. First, the direction is reversed: syslog counts down toward severity, where 0 is the emergency, while a TypeSafe Score counts up, where 0 is the lowest level in your criteria array and the highest index is the top of your spectrum. Second, a syslog severity is fixed by the device’s author at compile time, whereas a Score is assigned by Jev at request time from the level descriptions you supply — you write the severity rubric yourself, in prose, and let the model place content on it. Do any inversion in application code and comment it, because a reversed scale is one of the easiest bugs to ship and one of the hardest to notice.

Key Takeaway: Score is for judgments that live on an ordered spectrum — severity, urgency, frustration, blast radius — where each level is meaningfully more than the one below it. Do not fake a spectrum with a Noul, because a Noul of 0.5 means “I cannot tell,” not “medium.” Use the shuffle test: options that survive reordering belong in a Choice, options that do not belong in a Score.

Designing Levels

Plain-String Levels for Simple Scales

The simplest form of a Score question is an instruction plus an ordered array of strings. The array must contain at least two descriptions and at most ten, indexed from zero [Source: https://docs.typesafe.ai/sdk/javascript/api/type-aliases/ScoreCriteria.md].

from typesafe_sdk import Score, TypeSafeClient

client = TypeSafeClient()

frustration = Score(
    instructions="How frustrated the customer appears",
    criteria=[
        "Calm, just stating facts",
        "Frustrated but civil",
        "Very angry, strong language",
    ],
)

That is a three-level scale: index 0 is calm, index 1 is frustrated-but-civil, index 2 is very angry [Source: https://docs.typesafe.ai/introduction/quickstart.md]. A returned score can be anything from 0 to 2, including the space in between.

Two rules govern how you write those strings, and both come straight from the documentation. Describe situations concretely rather than degrees, and keep the question one-dimensional [Source: https://docs.typesafe.ai/primitives/score.md]. The first rule means writing “broken with a documented workaround” instead of “moderately severe.” Compare these two attempts at a severity scale for a NOC:

# Weak: the levels describe degrees of a word, not situations.
severity_vague = Score(
    instructions="How severe is this fault",
    criteria=["Minor", "Moderate", "Serious", "Very serious"],
)

# Strong: each level describes a situation an engineer could verify.
severity_concrete = Score(
    instructions="How severe is the service impact described in this alert",
    criteria=[
        "No user impact; counters or cosmetic log noise only",
        "Degraded performance on a redundant path; traffic still flowing",
        "One production service or path is down; users at one site affected",
        "Multiple sites or a core path down; widespread user impact",
    ],
)

The vague version forces Jev to guess what your organization means by “moderate.” The concrete version gives it verifiable facts to match against — is traffic still flowing? are users affected? how many sites? — which is exactly the kind of matching Jev is built to do consistently.

The one-dimensional rule is just as important and easier to violate accidentally. If you write a level that reads “affects two sites and has no rollback plan,” you have merged two spectrums into one, and Jev has no clean answer for an alert that affects two sites but does have a rollback plan. When a judgment genuinely has multiple dimensions — which network change risk always does — ask multiple Score questions instead. We will build exactly that pattern in the first worked example.

One mechanical detail: a level may be null to leave that score undescribed [Source: https://docs.typesafe.ai/sdk/javascript/api/type-aliases/ScoreCriteria.md]. In practice, most NOC rubrics describe every level, because an undescribed level is a level nobody on your team can defend in a post-incident review.

Structured Levels: A Summary and Its Signals

Plain strings work well until the scale gets subtle. The documented symptom is specific: when the model consistently scores between two levels on inputs you consider clear-cut, the fix is to enhance each level with an object containing a what field and an examples array showing representative scenarios [Source: https://docs.typesafe.ai/primitives/score.md].

Think of those two fields by their roles. The what field is the summary — one sentence defining the level. The examples array holds the signals — the concrete evidence that tells you this level applies rather than its neighbor. A level with only a summary asks Jev to interpret your definition; a level with signals shows it what the definition looks like in your data.

A note on field names: the Score reference names these fields what and examples [Source: https://docs.typesafe.ai/primitives/score.md], while the advanced-structure page’s pull-request example writes each level as a summary sentence plus a signals array [Source: https://docs.typesafe.ai/primitives/advanced.md]. Both are accepted because level objects are free-form JSON. What the docs insist on is consistency: use the same field names on every level so the model can compare like with like. Throughout this chapter, “summary” and “signals” describe the role each field plays; the code uses what and examples.

Here is the same frustration scale upgraded from plain strings to structured levels, with signals drawn from the kind of language that actually shows up in a Salesforce case about a WAN outage:

frustration = Score(
    instructions="How frustrated the customer contact appears in this case thread",
    criteria=[
        {
            "what": "Reporting facts with no complaint language",
            "examples": [
                "Circuit at the Dallas branch went down at 09:12, opening a case",
                "Attaching the traceroute you asked for",
            ],
        },
        {
            "what": "Impatient; asking for an update or a timeline",
            "examples": [
                "Any ETA on this? We have not heard back since yesterday",
                "Can someone confirm this is still being worked?",
            ],
        },
        {
            "what": "Explicitly frustrated; cites repeat contacts or missed commitments",
            "examples": [
                "This is the third outage on this circuit this month",
                "I was told Tuesday that this was fixed and it clearly is not",
            ],
        },
        {
            "what": "Threatening escalation, contract review, or leaving",
            "examples": [
                "Put me through to your director or we begin reviewing the contract",
                "We are pricing out a second provider because of this",
            ],
        },
    ],
)

One documented behavior shapes how you write these: the model evaluates each level independently and does not see neighboring levels [Source: https://docs.typesafe.ai/primitives/score.md]. That has a direct consequence. You cannot write level 2 as “more frustrated than level 1,” because when Jev is assessing level 2 it has no idea what level 1 said. Every level has to stand on its own as a complete description of a situation. Read each of your levels in isolation and ask: could an engineer who had never seen the other levels decide whether this one applies? If not, rewrite it.

This is also why overlapping levels are so damaging. If levels 1 and 2 both plausibly describe the same case, Jev will genuinely split its probability between them, and you will get a between-level score not because the input is ambiguous but because your rubric is.

How Many Levels

The floor is two and the ceiling is ten; the documentation adds that you should use up to ten levels only if each one is distinctly describable [Source: https://docs.typesafe.ai/primitives/score.md]. Within that range, the tradeoff is straightforward: too few levels throws away resolution you needed, and too many blurs the boundaries so badly that neighboring levels stop separating.

The practical rule: the number of levels should match the number of distinct actions your system can take. If your ServiceNow workflow routes a ticket to exactly four queues, a seven-level urgency score buys you nothing — three of those levels collapse into the same outcome, and you have added boundary ambiguity with no operational payoff. If a level cannot change what happens next, delete it.

Four levels is the common landing spot for network operations precisely because the industry already converged there. Network teams classify changes as Low, Medium, High, and Critical based on infrastructure context, redundancy, maintenance window, and dependencies, and ServiceNow’s change management module computes an overall risk rating from risk conditions and an optional assessment questionnaire, applying the highest value when both are active [Source: https://www.servicenow.com/community/developer-articles/getting-started-with-servicenow-change-request-risk-calculation/ta-p/2362172]. When your Score levels line up with the levels already in your ITSM tool, writing the answer back becomes a lookup instead of a translation.

Key Takeaway: Write levels as concrete, verifiable situations rather than degrees of an adjective, and keep each question one-dimensional. When plain strings stop separating cleanly, upgrade each level to a what summary plus an examples array of signals — remembering that Jev evaluates each level independently and never sees its neighbors. Pick a level count that matches the number of distinct actions your pipeline can actually take; four is the usual fit for network operations.

Reading Score Answers

The Score Value and Between-Level Results

The score field is a position on the spectrum, ranging from 0 to the highest level number, and it is calculated as the probability-weighted mean across the levels [Source: https://docs.typesafe.ai/primitives/score.md]. That single sentence explains everything surprising about Score responses.

Because it is a weighted mean, the score is almost never a whole number. A score of 1.3 means the answer falls between levels — mostly level 1, with some weight on level 2 [Source: https://docs.typesafe.ai/primitives/score.md]. This is a feature, not noise. When a Juniper MX change genuinely sits between “single site” and “regional,” a rubric that forced a hard pick would throw away the most useful thing Jev knows about it.

Figure 6.1: The ordered-levels spectrum with a between-level score

flowchart LR
    L0["Level 0: single interface or port"] --> L1["Level 1: one site or wiring closet"]
    L1 --> L2["Level 2: regional, multiple sites"]
    L2 --> L3["Level 3: core or internet edge"]
    S["Score 1.6, between Level 1 and Level 2"] -.-> L1
    S -.-> L2

Here is a full Score answer for a blast radius question with four levels, shown as JSON:

{
  "score": 1.6,
  "legend": {
    "0": "Single interface or port on one device; no transit traffic",
    "1": "One site or wiring closet; users at a single location",
    "2": "Regional; multiple sites or an aggregation layer",
    "3": "Core or internet edge; organization-wide or customer traffic"
  },
  "probabilities": {
    "0": 0.02,
    "1": 0.41,
    "2": 0.52,
    "3": 0.05
  },
  "confidence": 0.54
}

Work the arithmetic yourself once and the concept sticks for good: (0 × 0.02) + (1 × 0.41) + (2 × 0.52) + (3 × 0.05) = 0 + 0.41 + 1.04 + 0.15 = 1.60. The score is not a rounding of level 2 and it is not a rounding of level 1. It is the center of mass of Jev’s belief, sitting slightly above the midpoint between “one site” and “regional.”

Operationally, a 1.6 reads as: bigger than a single site, not clearly regional. That is an actionable answer — route this change to the CAB rather than the standard-change auto-approve path, because it sits on a boundary and a human should check the dependency list. A rubric forced to emit “level 2” would give you the same routing with no signal that it was a close call.

The Legend Maps Numbers Back to Your Levels

The legend maps level numbers back to their descriptions [Source: https://docs.typesafe.ai/primitives/score.md]. In the TypeScript SDK this is formalized as ScoreLegend<T>, a read-only mapped type described as “rubric descriptions keyed by score,” generated from the criteria you supplied [Source: https://docs.typesafe.ai/sdk/javascript/api/type-aliases/ScoreLegend.md].

The legend exists because raw scores are unreadable to humans. Nobody in a change review wants to see “blast radius: 1.6.” They want to see “between one site or wiring closet and regional; multiple sites or an aggregation layer.” The legend gives you the text to render, so the CAB argues about the rubric rather than about the number.

Use the legend for three things. Human-readable output: look up the level text before writing anything into a ServiceNow work note or a Salesforce case comment. Auditability: log the legend alongside the score, so that six months later — after the rubric has been revised twice — you still know which definitions produced that number. Bracketing a between-level score: for a 1.6, show both legend["1"] and legend["2"] and label the result as sitting between them, which is far more honest than rounding to 2 and pretending the answer was clean.

Probabilities and the Confidence Summary

probabilities gives the distribution across each level and sums to 1.0; confidence ranges from 0 to 1, with higher values indicating that probability is concentrated on one level [Source: https://docs.typesafe.ai/primitives/score.md]. In the Python SDK’s response types, a ScoreAnswer carries the expected score, confidence, rubric legend, and probabilities per integer score [Source: https://docs.typesafe.ai/sdk/python/api/types/responses.md].

Figure 6.2: The structure of a Score response

graph TD
    R["Score response"] --> Sc["score: probability-weighted mean position on the spectrum"]
    R --> Lg["legend: level number mapped back to its description"]
    R --> Pr["probabilities: distribution across levels, summing to 1.0"]
    R --> Cf["confidence: 0 to 1, how concentrated the probability mass is"]

The reason you must look at the distribution rather than the score alone is the documented warning that different probability distributions can produce identical scores, which is why examining probabilities and confidence alongside score provides fuller interpretation [Source: https://docs.typesafe.ai/primitives/score.md]. Here is a second distribution with the exact same score as the one above:

{
  "score": 1.6,
  "probabilities": {
    "0": 0.30,
    "1": 0.00,
    "2": 0.50,
    "3": 0.20
  },
  "confidence": 0.31
}

Check the arithmetic: (0 × 0.30) + (1 × 0.00) + (2 × 0.50) + (3 × 0.20) = 0 + 0 + 1.0 + 0.6 = 1.60. Same score. Completely different meaning.

The first distribution was unimodal — Jev believed the answer was level 1 or level 2 and split between two adjacent levels. That is an ordinary boundary case. The second distribution is bimodal: Jev put 30% on “single interface, no transit traffic” and 70% on “regional or worse,” with zero weight on the level in between. That is not a boundary case. That is a request where two readings of the same change are both plausible and they are far apart — usually a sign that the state you sent is missing a critical fact, like which interfaces the MX change actually touches. The score of 1.6 is arithmetically correct and operationally meaningless.

The confidence value is your fast screen for this. In the first response confidence is 0.54, consistent with weight spread over two neighboring levels. In the second it is 0.31, because the mass is split across non-adjacent levels. A practical pipeline rule for the NOC triage service: if confidence falls below your threshold, do not act on the score — enrich the state and ask again, or route to a human. Chapter 3’s response fields give you everything you need to build that gate, and it costs one comparison.

Key Takeaway: The score is a probability-weighted mean, so fractional values are the normal case and mean the answer sits between levels. Always read probabilities and confidence alongside it, because different distributions produce identical scores and a bimodal 1.6 means something entirely different from a unimodal 1.6. Use the legend to render both bracketing levels to humans and to keep an audit trail of the rubric that produced the number.

Worked Examples

Rating the Blast Radius of a Juniper MX Change Request

Blast radius describes the scope of systems, services, users, or business processes affected when a configuration change, deployment, or patch goes wrong, and it belongs at the start of the change management process — before the change request is approved and before the window opens [Source: https://virima.com/blog/what-is-blast-radius-in-it]. Three variables shape it: dependency depth (how many downstream systems rely on the configuration item being changed), service criticality (business-critical versus lab or development), and CMDB accuracy (an outdated CMDB produces underestimated blast radius and false confidence). Gartner research indicates that over 80% of unplanned IT outages originate from planned changes, with the primary cause being invisible downstream impacts rather than careless engineering [Source: https://virima.com/blog/what-is-blast-radius-in-it].

That statistic is the business case for this example: if most outages come from changes whose scope was misjudged, a fast, consistent, auditable scope rating in front of the CAB is worth building.

The change request below is the kind your NOC triage service receives from ServiceNow every day. Junos uses a candidate configuration model where changes are prepared but not applied until explicitly committed, which is why a well-formed Juniper change request can include the exact candidate diff for review before activation [Source: https://www.juniper.net/documentation/us/en/software/junos/network-mgmt/network-mgmt.pdf].

from typesafe_sdk import Score, TypeSafeClient

client = TypeSafeClient()

change_request = """
CHG0041288 - Juniper MX960, DEN-EDGE-01
Requested by: nfahey  Window: Sat 02:00-04:00 MT
Summary: Add BGP export policy to shift transit egress from AS64512 to AS64500
  during carrier maintenance. Applies to ge-0/0/6 (transit) only.
Candidate diff:
  [edit policy-options policy-statement TRANSIT-OUT]
  +    term prefer-64500 { then { as-path-prepend "64512 64512"; accept; } }
  [edit protocols bgp group TRANSIT]
  +    export TRANSIT-OUT
Devices in scope: DEN-EDGE-01 (1 of 2 edge routers, active/active)
Rollback: rollback 1 && commit; previously tested in lab 2026-09-02
Downstream CIs per CMDB: 3 aggregation switches, 2 branch sites, 1 payment gateway VRF
"""

questions = {
    "blast_radius": Score(
        instructions=(
            "How large is the blast radius if this change goes wrong - "
            "how much of the network and how many users would be affected"
        ),
        criteria=[
            "A single interface or port on one device; no transit or user traffic crosses it",
            "One site or wiring closet; users at a single location would be affected",
            "Regional; multiple sites or an aggregation layer would be affected",
            "Core or internet edge; organization-wide or customer-facing traffic would be affected",
        ],
    ),
    "rollback_difficulty": Score(
        instructions="How hard would it be to back this change out if it fails",
        criteria=[
            "One documented command, previously tested, restores the prior state",
            "A documented procedure exists but has not been tested on this platform",
            "Rollback requires manual reconstruction of configuration from backups",
            "No practical rollback; a failure requires a hardware or firmware recovery",
        ],
    ),
    "timing_risk": Score(
        instructions="How much additional risk does the timing of this change carry",
        criteria=[
            "Approved maintenance window, low-traffic period, full staffing",
            "Off-peak but outside a formal window, or reduced staffing",
            "Business hours on a redundant system",
            "Peak business hours, a change blackout period, or a live customer event",
        ],
    ),
}

response = client.system_one(state=change_request, questions=questions)

print(response.answers["blast_radius"].score)  # e.g. 1.6

Three separate questions, not one. This is the one-dimensional rule from the previous section applied to a judgment that network teams have always treated as multi-dimensional. ITIL evaluates changes across risk (impact combined with probability), cost analysis, and service impact assessment, with risk itself plotted on a two-dimensional matrix of impact against probability of a negative outcome [Source: https://advisera.com/20000academy/blog/2015/06/30/three-key-elements-of-assessment-and-evaluation-of-changes-according-to-itil/]. Trying to compress blast radius, rollback feasibility, and timing into one Score would force Jev to weight those dimensions for you, invisibly and inconsistently.

The documented pattern for putting them back together is composite scoring: ask multiple Score questions together, normalize each by dividing by len(criteria) - 1, then combine them with weights in your application code [Source: https://docs.typesafe.ai/primitives/score.md].

# Keep your own record of how many levels each rubric has, rather than
# reading it back off the question object.
level_counts = {"blast_radius": 4, "rollback_difficulty": 4, "timing_risk": 4}

weights = {
    "blast_radius": 0.55,       # scope of impact dominates CAB routing
    "rollback_difficulty": 0.30,
    "timing_risk": 0.15,
}

def normalized(name):
    """Map a raw score onto 0.0-1.0 so scales of different lengths can combine."""
    return response.answers[name].score / (level_counts[name] - 1)

composite = sum(weights[name] * normalized(name) for name in weights)

# Map the composite back onto the risk levels the change process already uses.
if composite < 0.25:
    risk = "Low"
elif composite < 0.50:
    risk = "Medium"
elif composite < 0.75:
    risk = "High"
else:
    risk = "Critical"

Note where the arithmetic happens: in Python, not in the prompt. Jev’s job is to place content on each rubric; combining and weighting those placements is deterministic math that belongs in code you can unit-test and a CAB member can read. It also means changing the weight on rollback difficulty is a code review, not a prompt rewrite.

Figure 6.3: Composite scoring across three one-dimensional Score questions

flowchart TD
    BR["Blast radius score, weight 0.55"] --> N["Normalize each score by dividing by levels minus 1"]
    RD["Rollback difficulty score, weight 0.30"] --> N
    TR["Timing risk score, weight 0.15"] --> N
    N --> W["Combine normalized scores with weights"]
    W --> C["Composite value, 0.0 to 1.0"]
    C --> Low["Low: below 0.25"]
    C --> Medium["Medium: 0.25 to 0.50"]
    C --> High["High: 0.50 to 0.75"]
    C --> Critical["Critical: above 0.75"]

Those four output labels are deliberate. Network teams already classify changes as Low (lab environments, isolated devices with backup circuits, maintenance windows), Medium (production devices with full N+1 or N+2 redundancy, off-peak windows, localized impact, proven rollback), High (core infrastructure, no redundancy, peak hours, minimal rollback feasibility), and Critical (internet edge or core routing, revenue-generating services, zero redundancy, organization-wide impact) [Source: https://www.servicenow.com/community/developer-articles/getting-started-with-servicenow-change-request-risk-calculation/ta-p/2362172]. Emitting those same four words means the Change Advisory Board — the cross-functional body of operations staff, network engineers, security representatives, and business stakeholders that evaluates changes before approval [Source: https://www.atlassian.com/itsm/change-management/change-advisory-board] — reads a familiar rating instead of learning a new scale. Organizations adopting structured CAB evaluation report that 60–70% of major incidents were previously linked to poorly assessed changes [Source: https://www.freshworks.com/freshservice/change-advisory-board/].

For this MX change, a blast radius of 1.6 is the honest answer. It touches one transit interface on one of two active/active edge routers, but the CMDB lists a payment gateway VRF downstream — and the CMDB dependency list is exactly the thing most likely to be wrong [Source: https://virima.com/blog/what-is-blast-radius-in-it]. A 1.6 at confidence 0.54 routes to the CAB with a note reading “boundary case, verify downstream CIs,” which is what a senior engineer would have written.

Rating Customer Frustration in a Salesforce Case About a WAN Outage

The second example crosses from network data into customer data. When a WAN circuit fails at a customer site, two systems light up: ServiceNow gets the incident, and Salesforce gets the case. The ServiceNow side is about restoring the circuit. The Salesforce side is about whether you are about to lose the account.

Before writing the rubric, know the fields already on the Salesforce Case object, because the Score has to add something they do not provide. The Priority picklist (API name Priority) carries standard values of High, Medium, and Low and directly drives queue assignment, SLA application through entitlement milestones, escalation trigger rules, and response time expectations [Source: https://medium.com/@aleksej.gudkov/define-entitlements-and-milestones-in-salesforce-873f78dfe6cd]. Milestone timers are priority-sensitive: if a case’s priority changes, the clock keeps running but the SLA duration changes accordingly [Source: https://www.salesforceben.com/complete-guide-to-salesforce-entitlements-and-milestones-in-service-cloud/]. The IsEscalated field is a checkbox on the Case object, set to true when an escalation rule fires, with an escalation icon appearing on the record; the flag clears automatically when the case closes or no longer meets escalation criteria [Source: https://www.infallibletechie.com/2021/11/isescalatedescalated-field-in-salesforce.html].

Here is the critical point, and the documentation states it plainly: sentiment and priority are independent dimensions. A case can be low priority (a feature request from a small account) yet carry highly negative sentiment from an extremely frustrated customer, or be high priority (a production outage) with neutral sentiment from a calm technical contact doing troubleshooting [Source: https://developer.salesforce.com/docs/analytics/einstein-vision-language/guide/use-pre-built-models-sentiment.html]. Priority tells you the operational urgency. IsEscalated tells you that a rule already fired. Neither tells you how the human on the other end feels, which is why organizations use sentiment to identify at-risk customers before churn, route dissatisfied cases to senior agents, and trigger proactive outreach [Source: https://www.exavalu.com/client_stories/innovating-customer-interaction-sentiment-analysis/].

A Score on frustration gives you that missing dimension with more resolution than a three-way positive/negative/neutral classification, because it is ordered and it returns a distribution.

case_thread = """
Case 00184402  Account: Ridgeline Logistics (Enterprise entitlement)
Priority: High    IsEscalated: false    Status: In Process    Origin: Email
Subject: Denver DC WAN link down again

--- 09:18 customer ---
Our Denver site lost its primary WAN link at 09:12. Secondary is up but everything
is crawling. Ticket please.

--- 11:40 customer ---
Any update? We are four hours into the window you quoted for first response.

--- 14:05 customer ---
This is the third outage on this circuit in five weeks. Your team told us on
August 28 that the line card was replaced and this was resolved. It clearly is not.
I need someone senior on this today, and I need a written explanation of what is
actually wrong with this circuit before we renew in November.
"""

frustration = Score(
    instructions="How frustrated the customer contact appears across this case thread",
    criteria=[
        {
            "what": "Reporting facts with no complaint language",
            "examples": [
                "Circuit at the Denver site went down at 09:12, opening a case",
                "Attaching the traceroute you asked for",
            ],
        },
        {
            "what": "Impatient; asking for an update or a timeline",
            "examples": [
                "Any ETA on this? We have not heard back since yesterday",
                "Can someone confirm this is still being worked?",
            ],
        },
        {
            "what": "Explicitly frustrated; cites repeat contacts or missed commitments",
            "examples": [
                "This is the third outage on this circuit this month",
                "I was told Tuesday that this was fixed and it clearly is not",
            ],
        },
        {
            "what": "Threatening escalation, contract review, or leaving",
            "examples": [
                "Put me through to your director or we begin reviewing the contract",
                "We are pricing out a second provider because of this",
            ],
        },
    ],
)

response = client.system_one(state=case_thread, questions={"frustration": frustration})

A plausible answer for this thread:

{
  "score": 2.7,
  "legend": {
    "0": "Reporting facts with no complaint language",
    "1": "Impatient; asking for an update or a timeline",
    "2": "Explicitly frustrated; cites repeat contacts or missed commitments",
    "3": "Threatening escalation, contract review, or leaving"
  },
  "probabilities": { "0": 0.00, "1": 0.04, "2": 0.22, "3": 0.74 },
  "confidence": 0.74
}

A 2.7 at confidence 0.74 reads as: firmly in “threatening contract review,” with residual weight on “explicitly frustrated.” The thread earns it — a repeat outage, a named broken commitment, and the renewal raised. That is a between-level score you can act on, because the distribution is unimodal at the top and the confidence is solid.

The payoff is what your pipeline does with it. Priority is already High and IsEscalated is still false, so no escalation rule has fired. Escalation rules typically fire on age, priority, sentiment, repeat issues, revenue impact, or approaching milestone deadlines [Source: https://help.salesforce.com/s/articleView?id=service.rules_escalation_rule_entry.htm&language=en_US&type=5], but nothing in the standard set reads “the customer mentioned the renewal.” A frustration score of 2.7 is that missing trigger: write it to a custom field and let the case escalate on frustration even while the SLA clock is still green.

score = response.answers["frustration"].score

if score >= 2.5:
    action = "notify account team + assign senior engineer + draft RCA commitment"
elif score >= 1.5:
    action = "personal update from a named engineer within the hour"
else:
    action = "standard queue handling"

Rating the Urgency of a Splunk Alert About Interface Errors

The third example is the one your NOC triage service runs thousands of times a day: a Splunk alert fires on rising interface errors, and something has to decide whether it interrupts a human.

Splunk Alert: "NOC - Interface Error Rate Rising"
Trigger: input_errors delta > 500 over 15m, 2 consecutive windows
host=core-agg-02.den (Arista EOS 4.31)  interface=Ethernet49/1

  Ethernet49/1 is up, line protocol is up (connected)
    10Gb/s, full-duplex, link speed negotiated
    5 minute input rate 3.94 Gbps, output rate 2.11 Gbps
       1,204,880 packets input, 8,441,290,112 bytes
       0 runts, 0 giants, 14,402 input errors, 14,402 CRC, 0 frame
       0 output errors, 0 collisions, 3 interface resets
    Last clearing of counters: 6d 04:11:22

Context: Ethernet49/1 is one of two members of Port-Channel1 to core-den-01.
Peer member Ethernet50/1: 0 input errors, carrying 2.02 Gbps.

CRC errors on a 10Gb link point at the physical layer — a marginal optic, a dirty connector, a damaged fiber. The counters here have been climbing for days on one member of a two-member port channel. The link is still up, traffic is still flowing, and nobody is paged. This is exactly the kind of alert where the right answer is “not now, but definitely before it fails.”

splunk_alert = """...the alert payload above, passed through verbatim..."""

urgency = Score(
    instructions=(
        "How urgently does a network engineer need to act on this alert, "
        "based on the current user impact and how fast it is degrading"
    ),
    criteria=[
        "Informational; counters within normal range and no user-visible effect",
        "Degradation confined to a redundant path; the backup member or link is carrying traffic normally",
        "Errors rising steadily on a production path; users may be seeing retransmits or slowness",
        "A production path is failing now; traffic is being lost on that path",
        "Multiple links or a core node are down; a site or region is offline",
    ],
)

response = client.system_one(state=splunk_alert, questions={"urgency": urgency})

A plausible answer:

{
  "score": 1.4,
  "legend": {
    "0": "Informational; counters within normal range and no user-visible effect",
    "1": "Degradation confined to a redundant path; the backup member or link is carrying traffic normally",
    "2": "Errors rising steadily on a production path; users may be seeing retransmits or slowness",
    "3": "A production path is failing now; traffic is being lost on that path",
    "4": "Multiple links or a core node are down; a site or region is offline"
  },
  "probabilities": { "0": 0.03, "1": 0.61, "2": 0.30, "3": 0.06, "4": 0.00 },
  "confidence": 0.61
}

The score is 1.4 — mostly “degradation on a redundant path,” leaning toward “errors rising on a production path.” That lean is the whole value of the answer. The port channel is protecting users right now, so this is not a page. But errors are accumulating on a link still carrying 3.94 Gbps while the peer member sits at zero, which rules out a shared upstream cause and points at this specific optic or fiber run. The 0.30 on level 2 is Jev registering that this is degrading, not stable.

Turning that into policy is a threshold on the score plus a gate on confidence:

answer = response.answers["urgency"]

if answer.score >= 3.0:
    route = "page on-call now"
elif answer.score >= 2.0:
    route = "assign to the active shift queue"
elif answer.score >= 1.0:
    route = "create a P3 incident; schedule optic replacement in the next window"
else:
    route = "suppress; log to the daily digest"

Notice that 1.4 and 1.9 land in the same bucket while 2.0 does not. That is the level-count rule doing its job: this pipeline has four distinct actions, so a five-level rubric is already one level richer than the routing needs. If level 4 never changes the outcome in your environment — because a site-down condition arrives on a different alert path entirely — drop to four levels and the remaining boundaries get sharper.

One last habit. Because the same score can come from different distributions, log score, confidence, and the full probabilities object on every automated routing decision, not just the rounded level. When someone asks in a post-incident review why the optic on Ethernet49/1 was not replaced sooner, “urgency 1.4, confidence 0.61, 30% weight on errors rising on a production path” is an answer. “Urgency: low” is not.

Figure 6.4: End-to-end flow of the Splunk urgency worked example

sequenceDiagram
    participant Splunk
    participant Service as NOC Triage Service
    participant TypeSafe as TypeSafe Jev
    participant ServiceNow

    Splunk->>Service: Interface error rate alert on Ethernet49/1
    Service->>TypeSafe: system_one with state and urgency Score question
    TypeSafe-->>Service: score 1.4, confidence 0.61, probabilities per level
    Service->>Service: Apply score threshold and confidence gate
    Service->>ServiceNow: Create P3 incident, schedule optic replacement

Key Takeaway: Multi-dimensional judgments like change risk should be several one-dimensional Score questions, normalized by dividing by len(criteria) - 1 and combined with weights in your own code rather than in the prompt. Map your top-level output onto the vocabulary the receiving system already uses — Low/Medium/High/Critical for ServiceNow change risk, a custom frustration field for Salesforce — and always log the score, confidence, and probability distribution behind every automated routing decision.

Chapter Summary

Score is the primitive for ordered judgments. Where Choice assigns content to one of several unordered labels and Noul returns the probability that a single proposition is true, Score places content on a spectrum you define as an ordered array of two to ten level descriptions, returning a probability-weighted position, a distribution across the levels, a legend, and a confidence value. The shuffle test decides between them: if reordering your options loses information, you have a spectrum — and a Noul of 0.5 is never a substitute, because it means Jev cannot tell, not that the answer is medium.

Good levels are concrete situations rather than degrees of an adjective, each readable in isolation because Jev evaluates every level without seeing its neighbors. When plain strings stop separating cleanly on inputs you consider obvious, upgrade each level to an object with a what summary and an examples array of signals drawn from your real data. Keep each question one-dimensional, and let the level count match the number of distinct actions your pipeline can take — four is the natural fit for network operations, because Low, Medium, High, and Critical are already the vocabulary of change risk in ServiceNow and in every CAB you have sat through.

Reading the answer is where the discipline lives. Fractional scores are normal and informative: a 1.6 blast radius genuinely means “bigger than one site, not clearly regional,” which is exactly the finding that should route a Juniper MX change to human review instead of auto-approval. But identical scores can come from very different distributions, so probabilities and confidence are not optional extras — a unimodal 1.6 is a boundary case and a bimodal 1.6 is a warning that your state is missing something. For multi-dimensional judgments, ask several Score questions, normalize each by len(criteria) - 1, and weight them in application code where the math is testable and the weights are reviewable. The next chapter takes up the Noul primitive in depth, where that 0-to-1 probability finally gets the treatment it deserves.

Key Terms

Chapter 7: The Noul Primitive: Yes/No with Real Probabilities

Learning Objectives

• Explain what a Noul truth probability between 0 and 1 represents
• Use optional criteria to clarify what yes and no mean
• Compose several Nouls as independent signals for a larger decision
• Select thresholds for acting on a Noul based on risk

What a Noul Is

A Probability That a Statement Is True, From 0 to 1

A Noul is the TypeSafe primitive for yes/no questions. You hand Jev a statement or a question about your state, and it returns a single number between 0 and 1: the probability that the answer is yes [Source: https://docs.typesafe.ai/primitives/noul.md]. That is the whole answer. There is no label, no rubric, no prose explanation — just a truth probability.

In the request body, a Noul question has three parts: type set to "noul", a required instructions field holding the yes/no question, and an optional criteria field holding descriptions that clarify what the true and false outcomes mean [Source: https://docs.typesafe.ai/primitives/noul.md]. In Python, you never write the type field yourself — the SDK class supplies it:

from typesafe_sdk import Noul, TypeSafeClient

client = TypeSafeClient()

response = client.system_one(
    state=syslog_line,
    questions={
        "is_flap": Noul(
            instructions="The message reports an interface repeatedly changing state",
        ),
    },
)

print(response.answers["is_flap"].noul)  # e.g. 0.94

The .noul attribute on the answer object is the probability of yes [Source: https://docs.typesafe.ai/sdk/python/api/types/responses.md]. Note the ordering of the sentence in instructions. The docs are explicit about this: phrase the question so that “a high probability means ‘yes’, so that the returned answer is unambiguous in its meaning” [Source: https://docs.typesafe.ai/primitives/noul.md]. If you write the instruction as “The interface is stable” and then threshold at 0.7 to mean “flapping,” you have built a bug into your control flow that no amount of testing the model will find.

Reach for a Noul whenever the judgment is genuinely binary: does this configuration contain a telnet line, does this Splunk alert describe a customer-facing outage, does this change ticket name a rollback owner. The official guidance lists three families of fit — binary classification, presence or absence detection, and situational assessment — and directs you to Choice when the answer is one of several named options and to Score when the answer is a position on a spectrum [Source: https://docs.typesafe.ai/primitives/noul.md]. All three primitives can be evaluated in parallel against the same state in a single request, which makes small atomic questions cheap to ask in bulk [Source: https://docs.typesafe.ai/primitives.md].

Key Takeaway: A Noul answers one yes/no question with one number between 0 and 1 — the probability that the answer is yes. Write instructions as a statement whose truth you want measured, so that high always means yes, and reach for Choice or Score instead when the answer is not genuinely binary.

0.5 Means Equal Uncertainty, Not a Medium Value

This is the single most important thing to internalize about the Noul, and it is where engineers coming from monitoring thresholds get burned.

A Score answer returns an expected score plus a separate confidence value, and a Choice answer returns a selected label plus confidence and a probabilities distribution across labels [Source: https://docs.typesafe.ai/sdk/python/api/types/responses.md]. A Noul returns neither. It returns only the noul field — “a value between 0 and 1 representing the likelihood of ‘yes,’ with no separate confidence calculation” [Source: https://docs.typesafe.ai/primitives.md]. The probability itself encodes both the direction of the judgment and the degree of certainty in it, which is why the documentation states flatly that “a value of 0.5 does not mean medium skill” [Source: https://docs.typesafe.ai/primitives/noul.md].

Think of a QoS DSCP marking. AF31 is not “half of EF.” It is a different class with different meaning. A Noul of 0.5 is not “half true” or “medium severity.” It means Jev assigns roughly equal probability to yes and to no — the evidence in your state genuinely does not settle the question. The right response to 0.5 is almost never “treat it as a moderate hit.” It is “a human needs to look at this, or my question was badly written, or my state was missing the field that would decide it.”

Here is how to read the scale in NOC terms:

Those cut points are not arbitrary. The TypeSafe guardrails cookbook builds its screening pipeline on exactly two levels — a review threshold around 0.35 that sends a case to human review, and an action threshold around 0.70 that triggers the configured automatic response [Source: https://docs.typesafe.ai/cookbooks/llm_guardrails.md]. The consistency cookbook describes the same idea with its lower edge at 0.30 [Source: https://docs.typesafe.ai/cookbooks/consistency_noul_cookbook.md]. The gap between 0.30 and 0.35 is not meaningful — both are saying “below roughly a third, stop looking.” This book uses 0.35 as the review floor throughout so that the number in the figures matches the number in the code, and we scale both cut points to network risk later in this chapter.

Figure 7.1: The Noul truth probability scale, with the neutral band and review/action thresholds

flowchart LR
    A["Below 0.35: Strong No"] --> B["0.35 to 0.69: Neutral Band - Route to Human Review"]
    B --> C["0.70 to 0.94: Probably Yes - Act on Low Risk"]
    C --> D["0.95 to 1.0: Strong Yes - Safe for Automated Action"]

Key Takeaway: The Noul probability carries both direction and certainty in one number; there is no separate confidence field to consult. A 0.5 is a statement of genuine ambiguity, not a medium-strength finding, and your code should route it to a person rather than splitting the difference.

Analogy: A Route-Map Match With a Confidence Attached

Network engineers already think in binaries with a fuzzy edge, so the Noul has a natural analogy.

A BGP prefix either matches a route-map clause or it does not. 10.1.0.0/16 either falls inside the prefix-list or it falls outside. There is no 0.7 match — the routing table is deterministic, and that determinism is exactly what makes it safe to build policy on.

Now imagine the same match, but the thing being matched is not a prefix. It is a sentence in a change ticket, or a paragraph of vendor release notes, or a free-text incident description typed by a field tech at 2 a.m. The question is still perfectly binary — “does this describe a hardware fault?” — but the input is not. A Noul gives you the route-map answer you wanted, with an honest label on how clean the match was.

That extra number changes what you are allowed to build. A regex-based syslog rule either fires or it does not, and when a message is worded slightly differently than your pattern expected, it silently does not fire. A Noul distinguishes three cases that a regex collapses into two: a clear match (0.95), a clear miss (0.05), and a message your rule genuinely cannot classify (0.5). That third case is where NOC automation historically fails quietly, and where a Noul lets you fail loudly and hand the case to a person instead.

The consistency of those numbers is what makes them usable as control-flow inputs. TypeSafe’s consistency cookbook measures Noul stability by repeating identical calls and computing the standard deviation of each question’s probability, reporting a mean standard deviation of 0.0102 against higher variation from general reasoning models [Source: https://docs.typesafe.ai/cookbooks/consistency_noul_cookbook.md]. A threshold at 0.70 is only meaningful if repeated evaluations of the same syslog line cluster tightly; if the answer wandered between 0.4 and 0.9 on identical input, no threshold would hold. When you measure repeatability yourself, include a throwaway unique identifier per call so each draw is independent and you are not reading a cached answer back [Source: https://docs.typesafe.ai/cookbooks/consistency_noul_cookbook.md].

Key Takeaway: A Noul is a route-map match for inputs that are not prefixes — a binary question answered over messy text, with a number attached saying how clean the match was. That number is stable enough to threshold on, which is what separates a Noul from a regex that silently misses reworded messages.

Clarifying Yes and No

Optional Criteria: Describing What Counts as Yes and No

The instructions field carries the question. The optional criteria field carries the definitions. The docs describe criteria for a Noul as “optional descriptions clarifying what true and false outcomes mean,” and the best-practice guidance is to include them “when yes/no boundaries are subtle” [Source: https://docs.typesafe.ai/primitives/noul.md].