Chapter 2: AI Fundamentals for Network Engineers

Models, Tokens, and Inference

A token is the basic unit a language model processes — roughly a word or word fragment ("inference" might be one token, "engineering" might split into two). A model reads a sequence of tokens, and every economic property of a request — size limits, price, speed — is expressed in tokens, the same way your switch thinks in frames while your billing thinks in bytes.

Two consequences matter operationally. First, a token is not a networking object: tokenizers run inside the AI stack, so tapping the wire shows JSON text, not tokens. TypeSafe reports counts back explicitly, so every response carries a usage object your service can log like an interface counter. Second, different models tokenize differently — an accurate count requires that specific model's tokenizer, so token budgets do not transfer between models.

Inference is a stateless request/response call: you send a payload, you get a payload back, and the server remembers nothing afterward. TypeSafe exposes this as POST /v1/systemone, wrapped by client.system_one(...). Unlike a chat assistant — which only appears to remember earlier turns because the client resends the whole conversation — a typed decision call carries its own state and questions and is judged on its own merits. If the model needs to know this is the fourth flap today, that fact must be placed in state.

Figure 2.1: Inference as a stateless request/response call

Because each call is independent, retries are safe and re-running a failed triage is idempotent. Every response also carries a request_id and the model that answered — log both next to your ServiceNow ticket number as a correlation pair.

Inference runs in two internal phases with different bottlenecks. Prefill pushes the whole input through the model in parallel to produce the first token — compute-bound, measured by TTFT, and analogous to serialization delay that scales with how much you sent. Decode then emits each subsequent token one at a time, each depending on all previous ones — memory-bandwidth-bound, measured in TPS, and paid once per output token. A model that emits a long essay pays decode cost on every token of it; a model that emits a label and a probability barely enters decode at all. That is the structural reason bounded answers are fast.

Three numbers describe a model dependency in production. Latency is round-trip time for one call — the tail (p99) matters more than the average, the same way a BFD timer cares about worst-case path latency, not typical latency. Throughput is tokens per second and requests per minute the endpoint will accept, like a circuit's committed information rate: traffic above it should be shaped, not blasted. Cost is quoted per million tokens and usually dominated by input size, so cost control is a data-shaping problem — sending a full show tech-support dump when three syslog lines would do is the AI equivalent of leaving a packet capture running on a metered circuit.

A token is the model's unit of accounting, produced inside the AI stack and never visible on your wire, so token counts come from the API's usage field rather than from your monitoring. Inference is a stateless request/response call, with the timeouts, retries, and request IDs that implies. Latency, throughput, and per-token cost define the dependency, and all three are shaped mainly by how much you choose to send.

Classification Versus Generation

Classification is choosing one option from a defined set; generation is producing open-ended text. Confusing the two is the most common reason AI projects in network operations go sideways. Generation is what most people have used — you ask, you get paragraphs, and the output space is effectively infinite, with nothing telling you how sure the model was. It suits tasks a human will read and judge: a post-incident summary, an explanation of protocol behavior, a design document sketch.

Figure 2.2: Classification versus generation

Classification constrains the output before the model answers. The TypeSafe SDK declares this per question:

• Choice(instructions=..., criteria={...}) — pick one label from a dictionary of labels and meanings; classification in its purest form.
• Score(instructions=..., criteria=[...]) — place the input on an ordered rubric you define, low to high.
• Noul(instructions=...) — a yes/no judgment, returned as a probability of "yes."

For a ServiceNow ticket correlating a Cisco IOS-XE BGP flap with an Arista EOS port flap:

response = client.system_one(
    state=ticket,
    questions={
        "team": Choice(
            instructions="Which NOC team should own this incident",
            criteria={
                "routing": "BGP, OSPF, route policy, and WAN reachability",
                "switching": "Campus and data-center L2, VLANs, port flaps, STP",
                "wireless": "APs, WLAN controllers, RF and client association",
                "transport": "Circuits, optics, carrier faults, physical plant",
            },
        ),
        "severity": Score(
            instructions="How severe is the business impact described",
            criteria=[
                "Single user or cosmetic",
                "One site degraded, workaround exists",
                "Multiple sites or a business-critical app down",
            ],
        ),
        "customer_facing": Noul(
            instructions="The incident affects an externally visible service",
        ),
    },
)

The team set is closed — the model cannot invent a fifth team. The severity rubric is the team's own operational language, not a generic scale the model imagined. The yes/no question returns a number between 0 and 1, because the useful output is a degree of belief, not a verdict.

A bounded output is testable in the ordinary software sense: run three hundred historical tickets with known owning teams, compute accuracy, diff month over month, fail CI on regression. None of that works on a paragraph — there is no assertion for "the answer sounded reasonable." Bounded outputs are also comparable and aggregable, while free text needs a second parsing layer that adds its own errors. There is a performance dividend too: a classification answer is a handful of output tokens, while a generated explanation is hundreds or thousands, each paid sequentially in the decode phase.

The cleanest way to hold the distinction: an ACL entry matches a packet against a finite ordered list of conditions and returns permit or deny — testable, diffable, provable, auditable, because the answer space was defined before the packet arrived. A network design document is open-ended prose, varies by author, and is reviewed by judgment, not assertion. A Choice question is an ACL match; a chatbot prompt is a design document. If generated text is about to feed a field another system will act on, a design document has been written where an ACL belonged.

Classification picks one option from a set you defined in advance; generation produces open-ended text with no bounded answer space. Bounded outputs can be unit-tested against labeled history, compared across time, aggregated into metrics, and returned in a few tokens instead of thousands. The test is simple: if another system will act on the answer, it needs to be an ACL match, not a design document.

Probabilities and Calibration

A classification model does not really pick one answer — it produces a probability distribution, a confidence value for every allowed option, and the "answer" is whichever came out on top. For the ticket example:

{
  "team": {
    "choice": "routing",
    "confidence": 0.71,
    "probabilities": {
      "routing": 0.71,
      "switching": 0.19,
      "transport": 0.07,
      "wireless": 0.03
    }
  }
}

Read it like a routing table, not a verdict. Routing won at 0.71; switching is a genuine runner-up at 0.19, unsurprising since the ticket mentions an Arista leaf port flapping alongside the BGP adjacency change; transport at 0.07 fits "flapping could be an optic"; wireless is effectively ruled out at 0.03. The distribution summarizes real ambiguity in the ticket.

Figure 2.3: Probability distribution across four NOC teams

With a distribution instead of a bare label, code gets graduated behavior that mirrors how a NOC already tiers severity:

confidence = 0.71  # the ChoiceAnswer's confidence value

if confidence >= 0.85:
    assign_to_queue(team)                 # auto-assign
elif confidence >= 0.60:
    assign_to_queue(team, flag="review")  # assign, mark for review
else:
    assign_to_queue("NOC-Triage")         # a human decides

Calibration is the alignment between predicted probability and observed outcome frequency: a well-calibrated model assigns 80% confidence to decisions correct about 80% of the time. It is the same property as a link-utilization gauge reading true — 80% on the dial really is 80% of line rate. To check it, bin decisions by stated confidence and compare each bin's actual accuracy:

Every bucket's observed accuracy lands close to its stated confidence — that is the whole test, formalized as Expected Calibration Error (ECE), a single scalar tracked over time. Modern deep neural networks are systematically overconfident by default — a network may express 90% confidence while being right only about 70% of the time. For automation the consequence is direct: a model claiming "95% safe" while its real accuracy at that level is 70% will approve unsafe changes at a catastrophic rate. This is precisely what TypeSafe's RLCD (Reinforcement Learning for Calibrated Decisions) is designed against, in contrast to RLHF, which optimizes for human-preferred responses and can reinforce confident-sounding hallucinations, and can also cause mode dropping — narrowing the output distribution and suppressing the runner-up alternatives that signal ambiguity. When calibration drifts, temperature scaling and similar post-hoc techniques can often repair it without retraining.

Calibration is a property of a population of decisions, not any single decision — a model that is never wrong at 90% confidence is under-confident, not accurate. Three implications follow: you cannot audit calibration from one incident, you need the bucket table; a threshold is a statement about an acceptable error rate, not a promise of correctness (auto-assigning at 0.85 accepts roughly one mis-assignment in seven at the boundary, a deliberate business tradeoff); and calibration must be monitored continuously as vendor mix and alert shapes drift, the same way an interface error-rate baseline is re-examined, not set once.

A classification answer is a probability distribution across every allowed option, and the runner-up values tell you how ambiguous the input was. Calibration means those numbers correspond to reality — 80% confidence is right about 80% of the time — and it is verified across buckets of many decisions, never on a single case. Thresholds are only safe on a calibrated model, so measure calibration before you automate and monitor it after.

Where AI Belongs in a Network Workflow

Deterministic logic is any computation whose answer follows from the input by fixed rules — same input, same output, provably. Interface status checks, threshold comparisons, date arithmetic, regex matching, unit conversions: these stay in code permanently. Not because a model would necessarily get them wrong, but because it might, and there is no way to prove otherwise. if utilization > 80 is verified by inspection and a two-line unit test; "ask the model whether utilization is high" can only be validated statistically — a proof traded for a probability, for nothing gained. Language models are pattern-matching engines, not calculators or clocks: date math and arithmetic belong in Python, with the computed value placed into state as a fact if the model needs it. This is the same division of labor as BGP and a route-map: the protocol computes best paths by deterministic rules, the route-map encodes policy judgment.

What benefits from a model is language interpretation a reasonable engineer would judge rather than compute: interpreting a vague ticket description no regex can parse; ranking severity from a caller's words against a rubric; spotting a risky change touching core policy or a shared-services VLAN; and reconciling four vendors' different log dialects describing one physical fault. In every case, the input is language, the output is a bounded label or score, and the alternative is a human reading the text.

Figure 2.4: Deciding what stays in code versus what goes to the model

The documented consensus is direct: LLMs should not operate as autonomous pilots in network automation, for four reasons. Semantic blindness — an LLM has no awareness of network history, unstated business rules, or dependencies, so it can produce output that is syntactically correct but semantically unsafe, such as reading "optimize BGP for the new Dallas link" as license to disrupt long-standing peering policy. Amplified blast radius — AI-driven mistakes trade many small, localized errors for rare, catastrophic ones, such as an AI updating ACLs across hundreds of firewalls on flawed logic in a single push. Hallucination and undetectable error — invented commands or nonexistent parameters hide inside a thousand-line generated script that nobody can audit line by line. Context drift and structural violations — LLMs struggle to hold state across sequential steps, and research on LLM-generated topologies found violations (duplex mismatches, ring loops, oversubscription) that worsened sharply as network size grew, meaning a small lab test can mask failures that appear only at production scale.

Figure 2.5: Failure path of an LLM holding control authority

The responsible pattern is human-in-the-loop with automated safeguards: the model generates a candidate judgment, and diffing, rollback, staged deployment, and audit trails govern what happens next. A model that can only return one of four team labels with a probability distribution cannot hallucinate a nonexistent command and cannot push anything to a device — it has no control authority, and deterministic code decides what happens next.

Anything provable — status, thresholds, arithmetic, dates, string matching, and every write action — stays in deterministic code, because a proof beats a probability. The model handles interpretation of language: which team owns this, how bad the described impact is, whether a change looks risky. Keeping the model out of the control path defuses the documented failure modes of generative automation, because a bounded label with a probability cannot execute anything.

Key Terms