Chapter 10: Advanced Structure and Model Limitations

Structured Instructions and Rubrics

Every question so far used plain English strings, and for most NOC triage questions that is still the right choice. But TypeSafe's System One models support JSON structure across question types: instructions, options, levels, and criteria are all EntryType fields, accepting strings, objects, arrays, or null. Anywhere you can put a sentence, you can put a data structure — the same shift as moving from a raw syslog line to a structured event with explicit facility, severity, and interface keys.

response = client.system_one(
    state=syslog_line,
    questions={
        "blast_radius": Choice(
            instructions={
                "task": "Classify the blast radius of this device event",
                "site_context": "Branch site, single uplink, 40 users",
            },
            criteria={
                "single_port": "One access port, one host affected",
                "single_device": "The whole device is down or isolated",
                "site_wide": "The site loses connectivity to the core",
            },
        ),
    },
)

The same applies to criteria. Score and Noul questions can use structured entries with definitions, signals, and "not for" descriptions, turning a vague rubric into an auditable classification table with an explicit exclusion list for each level.

Field Objects

A field object has three documented components:

The payoff is reuse: write one field definition once, then drive three different question types from it.

Figure 10.1: One field object driving Noul, Choice, and Score

OPTICAL_RX = {
    "name": "optical_rx_power",
    "type": "number",
    "description": "Receive optical power on the transceiver, in dBm",
}

response = client.system_one(
    state=show_interface_transceiver_output,
    questions={
        "rx_present": Noul(
            instructions={"field": OPTICAL_RX, "task": "Is a receive power value reported?"},
        ),
        "rx_band": Choice(
            instructions={"field": OPTICAL_RX, "task": "Which alarm band applies"},
            criteria={
                "normal": "Within operating range",
                "warning": "Outside warning, inside alarm threshold",
                "alarm": "Outside the low or high alarm threshold",
                "not_reported": "No value present",
            },
        ),
        "rx_risk": Score(
            instructions={"field": OPTICAL_RX, "task": "How concerning for link stability"},
            criteria=["Comfortably inside range", "Near a threshold", "Past a threshold"],
        ),
    },
)

Reuse also keeps the answers consistent: when a field description drifts between the Noul and the Choice, you get contradictions and lose an afternoon finding out why.

Key Takeaway: instructions, options, levels, and criteria are all EntryType fields accepting objects and arrays, not just strings, so you can label context instead of burying it in prose. A field object with name, type, and description defines a data point once and drives Noul, Choice, and Score questions about it, keeping the answers consistent and the definition in one place.

Hierarchical Classification

A flat Choice with sixty labels is the prompt-engineering equivalent of a sixty-line ACL with no remarks: it works until it doesn't, and when it doesn't you cannot tell which line matched. Hierarchical classification replaces that with a tree walk — organise the labels into a taxonomy and navigate from root to leaf, one Choice per level. It mirrors how routing tables narrow by longest match instead of comparing against every prefix at once.

Three Choice questions of four to six labels each is far easier than one Choice of thirty, and it buys per-node observability: when accuracy drops you learn the vendor level is fine and the subsystem level under nx-os is where things go wrong — a fixable, localised problem.

Figure 10.2: Vendor to platform to subsystem taxonomy

Showing Subtree Children

The key implementation detail: at each step the options represent child nodes, with their values containing the entire subtree beneath them — not just {"cisco": "Cisco devices", "arista": "Arista devices"}. Showing the subtrees lets the model see that both options exist below and weigh competing classifications. That matters when a leaf category is not obvious from parent names alone: an alert mentioning "peer-link down" is ambiguous from four vendor names, but if the model can see vpc under cisco → nx-os and mlag under arista → eos, it has evidence to commit correctly at level one.

Walking the Tree in Code

Code owns the loop; the model owns each decision. The process loops through the nested structure, using the current node as criteria, until reaching a leaf node.

Figure 10.3: Taxonomy-walking loop

Those numbers come from testing across four taxonomies (CPC patents, Shopify products, MeSH biomedical subjects, source code). Greedy has no recovery path: guess arista on a Nexus alert and every later decision is confined to the wrong subtree. Beam search keeps cisco alive so deeper evidence can correct the early ambiguity.

Compare paths fairly with a length-normalised geometric mean: product(edge_probabilities) ** (1 / decisions). Without normalisation, a shallow leaf always beats a deep one by multiplying fewer numbers below 1.0 together. The companion metric is the separation ratio — the top path's score divided by its nearest competitor's. Near 1.0 means ambiguous; a large ratio means clear separation. Wire it as a routing rule: above threshold auto-assigns the ServiceNow ticket, near 1.0 routes to a human with both candidate paths shown.

Key Takeaway: Replace one wide Choice with one Choice per taxonomy level, passing each node's children as criteria with their subtrees as values so the model can see what lives below each branch before committing. Greedy search cannot recover from an early mistake; beam search with K=3 scored 100% against greedy's 50% in the documented tests, with paths compared by length-normalised geometric mean and ambiguity flagged by the separation ratio.

Bounded Extraction

Bounded extraction is the pattern where code, not the model, determines the set of possible answers, and the model's only job is to pick from that set: a regex finds candidate values in the text, a Choice question picks which candidate is wanted, and code copies the picked value and normalises it.

Figure 10.4: Bounded-extraction pipeline

Because the model selects exclusively from regex-discovered candidates, the value you get back is one of those spans, copied unchanged — it cannot invent a value or transpose a digit. This is a structural improvement, not a probabilistic one. Research on LLM numeric handling shows multi-digit numbers are tokenized and can be reconstructed incorrectly, so "1200" can come back as "1002" or "200" — a fundamental limitation of token-based processing. Now imagine that transposition hitting a VLAN ID or a BGP AS number.

Date Components with Choice, Then Compute in Code

"The model reads what the text says and never does the calendar math." The model answers which date component is present, represented as an enumerated choice rather than free-form parsing; code performs ordering, duration, and offset computation.

An email that never states an end time and an email whose end time your pipeline failed to read are completely different situations, and the ticket should say which one occurred.

Pre-Parsed Value Extraction and Structure Recovery

Normalisation and parsing stay in application logic. For emails, code lowercases the selected address. For phone numbers, code copies the picked value and normalises it with the model-supplied country using the phonenumbers library. For monetary amounts, code parses the number with Decimal after the model identifies which locale-specific grouping convention ("1,000.50" vs. "1.000,50") applies — the model identifies what is needed, code handles how to normalise it.

date_options = {c: f"The literal span '{c}' as it appears" for c in candidates}
date_options["not_stated"] = "The email does not state this date"

Choice(
    instructions="Which candidate is the date maintenance begins",
    criteria=date_options,
)

The same discipline scales to whole documents. The autoformat pattern asks narrow yes/no questions (is this line CLI output? does it continue the previous paragraph?) to identify structure, then code renders the markdown from the original characters. "The model never generates text: it answers narrow questions about the document." That is how you clean a pasted wall of device output in a ServiceNow work note without asking Jev to "reformat this nicely" — which is text generation, and not what a System One model is for.

Key Takeaway: In bounded extraction, code enumerates the candidates with a regex and the model only picks one, so the returned value is a span copied unchanged — it cannot be invented or have a digit transposed. Apply it to dates by extracting components as enumerated choices and doing every comparison, duration, and offset in code, gating anything below 0.60 confidence to human review.

Jev 1.13 Jaggedness

Jaggedness is TypeSafe's word for a model's uneven capability profile: excellent at some things, documented-bad at others. TypeSafe publishes a jaggedness page for Jev 1.13 so you design around the weak spots instead of discovering them in production — the way you would not deploy 8,000 VLANs on a switch rated for 4,094 and act surprised.

Literal Interpretation and Boundary Cases

Jev applies your criteria as written, not as intended. If your critical criterion says "the site is down" and an alert describes a site on its backup uplink at degraded throughput, the model has no basis for deciding whether "down" includes "degraded" — because you did not say. Structured criteria fix this by writing the boundary case into the rubric:

Choice(
    instructions="Severity of this site event",
    criteria={
        "critical": {
            "definition": "The site has no working path to the core",
            "not_for": "Site is on a backup uplink and still forwarding",
        },
        "major": {
            "definition": "Lost redundancy but still forwarding",
            "signals": ["running on backup uplink", "one core unreachable"],
        },
    },
)

This is ACL discipline: an ACL does exactly what the lines say, in order, with no interpretation of intent. A rubric is the same artifact — write the exceptions down.

Distraction and Injected Instructions

Because Jev does not treat state as hostile by default, any untrusted text placed there carries injection risk. OWASP distinguishes direct injection (user prompts manipulate behavior) from indirect injection (external sources — websites, configs, syslog, tickets, email — carry hidden attacker instructions). Indirect injection is the one that matters for a NOC pipeline, since almost everything in state is externally authored.

Six layers of defense, in build order: data is data (labelled envelope, e.g. {"untrusted_ticket_body": "..."}); input filtering (allowlists, strip attack keywords); human-in-the-loop for high-risk actions; constrain model scope to enumerated criteria; validate output schema strictly in code; least privilege for any invoked APIs.

Figure 10.5: Prompt-injection mitigation layers

This composes with bounded extraction: if the model's only legal answer is one of the spans your regex found, an injected sentence can at worst cause the wrong span to be selected — it cannot cause an arbitrary instruction to execute, because the answer space never contained one.

Key Takeaway: Jev 1.13's documented weaknesses — arithmetic, counting, numeric representations, date ordering, distraction from large state, and neutral treatment of hostile input — are design constraints, not bugs to work around with cleverer prompts. Keep the math in code, keep state narrow, write boundary cases explicitly into structured criteria, and treat every syslog line and ticket body as untrusted data behind filtering, strict output validation, and a human gate on risky actions.

Worked Examples

Example 1: Walking a Vendor → Platform → Subsystem Taxonomy

An Arista EOS alert arrives at the triage service. Instead of one thirty-label Choice, the pipeline walks a three-level tree defined as a nested dict — interior nodes map to children, leaves map to description strings.

def walk_taxonomy(alert_text, taxonomy=TAXONOMY, level_names=LEVEL_NAMES):
    node = taxonomy
    path, edge_probs = [], []
    for level in level_names:
        if not isinstance(node, dict) or not node:
            break
        response = client.system_one(
            state={"untrusted_alert_text": alert_text},
            questions={f"{level}_choice": Choice(
                instructions={"task": f"Which {level} this alert belongs to"},
                criteria=node,  # children, values are their subtrees
            )},
        )
        answer = response.answers[f"{level}_choice"]
        path.append(answer.choice)
        edge_probs.append(answer.probabilities[answer.choice])
        node = node[answer.choice]
    score = math.prod(edge_probs) ** (1 / len(edge_probs)) if edge_probs else 0.0
    return path, edge_probs, score

Running it on an MLAG peer-link alert returns ['arista', 'eos', 'mlag'] with a length-normalised score around 0.966. The subtree trick earns its keep at level one: a model shown only four vendor names has less to go on than one that can see mlag under arista → eos.

The separation ratio then gates the ServiceNow assignment: at or above threshold, auto-assign to "-".join(path); below threshold, route to noc-triage-review with a note that the classification was ambiguous. Upgrading to beam search is a contained change — keep the top K children at each level, expand all K frontiers in parallel, rank survivors by the same geometric-mean score — the change that moved documented accuracy from 50% to 100% at K=3.

Example 2: Extracting Maintenance-Window Dates Safely

A carrier sends a maintenance notification into a Salesforce case for a Juniper MX uplink. The pipeline needs start, end, duration, and freeze overlap — the last two are arithmetic, so neither goes to the model.

DATE_RE = re.compile(r"\b\d{1,2}/\d{1,2}/\d{4}\b")
candidates = DATE_RE.findall(email_body)
# ['12/01/2026', '12/02/2026', '15/01/2026', '28/11/2025']

date_options = {c: f"The literal span '{c}'" for c in candidates}
date_options["not_stated"] = "The email does not state this date"

The model picks which candidate answers start_date, end_date, and a separate date_order Choice (day_first vs. month_first). 12/01/2026 is 12 January under day-first and 1 December under month-first — eleven months apart — so the pipeline asks the model the question it can answer (which convention?) rather than asking it to compute the calendar math. Two signals support day-first here: a European sender, and 15/01/2026, which cannot be month-first since there is no fifteenth month.

def parse_span(date_span, time_span, day_first):
    a, b, year = (int(p) for p in date_span.split("/"))
    day, month = (a, b) if day_first else (b, a)
    hour, minute = (int(p) for p in time_span.split(":"))
    return datetime(year, month, day, hour, minute, tzinfo=timezone.utc)

needs_human = min(confidences) < CONFIDENCE_FLOOR or "not_stated" in dates
if not needs_human:
    try:
        start, end = parse_span(start_d, start_t, day_first), parse_span(end_d, end_t, day_first)
    except ValueError:
        start = end = None  # impossible dates like Feb 30, flagged not guessed
        needs_human = True

Code then computes duration and freeze overlap deterministically. The first run returns a 750-hour "maintenance window" for what the email clearly describes as an overnight job — the model picked the correct spans, but the parsing interpretation produced an implausible result. A sanity rule (duration_hours > 48 on an overnight window routes to human review) catches it, turning a silent thirty-one-day outage on the customer record into a flagged case for review instead.

Key Takeaway: A taxonomy walk is a loop in your code where the model makes one narrow Choice per level with each node's subtree visible as evidence, scored by a length-normalised geometric mean and gated on separation ratio. A safe date pipeline is the same shape: regex enumerates the candidate spans, the model picks which span and which locale convention applies, and code does every parse, comparison, and duration behind confidence, structural, and plausibility gates.

Key Terms