Study Guide: Chapter 9: Designing Decisions: Atomic Questions, Fan-Out, and Composite Scoring

The Five-Step Build Process

Key Points

TypeSafe documents a five-step process for building with System One: each step narrows the problem before the next begins.
The underlying philosophy is "code in control" — the model supplies narrow, structured decisions, not autonomous behavior.
Step 1: keep deterministic work (regex, CMDB lookups, datetime math) in code — only genuine judgments become questions.
Step 2: decompose the input state so a request includes only the fields the current questions need, not an entire device config.
Steps 3–5: ask narrow, atomic questions; batch them into one request; combine the typed answers with rules or weighted sums written in code.

This chapter turns a messy operational judgment — "should we let this change go tonight?" — into a set of typed questions plus a few lines of Python you can defend to a change advisory board (CAB). TypeSafe documents this as five deliberately ordered steps.

Step	What it means in the NOC
1. Use code when you can	Parse the diff, look up the device role in the CMDB, resolve the vPC peer, compute whether the requested time falls inside the approved window — no model call.
2. Decompose the input state	Send the ticket's summary, implementation plan, backout plan, test plan, and the diff — not the 9,000-line running config.
3. Use structure in questions	Replace "is this change risky?" with "does the diff modify BGP or OSPF configuration?"
4. Ask many questions together	Six risk questions, an intent classifier, and speculative extras in a single request.
5. Combine outputs in code	`risk = sum(weight * factor)`, then thresholds that pick auto-approve, CAB review, or reject.

Figure 9.1: The five-step build process

flowchart TD A["Step 1: Use code when you can"] --> B["Step 2: Decompose the input state"] B --> C["Step 3: Use structure in questions"] C --> D["Step 4: Ask many questions together"] D --> E["Step 5: Combine outputs in code"]

Step 1 is a discipline: anything a for loop, a regex, or a CMDB query can answer exactly should never become a question — deterministic work "is reliable and cheap." You would not ask a colleague whether 10.42.7.19 falls inside 10.42.0.0/16; you do the mask arithmetic. But when a window is described in free text ("Sunday night after the batch run"), no datetime object exists to compare — that sentence has to be interpreted, and interpretation is a judgment. The boundary between step 1 and step 3 is exactly this: structured data goes to code, prose goes to questions. Your service owns control flow; a System One request is a leaf in your program, not a driver of it.

Step 2 exists for two reasons. First, cost and latency: in a batched request the state dominates request size, so attaching a full config to every triage call is the single most expensive mistake you can make. Second, interpretation: paste an entire running config next to a 12-line diff and ask "does this change modify BGP?" and the model sees a config full of unrelated BGP stanzas. Think of state construction as building a SPAN session — mirror only the interfaces you care about.

def build_state(ticket: dict, diff_text: str, cmdb: dict) -> dict:
    """Assemble only the fields the risk questions actually need."""
    return {
        "change_number": ticket["number"],
        "short_description": ticket["short_description"],
        "implementation_plan": ticket["implementation_plan"],
        "backout_plan": ticket["backout_plan"],
        "test_plan": ticket["test_plan"],
        "requested_window_text": ticket["window_text"],
        "approved_window_text": cmdb["maintenance_window"],
        "device": {
            "hostname": cmdb["hostname"],
            "platform": cmdb["platform"],
            "role": cmdb["role"],
            "vpc_peer": cmdb["vpc_peer"],
            "supervisors": cmdb["supervisor_count"],
        },
        "config_diff": diff_text,
    }

Note what is absent: the full config, the comment thread, months of syslog. Note also what is present because code put it there — role, vpc_peer, and supervisors came from the CMDB in step 1, not from the model. Steps 4 and 5 are the payoff: many narrow questions about the same state go out in one request with no latency penalty, and the answers come back as independent typed values your code composes with rules or weighted sums. This auditability matters: when a composite score says 0.64 and the change goes to CAB, you can print the contributing factors and their weights — nobody has to trust a single opaque number.

Key Takeaway: The five-step process is an order of operations: do everything deterministic in code, send only the state the questions need, ask narrow atomic questions rather than broad ones, batch them into a single request, and combine the typed answers with rules or weights you control. The model contributes judgments; your code contributes control flow and arithmetic.

Atomic Questions

Key Points

"Is this change risky?" silently combines blast radius, protocol impact, timing, rollback quality, and testing into one hidden weighting — it cannot be answered, audited, or tested.
Decomposed, it becomes six answerable questions: five Nouls and one Score (lab validation is genuinely ordinal, not yes/no).
Test for a good question: could a competent engineer answer it in about five seconds from the state alone, without predicting the future?
Phrase questions positively (e.g., rollback_documented) and invert them in code, next to the weight, so polarity flips stay visible instead of buried in prompt text.
Independence: each question is scored on its own against the state, so you can add, test, and reorder questions without disturbing the others.

A real change advisory board's reasoning about a Cisco NX-OS change converges on recurring dimensions: impact scope, operational risk factors (rollback, testing), scheduling, and redundancy posture — plus, for Nexus platforms, vPC peer consistency and protocol convergence. Decomposed, "is this change risky?" becomes six answerable questions:

Question key	Primitive	What it asks
`core_device`	Noul	Targets a core, spine, or aggregation switch rather than a single access leaf
`routing_protocol`	Noul	The diff adds, removes, or modifies BGP or OSPF configuration
`vpc_pair`	Noul	The device is one half of a vPC pair and the change affects vPC state
`outside_window`	Noul	The requested time falls outside the device's approved maintenance window
`rollback_documented`	Noul	The backout plan is a concrete, executable procedure, not boilerplate
`lab_validation`	Score	How thoroughly the change was validated before production

Figure 9.2: Decomposing "is this change risky?" into six atomic questions

graph TD Root["Is this change risky?"] --> Q1["core_device: targets core or aggregation switch"] Root --> Q2["routing_protocol: modifies BGP or OSPF"] Root --> Q3["vpc_pair: affects vPC peer consistency"] Root --> Q4["outside_window: outside approved maintenance window"] Root --> Q5["rollback_documented: concrete backout procedure"] Root --> Q6["lab_validation: how thoroughly tested"]

lab_validation is a Score because lab validation is genuinely ordinal — "syntax checked by a peer" sits between "nothing" and "validated on matching hardware at the target release," and collapsing that gradient into yes/no throws away the distinction a CAB spends the most time on. Notice also the positive phrasing of rollback_documented: write questions positively and invert them in code, next to the weight, where the polarity flip is visible — scattering negations through question text is how risk models quietly acquire sign errors.

The test to apply before shipping any question: could a competent engineer, handed only the state, answer this in about five seconds without looking anything up? "Does this diff touch BGP?" — yes, the engineer scans for router bgp. "Will this change cause an outage?" — no, that requires knowing traffic patterns and the future; it is a prediction, not a judgment. This test also catches questions that are really lookups in disguise — "does this device have dual supervisors?" is a CMDB field, not a judgment, and belongs in the state.

The property that makes the whole design work is independence: each question is scored on its own against the document, so its answer doesn't depend on what else is in the request. TypeSafe validated this empirically across five runs: answers matched whether asked alone or alongside a dozen others, regardless of document size. Operationally this means you can add a question without re-validating the others, unit-test one question at a time against labeled historical tickets, and reorder questions for readability without wondering whether answers shifted.

Key Takeaway: Replace one unanswerable composite judgment with a handful of questions that a knowledgeable engineer could each answer in seconds from the state alone. Because each question is scored independently against the state, you can add, test, and reorder questions without disturbing the others — which is what makes a risk model maintainable rather than a prompt you are afraid to touch.

Pre-Quiz: Speculative Fan-Out

A team currently issues six separate System One calls for six risk questions on the same change ticket, resending the full state each time. Why does batching them into one call save so much, per the documented GDPR case study (12.2x cheaper, 10.0x faster for 13 questions)?

Because each question is answered by a smaller, cheaper model when batched.

Because the state is sent and counted once, and each additional question adds only the small number of tokens in its own instructions, rather than the whole state being resent per question.

Because the API caches answers from prior questions and reuses them for later ones.

Because batched requests skip the confidence-scoring step entirely.

Before knowing whether a change ticket is an OS upgrade, a VLAN edit, or a SPAN session, the NOC service asks a change_intent classifier plus issu_claimed and orphan_ports_addressed questions all in the same request, even though the latter two only matter for OS upgrades. What pattern is this, and what does it save?

Composite scoring — it saves by combining the answers with weights.

Speculative fan-out — it saves a second round trip, since if the ticket turns out to be an OS upgrade, the answers are already there at the cost of roughly the extra questions' tokens.

Intent routing — it saves by sending the request to a specialist LLM instead of System One.

Decomposition — it saves by shrinking the state sent to the model.

A risk-scoring service classifies change_intent as os_upgrade and then needs to attach a 4,000-line release-notes document to check for known compatibility issues on the target platform. Should this be handled as more speculative questions in the original request, or a second request?

More speculative questions in the original request, since fan-out removes all round trips.

A second request, because this is state that wasn't in the original request and would be too expensive to attach speculatively to every ticket.

More speculative questions, since the release notes are a question, not state.

Neither — release-note compatibility should never be checked by System One.

Speculative Fan-Out

Key Points

Speculative fan-out sends many questions — including ones you might not need — in one call, and lets code decide afterward what's relevant.
Parallel evaluation means extra questions add roughly no latency; the state is sent once, so each extra question costs only the tokens of its own instructions.
Documented case study: 13 questions in one call was 12.2x cheaper and 10.0x faster than 13 separate queries over the same document.
Rule of thumb: more questions over the same state, always batch; different state, new request.
A second request is warranted when the first answer forces you to gather genuinely new data, when speculative state would be expensive to attach to every ticket, or when a human resubmits changed state.

Speculative fan-out is one of the core System One patterns, alongside composite scoring and intent routing. Because all questions in a request are evaluated simultaneously, adding more questions typically adds no latency — two questions and fifteen questions over the same state run in roughly the same wall-clock time. The cost savings come from token accounting: the state is sent once and counted once, and each additional question adds only a sentence or two of instructions. Translated to NOC triage: a ticket plus diff might be 1,200 tokens; six risk questions add perhaps 150 tokens between them. Asking them separately means resending 7,200 tokens of state for identical answers.

Figure 9.3: Speculative fan-out versus serial round trips

flowchart LR subgraph Serial["Serial round trips"] direction LR S1["State sent"] --> S2["Question 1"] S2 --> S3["State sent again"] S3 --> S4["Question 2"] S4 --> S5["...repeat per question"] end subgraph FanOut["Speculative fan-out"] direction LR F1["State sent once"] --> F2["All questions in parallel"] F2 --> F3["Code reads relevant answers"] end

Visual animation — coming soon

The principle: speculative questions are ignored when irrelevant and save a round trip when they are not. Before knowing whether a ticket is an OS upgrade, a VLAN edit, or a SPAN session, the fan-out request asks everything at once:

FANOUT_QUESTIONS = {
    **RISK_QUESTIONS,                # Always used.

    "change_intent": CHANGE_INTENT,  # Classifier

    # Speculative: only read when change_intent == "os_upgrade".
    "issu_claimed": Noul(instructions=(
        "The implementation plan states the upgrade will be performed "
        "nondisruptively using ISSU rather than a traditional reload."
    )),
    "orphan_ports_addressed": Noul(instructions=(
        "The plan acknowledges devices connected to orphan ports and "
        "describes how their connectivity is handled during the reload."
    )),

    # Speculative: only read when change_intent == "emergency_patch".
    "cve_referenced": Noul(
        instructions="The ticket references a specific CVE or field notice.",
    ),
}

If the ticket turns out to be a SPAN session, the two ISSU-related answers come back unused at a cost of roughly sixty tokens. If it is an upgrade, the answers are already there — no second request, no second latency hit. The routing code stays clean because the conditionals live in one place:

def upgrade_flags(response, intent: str) -> dict:
    """Read speculative answers only when the intent makes them meaningful."""
    a = response.answers
    if intent == "os_upgrade":
        return {
            "issu_claimed": a["issu_claimed"].noul,
            "orphan_ports_addressed": a["orphan_ports_addressed"].noul,
        }
    if intent == "emergency_patch":
        return {"cve_referenced": a["cve_referenced"].noul}
    return {}

Fan-out removes round trips caused by question dependencies, not state dependencies. A second request is genuinely warranted when the first answer changes what data you must gather (e.g., pulling release notes or a compatibility matrix once os_upgrade is known), when the speculative state would be expensive for everyone (don't attach a 4,000-line document to every ticket on the chance one in twenty needs it), or when a human resubmits changed state after a request for a real backout plan.

Key Takeaway: Because questions are evaluated in parallel and the state is sent once, extra questions cost a few tokens and essentially no latency — batching thirteen questions instead of issuing thirteen calls measured 12.2x cheaper and 10.0x faster. Ask speculative questions up front and let code ignore the irrelevant ones; reserve a second request for the case where the first answer forces you to gather genuinely new state.

Post-Quiz: Speculative Fan-Out