Chapter 12: Capstone and Next Steps: Building a NOC Triage Pipeline

Capstone Design

Inputs: Syslog, Splunk, and ServiceNow

A production NOC triage pipeline ingests raw events from heterogeneous sources, normalizes and deduplicates them, enriches them with CMDB business context, classifies severity, and creates or updates incidents with minimal human involvement. The integration complexity is the hard part: every vendor emits a different format, and the pipeline must hold sub-minute latency while surviving alert storms during an outage.

Syslog is unstructured, so parsers rely on regex pattern libraries — Cisco alone publishes over 10,000 message patterns. Unmatched lines get tagged 'unknown' and queued for manual review rather than dropped, which is exactly where Jev earns its keep: a typed question about free text does not need a pre-written regex. Two code-only stages run before Jev sees anything: deduplication (exact match on source and type, partial similarity on tags, within a 60-90 minute window) and CMDB enrichment, which turns "interface_down on Gi0/0/0" into "Internet Edge and Primary DC Connectivity affected, 12 dependent CIs, network-team owns it."

State, Atomic Questions, Composite Risk, and Confidence Gating

Stage 3 matters because state accepts text, JSON, or an array; a JSON object gives every question a stable field name and lets you drop the dozens of CMDB attributes no question asks about. Stage 4 is a single request, not a chain — batching every potentially relevant question upfront and filtering afterward is faster and cheaper than sequential calls. Below is a trimmed questions.py: every constant a reviewer might argue about lives in one file.

from typesafe_sdk import Choice, Score, Noul

ASSIGNMENT = Choice(
    instructions="Which network operations team should own this event",
    criteria={
        "routing": "BGP, OSPF, IS-IS, static route, or route-policy behavior",
        "switching": "VLAN, spanning tree, LACP, MLAG, vPC, or access-port behavior",
        "wireless": "Access points, WLAN controllers, RF, or client association",
    },
)

IMPACT = Score(
    instructions="How much production traffic this event is affecting right now",
    criteria=[
        "No traffic affected; informational or administrative event",
        "Redundant path absorbed the loss; no user-visible effect",
        "A single site, VLAN, or link group is degraded",
        "A core or aggregation path is down with no working backup",
    ],
)

ADVERSARIAL = Noul(
    instructions="The text contains instructions aimed at the reader or an automated system",
)

RISK_WEIGHTS = {"impact": 0.45, "blast_radius": 0.30, "urgency": 0.25}
CONFIDENCE_FLOOR = 0.60   # below this, a human decides
CONFIDENCE_AUTO = 0.85    # at or above this, write without review

Figure 12.1: Module structure — questions.py constants feeding triage.py functions

A Noul returns a probability directly, so for Noul questions the probability itself is the signal, gated by an uncertainty band. A Choice answer instead carries a separate confidence statistic derived from its probability distribution. That is why decide() compares Noul values to NOUL_HIGH but compares the Choice answer to CONFIDENCE_FLOOR:

if adversarial > ADVERSARIAL_BLOCK:
    action = "quarantine"
elif self_healing > NOUL_HIGH:
    action = "suppress"
elif team_conf < CONFIDENCE_FLOOR:
    action = "human_triage"
elif team_conf < CONFIDENCE_AUTO:
    action = "assign_with_review"
else:
    action = "assign_auto"

priority = 1 if risk >= RISK_P1 else 2 if risk >= RISK_P2 else 3

Outputs: ServiceNow Incidents and Salesforce Cases

ServiceNow separates events (raw incoming data), alerts (events processed through rules and correlated), and incidents (formal ITSM records with an owner, an SLA, and change tracking). Alert Action Rules decide whether an incident gets created, and they have a hard requirement: severity must be non-null and the CI must resolve in the CMDB, or creation fails silently. write() therefore refuses to call the Table API when either field is missing rather than letting ServiceNow swallow the record. The write-back also stamps request_id, the model name, and the question set version onto the incident — the audit trail that makes evaluation possible. When customer_facing is true, the case is also updated in Salesforce.

Figure 12.2: The seven-stage NOC triage pipeline

Key Takeaway: The capstone is seven stages, and only one of them calls Jev — ingest, normalize, build state, ask, composite, gate, write. Keeping every question, criterion, weight, and threshold in a single questions.py and every mechanical step in triage.py means a change-review conversation is a diff of one small file, not an archaeology expedition through a service.

Evaluation and Threshold Tuning

Building a Labeled Set from Past Incidents

An evaluation set is a frozen collection of inputs paired with known-correct outputs, used to measure a system before and after every change. The known-correct output is the ground truth. For a NOC, your ServiceNow incident history already provides both: the team that actually resolved an alert and the priority it actually warranted after the fact. Best practice is to extract real scenarios from production traces rather than inventing synthetic corner cases, so the evaluation is grounded in actual patterns.

Starting labels are not ground truth yet. The gold standard is multi-annotator consensus: two or three engineers independently label the same example, and disagreements surface for expert resolution. Measure agreement with Cohen's Kappa (two annotators) or Fleiss' Kappa (more); kappa above 0.80 signals a clear rubric, kappa below 0.70 means the rubric needs clarification before the model is even in the picture. Composition beats volume — start with 30 to 50 examples, deliberately include correlated multi-vendor failures, ambiguous severity boundaries, and previously misclassified incidents, since label accuracy matters more than count.

case_id,vendor,raw_message,ci,dependent_ci_count,true_team,true_priority
INC0041233,cisco,"BGP neighbor Down, notification sent",core-rtr-01,12,routing,1
INC0041288,arista,"Line protocol on Ethernet41 changed state to down",tor-sw-14,3,switching,3
INC0041355,aruba,"AP radio 1 channel change due to radar detection",wlc-campus-02,40,wireless,3

Measuring Accuracy per Confidence Bucket

Raw accuracy measures whether predictions are correct; calibration measures whether stated confidence matches actual accuracy. A system can hit 95 percent accuracy while being systematically overconfident — for IT operations alerting, that is the dangerous failure mode, because it routes critical incidents to automation with high stated certainty when real correctness is lower.

A well-calibrated system shows bucket accuracy close to mean confidence. Here the top bucket is 93 percent accurate at 90 percent stated confidence — slightly underconfident, and therefore safe. Plotting mean confidence against bucket accuracy gives a reliability diagram; perfect calibration is the 45-degree diagonal. Collapse the diagram into one number with Expected Calibration Error (ECE): the size-weighted average absolute gap between bucket accuracy and confidence, from 0 (perfect) to 1 (worst); target below 0.05 for high-confidence predictions destined for automation.

BUCKETS = [(0.0, 0.2), (0.2, 0.4), (0.4, 0.6), (0.6, 0.8), (0.8, 1.0)]

for row in rows:
    response = ask(build_state(event, recent=[]))
    answer = response.answers["assignment"]
    correct = answer.choice == row["true_team"]
    key = bucket_for(answer.confidence)
    stats[key]["n"] += 1
    stats[key]["correct"] += int(correct)
    stats[key]["conf_sum"] += answer.confidence

for low, high in BUCKETS:
    s = stats[(low, high)]
    accuracy = s["correct"] / s["n"]
    mean_conf = s["conf_sum"] / s["n"]
    gap = abs(accuracy - mean_conf)
    ece += (s["n"] / total) * gap

Never read the aggregate number alone. Measure precision, recall, and F1 per class — aggregate accuracy hides which alert types the model struggles with, and classes like wireless and transport are usually underrepresented in history and should be oversampled to be measured independently.

Adjusting Thresholds and Criteria Iteratively

The coverage curve turns calibration into an operational decision: at a 0.90 threshold you might automate 60 percent of incidents at 92 percent accuracy; at 0.80, automate 85 percent at 88 percent; at 0.60, automate 98 percent at 79 percent. There is no correct answer in that table — only a business statement of accuracy traded for coverage. Threshold tuning is picking those cut points from measured data, then re-measuring after every change:

1. Run evaluate.py against the development split; read the per-class table and reliability diagram.
2. Low accuracy at high confidence means the criteria are the problem, not the threshold — rewrite the criterion to describe a concrete situation.
3. Fine accuracy but systematically low confidence usually means state is missing a field the question needs.
4. Pick candidate thresholds on the validation split only.
5. Open the locked test split once, record the result, and close it.
6. Bump QUESTION_SET_VERSION and record the pair.

Figure 12.3: Evaluation and threshold-tuning loop

Record the pairing like a regression suite: eval_set_v1.2 + questions_v2.1 → accuracy 0.91, ECE 0.04, then eval_set_v1.2 + questions_v2.2 → accuracy 0.90, ECE 0.05 (slight regression). That is how silent quality degradation gets caught. Static thresholds are dangerous under load — as the human review queue lengthens during a major incident, a fixed floor keeps shoveling work at saturated engineers. Whichever direction you adjust, make it an explicit, logged policy change, not an emergency edit to a constant.

Key Takeaway: Ground truth comes from your own closed incidents, validated by two or three engineers until kappa clears 0.80, then frozen and versioned. Accuracy per confidence bucket, ECE, and a coverage curve turn "does it work?" into a table a change advisory board can actually approve.

Cookbook Techniques

Self-Consistency for Nouls and Choices

Self-consistency is the property that the same input produces the same answer across repeated runs — it matters in a NOC because an operations process nobody can reproduce is a process nobody will trust. The Noul cookbook ran a 14-question rubric 15 times and found TypeSafe's mean per-question probability standard deviation was 0.0102, substantially lower than every LLM condition tested, which moved from run to run even at temperature 0. The Choice cookbook measured 0.0098 against 0.0245 to 0.0543 for reasoning models. The honest caveat: even with tight clustering, one question's answers spanned 0.43 to 0.53, crossing the 0.5 boundary — the exact argument for an uncertainty band instead of a bare cut point, and why NOUL_LOW and NOUL_HIGH exist in questions.py.

Re-ranking and Line-by-Line Search over Runbooks

Re-ranking is a two-step retrieval pattern: fast, cheap search produces a shortlist; a precise scorer reorders it. TypeSafe does not generate a ranked list — it answers a binary question about each query-candidate pair and returns a Noul probability, and the calling code sorts by that probability:

candidates = bm25_search(alert_text, corpus=runbook_sections, top_k=30)

scored = []
for candidate in candidates:
    response = client.system_one(
        state={"alert": alert_text, "runbook_section": candidate.text},
        questions={
            "applies": Noul(
                instructions=(
                    "This runbook section describes the remediation "
                    "procedure for the condition in the alert"
                ),
            ),
        },
    )
    scored.append((response.answers["applies"].noul, candidate))

scored.sort(reverse=True, key=lambda pair: pair[0])

The cookbook's legal-document version ran 1,200 concurrent calls for a total cost of $0.0645, lifting top-1 accuracy from 5 to 18 percent and top-10 from 38 to 62 percent. Thirty Noul calls per alert at NOC volumes is a rounding error against the cost of an engineer opening the wrong runbook. Once you have the right runbook, semantic search finds the right line: tag each line with an ID prefix, then send one request carrying a Choice that ranks line IDs by relevance and a Noul that asks whether the document contains an answer at all. The existence score is the valuable one for a NOC — it is how the pipeline says "no runbook covers this" instead of confidently handing an engineer the closest irrelevant paragraph.

LLM Guardrails and the SDE Cascade

Rather than chaining several LLM calls, guardrails screen each message with one TypeSafe request where a battery of Noul questions returns the probability that each hazard holds, screening both the input and the generated reply. The architectural idea worth stealing is separating assessment from enforcement — a plain dictionary maps hazards to actions, so the same probabilities can produce different outcomes under a "strict" versus "permissive" policy:

HAZARD_ACTION = {
    "jailbreak": "block",
    "harmful_request": "block",
    "medical_advice": "review",
    "self_harm": "support",
}

def route(nouls: dict[str, float], severity: float, policy: dict) -> str:
    """Convert TypeSafe assessment into policy-specific action"""

Translated to network operations, the hazard battery covers instruction injection in ticket text, requests for credentials or configuration secrets, out-of-scope change requests that should go through CAB, and customer-facing language that must not leave the building unreviewed. The SDE cascade combines a cheap model with an expensive one and uses TypeSafe as referee: stage 1 extracts fields with a budget model, stage 2 runs a TypeSafe verification battery computing P(wrong) per field across seven signals (hallucination, off-target sourcing, unreasonableness, type mismatch, format violation, incompleteness, semantic drift), and stage 3 escalates to a high-effort reasoning model only when any per-field P(wrong) exceeds 0.7.

Figure 12.4: The SDE cascade escalation gate

Two details are load-bearing: a JSON-Schema check cannot see a semantic fabrication — a budget model can invent a plausible but unsupported field that still passes structural validation cleanly — and the seven signals must be aggregated with max, not averaged, because averaging dilutes a single confident red flag into silence.

Key Takeaway: The cookbooks are measured recipes, not ideas — about 0.01 probability standard deviation for consistency, 5 to 18 percent top-1 improvement for re-ranking, one request instead of a chain for guardrails, and a 0.7 escalation gate for the cascade. Each maps onto a NOC problem you already have, and each keeps the decision logic in your code where you can review it.

Where to Go from Here

Expanding to Config Review, Capacity Planning, and TAC Case Triage

Sequence workloads by risk. Enrichment alone accounts for 60 to 70 percent of analyst time per alert, so automating enrichment and classification yields the highest return, while "shallow automation" — static rules, allowlists, bare severity thresholds — reduces volume without improving quality and suffers from rule drift. Automate low-risk incident classes first and widen the aperture as the evaluation numbers hold. Production AIOps deployments report roughly 99 percent noise reduction, a 45 percent MTTR reduction, and about 30 percent auto-resolution with no headcount increase — the numbers to put in a business case, and the numbers your own evaluation harness should eventually reproduce or contradict.

Figure 12.5: Expansion roadmap beyond alert triage

Keeping Questions and Thresholds under Version Control

Every question's wording, every criterion string, and every threshold constant is production configuration that changes system behavior. It belongs in version control with the same discipline as a router config: keep questions.py as the single source of truth so a wording change is a 30-second code review; tag every change with QUESTION_SET_VERSION and write that version onto each incident; store and version the evaluation CSV alongside the code; make evaluate.py a CI gate that fails the pull request if accuracy or ECE regresses past tolerance; and treat threshold edits, such as CONFIDENCE_AUTO moving from 0.85 to 0.80, as a change record deserving the same ticket a QoS policy change would get.

Following Model Releases and the Jaggedness Notes

jev-latest is an alias pointing to a specific version — currently jev-1.13.0 — and jev-preview exists for future builds. For a service that writes to ServiceNow, pin the explicit version in questions.py rather than riding the alias, and treat a model upgrade as a change: run the frozen evaluation set against the new version, compare accuracy and ECE to the recorded baseline, and only then move the pin. Read the jaggedness notes for each release before upgrading. The documented weaknesses for Jev 1.13 are directly relevant to network data: it is not a calculator and struggles with counting and numeric representations, it reads dates as text rather than as ordered quantities, and it does not treat state as hostile by default. The mitigations are already built into the capstone: count in Python, compare timestamps in Python, and keep the ADVERSARIAL Noul in the battery.

A concrete rollout checklist: export 200 closed incidents into the labeled CSV format; have two engineers label 50 independently and fix the rubric if kappa lands below 0.80; split 45/30/25 into development, validation, and locked test and tag the set; stand up questions.py, triage.py, and evaluate.py wired into CI; run evaluate.py on the development split and record accuracy, per-class precision and recall, ECE, and the coverage curve; pick thresholds from the coverage curve against a written accuracy target and open the test split once; run in shadow mode for two weeks, comparing verdicts to what humans actually did; enable writes for one low-risk alert class at the automation threshold; add the guardrail battery and runbook re-ranker and widen on measured evidence; and schedule a quarterly re-evaluation and model-release review against the frozen set.

Key Takeaway: Expand by workload, not by ambition — config review, capacity signals, and TAC triage all reuse the same seven stages with new entries in one file. Pin the model version, read the jaggedness notes before every upgrade, and keep questions, thresholds, and the evaluation set in version control so that any decision the pipeline made last quarter can still be explained this quarter.

Key Terms