Chapter 6: The Score Primitive: Severity, Risk, and Spectrums

When to Use Score

Score is the TypeSafe question type for evaluating content against ordered, descriptive levels. You hand Jev an ordered array of level descriptions running from low to high, and the response comes back with a numerical score, a probability distribution across those levels, and a confidence measurement.

The word that matters is ordered. A Score fits only when the levels form a meaningful sequence — level 2 is genuinely "more" of something than level 1. In a NOC, that covers most of the judgments made every shift: severity of a fault, urgency of an alert, customer frustration, blast radius of a change, confidence in a root cause, capacity headroom, staleness of a CMDB record.

The test to apply: if you can shuffle the options without losing information, it is a Choice. If shuffling them destroys meaning, it is a Score. "Routing / wireless / security" survives shuffling. "Single interface / single site / regional / core" does not.

Why a Noul of 0.5 Is Not Medium

Engineers new to System One often try to build a spectrum out of a Noul, reasoning that since a Noul returns a number between 0 and 1, 0.2 must mean "a little urgent." The documentation rejects this directly: a value of 0.5 does not mean medium, because the probability encodes both the direction of the judgment and the degree of certainty in it — there is no separate confidence metric.

A Noul of 0.5 means Jev thinks the proposition is about as likely true as false — a statement about uncertainty, not magnitude. A Score, by contrast, separates the two concerns: the score field tells you where on the spectrum content sits, and the confidence field tells you how sure Jev is about that placement. A blast radius score of 2.0 with confidence 0.91 and one with confidence 0.31 are both "regional," but only the second should wake somebody up for a second opinion.

An Analogy You Already Use: Syslog Severity 0 to 7

Every network engineer already thinks in ordered levels because of syslog: pick a severity number from 0 to 7 — emergency, alert, critical, error, warning, notice, informational, debug — and the device sends that level and everything more severe. A Score is the same idea with two differences. First, direction is reversed: syslog counts down toward severity (0 is the emergency), while Score counts up (0 is the lowest level in your criteria array). Second, a syslog severity is fixed by the device vendor at compile time, whereas a Score is assigned by Jev at request time from level descriptions you write yourself. Do any inversion in application code and comment it — a reversed scale is one of the easiest bugs to ship and one of the hardest to notice.

Key Takeaway: Score is for judgments that live on an ordered spectrum — severity, urgency, frustration, blast radius — where each level is meaningfully more than the one below it. Do not fake a spectrum with a Noul, because a Noul of 0.5 means "I cannot tell," not "medium." Use the shuffle test: options that survive reordering belong in a Choice, options that do not belong in a Score.

Designing Levels

Plain-String Levels for Simple Scales

The simplest form of a Score question is an instruction plus an ordered array of strings, containing at least two descriptions and at most ten, indexed from zero.

frustration = Score(
    instructions="How frustrated the customer appears",
    criteria=[
        "Calm, just stating facts",
        "Frustrated but civil",
        "Very angry, strong language",
    ],
)

Two rules govern how you write those strings. Describe situations concretely rather than degrees, and keep the question one-dimensional. The first rule means writing "broken with a documented workaround" instead of "moderately severe":

# Weak: degrees of a word, not situations.
severity_vague = Score(
    instructions="How severe is this fault",
    criteria=["Minor", "Moderate", "Serious", "Very serious"],
)

# Strong: each level is a verifiable situation.
severity_concrete = Score(
    instructions="How severe is the service impact",
    criteria=[
        "No user impact; cosmetic log noise only",
        "Degraded performance on a redundant path",
        "One production service or path is down",
        "Multiple sites or a core path down",
    ],
)

The vague version forces Jev to guess what "moderate" means to your organization. The concrete version gives verifiable facts to match against. The one-dimensional rule is just as important: a level reading "affects two sites and has no rollback plan" merges two spectrums into one, and Jev has no clean answer for a change that affects two sites but does have a rollback plan. When a judgment genuinely has multiple dimensions, ask multiple Score questions instead.

Structured Levels: A Summary and Its Signals

When the model consistently scores between two levels on inputs you consider clear-cut, enhance each level with an object containing a what field and an examples array of representative scenarios. The what field is the summary — one sentence defining the level. The examples array holds the signals — concrete evidence that this level applies rather than its neighbor. (Some documentation pages instead name these fields summary/signals; both are accepted because level objects are free-form JSON — what matters is using the same field names on every level.)

frustration = Score(
    instructions="How frustrated the customer contact appears",
    criteria=[
        {
            "what": "Reporting facts with no complaint language",
            "examples": ["Circuit went down at 09:12, opening a case"],
        },
        {
            "what": "Threatening escalation, contract review, or leaving",
            "examples": ["We are pricing out a second provider"],
        },
    ],
)

One documented behavior shapes how you write these: the model evaluates each level independently and does not see neighboring levels. You cannot write level 2 as "more frustrated than level 1," because Jev has no idea what level 1 said when assessing level 2. Every level must stand on its own as a complete description. This is also why overlapping levels are so damaging — if levels 1 and 2 both plausibly describe the same case, Jev will genuinely split its probability between them, producing a between-level score not because the input is ambiguous but because the rubric is.

How Many Levels

The floor is two and the ceiling is ten, with the caveat that you should use up to ten levels only if each is distinctly describable.

The practical rule: the number of levels should match the number of distinct actions your system can take. If a ServiceNow workflow routes a ticket to exactly four queues, a seven-level urgency score buys nothing — three levels collapse into the same outcome while adding boundary ambiguity. If a level cannot change what happens next, delete it. Four levels is the common landing spot for network operations because the industry already converged there: teams classify changes as Low, Medium, High, and Critical, and ServiceNow's change management module computes risk from conditions and an assessment questionnaire.

Key Takeaway: Write levels as concrete, verifiable situations rather than degrees of an adjective, and keep each question one-dimensional. When plain strings stop separating cleanly, upgrade each level to a what summary plus an examples array of signals — remembering that Jev evaluates each level independently and never sees its neighbors. Pick a level count that matches the number of distinct actions your pipeline can actually take; four is the usual fit for network operations.

Reading Score Answers

The Score Value and Between-Level Results

The score field ranges from 0 to the highest level number and is calculated as the probability-weighted mean across the levels. Because it is a weighted mean, the score is almost never a whole number — a score of 1.3 means the answer falls mostly on level 1, with some weight on level 2. This is a feature, not noise.

Figure 6.1: The ordered-levels spectrum with a between-level score

{
  "score": 1.6,
  "legend": {
    "0": "Single interface or port on one device",
    "1": "One site or wiring closet",
    "2": "Regional; multiple sites or aggregation layer",
    "3": "Core or internet edge; organization-wide impact"
  },
  "probabilities": {"0": 0.02, "1": 0.41, "2": 0.52, "3": 0.05},
  "confidence": 0.54
}

Work the arithmetic once and it sticks: (0 × 0.02) + (1 × 0.41) + (2 × 0.52) + (3 × 0.05) = 1.60. The score is not a rounding of level 2 or level 1 — it is the center of mass of Jev's belief. Operationally, a 1.6 reads as "bigger than a single site, not clearly regional" — route to the CAB rather than auto-approve, because it sits on a boundary a human should check.

The Legend Maps Numbers Back to Your Levels

The legend maps level numbers back to their descriptions. In the TypeScript SDK this is formalized as ScoreLegend<T>, "rubric descriptions keyed by score," generated from the criteria you supplied. The legend exists because raw scores are unreadable to humans — nobody wants "blast radius: 1.6" in a change review; they want "between one site or wiring closet and regional." Use the legend for three things: human-readable output (look up the level text before writing a work note), auditability (log the legend alongside the score so definitions are traceable months later), and bracketing a between-level score (show both neighboring levels rather than rounding and pretending the answer was clean).

Probabilities and the Confidence Summary

probabilities gives the distribution across each level and sums to 1.0; confidence ranges from 0 to 1, with higher values indicating probability is concentrated on one level.

Figure 6.2: The structure of a Score response

The reason you must look at the distribution and not the score alone: different probability distributions can produce identical scores. A second distribution with the exact same 1.6 score: probabilities 0.30 / 0.00 / 0.50 / 0.20 across levels 0–3, confidence 0.31. The arithmetic still lands on 1.60, but this distribution is bimodal — 30% on "single interface" and 70% on "regional or worse," with zero weight on the level in between. That is not a boundary case; it is a sign the state is missing a critical fact. A practical pipeline rule: if confidence falls below your threshold, do not act on the score — enrich the state and ask again, or route to a human.

Key Takeaway: The score is a probability-weighted mean, so fractional values are the normal case and mean the answer sits between levels. Always read probabilities and confidence alongside it, because different distributions produce identical scores and a bimodal 1.6 means something entirely different from a unimodal 1.6. Use the legend to render both bracketing levels to humans and to keep an audit trail of the rubric that produced the number.

Worked Examples

Rating the Blast Radius of a Juniper MX Change Request

Blast radius describes the scope of systems, services, users, or business processes affected when a change goes wrong, and belongs at the start of change management, before the window opens. Over 80% of unplanned IT outages originate from planned changes, most often from invisible downstream impacts rather than careless engineering — which is the business case for a fast, consistent, auditable scope rating in front of the CAB.

questions = {
    "blast_radius": Score(
        instructions="How large is the blast radius if this change goes wrong",
        criteria=[
            "A single interface or port; no transit or user traffic",
            "One site or wiring closet",
            "Regional; multiple sites or an aggregation layer",
            "Core or internet edge; org-wide or customer-facing traffic",
        ],
    ),
    "rollback_difficulty": Score(
        instructions="How hard would it be to back this change out",
        criteria=[
            "One documented, tested command restores prior state",
            "Documented but untested on this platform",
            "Requires manual reconstruction from backups",
            "No practical rollback; needs hardware/firmware recovery",
        ],
    ),
    "timing_risk": Score(
        instructions="How much additional risk does the timing carry",
        criteria=[
            "Approved window, low traffic, full staffing",
            "Off-peak but outside a formal window",
            "Business hours on a redundant system",
            "Peak hours, blackout period, or a live event",
        ],
    ),
}

Three separate questions, not one — the one-dimensional rule applied to a judgment network teams have always treated as multi-dimensional. Trying to compress blast radius, rollback feasibility, and timing into one Score would force Jev to weight those dimensions invisibly and inconsistently. The documented fix is composite scoring: ask multiple Score questions together, normalize each by dividing by len(criteria) - 1, then combine with weights in application code.

level_counts = {"blast_radius": 4, "rollback_difficulty": 4, "timing_risk": 4}
weights = {"blast_radius": 0.55, "rollback_difficulty": 0.30, "timing_risk": 0.15}

def normalized(name):
    return response.answers[name].score / (level_counts[name] - 1)

composite = sum(weights[name] * normalized(name) for name in weights)

if composite < 0.25: risk = "Low"
elif composite < 0.50: risk = "Medium"
elif composite < 0.75: risk = "High"
else: risk = "Critical"

The arithmetic happens in Python, not the prompt — Jev places content on each rubric; combining and weighting is deterministic math a CAB member can read, and changing a weight becomes a code review, not a prompt rewrite. Those four output labels are deliberate: they match the Low/Medium/High/Critical vocabulary network teams and ServiceNow's change risk calculation already use. For this MX change, a blast radius of 1.6 at confidence 0.54 is the honest answer — it touches one transit interface on an active/active edge router, but the CMDB lists a downstream payment gateway VRF, and the CMDB dependency list is exactly what's most likely to be wrong.

Figure 6.3: Composite scoring across three one-dimensional Score questions

Rating Customer Frustration in a Salesforce Case About a WAN Outage

When a WAN circuit fails, ServiceNow gets the incident and Salesforce gets the case. The Salesforce Case object's Priority picklist (High/Medium/Low) drives queue assignment, SLA milestones, and escalation rules. IsEscalated is a checkbox set true when an escalation rule fires. The critical point: sentiment and priority are independent dimensions. A case can be low priority yet carry highly negative sentiment, or high priority with a calm technical contact. A frustration Score supplies that missing dimension with more resolution than a three-way classification, because it is ordered and returns a distribution.

{
  "score": 2.7,
  "legend": {
    "2": "Explicitly frustrated; cites repeat contacts or missed commitments",
    "3": "Threatening escalation, contract review, or leaving"
  },
  "probabilities": {"0": 0.00, "1": 0.04, "2": 0.22, "3": 0.74},
  "confidence": 0.74
}

Priority is already High and IsEscalated is still false, so no standard rule has fired — escalation rules typically key on age, priority, sentiment, repeat issues, or revenue, not "the customer mentioned the renewal." A frustration score of 2.7 is that missing trigger:

score = response.answers["frustration"].score
if score >= 2.5:
    action = "notify account team + assign senior engineer + draft RCA commitment"
elif score >= 1.5:
    action = "personal update from a named engineer within the hour"
else:
    action = "standard queue handling"

Rating the Urgency of a Splunk Alert About Interface Errors

CRC errors climbing on one member of a two-member port channel point at the physical layer — a marginal optic or damaged fiber. The link is up, traffic is flowing, and nobody is paged. This is the alert where the right answer is "not now, but definitely before it fails."

urgency = Score(
    instructions="How urgently must a network engineer act on this alert",
    criteria=[
        "Informational; no user-visible effect",
        "Degradation confined to a redundant path",
        "Errors rising on a production path; possible retransmits",
        "A production path is failing now; traffic is being lost",
        "Multiple links or a core node down; a site is offline",
    ],
)

A plausible answer: score 1.4, confidence 0.61, probabilities 0.03 / 0.61 / 0.30 / 0.06 / 0.00 across levels 0–4 — mostly "degradation on a redundant path," leaning toward "errors rising." The port channel is protecting users now, so this is not a page, but the 0.30 weight on level 2 shows this is degrading, not stable, which rules out a shared upstream cause and points at this specific optic.

if answer.score >= 3.0:
    route = "page on-call now"
elif answer.score >= 2.0:
    route = "assign to the active shift queue"
elif answer.score >= 1.0:
    route = "create a P3 incident; schedule optic replacement"
else:
    route = "suppress; log to the daily digest"

Notice 1.4 and 1.9 land in the same bucket while 2.0 does not — the level-count rule doing its job. This pipeline has four distinct actions, so the five-level rubric is already one level richer than routing needs; if level 4 never changes the outcome in your environment, drop to four levels and the remaining boundaries sharpen. Because the same score can come from different distributions, log score, confidence, and the full probabilities object on every automated decision — "urgency 1.4, confidence 0.61, 30% weight on errors rising" is an answer a post-incident review can use; "urgency: low" is not.

Figure 6.4: End-to-end flow of the Splunk urgency worked example

Key Takeaway: Multi-dimensional judgments like change risk should be several one-dimensional Score questions, normalized by dividing by len(criteria) - 1 and combined with weights in your own code rather than in the prompt. Map your top-level output onto the vocabulary the receiving system already uses, and always log the score, confidence, and probability distribution behind every automated routing decision.

Key Terms