Study Guide: Chapter 11: Integrating with the Tools You Already Run

Pre-Quiz: Splunk, ServiceNow & Salesforce

A Splunk saved search fires and needs to notify an external triage service. Which statement correctly distinguishes a webhook alert action from the HTTP Event Collector (HEC)?

A webhook alert action and HEC both send data out of Splunk, but HEC uses a different payload format

A webhook alert action sends data out from Splunk when a search triggers; HEC is the inbound listener that receives data into Splunk

HEC is only available for cloud-hosted Splunk instances, while webhook alert actions work everywhere

A webhook alert action requires the Splunk Add-on for ServiceNow, while HEC works with any external service

A team enables the [webhook] allow list in alert_actions.conf on Splunk 9.0+ but forgets to add any allowlist.* entries. What happens?

Splunk blocks all outbound webhooks until at least one entry is added

Splunk permits webhooks to any destination URL, because an enabled-but-empty allow list authorizes everything

Splunk falls back to the previous version's behavior and ignores the allow list entirely

Splunk sends an admin alert but still delivers the webhook to its configured URL

A Splunk search that flags forty flapping interfaces triggers a webhook alert. Why might the downstream triage service only ever see one interface per alert, and what is the recommended fix?

Splunk truncates the payload at 1KB, so add pagination to the receiver

The webhook payload's result field carries only the first result row of the search; aggregate with SPL like stats count by host before alerting so the single row is already a summary

The receiver's Pydantic model only accepts one interface per request; loosen the schema to accept a list

Splunk deduplicates identical alerts within a rolling window, so only the newest interface survives

A triage service enriches ServiceNow incidents after a human has already filled in some ticket fields. Why does the chapter recommend PATCH over PUT for this enrichment step?

PATCH is faster than PUT because it uses a smaller HTTP header

PUT requires OAuth while PATCH accepts basic auth

PATCH updates only the fields you send, while PUT replaces the entire record — a PUT from the triage service would blank out fields a human already filled in

PATCH automatically resolves field-name conflicts between the human and the service

A support manager wants the triage pipeline to compute an SLA credit dollar amount directly from Jev's severity Score. What does the chapter say about this approach?

This is the intended use of Score — it was designed to output currency amounts

Don't ask Jev to compute arithmetic like SLA credits; instead ask which rubric band the evidence falls into, then compute the amount in testable Python code

SLA credits should use a Choice question instead of a Score question so the answer is categorical

This is fine as long as the confidence on the Score answer is above 0.85

Splunk Alert Actions and Webhooks

Key Points

Splunk webhook alert actions send data OUT when a saved search fires; HEC receives data IN — opposite directions on the same platform.
Since Splunk 9.0, webhook URLs must match a regex in the [webhook] allow list in alert_actions.conf — an enabled-but-empty allow list permits everything, not nothing.
The webhook payload has six fields (result, sid, results_link, search_name, owner, app), and result carries only the FIRST result row — aggregate with SPL before alerting.
The FastAPI receiver validates the payload with a Pydantic BaseModel, builds a labeled JSON state (not a flattened string), and asks all triage questions in one system_one call (Speculative Fan-Out).
Every typed answer carries confidence and probabilities alongside its headline value — confidence is the second axis routing logic branches on.

Triggering a webhook from a Splunk saved search

An alert action is what Splunk does when a saved search matches its trigger condition. A webhook is the simplest one: Splunk makes an outbound HTTP POST to a URL you specify, configured from Settings > Searches, reports, and alerts. Direction matters here because Splunk has two HTTP features that point opposite ways: webhook alert actions send data out from Splunk with Splunk as the initiator, while the HTTP Event Collector (HEC) is the inbound listener that receives data into Splunk. For a NOC triage service, we want the outbound direction — Splunk detects, our service decides.

Since Splunk Enterprise 9.0, a webhook URL will not fire unless it matches an entry in an allow list — the single most common reason a newly built integration silently does nothing:

[webhook]
enable_allowlist = true
allowlist.webhook1 = ^https:\/\/10\.201\..*\/
allowlist.webhook2 = ^https:\/\/(.*\.|)company.com\/?.*\/
allowlist.webhook3 = ^https:\/\/typesafe\.api\.endpoint\/webhooks

There is a trap in the failure mode: if you enable the allow list but specify no URLs, Splunk authorizes webhooks to send to any endpoint. It is a permit-list ACL, and an empty one here permits rather than denies — always define restrictive expressions anchored on https://.

The payload Splunk delivers is small and fixed — six fields, of which result holds only the first result row of the triggering search:

Field	What it carries	Why the decision service cares
`result`	Fields from the first result row only	This is the evidence you pass as `state`
`sid`	Search ID, formatted `scheduler_{owner}_{search_name}_{timestamp}_{sequence}`	Natural idempotency key and audit correlator
`results_link`	URL to the full results in Splunk's UI	Paste into the ticket so a human can see everything
`search_name`	Name of the saved search that fired	Selects which question set to ask
`owner`	Username of the search owner	Fallback routing and ownership
`app`	Splunk app context	Separates lab from production pipelines

The workaround for the one-row limitation is SPL that aggregates before it alerts — stats count by host with the summary fields you want in result — so the row you receive is already a summary rather than an arbitrary sample.

Key Takeaway: A Splunk webhook alert action is an outbound HTTP POST carrying six fields, of which result holds only the first row of the triggering search. Before anything works on Splunk 9.0+, the destination URL must match a regex in the [webhook] allow list, and an enabled-but-empty allow list permits everything rather than nothing.

A FastAPI receiver that builds state and calls system_one

If you have not used FastAPI before

The receiver below is a small web service, and three lines of it are framework mechanics rather than TypeSafe concepts. app = FastAPI() creates the service. @app.post("/webhooks/splunk") is a decorator that says "run this function when a POST arrives at that path." class SplunkWebhook(BaseModel) is a Pydantic model — a declaration of the JSON shape you expect, which FastAPI validates automatically and rejects with a 422 if the body does not match. Everything else is ordinary Python. If you would rather not run a web framework, the same logic works unchanged inside a Flask route, an AWS Lambda handler, or a plain http.server — only the four framework lines change, and build_state plus the system_one call are the parts that matter.

The receiver's job is narrow: validate the payload, turn it into state, ask a fixed set of typed questions, and hand the typed answers to whatever writes the ticket. Keeping the question set in one module-level constant matters more than it looks — treat that file like a routing policy: reviewed, versioned, diffed in a pull request.

QUESTION_SET_VERSION = "noc-triage-v3"

TRIAGE_QUESTIONS = {
    "owning_team": Choice(
        instructions="Which NOC team owns the first response to this alert",
        criteria={
            "routing": "BGP, OSPF, EIGRP, route flaps, next-hop or peering problems",
            "switching": "VLANs, spanning tree, port-channels, interface errors",
            "wireless": "Aruba AOS-CX or controller-attached APs, RF, client association",
            "transport": "Optical, WAN circuits, carrier-facing links, LOS or SD alarms",
            "security": "ACL denies, authentication failures, unexpected config change",
        },
    ),
    "service_affecting": Noul(instructions="...currently degrading user traffic"),
    "redundancy_lost": Noul(instructions="...redundant path or peer is down"),
    "severity": Score(
        instructions="How severe is this alert for the production network",
        criteria=[
            "Informational; counters or state noted, no user impact",
            "Degraded; a single link or client group is affected",
            "Outage; a core path, peering session, or site is down",
        ],
    ),
}

class SplunkWebhook(BaseModel):
    result: dict
    sid: str
    results_link: str | None = None
    search_name: str | None = None
    owner: str | None = None
    app: str | None = None

@app.post("/webhooks/splunk")
def receive(payload: SplunkWebhook, authorization: str = Header(default="")):
    if authorization != f"Bearer {SHARED_SECRET}":
        raise HTTPException(status_code=401, detail="unauthorized")
    response = client.system_one(state=build_state(payload), questions=TRIAGE_QUESTIONS)
    return {"sid": payload.sid, "model": response.model}

Two design points matter. First, state is passed as a JSON object rather than a flattened string, so Jev sees interface and error_count as distinct facts. Second, all four questions ride in a single call — the Speculative Fan-Out pattern: send many questions, including speculative ones, in one call and let your code decide what's relevant. You pay for one round trip whether you ask one question or four.

Key Takeaway: The receiver parses the six-field webhook, promotes the result fields into a labelled JSON state, and asks its entire question battery in one system_one call. Keeping the questions and thresholds in one reviewable constant turns your triage policy into something a change board can read.

Enriching the event with team, severity, and confidence

A SystemOneResponse gives you answers keyed by question name, plus the model used and token usage. A ChoiceAnswer includes the selected label, a confidence score, and probabilities per label; a ScoreAnswer includes the expected score, confidence, a rubric legend, and probabilities per integer score.

Confidence is "a statistic computed from the probability distribution the answer already gives you," ranging 0 to 1 — a concentrated distribution means high confidence, a spread-out one signals uncertainty. The documentation's own published bands are practical starting points: above 0.9, act automatically even on high-stakes decisions; between 0.5 and 0.9, proceed cautiously and seek confirmation; below 0.5, route to humans. That general banding is Confidence-Gated Routing — one of TypeSafe's architectural patterns.

Important nuance carried through this chapter: those documented bands (above 0.9 / 0.5–0.9 / below 0.5) are the starting point, not the final word. Later in this chapter, the worked example replaces them with narrower cut points — 0.85 and 0.60 — measured from Chapter 8's own confidence-versus-accuracy data for the specific assign/route decision this pipeline makes. Chapter 12 ships that same narrower pair as named constants. Keep the wide documented bands in mind as background, but the operative numbers for this pipeline are 0.85 and 0.60.

Key Takeaway: Enrichment means lifting choice, score, and noul values out of the response along with their confidence and probabilities, and stamping the model and question-set version alongside them. Confidence is not decoration — it is the second axis your routing logic branches on.

ServiceNow and Salesforce

Key Points

POST /api/now/table/incident creates a record and returns its sys_id; PATCH updates only the fields sent; PUT replaces the entire record — use PATCH for enrichment.
sysparm_input_display_value lets you send label values instead of database values; X-no-response-body suppresses the echoed record for bulk operations.
Flow Designer's REST activity step lets ServiceNow pull from an external decision service — the inverse direction of the Splunk webhook push, useful for human-raised tickets.
Push (Splunk webhook) handles machine-generated alert volume; Pull (Flow Designer) handles human-raised tickets — both should call the same versioned question set.
Salesforce case triage reuses the same typed answers: Score drives priority, Choice drives owner queue, confidence decides auto-assign vs. escalation; arithmetic belongs in code, not the model.

Creating incidents via the Table API with typed fields

The ServiceNow Table API is a REST interface over any table; for incidents, POST to /api/now/table/incident with field name-value pairs, and the response returns the new record including its assigned sys_id. Updates use two verbs with two different meanings:

Operation	Method and path	Behavior
Create incident	`POST /api/now/table/incident`	Inserts a record, returns it with a new `sys_id`
Partial update	`PATCH /api/now/table/incident/{sys_id}`	Updates only the fields present in the body
Full replacement	`PUT /api/now/table/incident/{sys_id}`	Replaces the whole record with the values provided

For an enrichment pipeline you almost always want PATCH; a PUT from a triage service will happily blank out fields a human filled in while you were deciding. The mental model is merge versus overwrite in a configuration push. Two response controls are worth knowing: X-no-response-body: true suppresses the returned record for bulk operations, and sysparm_input_display_value=true lets you send display-value labels and have the API convert them.

Key Takeaway: POST to /api/now/table/incident creates a record and returns its sys_id; PATCH updates only the fields you send while PUT replaces the record wholesale. Typed answers map cleanly onto assignment_group, urgency, impact, and work_notes.

Flow Designer and REST steps that call a decision service

Flow Designer is ServiceNow's low-code automation builder, and it can call outward as well as react inward. A REST activity step lets a flow invoke an external REST service: choose the HTTP method, set the target URL, define authentication, map request headers and body, and map the JSON response into flow variables with conditional logic on HTTP status codes.

Figure 11.1: ServiceNow Flow Designer calling the decision service

flowchart LR A["Incident Created in ServiceNow"] --> B["Flow Extracts Incident Details"] B --> C["REST Step Posts to Decision API"] C --> D["Flow Parses Choice, Score, and Noul Results"] D --> E["Incident Updated with Recommendation"] E --> F{"Confidence Score"} F --> G["Notify Assignment Group"]

Direction	Trigger	Who calls whom	Best for
Push (webhook receiver)	Splunk saved search fires	Splunk → your service → Table API	Machine-generated alerts arriving at volume
Pull (Flow Designer)	Incident created or updated	ServiceNow flow → your decision API	Human-raised tickets and email-to-incident

Key Takeaway: Flow Designer's REST activity turns ServiceNow into a client of your decision service, covering human-raised tickets the way the Splunk webhook covers machine-raised alerts. Both directions should call the same versioned question set.

Salesforce case triage from Score and Choice answers

Customer-facing outages produce Salesforce cases as well as ServiceNow incidents, and the same typed answers drive both. TypeSafe documents its REST API as the integration surface for "any HTTP-capable system" rather than shipping a Salesforce-specific connector, so treat Salesforce as one more REST consumer with field names in your org's own mapping layer.

Typed answer	Case attribute it drives	Rule
`severity` (Score)	Priority tier	Expected score near the top rubric band → highest priority
`owning_team` (Choice)	Owner queue	Selected label maps to a queue name in your org
`customer_visible` (Noul)	Public-facing flag	High yes-probability marks the case for customer communications
`severity_confidence`	Escalation path	Below the floor threshold, route to a human triage queue rather than auto-assigning

Computing priority from a Score beats a hand-written if ladder over syslog severities because the rubric is explicit and editable. One caution carried forward from earlier chapters: Jev is built for structured decisions, not arithmetic. Do not ask it to compute an SLA credit or total minutes of downtime; ask which rubric band the evidence falls into, then do the arithmetic in Python where it is testable.

Key Takeaway: Salesforce case triage reuses the same Score and Choice answers that drive ServiceNow, with the Score setting priority, the Choice setting the owner queue, and confidence deciding whether the assignment is automatic. Keep arithmetic in code rather than in the model.

Post-Quiz: Splunk, ServiceNow & Salesforce