Pipeline: listen, triage, enrich, resolve

Four stages ran between a source poll and a graph row.

 sources: RSS, vendor APIs, search APIs
        │
   LISTEN     continuous · systemd service · dedupe · store raw
        │
   TRIAGE     batched · cheapest model · relevance classification
        │
   ENRICH     batched · mid-tier model · structured extraction
        │
   RESOLVE    deterministic-first entity resolution
        │
   CORPUS     graph write · provenance stamped per edge

Stage ordering

Each stage rejects work before a more expensive stage sees it. Triage ran 8,584 calls for $106.71 on the cheapest available model and rejected 62.1% of items at $0.0124 per call, upstream of an extractor costing roughly four times more per item. The same corpus assembled with the stages in the reverse order carries the enrichment cost on every rejected item as well as every accepted one.

Triage was tuned toward rejection. A false reject is a signal recoverable on a later pass. A false accept is enrichment spend on noise, and is not recoverable.

The same ordering rule governs the stages below triage, stated in the enrichment spike as: never spend an LLM call on a fact the deterministic path already produced. Applied to attack events, the deterministic systems-inheritance pass covered 2,303 of 4,362 events (53%) at zero model cost, leaving approximately 2,060 events in the systems gap and 4,350 in the impacts gap. The extraction handler’s candidate query selected only rows in one of those gaps.

Listen

The listen stage ran as a persistent systemd user service polling feeds and APIs on a fixed interval, deduplicating on source URL and content hash, and writing raw payloads before any interpretation.

The source registry

feed_sources held 309 rows at teardown, 26 of them still enabled. The 2026-05-31 feed-health audit recorded 159 enabled feeds at that point in the run, of which 157 had produced a record in the preceding 30 days and one — the Stimson Center RSS feed — was genuinely broken.

source_type	rows	enabled at teardown	poll interval range
RSS	210	3	3,600 s – 604,800 s
GROK_SEARCH	72	0	43,200 s
STRUCTURED_REPORT	11	11	21,600 s – 604,800 s
X_SEARCH	10	8	28,800 s
FEDERAL_REGISTER	1	1	86,400 s
PR_WIRE	1	1	3,600 s
OPENALEX	1	0	86,400 s
SEC_EDGAR	1	0	21,600 s
USA_SPENDING	1	1	86,400 s
dvids_api	1	1	21,600 s

Cadence clustered at four values: 167 rows at hourly, 72 at twelve-hourly, 36 at six-hourly, 13 daily. The remainder sat at two-hourly, eight-hourly, weekly, or longer. Cadence was a per-row integer column, not a code constant, so retuning a feed was a feed_sources update rather than a deploy.

Each listener implemented one poll method returning a list of raw records. base.py declares source_url as the dedup key. Eleven listener modules existed: rss, grok_search, xai_search, sam_gov, sec_edgar, federal_register, usaspending, openalex, dvids, youtube, and structured_report.

Source classes

The collection decision replaced a free-text category column — 25 ad-hoc labels, 127 enabled rows carrying an empty string — with a closed vocabulary of seven classes. The class is what relevance-gating, prioritisation, and the provenance source_category field all key off.

class	collects	rows	enabled at teardown
incident-threat	attacks, strikes, intrusions, sightings, downings	62	5
policy-analysis	think-tank, doctrine, regulatory, arms-control framing	41	0
cuas-vendor	counter-drone and drone OEM commercial activity	40	1
social-osint	curated X/Telegram-style OSINT accounts	29	4
adversary-supply-chain	sanctions, seizures, component tracing, export prosecutions	9	5
procurement	government solicitations and awards	8	6
research	academic and technical literature	5	0

The migration did not complete. 102 rows still carried an empty or null category at teardown and a further 13 carried pre-vocabulary labels (allied_mod, defense-security, robotics-industry, conflict, and others). All but five of those 115 rows were disabled, so the residue affected the registry rather than the running collection.

The audit that produced this vocabulary measured yield per source class against the drone product: RSS at 26% drone-relevant over 10,687 records in 30 days, SAM.gov at 3% over 1,036 records, and 26 inspection feeds at approximately 1%.

Source trust tiers

Class describes what a source collects. A separate six-value ladder, rp_source_tiers, describes how far an assertion from it can be trusted, and carries a numeric weight for downstream confidence:

tier	weight	definition	examples
primary	1.00	Direct source: company filing, government notice, primary court document, official press release from named party	SAM.gov, company press release, SEC filing
tier1	0.90	Established journalism with editorial standards, named byline, fact-checked	Reuters, AP, WSJ, FT, Nikkei, Defense News
tier2	0.75	Trade publications and analyst firms with domain credibility	C4ISRNET, Breaking Defense, The Drive, Janes, RAND
tier3	0.60	Blogs, specialist newsletters with named authorship and track record	The Warzone, Naval News, War on the Rocks
tier4	0.35	Aggregators, anonymous sources, OSINT without corroboration	Telegram channels, unattributed OSINT
unverified	0.10	Anecdotal, social media, single-source without verification	X/Twitter posts, forum threads

The weights are ordinal, not calibrated — the ratio between tier1 at 0.90 and tier4 at 0.35 was set by judgment, not measured against outcomes. The table’s function is to make the judgment explicit and queryable rather than implicit in whichever code path last touched a record.

Deduplication

Three keys operated at different points.

source_url is the intake gate. Each poll cycle loads the full set of URLs already present in listening and drops any raw record whose source_url is a member:

existing_urls = client.get_existing_source_urls()
new_records = [
    r for r in raw_records if r.get("source_url") and r["source_url"] not in existing_urls
]

canonical_id — the source’s own stable identifier (RSS <guid>, DOI, SAM notice id, OFAC action id) — is captured into provenance to make dedup recoverable when a URL drifts.

content_sha256, a hash of title plus content as ingested, detects silent edits and retractions against a record already held, and is what the re-verification loop compares on a later pass.

Signal creation applies a fourth check: check_duplicate_signal(source_url) runs before any signal row is written, so a triage pass over a re-ingested item produces no second signal.

The keyword pre-filter

engine/listen/filters.py compiles a keyword list into one case-insensitive regex and applies it to title plus content after dedup and before extraction. Single words get word boundaries; multi-word phrases match as written.

if " " in kw or "-" in kw:
    parts.append(escaped)
else:
    parts.append(rf"\b{escaped}\b")

The gate is per-feed, read from feed_sources.config.filter_keywords, with a 40-term default list applied to PR_WIRE. Gate policy differed by class: a hard drone gate on broad RSS and policy-analysis; a narrowed source query plus keyword gate on procurement; no gate on social-osint and incident-threat, where account curation already produced 81% drone relevance and an over-eager gate would suppress breaking incidents; no gate on research or adversary-supply-chain, the first because needs_extraction = False means there is no model cost to protect, the second because a component shipment may never use the word “drone”.

The gate sits before extraction because that is where the cost is. Extraction runs per record during listen, upstream of triage, so a triage-level filter would let non-drone records burn the extraction budget first.

Provenance at capture

Provenance is born at collection. Carried through, never reconstructed at publish time.

Every listener emits a uniform object on raw["metadata"], persisted into listening.metadata with no schema change:

{
  "provenance": {
    "feed_source_id":  "<uuid>",              // FK to feed_sources.id
    "feed_name":       "Defense One",
    "source_category": "policy-analysis",
    "retrieved_at":    "2026-05-30T14:22:01Z", // when we pulled it
    "published_at":    "2026-05-29",           // the source's own date
    "canonical_id":    "<rss-guid | doi | sam-notice-id | ofac-id>",
    "canonical_url":   "https://...",
    "content_sha256":  "<hex>",
    "author":          "Jane Doe | null",
    "publisher":       "Defense One | null",
    "redirect_chain":  ["http://old", "https://new"],
    "retrieval_tool":  "rss_listener@<commit>"
  }
}

Before this object existed, the producing feed was unrecoverable from a listening row: _feed_name was a logging string, and downstream citations were reconstructed from a flat source_url — inferred rather than captured. The spike verified the gap by count: 0 of 10,687 RSS rows, 0 of 1,624 GROK, 0 of 1,036 SAM.gov, and 0 of 19 SEC rows carried non-empty metadata in the preceding 30 days. Only OpenAlex (8,391) and DVIDS (93) did.

After the change, coverage is complete: of 11,740 listening rows captured after 2026-06-01, 11,740 carry a provenance key.

Storing the raw payload is what makes provenance reconstructable. Sources mutate and disappear; the captured bytes do not.

Volume

The listen worker collected 55,406 signals over the run. The listening table — raw captures before triage — holds 74,368 rows:

source_type	rows
RSS	40,027
OPENALEX	11,826
GROK_SEARCH	10,994
X_SEARCH	5,345
SAM_GOV	3,749
dvids_api	1,620
MANUAL	479
STRUCTURED_REPORT	193
FEDERAL_REGISTER	85
SEC_EDGAR	31
USA_SPENDING	19

Per-record extraction during listen — the listen_extract agent — ran 2,333 calls for $3.63 total, one error. It is the cheapest paid stage in the pipeline by two orders of magnitude.

Collection ran continuously while processing ran in batches. Decoupling them means a provider outage or a rate limit stalls the queue rather than losing data.

Triage

Triage read PENDING rows from listening, in sub-batches of ten, on a 300-second recurring cadence. One Haiku call per batch, temperature=0.1, max_tokens=4096, with the full list of known company names (capped at 500) injected for entity matching.

The prompt is three ordered gates, not a single classification request. Steps run in sequence and an item that fails an early step is never classified:

Step 1 — scope check. Does the article specifically mention or focus on robotics, drones/UAS/UAV, autonomous systems, unmanned vehicles, counter-UAS, AI-enabled weapons/defense autonomy, or industrial automation in a security, defense, or infrastructure context? Five explicit reject classes follow, each phrased as a rule rather than a preference:

- General defense policy, strategy, or geopolitical commentary
  WITHOUT specific robotics/autonomous systems content → NOT relevant
- Historical military analysis, opinion/editorial, political commentary → NOT relevant
- General cybersecurity, IT, or software news without autonomy angle → NOT relevant
- Consumer electronics, entertainment, pure business news → NOT relevant
- Broad market reports with no specific company action → NOT relevant

Step 2 — event suppression. Trade-show attendance, exhibition announcements, and market-size projections are rejected unless the item carries a product launch, a contract or partnership, or a deployment.

Step 3 — classify. Only items surviving both gates receive a signal_type from a fixed twelve-value enum, a significance of HIGH/MEDIUM/LOW with HIGH bound to a stated threshold (major contract, deployment, or funding above $10M), a one-sentence summary, and a company list matched against the injected names.

Rejected items must carry a skip_reason. The response schema is fixed:

{"items": [{"index": 1, "relevant": true, "signal_type": "CONTRACT_AWARD",
            "significance": "HIGH", "summary": "...", "companies": ["..."],
            "skip_reason": null}]}

Measured outcome across the run: 74,368 listening rows resolved to 46,210 IGNORED and 28,158 PROCESSED — a 62.1% reject rate. The 8,584 triage calls carried 63,591 items, a mean batch of 7.4 against a cap of 10, because the pending queue was usually shorter than a full batch at the 300-second cadence.

Accepted items produced 55,406 signals, distributed as:

signal_type	signals		significance	signals
PRODUCT_LAUNCH	13,530		MEDIUM	30,590
CONFLICT_USE	12,022		HIGH	17,935
DEPLOYMENT	8,366		LOW	6,881
PARTNERSHIP	4,886
POLICY_CHANGE	4,600
REGULATORY	3,440
FUNDING	2,257
CONTRACT_AWARD	1,994
LEADERSHIP_CHANGE	1,508
EARNINGS	1,360
RFP	724
ACQUISITION	683
BANKRUPTCY	10

A further 26 signal rows carry no signal_type.

Triage carried the highest error rate of any pipeline stage: 1,084 of 8,584 calls returned status='error', 12.63%, against 0.24% for attack_event_enrich, 0.11% for classify, 0.04% for listen_extract, and 0.00% for resolution_fallback. The dominant cause was provider-side, not malformed output: across the whole trace log 2,185 calls returned 401 Unauthorized from the router, 265 returned 403 Forbidden, and 246 more carried an explicit daily key-limit message. Credential and quota failures land hardest on the stages that run most often against the shortest cadence, which is why the 300-second loop carries 53× the error rate of the stage with the highest call volume in the project.

A failed batch logged the error and continued to the next batch; the ten listening rows kept triage_status='PENDING' and were picked up on a later tick.

Enrich

Enrichment read gap-gated candidates and asked only for the facts a given row was missing. Output is raw mentions, not resolved entities.

Three mechanisms handled partial extraction and schema violations, all deterministic and all applied after the model returned.

Confidence floor. Every extracted system, weapon, and impact carries a self-reported confidence, clamped to [0,1]. MIN_EXTRACTION_CONFIDENCE = 0.55; anything below is dropped and counted in mentions_skipped rather than written.

Controlled-vocabulary rejection. impact_type must be one of six values (deaths, injuries, damage, disruption, financial, unknown) and weapon category one of a fixed set. A value outside the vocabulary is discarded — the row is not written with a coerced or nearest-match value.

Shape rejection. A response that parses as JSON but is not a dict is logged, the event is stamped as attempted, and the batch continues:

attack_event_enrich: event_id=%s returned non-dict JSON (%s); stamping and skipping

Partial extraction is therefore normal rather than exceptional: an event can contribute two impacts and zero systems, and it still exits the candidate set, because the exit condition is “attempted”, not “yielded rows”.

The JSON retry path

call_llm_json is the single wrapper for structured output. It takes max_json_retries: int = 2 and re-prompts on a parse failure with a fixed appended instruction:

prompt += (
    "\n\nYour previous response was not valid JSON. "
    "Return ONLY valid JSON. No markdown, no trailing text, no comments."
)

Cost accumulates across attempts and the trace records the total, not the last attempt. The status column distinguishes three outcomes:

status	meaning	rows
ok	parsed on the first attempt	82,281
success	legacy value, pre-dating the ok default	211
error	all attempts failed, or the call itself failed	3,105
json_retry	parsed on attempt 2 or 3 after a first-attempt parse failure	16

The 16 json_retry rows are the calls the retry recovered. They spread across nine agents and four models, with no clustering by either:

agent	model	rows	first	last
conflict_extractor	anthropic/claude-sonnet-4-6	4	2026-04-17	2026-06-29
editorial_review	anthropic/claude-sonnet-4-6	3	2026-05-05	2026-05-30
content_remediation	anthropic/claude-haiku-4.5	2	2026-05-08	2026-05-28
seo_rewrite	anthropic/claude-haiku-4-5	2	2026-04-17	2026-04-17
attack_event_enrich	anthropic/claude-haiku-4.5	1	2026-07-05	2026-07-05
signal_scan	anthropic/claude-haiku-4.5	1	2026-05-08	2026-05-08
discovery	anthropic/claude-haiku-4.5	1	2026-06-06	2026-06-06
freshness_checker	anthropic/claude-haiku-4.5	1	2026-05-12	2026-05-12
chief-engineer	anthropic/claude-haiku-4.5	1	2026-07-16	2026-07-16

Retries that exhausted all three attempts are recorded as status='error' with an error string prefixed json_parse_failed. There are 372 of those, and unlike the recovered set they do cluster:

agent	exhausted retries
signal_enricher	150
content_remediation	110
qa_remediation	108
conflict_extractor	2
cluster_conflict	1
rp_models_smoke	1

368 of the 372 came from three agents, all of them long-output stages. The error strings carry the response length, and they concentrate in a narrow band — repeated failures at 6,796 to 7,150 characters, with a second cluster near 9,000. signal_enricher averaged 3,031 output tokens per successful call, the highest of any stage. The failures are truncation at the output ceiling, which a re-prompt for “valid JSON only” cannot fix: the model was not producing malformed JSON, it was producing JSON that ran out of room. Two retries at full prompt cost were spent on each one before the call was abandoned.

A recovered retry is 0.02% of calls. An exhausted one is 0.43%. The retry loop paid for itself only in the sense that it was cheap; it did not address the failure that actually occurred.

Resolve

The deterministic pass owns exact alias matches. resolution_fallback operates only on the misses, across four entity kinds — system, actor, manufacturer, weapon_system — with a separate and higher confidence floor than extraction: MIN_CONFIDENCE = 0.72. Class and type values are constrained to closed sets (fourteen system classes, three system roles, seven actor types), and the resolver returns one of three actions per mention: bind to an existing entity, propose a new one, or skip.

Each resolved edge carries resolution_method and resolution_confidence, and extraction-sourced edges are tagged resolution_method='llm_extraction' so a later audit can separate a mention produced by a model from a mention read off a source. The governing tenet:

Entity resolution is one pattern. Capture raw; deterministic-first; LLM-fallback only on misses; every edge provenance-tagged.

Part 1 carries the normalization function, the alias tables and the closed vocabularies. Resolution ran 913 calls with zero errors, the only stage in the pipeline with a clean error record.

Queue and idempotency

Re-running a stage is safe because of four mechanisms, each of which was added after the failure it prevents was observed.

Claim-on-fetch. Selection and claim are one committed statement. The inner CTE locks the head-N candidates with FOR UPDATE SKIP LOCKED; the outer UPDATE stamps the attempt timestamp and returns the claimed rows:

WITH claimed AS (
    SELECT ae.id
    FROM attack_events ae
    ...
    FOR UPDATE OF ae SKIP LOCKED
)
UPDATE attack_events SET enrich_attempted_at = now()
FROM claimed c
...
RETURNING ...

Before this, every concurrent worker fetched the same head-N candidates and re-enriched each one roughly N times before any stamp landed.

Stamp-on-attempt. The gap query excludes rows where enrich_attempted_at IS NOT NULL, and every event a batch touches is stamped — extracted, empty, and skipped alike. Without it, an event whose extraction yielded no insertable rows stayed in the candidate set permanently and was re-fetched every batch. The observed cost of that loop before it was closed: 6,846 calls hitting 130 distinct events in 24 hours, approximately $22. At teardown, 6,996 of 7,218 attack events carried the stamp.

Advisory locks. Recurring tasks that must not overlap take a transaction-scoped advisory lock on a hashed task name and return a locked status rather than queueing behind the running instance:

cur.execute("SELECT pg_try_advisory_xact_lock(hashtext(%s)) AS locked", (LOCK_KEY,))
if not (cur.fetchone() or {}).get("locked"):
    return {"summary": "... skipped: already running", "status": "locked", "cost_usd": 0.0}

Stuck-row reaping. A recurring task scans three queue tables — jobs, generation_queue, ops_event_deliveries — for rows sitting in a claimed state (running, processing) past a staleness threshold, and either resets them to pending or fails them once an attempt ceiling is reached. Defaults: 45 minutes stale, 3 attempts, 50 rows per table per pass, on a 1,800-second cadence.

Idempotency is the property that makes “re-run it” a valid response to most failures, and it is expensive to retrofit. Three of the four mechanisms above were retrofits, each carrying a backlog reference and a measured cost for the incident that prompted it.

Loop cadence

Three loops, separated by frequency and cost profile.

loop	cadence	cost profile	function
collection	continuous	near-zero	poll, dedupe, capture raw
enrichment	batched, scheduled	dominant	drain queue, extract, resolve
re-verification	slow, decay-driven	zero by design	re-check facts whose freshness decayed

The re-verification loop performs deterministic checks only — HTTP status, content hash comparison against last capture — and escalates to a model only when a change is detected. Its per-check cost is zero by construction, which is what allows it to run on a 30-minute cadence against the whole published set.

Re-verify ran on a 30-minute cadence with a per-entity cooldown stamp to prevent re-checking the same record repeatedly.

Cadence was registry data, not code. eng_recurring_tasks held 40 rows, 29 enabled at teardown, each with a handler dotted path, an interval_seconds, and a worker_group. The intervals span five orders of magnitude:

task	handler	interval
health_monitor	engine.tasks.health_monitor.run	300 s
triage	engine.agents.executor.run_triage	300 s
signal_enricher	engine.agents.executor.run_signal_enricher	900 s
signal_embedder	engine.tasks.embed_signals.run_embed_signals	900 s
job_health	engine.tasks.stuck_row_reaper.run	1,800 s
directory_reverify	engine.tasks.directory_reverify.run	1,800 s
entity_resolution_groom	engine.handlers.entity_resolution_groom.run	3,600 s
attack_event_enrich	engine.handlers.attack_event_enrich.run	3,600 s
discovery	engine.agents.executor.run_discovery	7,200 s
conflict_extractor	engine.agents.executor.run_conflict_extractor	7,200 s
directory_freshness_slo	engine.tasks.directory_freshness_slo.run	86,400 s
directory_count_rollup	engine.tasks.directory_count_rollup.run	604,800 s

Retuning a loop was an UPDATE. Disabling one was a boolean flip, which is also how the directory tasks shipped: registered disabled, enabled by the operator at go-live.

Freshness decay

Decay was implemented as a per-cut age threshold plus a fraction-of-population service level, both constants in engine/tasks/directory_ops.py, both environment-overridable.

FRESHNESS_TARGET_DAYS: Final[dict[str, int]] = {
    "drone-detection": 45,       # Fast: regulatory + grant churn
    "public-safety-robots": 30,  # Fast: grant cycles + active collection
    "security-robots": 60,       # Medium: steady vendor set
    "asset-inspection": 120,     # Slow: long asset lifecycles, stable vendors
}
MAX_STALE_FRACTION: Final[float] = float(os.environ.get("RP_DIRECTORY_MAX_STALE_FRACTION", 0.25))

A published company is stale when last_verified_at is null or older than its cut’s target. A cut breaches when more than 25% of its published members are stale. A cut with zero members never breaches.

The re-verification worker selects the stalest members oldest-first with nulls first and UUID as the tie-breaker, ten per cut per run, tightest SLO target first. The verification pass is deterministic and free: fetch the company website, require the extracted page to identify the company, require one provenance source to still identify it, then stamp last_verified_at.

One clause in the selection query carries most of the operational weight:

AND (
    c.provenance ->> 'reverify_last_attempt_at' IS NULL
    OR (c.provenance ->> 'reverify_last_attempt_at')::timestamptz
       < now() - interval '7 days'
)

A failed attempt stamps reverify_last_attempt_at, which removes the company from selection for seven days. Without it, permanently unverifiable companies — no website, dead website — sit at the head of a nulls-first queue and every run re-attempts the same failing set forever. The comment in the source records where that was learned: observed on the first live run (22/23 failures).

Measured across 242 runs between 2026-07-19 and 2026-07-24: 510 attempts, 118 verified, 392 failed, and 200 runs that attempted nothing at all. The cooldown that prevented the starvation loop also emptied the candidate pool: once the stalest cohort had failed once, it was excluded for a week, and the worker had nothing eligible to do on 83% of its ticks.

The dominant failure reason on the first live pass was missing_or_unsafe_website — the stalest members were companies without a website field at all. Freshness recovery was therefore gated on a field backfill rather than on verification capacity, which is not a conclusion the freshness metric itself could produce.

Self-healing and the monitoring chain

Recovery is three mechanisms: Restart=always on the worker units, idempotent stages so any re-run is safe, and a durable queue so a crash loses position rather than data. The requirement they serve:

The engine self-heals. A solo operator must never be the recovery mechanism.

Above them sits a monitor, a remediation worker and an escalation path. The measured behaviour of the three, over the six days before teardown, differs at each link.

The monitor fired, daily, correctly. directory_freshness_slo ran on its 86,400-second cadence and emitted a signal against code:tool_directory_freshness with outcome='error' on six consecutive days. The breach spanned three of four coverage areas throughout:

date	drone-detection (45 d)	public-safety (30 d)	security (60 d)	asset-inspection (120 d)
07-18	78.2%	82.9%	30.5%	0.0%
07-19	63.8%	56.1%	28.6%	0.0%
07-20	50.8%	52.4%	28.1%	0.4%
07-21	50.8%	52.4%	28.1%	0.4%
07-22	54.4%	52.4%	28.1%	0.4%
07-23	55.0%	53.7%	28.1%	0.4%

Against the 25% ceiling, drone-detection (307 published members), public-safety-robots (82), and security-robots (531) breached on every pass. asset-inspection (260) never did.

The remediation worker reported success while doing nothing. Over the same window, directory_reverify emitted outcome='ok' on all 242 runs, including the 200 on which it attempted zero companies. Its health signal measured whether the process completed, not whether the metric moved. On the health board this reads as a green worker beside a red metric, with no edge between them.

Nothing escalated off the board. The alarm was visible to anything that queried the live health board and to nothing else. There was no push path. code:tool_directory_freshness still reads [failing] at teardown, last signal 2026-07-23 20:44 — the breach outlived the system that measured it.

The monitoring path had also shared a failure mode with the path it monitored. The HQ-side telemetry adapters translate engine tables into signals, and nothing scheduled them, so they went stale and the board reported false alarms — wrapper_llm failing and the dependency probes silent — from a two-day-old refresh rather than a real outage. Later, the cron entry that ran the refresh stopped executing entirely: the last heartbeat before the gap is 2026-06-09 20:44, the next is 2026-06-11 09:23, a 36.65-hour hole in the record of whether anything was working.

The fix is a watcher on the watcher, and it is the last step of the refresh script rather than the first:

# 3. Forcing function: heartbeat THIS job last, only after the refresh ran.
$RUN scripts/signals.py --self-heartbeat        >> "$LOG" 2>&1

The job heartbeats itself only after its own work completes, and it is registered with an expected interval of 7,200 seconds. If it stops running, code:telemetry_refresh goes silent and the board alarms on its own staleness. Cron was subsequently replaced with a systemd user timer carrying Persistent=true and RandomizedDelaySec=120. That fix broke silently for eight hours on introduction, because systemd user services get a minimal PATH without ~/.local/bin, where uv lives.

Measured latency by stage

Cost and latency rank the stages differently, and neither ranks them the way call volume does. attack_event_enrich ran 29,805 calls, 34.8% of every call in the project, at $0.0032 each; business_events ran 8,177 for $262.15, 26.3% of total spend on 9.6% of calls, because it reads a full document and emits structured output containing several linked entities. Part 5 carries spend by stage in full.

Percentiles below are over successful calls only, for agents with at least 150 of them.

stage	calls	p50	p95	p99	max	mean prompt tok	mean output tok
attack_event_enrich	29,732	4,229 ms	8,900 ms	11,186 ms	41,764 ms	1,053	441
business_events	7,629	1,530 ms	4,780 ms	14,359 ms	39,981 ms	—	—
triage	7,500	5,770 ms	9,001 ms	16,274 ms	307,793 ms	11,747	603
attack_event_sector	7,139	1,448 ms	2,727 ms	3,697 ms	8,616 ms	744	101
signal_enricher	5,187	4,392 ms	25,755 ms	28,153 ms	39,349 ms	4,318	3,031
listen_extract	2,332	2,348 ms	3,864 ms	5,069 ms	64,979 ms	696	176
classify	1,817	2,815 ms	4,040 ms	5,659 ms	306,374 ms	6,178	188
editorial_review	1,542	19,158 ms	44,789 ms	54,991 ms	593,905 ms	—	—
conflict_extractor	1,211	35,827 ms	45,517 ms	51,832 ms	153,976 ms	4,290	3,213
x_search_listener	1,160	17,667 ms	33,886 ms	46,218 ms	64,914 ms	8,336	2,523
resolution_fallback	913	2,479 ms	9,028 ms	14,376 ms	29,662 ms	13,046	196
content_remediation	589	37,255 ms	79,305 ms	106,889 ms	516,071 ms	—	—

The p95/p50 ratio separates two shapes. Stages with a bounded output — attack_event_sector at 1.9×, attack_event_enrich at 2.1×, triage at 1.6× — are predictable enough to schedule against. Stages with an unbounded output — signal_enricher at 5.9× — are not: the median call is fast and the tail is six times longer, because output length is a function of how much the source document contained. signal_enricher is also the stage with the most exhausted JSON retries. Long tail and truncation failure are the same underlying property measured two ways.

Maxima are dominated by the retry loop rather than by any single call. A triage call recorded at 307,793 ms and a classify call at 306,374 ms are three-attempt sequences: call_llm_json starts its clock before the first attempt and records total elapsed on the final one.

Per-model latency, over successful calls:

model	calls	p50	p95
anthropic/claude-haiku-4.5	62,344	3,779 ms	11,466 ms
anthropic/claude-sonnet-4-6	14,211	2,693 ms	41,073 ms
anthropic/claude-haiku-4-5	2,299	4,514 ms	6,168 ms
text-embedding-3-small	1,260	1,131 ms	1,688 ms
grok-4.3	1,160	17,667 ms	33,886 ms
anthropic/claude-opus-4.6	612	28,505 ms	36,181 ms
anthropic/claude-sonnet-4.5	282	110,879 ms	139,431 ms
anthropic/claude-opus-4	177	72,796 ms	147,876 ms

Sonnet’s median is faster than Haiku’s while its p95 is 3.6× Haiku’s. Model choice did not determine latency; the work assigned to each model did. The tiers with the slowest medians — Opus, and Sonnet 4.5 — were routed only to design and review work, where a two-minute call is not on any critical path.

Throughput

Throughput is measured over active hours, meaning clock hours in which a given stage made at least one call. Idle hours are excluded; a stage on a 300-second cadence with an empty queue does no work and should not be averaged in.

stage	active hours	calls	mean calls/active hour	median	peak hour
attack_event_enrich	726	29,805	41.1	5	5,126
triage	2,140	8,584	4.0	3	12
business_events	623	8,177	13.1	12	24
signal_enricher	1,941	5,487	2.8	3	5
listen_extract	546	2,333	4.3	2	20

In item terms rather than call terms, triage handled a median of 16 items per active hour, a mean of 29.7, and a peak of 120 — against 2,140 active hours and 63,591 items total.

The gap between attack_event_enrich’s median of 5 calls per hour and its peak of 5,126 is the difference between the recurring mode and the backfill mode. The recurring path used RECURRING_BATCH_SIZE = 5; the backfill path used BATCH_SIZE = 50 and ran until the gap closed. The same handler produced both numbers.

The bottleneck was never model throughput. Summing latency across every non-error call gives the total time each stage spent waiting on a model:

stage	calls	total model-wait
attack_event_enrich	29,733	39.3 h
conflict_extractor	1,215	12.4 h
triage	7,500	11.9 h
signal_enricher	5,187	10.6 h
editorial_review	1,545	9.9 h
x_search_listener	1,160	6.2 h
business_events	7,629	4.4 h

The trace log spans 2026-04-09 to 2026-07-24, 106 days. The seven stages above account for 94.7 hours of model wait across that window. The constraint was cadence and candidate supply, not inference speed: signal_enricher was active in 1,941 hours and made 2.8 calls in each, because a 900-second loop with a small eligible set has nothing to do most of the time. business_events inverts the relationship between spend and time entirely — 26.3% of project spend against 4.4 hours of model wait, the cheapest stage per unit of time and the most expensive per unit of work.

Model tiering

The routing rule:

Agent dispatch rubric. Route work by class: strategic agents for judgment, Sonnet/Haiku tiers for execution and bulk, Codex via dispatch.py for self-contained implementation.

Applied across the run: Haiku ran 4.4× the call volume of Sonnet at lower total cost; Opus averaged 17× Haiku’s per-call cost across 820 calls, all of them schema design, architectural decisions, or review passes. Aggregate token consumption was 221.7M prompt tokens and 28.7M output tokens, at a mean latency for successful calls of 6,816 ms. Part 5 carries spend by model.

Two Haiku identifiers coexist in the trace log — anthropic/claude-haiku-4.5 and anthropic/claude-haiku-4-5 — differing only in a separator. They are the same tier reached through two spellings, split across 64,437 and 2,301 calls. Model identifiers were configuration strings rather than validated enum values, so a typo routed successfully and cost-by-model reporting silently split a row in two.

Spend ceilings and what they measured

The spend cap is a daily envelope — DAILY_ENVELOPE_USD = 2.70, against a stated ~$80/month steady-state pace — read from the trace log at the top of each task and checked before committing to paid work. The lever the policy names is cadence, not quality. It was added in June, after a $528 month.

What the envelope reads is spend. Stage yield — the fraction of a stage’s output that survived into the corpus and was subsequently read — has no column anywhere in the trace log. attack_event_enrich produced 34.8% of all calls into a layer with no downstream consumer. Every row it produced was dropped at teardown. The trace log recorded its cost exactly and could not represent its value. directory_reverify emitted a success signal on every run including the 200 that attempted nothing, because the signal recorded that the handler returned, not that the metric it exists to move had moved. Both instruments measure the act rather than the effect.

---

Next: Part 3 — Harness.