Silently-ignored OUTGOING_PAYMENT.* events — Reconciliation & alerting

The Grid adapter classifies an uncatalogued OUTGOING_PAYMENT.<token> — a money-movement status Grid may add after we ship — as family: "ignored", the same bucket as benign CUSTOMER.* / VERIFICATION.* events (lightspark.ts:939-950, slice/761). That was the correct fix for an earlier "unknown token wedges the receiver" bug — but it trades a wedge for a silent stall: if Grid ships a new terminal token before we update the adapter, a real settlement transition lands as an ignored receipt, nothing pages, and the offramp can sit in provider_submitted / provider_processing forever with no reconciliation.

The question is what to build and where: an alert that distinguishes a money-shaped ignore from a benign one, an independent reconcile cron that un-strands non-terminal offramps via the authoritative getProviderTransaction read, or both — and how it coordinates with the #763 webhook-receiver contract that already owns "records ignored events" and "reconciles authoritative facts". This page maps the code as it is today, presents three genuinely different shapes with diagrams, scores them on an equal-weight rubric, recommends one with a file-by-file blast radius, and ends with an acceptance gate you can run.

1 · What's in the code today

Two tables carry the state. providerEvents journals every webhook — including its real eventType and family — so the discriminator we need ("ignored and money-shaped") is already recoverable with no new field. offramps holds the durable money state and already has a by_status index — which no code uses.

Current schema / shape

schema.ts:555-574 (providerEvents) & schema.ts:469-499 (offramps). The by_status index exists at schema.ts:498; grep finds zero withIndex("by_status") readers on this table.

The inconsistency

Today's flow — the silent stall

A verified, real settlement transition dead-ends as an ignored receipt. The offramp stays non-terminal with no path back to settled / failed.

2 · The three options

Each option shows its data touch, the lifecycle it drives, the new backend surface, and how it is exercised. All three land on claude/epic-jepsen-1077af (the reconciliation base that slice/761 sits on top of), not on the adapter slice.

moneyMovement/
  webhookHttp.ts        ignored-fan-out branch (part of #763 §17.5)
    └─ if eventType.startsWith("OUTGOING_PAYMENT.")
         void reportError(new Error("money-shaped event ignored — stale catalog"),
           { source, phase: "ignoredMoneyShaped", eventType })

# hermetic: force an ignored money-shaped receipt, spy the reporter
setReporter(spy); postWebhook("OUTGOING_PAYMENT.NEW_TOKEN")
expect(spy).toHaveBeenCalledWith(/stale catalog/, { phase: "ignoredMoneyShaped" })

moneyMovement/
  reconciliation.ts               new  internalAction reconcileNonTerminalOfframps
    ├─ runQuery listNonTerminalOfframps      (offramps.by_status, age-gated)
    ├─ per offramp: build orchestrator (same deps as webhookHttp.ts:180)
    ├─ getProviderTransaction({ transactionRef })
    ├─ synthesize ProviderWebhookEvent { family: money_movement, transactionStatus }
    └─ applyVerifiedEvent(event, { offrampId, expectedStatus, expectedVersion })
  queries.ts                            +  internalQuery listNonTerminalOfframps
  crons.ts                              +  crons.interval(... reconciliation ...)

= Option B  (reconciliation.ts + listNonTerminalOfframps + crons.interval)
+ Option A  alert wired into the §17.5 ignored fan-out
+ docs/planning/PROVIDER-MODULE-DESIGN-740.md  §17.5 acceptance: "ignored
            OUTGOING_PAYMENT.* MUST reportError" (so #763 cannot regress it)

3 · Comparison & scorecard

Five criteria, equal weight, constructed for this decision. Three dots = excellent, two = good, one = weak. Total out of 15.

4 · Recommendation & blast radius

Option C. Build the reconcile cron (B) now on claude/epic-jepsen-1077af where it's independently provable, and lock the money-shaped alert (A) as a §17.5 acceptance criterion so the #763 receiver-finalization slice can't ship the ignored fan-out without it.

Concrete decisions baked into the spec

1. Cron target — a new internalAction in moneyMovement/reconciliation.ts; getProviderTransaction needs action context, so it cannot be a query/mutation.
2. Scan — new internalQuery listNonTerminalOfframps over the existing offramps.by_status index, for provider_submitted + provider_processing.
3. Age gate — only offramps with updatedAt < now − 10 min, so a live webhook wins the race first and the cron never fights fresh deliveries (mirrors UNRESOLVED_REPORT_MIN_AGE_MS at webhookHttp.ts:58).
4. Capability gate — skip providers where capabilities().completion !== "provider_webhook"; directTransfer's chain_receipt has no authoritative read (directTransfer.ts:158).
5. Reuse, don't reinvent — the action synthesizes a ProviderWebhookEvent from the read and calls the existing applyVerifiedEvent; no new transition logic, no orchestrator change.
6. Alert channel — the existing reportError ([REPORT] today, inherits PostHog capture when that slice lands). No new pager.
7. Alert home — inside the §17.5 ignored fan-out branch in webhookHttp.ts, gated on eventType.startsWith("OUTGOING_PAYMENT."); filed as a #763 acceptance criterion, not shipped blind here.
8. Cadence — crons.interval({ minutes: 15 }); cheap, and the age gate makes the interval non-load-bearing.

Blast radius — files the implementer touches

Out of scope: no new schema/index (both indexes exist), no new pager, no change to the adapter's ignored classification (that fix stays), no live Grid-sandbox proof of the alert (hermetic here; live proof rides the #763 fan-out slice).

5 · Acceptance gate

Patterned on __tests__/webhookReceiver.test.ts (convex-test + the singleton fake provider). The cron-half proves live-in-test today; the alert-half proves hermetically now and rides #763 for its end-to-end proof. Runner: vp test <path> (package.json:9).

The walkthrough

# 1 — cron un-strands a stalled offramp (no webhook delivered)
seed offramp → provider_submitted            # driveToSubmission(), webhookReceiver.test.ts:164
fake.setTransactionStatus(ref, "completed")  # fake.ts:289 — authoritative read only
await t.action(internal.moneyMovement.reconciliation.reconcileNonTerminalOfframps, {})
expect(offramp).toMatchObject({ status: "settled" })     # ladder ran, no event

# 2 — idempotent + age gate: fresh offramp is skipped, second run is a no-op
seed offramp updatedAt = now              → not returned by listNonTerminalOfframps
run twice on a settled offramp            → no double-settle (CAS version guard)

# 3 — capability gate: chain_receipt provider is skipped, not thrown
directTransfer offramp in provider_submitted → skipped (completion !== "provider_webhook")

# 4 — alert (hermetic): money-shaped ignore pages; benign ignore is silent
setReporter(spy); finalize ignored eventType="OUTGOING_PAYMENT.NEW_TOKEN"
expect(spy).toHaveBeenCalledWith(match(/stale catalog/), { phase: "ignoredMoneyShaped" })
finalize ignored eventType="CUSTOMER.UPDATED"  → expect(spy).not.toHaveBeenCalled()

What gets automated

1. Cron unit/behaviour test in __tests__/reconciliation.test.ts — patterned on webhookReceiver.test.ts's persistenceFor + fake-override harness.
2. Alert test — setReporter spy (init.ts:28) asserting the money-shaped vs benign split; the first test to cover the family: "ignored" path at all (none exists today).
3. (Follow-up on #763) — the §17.5 fan-out slice inherits the alert as an acceptance criterion and adds the live Grid-sandbox proof.

6 · Open questions for the implementer (defaults stated)

1. Age threshold before the cron touches an offramp:
   • 10 min ← recommended. Long enough that a normal webhook settles first; matches the receiver's existing race window.
   • 1 min — more aggressive recovery, but risks racing live deliveries and doubling provider-read load.
2. What the cron does on an unexpected provider status (e.g. read says "processing" but no transition applies):
   • reportError + leave the offramp ← recommended. The cron is a net, not an authority; surface the divergence, don't force state.
   • Silently skip — loses the signal that a genuinely-stuck offramp exists.
3. Should the cron also emit the money-shaped ignore alert (fold A fully into B)?
   • No — keep A at the receiver seam ← recommended. The seam pages in seconds; the cron would page on the next tick. Cron divergence log is the backstop.
   • Yes — one mechanism, but slower detection and couples two concerns.
4. Batch size per cron run:
   • Bounded page (e.g. 50) oldest-first ← recommended. Predictable provider-read load; the next tick picks up the rest.
   • Unbounded .collect() — simpler, fine at current volume, but an unbounded scan is a latent footgun.

7 · Decision

╭─────────────────────────────────────────────────────────────────────────╮
│ Ignored OUTGOING_PAYMENT.* reconciliation · Final Recommendation        │
├─────────────────────────────────────────────────────────────────────────┤
│ Score          A:10   B:9   C:13 ★        (out of 15)                   │
│ Recommended    C — reconcile cron now + money-shaped alert as #763 gate │
│ Blast radius   ~7 files · ~350 LOC · no schema change                   │
│ Acceptance     hermetic convex-test: cron un-strands; alert splits      │
│                money-shaped vs benign ignores                           │
├─────────────────────────────────────────────────────────────────────────┤
│ NEXT GATE — pick:                                                       │
│  (1) Confirm Option C; produce implementer handoff & dispatch agent     │
│  (2) Pick A or B instead (state which)                                  │
│  (3) Modify Option C (e.g. change age gate, batch, or alert home)       │
│  (4) Dig into a specific section first (ladder, capability gate, #763)  │
╰─────────────────────────────────────────────────────────────────────────╯

After your call, I tighten this into a file-level implementer handoff with the exact test cases as the agent's prompt. The agent builds; you walk the acceptance gate; if it passes, we ship.