# PER_EXCHANGE — Exactly-Once-Per-Exchange Alerting — Design

**Date:** 2026-04-22
**Scope:** alerting layer only (webhook-delivery-level idempotency is out of scope — see "Out of scope" below).
**Preceding context:** `.planning/sse-flakiness-diagnosis.md` (unrelated); `docs/superpowers/specs/2026-04-19-alerting-design.md` (foundational alerting design — original PER_EXCHANGE intent).

## Motivation

A user wants to create an alert rule of the shape "for every exchange that ends in FAILED status, notify Slack — exactly once per exchange." Exchanges are **terminal events**: once in FAILED state they never transition back, unlike agents which toggle between LIVE / STALE / DEAD. So "exactly once" is well-defined and achievable.

Today's PER_EXCHANGE mode partially supports this but has three gaps that, in combination, either miss exchanges or flood the Inbox with duplicates:

1. **The cursor is dead code.** `ExchangeMatchEvaluator.evaluatePerExchange` at `cameleer-server-app/.../eval/ExchangeMatchEvaluator.java:141` computes a `_nextCursor` and stamps it onto the **last firing's context map**, but `AlertEvaluatorJob.applyBatchFiring` (`:206-213`) never reads it, and `reschedule` (`:259`) calls `releaseClaim(rule.id(), nextRun, rule.evalState())` — passing the **original, unmodified** `evalState`. So `lastExchangeTs` is never persisted; every tick runs with `timeFrom = null` and re-scans ClickHouse from retention's beginning. The partial unique index on `alert_instances(rule_id, context->'exchange'->>'id')` silently de-dups duplicate *instances* on each tick — but `enqueueNotifications` in `applyBatchFiring` runs unconditionally on the returned row, so **every tick enqueues a fresh PENDING `AlertNotification` for every matching exchange already in retention**. The user-visible symptom is not "a same-millisecond collision"; it is "Slack gets re-spammed for every historical failed exchange on every tick." Same-millisecond collision is also real, but strictly subsumed once a working cursor exists.
2. Alert-instance writes, notification enqueues, and `evalState` cursor advance are not coupled transactionally. Once §1 is fixed and the cursor *does* advance, a crash between the instance write and the cursor persist would produce either silent data loss (cursor advanced, instances never persisted) or duplicate instances on recovery (instances persisted, cursor not advanced). §2 makes this atomic before the cursor goes live.
3. The rule-configuration surface for PER_EXCHANGE admits nonsensical combinations (`reNotifyMinutes > 0`, mandatory-but-unused `perExchangeLingerSeconds`, `forDurationSeconds`) — a user following the UI defaults can build a rule that re-notifies hourly even though they want one-shot semantics.

The product rules, agreed with the user:

- The AlertInstance **stays FIRING** until a human acks or resolves it. No auto-resolve sweep.
- The action (webhook) **fires exactly once** per AlertInstance.
- The Inbox **contains exactly one AlertInstance** per failed exchange — never a duplicate from cursor errors, tick re-runs, or process restarts.

"Exactly once" here is at the alerting layer — one AlertInstance, one PENDING AlertNotification per (instance × webhook binding). The HTTP dispatch that follows is still at-least-once on transient failures; that's a separate scope.

## Non-goals

- Auto-resolve after linger seconds. The existing spec reserved `perExchangeLingerSeconds` for this; we're explicitly dropping the field (unused + not desired).
- Resolve-on-delivery semantics. Alert stays FIRING until human intervention.
- Webhook-level idempotency / exactly-once HTTP delivery to Slack. Rare duplicate Slack messages on timeout retries are accepted; consumer-side dedup (via `alert.id` in the payload) is a template concern, not a server change.
- Any change to COUNT_IN_WINDOW mode.
- Backfilling duplicate instances already created by the existing broken cursor in any running environment. Pre-prod, manual cleanup if needed.
- **Sealed condition-type hierarchy.** A follow-up refactor could replace the current "one `AlertRule` with mode-gated fields" model with a sealed hierarchy (`PerExchangeCondition` / `CountInWindowCondition` / `AgentLifecycleCondition`) where each type carries only the knobs it supports. The UI already sharded condition forms under `ui/src/pages/Alerts/RuleEditor/condition-forms/` — the backend is the laggard. §3 here only patches the three known field conflicts; the structural cleanup is a separate phase (see §Follow-ups).
- **Throttle / coalesce primitive for PER_EXCHANGE.** njams bakes `throttling` + `throttlingEventCount` into its CEP queries ("fire once, count the rest in W seconds"). If operators later find the Inbox unwieldy during incident storms, a `coalesceSeconds` knob is the right cure — one FIRING per (rule × signature) per window, with `occurrenceCount` maintained on the instance. Explicitly parked; see §Follow-ups.

## Design

Four focused changes. Each is small on its own; together they make PER_EXCHANGE fire exactly once per failed exchange.

### 1. Composite cursor for cursor monotonicity

**Current.** `evalState.lastExchangeTs` is a single ISO-8601 string the evaluator *reads* as a lower bound, but **never writes**. The advance is computed (`latestTs = max(startTime)`) and attached to the last firing's context map as `_nextCursor`, then discarded by `applyBatchFiring` — `reschedule` passes the untouched `rule.evalState()` to `releaseClaim`. Net effect today: every tick queries ClickHouse with `timeFrom = null` (first-run path). The same-millisecond-collision bug described in the original spec assumes the cursor works; in practice the cursor has never worked. Fixing the advance is therefore both a correctness fix and a dead-code-elimination.

**New.** Replace with a composite cursor `(startTime, executionId)`, serialized as `"<ISO-8601 startTime>|<executionId>"` in `evalState.lastExchangeCursor`.

- Selection predicate: `(start_time > cursor.ts) OR (start_time = cursor.ts AND execution_id > cursor.id)`.
  - This is trivially monotone: every consumed exchange is strictly-after the cursor in the composite ordering.
  - Handles two exchanges at the exact same millisecond correctly — both are selected on their turn, neither re-selected.
  - Uses the existing ClickHouse primary-key order `(tenant_id, start_time, …, execution_id)`, so the predicate is a range scan on the PK.
- Advance: set cursor to the `(startTime, executionId)` of the lexicographically-last row in the batch (last row when sorted by `(start_time asc, execution_id asc)`).
- First run (no cursor): today's behaviour is `cursor = null → timeFrom = null → unbounded scan of ClickHouse history` — any pre-existing FAILED exchange in retention would fire an alert on the first tick. That's broken and needs fixing. New rule: initialize `lastExchangeCursor` to `(rule.createdAt, "")` at rule creation time — so a PER_EXCHANGE rule only alerts on exchanges that fail *after* it was created. The empty-string `executionId` component is correct: any real execution_id sorts strictly after it lexicographically, so the very first matching exchange post-creation gets picked up on the first tick. No ambient lookback window, no retention dependency, no backlog flood.
- `evalState` schema change: retire the `lastExchangeTs` key, add `lastExchangeCursor`. Pre-prod; no migration needed. Readers that see neither key treat the rule as first-run.

**Affected files (scope estimate):**
- `ExchangeMatchEvaluator.evaluatePerExchange` — cursor parse/advance/selection.
- `SearchRequest` / `ClickHouseSearchIndex.search` — needs to accept the composite predicate. Option A: add an optional `afterExecutionId` param alongside `timeFrom`. Option B: introduce a dedicated `AfterCursor(ts, id)` type. Plan phase picks one — A is simpler.
- `evalState` JSON schema (documented in alerting spec).

### 2. Transactional coupling of instance writes + cursor advance

This section presumes §1 is landed — once the cursor actually advances, we need the advance and the instance writes to be atomic.

**Current (post-§1).** Per tick for a PER_EXCHANGE rule:
1. `applyResult` iterates the `EvalResult.Batch` firings and calls `applyBatchFiring` for each — one `AlertInstance` save + `enqueueNotifications` per firing, each its own transaction (or auto-commit).
2. After the rule loop, `reschedule(rule, nextRun)` saves the updated `evalState` + `nextRunAt` in a separate write.

Crash anywhere between steps 1 and 2 (or partway through the loop in step 1) produces one of two inconsistent states:
- Instances saved but cursor not advanced → next tick duplicates them.
- Cursor advanced but no instances saved → those exchanges never alerted.

(Note: today, pre-§1, the "cursor never advances" bug means only the first failure mode ever occurs. §2 prevents the second from appearing once §1 is live.)

**New.** Wrap the whole Batch-result processing for a single rule in one `TransactionTemplate.execute(...)`:

```
TX {
  persist all AlertInstances for the batch
  insert all PENDING AlertNotifications for those instances
  update rule: evalState.lastExchangeCursor + nextRunAt
}
```

Commit: all three land atomically. Rollback: none do, and the rule stays claimed-but-cursor-unchanged so the next tick re-processes the same exchanges. Combined with the monotone cursor from §1, that gives exactly-once instance creation: if a batch half-succeeded and rolled back, the second attempt starts from the same cursor and produces the same set.

Notification dispatch (`NotificationDispatchJob` picking up PENDING rows) happens outside the transaction on its own schedule — webhook I/O must never hold a DB transaction open.

**Affected files (scope estimate):**
- `AlertEvaluatorJob.applyResult` + `applyBatchFiring` — fold into one transactional block when the result is a `Batch`.
- No change to the COUNT_IN_WINDOW path (`applyResult` for non-Batch results keeps its current semantics).
- `PostgresAlertInstanceRepository` / `PostgresAlertNotificationRepository` / `PostgresAlertRuleRepository` — existing methods usable from inside a transaction; verify no implicit auto-commit.

### 3. Config hygiene — enforce a coherent PER_EXCHANGE rule shape

Three knobs on the rule are wrong for PER_EXCHANGE and trap the user into buggy configurations.

| Knob | Current state | New state for PER_EXCHANGE |
|---|---|---|
| `reNotifyMinutes` | Default 60 in UI; re-notify sweep fires every N min while FIRING | **Must be `0`.** API 400s if non-zero. UI forces to `0` and disables the input with tooltip "Per-exchange rules fire exactly once — re-notify does not apply." |
| `perExchangeLingerSeconds` | Validated as required by `ExchangeMatchCondition` compact ctor; unused anywhere in the code | **Removed.** Drop the field entirely — from the record, the compact-ctor validation, `AlertRuleRequest` DTO, form state, UI. Pre-prod; no shim. |
| `forDurationSeconds` | Applied by the state machine in the COUNT_IN_WINDOW / agent-lifecycle path | **Must be `0`/null** for PER_EXCHANGE. 400 on save if non-zero. UI hides the field when PER_EXCHANGE is selected. Evaluator path already ignores it for `Batch` results, so this is a contract-tightening at the API edge only. |

Net effect: a PER_EXCHANGE rule's configurable surface becomes exactly `{scope, filter, severity, notification title/message, webhooks, targets}`. The user can't express an inconsistent combination.

**Mode-toggle state hygiene (UX).** When the user flips `fireMode` PER_EXCHANGE ↔ COUNT_IN_WINDOW inside `ExchangeMatchForm`, the form state for the *other* mode's fields must be cleared — not just hidden. The njams Server_4 frontend mode-gates fields via `*ngIf` and silently retains stale values behind the toggle (`src/app/rules/.../rule-view.component.html:96–112`), which produces save-time surprises. In `form-state.ts` the `setFireMode` reducer must reset the fields that are no longer in scope for the new mode (to their type-appropriate zero, not to undefined — the record compact-ctor still runs). That keeps the API-layer cross-field validator (400-on-save) and the form shape permanently consistent.

**Builder-UX lessons worth adopting (tiny, in-scope).**
- **Disabled "Add" gating.** `AlertRuleController` accepts `webhooks: []` and `targets: []` as valid, which lets the user save a rule that never notifies anyone. The form already splits by step; the Notify step's "Add webhook" button should stay enabled, but the wizard's "Save rule" in `ReviewStep.tsx` should block-with-reason if `webhooks.length === 0 && targets.length === 0`. njams's pattern of disabling "Add X" until the last row is complete (`rule-view.component.ts:38–45`) is the right shape.
- **Preserve `/test-evaluate` and `/render-preview`.** `AlertRuleController` exposes POST `{id}/test-evaluate` and `{id}/render-preview`; the wizard should surface at least render-preview in `ReviewStep.tsx` before save. njams ships no in-builder preview and operators compensate with trial-and-error creation. We already have the endpoints; not wiring them up would be leaving value on the floor.

**Affected files (scope estimate):**
- `ExchangeMatchCondition` — remove `perExchangeLingerSeconds`.
- `AlertRuleController` / `AlertRuleRequest` — cross-field validation (reNotify + forDuration vs fireMode; empty webhooks+targets).
- `ui/src/pages/Alerts/RuleEditor/condition-forms/ExchangeMatchForm.tsx` + `form-state.ts` — clear fields on mode toggle; disable reNotify + hide forDuration when PER_EXCHANGE; remove the linger field.
- `ui/src/pages/Alerts/RuleEditor/ReviewStep.tsx` — block save on empty webhooks+targets; render-preview pane.
- Tests (§4).

### 4. Tests that lock the guarantees

Six scenarios: four on the correctness core, one red-test that reproduces today's actual bleed (turns green on fix), one on the builder-UX state-clearing contract.

**Test 1 — cursor monotonicity** (`ExchangeMatchEvaluatorTest`, unit)

- Seed two FAILED executions with identical `start_time`, different `executionId`.
- Tick 1: both fire, batch of 2.
- Tick 2: neither fires.
- Seed a third at the same timestamp. Tick 3: that third only.

**Test 2 — tick atomicity** (`AlertEvaluatorJobIT`, integration with real Postgres)

- Seed 3 FAILED executions. Inject a fault on the second notification-insert.
- Tick → transaction rolls back: 0 AlertInstances, cursor unchanged, rule `nextRunAt` unchanged.
- Remove fault, tick again: 3 AlertInstances + 3 PENDING notifications, cursor advanced.

**Test 3 — full-lifecycle exactly-once** (extends `AlertingFullLifecycleIT`)

- PER_EXCHANGE rule, dummy webhook.
- Seed 5 FAILED executions across two ticks (3 + 2). After both ticks: exactly 5 FIRING AlertInstances, exactly 5 PENDING notifications.
- Third tick with no new executions: zero new instances, zero new notifications.
- Ack one instance: other four unchanged.
- Additionally: POST a PER_EXCHANGE rule with `reNotifyMinutes=60` via the controller → expect 400.
- Additionally: POST a PER_EXCHANGE rule with `forDurationSeconds=60` → expect 400.

**Test 4 — first-run uses rule creation time, not unbounded history** (unit, in `ExchangeMatchEvaluatorTest`)

- Seed 2 FAILED executions dated *before* rule creation, 1 *after*.
- Evaluate a freshly-created PER_EXCHANGE rule whose `evalState` is empty.
- Expect: exactly 1 firing (the one after creation). The pre-creation ones must not appear in the batch.

**Test 5 — pre-fix regression reproducer: notifications do not re-enqueue for already-matched exchanges** (integration, `AlertEvaluatorJobIT`)

- Seed 2 FAILED executions. Tick 1 → 2 FIRING instances, 2 PENDING notifications. Dispatcher drains them → 2 DELIVERED.
- Tick 2 with **no new executions**: expect zero new PENDING notifications. (Today, without the §1+§2 fix, tick 2 re-enqueues both. This test should be written red-first against `main`, then go green when the cursor is actually persisted.)
- This test directly pins the bug the original spec text understated: instance-level dedup via the unique index is already working; notification-level dedup is what's broken.

**Test 6 — form state clears on fireMode toggle** (unit, Vitest, `condition-forms/ExchangeMatchForm.test.tsx`)

- Build an initial form state with `fireMode=COUNT_IN_WINDOW, threshold=5, windowSeconds=300`.
- Dispatch `setFireMode(PER_EXCHANGE)`.
- Expect: `threshold` and `windowSeconds` are cleared to their zero-values (not merely hidden), and the record compact-ctor doesn't throw when `form-state.ts` rebuilds the condition object.
- Dispatch `setFireMode(COUNT_IN_WINDOW)` — expect threshold/window come back as defaults, not as stale values.

Plus a small unit test on the new cross-field validator to isolate its logic from the IT setup.

## Out of scope

- **Webhook-level idempotency.** `WebhookDispatcher` still retries on 5xx / network / timeout. For Slack, that means a timeout mid-POST can produce a duplicate channel message. The consumer-side fix is to include a stable ID (e.g. `{{alert.id}}`) in the message template and drop duplicates on Slack's side — doable today via the existing Mustache editor, no server change. If in the future we want strict exactly-once HTTP delivery, that's a separate design.
- **Auto-resolve of PER_EXCHANGE instances.** Alerts stay FIRING until humans intervene. If operational experience shows the Inbox gets unwieldy, a later phase can add a manual "resolve all" bulk action or an opt-in TTL sweep.
- **Rule-level dedup of identical alerts in a short window** (e.g. "same failure signature fires twice in 5 s"). Out of scope; every failed exchange is its own event by design.
- **COUNT_IN_WINDOW changes.** Untouched.
- **Migration of existing PER_EXCHANGE rules.** Pre-prod; any existing rule using the retired `perExchangeLingerSeconds` field gets the value silently dropped by the API's unknown-property handling on next PUT, or rejected on create (new shape). If needed, a one-shot cleanup is easier than a shim.

## Risks

- **ClickHouse predicate performance.** The composite predicate `(start_time > ? OR (start_time = ? AND execution_id > ?))` must hit the PK range efficiently. The table PK is `(tenant_id, start_time, environment, application_id, route_id, execution_id)`, so the OR-form should be fine, but we'll verify with `EXPLAIN PIPELINE` against the IT container during plan-phase. Fallback: `(start_time, execution_id)` tuple comparison if CH has native support (`(start_time, execution_id) > (?, ?)`), which it does in recent versions.
- **Transaction size.** A single tick caps at `limit = 50` matches (existing behaviour), so the transaction holds at most 50 AlertInstance + 50 AlertNotification writes + 1 rule update. Well within safe bounds.
- **Cursor format churn.** Dropping `lastExchangeTs` in favour of `lastExchangeCursor` is a one-line `evalState` JSON change. Pre-prod; no shim needed. In practice the churn is even more benign than it looks: today no rule has ever persisted a `lastExchangeTs` value (the advance path is dead code — see §1 Current.), so every existing PER_EXCHANGE rule will hit the first-run path `(rule.createdAt, "")` on first post-deploy tick. Side effect: on deploy, long-standing PER_EXCHANGE rules will immediately scan from `rule.createdAt` forward and enqueue notifications for every FAILED exchange in retention that matches. This is a **one-time backlog flood** proportional to ClickHouse retention × failure rate × number of PER_EXCHANGE rules. For pre-prod with small history this is tolerable; if a rule was created years ago on a real environment, bound the first-run scan by clamping initial cursor to `max(rule.createdAt, now() - deployBacklogCap)` where `deployBacklogCap` is a config (default 24 h). Call this out explicitly in the plan-phase so deployment order is "deploy first, then create rules" or "accept the one-time flood."

## Follow-ups (parked — separate phases)

Explicit list of ideas that are valuable but deliberately not in this spec's scope.

1. **Sealed condition-type hierarchy (backend).** Replace `AlertRule` + `fireMode` field with a sealed `Condition` hierarchy where each type carries only its own knobs. The UI is already sharded (`condition-forms/*Form.tsx`); the backend would follow. Biggest win: kills the whole "mode-gated field" class of bug at the record level, so cross-field validators become compact-ctor invariants instead of controller-layer glue. Estimated scope: medium (DTO migration, Jackson polymorphism, request compatibility). Trigger: when a 4th condition kind lands or when the next "silently-ignored field" bug surfaces.
2. **`coalesceSeconds` primitive on PER_EXCHANGE.** "One FIRING per (rule × signature) per window; attach occurrenceCount." Addresses the Inbox-flood scenario during incident storms without breaking the exactly-once-per-exchange guarantee for the default case. njams bakes this into its CEP template as `throttling` + `throttlingEventCount`; we'd express it as a post-match coalescer on the AlertInstance write path. Trigger: first operator complaint about Inbox volume during a real incident, or when we onboard a tenant with >100 failed exchanges/min.
3. **Cross-phase: ingestion-time rule-matching.** Today's tick+cursor model is correct but latency-bound to `evaluationIntervalSeconds`. A streaming path (agent → ClickHouse ingest → publish → rule matcher) would drop alert latency to seconds. Not needed today; flagged because the spec's design explicitly chooses batch over streaming and future requirements may flip that.

## Verification

- `mvn -pl cameleer-server-app -am -Dit.test='ExchangeMatchEvaluatorTest,AlertEvaluatorJobIT,AlertingFullLifecycleIT,AlertRuleControllerIT' ... verify` → 0 failures.
- Manual: create a PER_EXCHANGE / FAILED rule via UI. Verify `reNotifyMinutes` is fixed at 0 and disabled; verify the linger field is gone; verify toggling `fireMode` clears COUNT_IN_WINDOW-specific fields. Produce failing exchanges. Verify Inbox shows one instance per exchange, Slack gets exactly one message each. **Wait three evaluation-interval ticks with no new exchanges**; verify no additional notifications arrive (the pre-fix bleed). Ack one instance. Produce another failure. Verify only the new one appears. Save a rule with empty webhooks+targets → expect blocked at the Review step with a reason shown.
