The atomic commit, and why simulation does not prove real broker semantics

The question outbox.mjs answers is: how do you persist "the extraction result exists" and "an event announcing it was created" in a way that never lets only one of the two happen? The classic answer, documented by AWS as the transactional outbox pattern, is to write both in the same database transaction and let a separate process (the relay) read the outbox table and publish - if the transaction fails, neither side happens; if it commits, both happened together, and the relay handles delivery of the event decoupled from whether the publish itself succeeds on the first try.

In this chapter, "same transaction" is simulated as a single synchronous JavaScript call with no await in the middle - true enough to prove the dedupe and conflict logic, but it does not prove what happens when a process dies mid-COMMIT on real disk, or what happens when two database connections race for the same row. examples/pipeline-runbook.md names this gap explicitly, along with the roadmap for closing it with real PostgreSQL (not executed in this session - see the cost/reversibility section). The pedagogical point survives the simplification: the bug the outbox prevents - losing the event, or publishing an event for a transaction that rolled back - is real and documented independent of which technology implements it.

The opening scenario, tested end to end in crash-recovery.test.mjs:

assert.throws(() => worker.processOnce(0, createCrashInjector('after_commit_before_ack')), CrashSimulated);
const resumed = worker.processOnce(5000); // a "fresh process," well past lease expiry
assert.equal(resumed.outcome, 'acked');
assert.equal(resumed.deduped, true, 'the second commit must dedupe, not create a second event');
assert.equal(extractCalls, 2, 'extraction really ran twice - at-least-once execution');
assert.equal(outbox.pendingRows().length, 1, 'but only one event was ever committed - exactly-once effect');

Extraction runs twice (the worker has no way of knowing it already finished before it died); the outbox commit is the only place that knows that key already has a result, so the second attempt returns the existing result instead of minting a second event. One review, not two.

DLQ, quarantine, and auditable reprocessing

Not every error deserves a retry. A document with an invalid schema is not going to pass on the second attempt just because it waited three hundred milliseconds - that is a permanent failure, and retry-policy.mjs sends it straight to the DLQ on the first attempt instead of burning maxAttempts on something that will not change. The reason recorded alongside the job in the DLQ is a short classified code (permanent_error, max_attempts_exceeded), never the raw error message - a parser's error message can contain a fragment of the document itself, and a queue-operations dashboard is not the place for that to leak.

Reprocessing a job from the DLQ is a command, not an automatic retry: it requires operatorId, creates a new job at operationVersion + 1 (never resurrects the dead job in place), and writes an audit entry linking the old job, the new job, and who made the call. This is deliberate: if a document genuinely needs another attempt after a manual fix, that decision has a name and a timestamp, the same way a review approval does in chapter 3.

Cancellation is a third bucket, separate from the DLQ. A job whose deadlineAt has already passed is cancelled the moment anyone tries to pick it up (leaseNext), and cancellation never enters the DLQ map - there is no reprocessFromDlq path for a cancelled job. If a cancelled document genuinely needs to be redone, that is a fresh human decision, with an explicit enqueue() call at a new version, not an automatic resurrection hiding behind a retry.

Simple queue vs. durable workflow: where Temporal actually fits

Temporal solves a broader problem than queue-plus-outbox: durability of execution, not just of a message. A Workflow Execution in Temporal can run indefinitely and, on failure, does not start over - it resumes via Replay, rebuilding its state from a recorded Event History by checking the commands it generates again against what was already recorded. That only works because Workflow code is deterministic by contract: given the same Event History, it has to generate the same commands every time it is re-executed. Anything non-deterministic - a network call, Date.now(), and especially an LLM call, whose response can vary between runs even with the same prompt - has to live inside an Activity, which Temporal treats as a re-executable black box that replay skips rather than reruns.

This chapter's queue-plus-outbox does not try to be that. It durably completes one step's effect (a result plus an event), not the entire execution of a multi-step pipeline with long-lived state between steps. If RelayOps ever needs a workflow that survives for weeks, with conditional steps that genuinely depend on document content and automatic resumption from wherever it stopped, that is exactly the use case that would justify adopting an engine like Temporal - but that adoption was not built, run, or measured in this session. What exists here is a reading of the official documentation and a conceptual comparison, not an implementation.

Four failures

Failure 1 - infinite retry

Symptom: a job never leaves the queue; the log shows the same key being attempted hundreds of times. Cause: a retry with no attempt cap, usually because someone treated "transient failure" as "always worth retrying" without ever asking how many times. Response: retry-policy.mjs has exactly three outcomes (retry, dead_letter, cancel), and retry only exists while attempt < maxAttempts. crash-recovery.test.mjs proves this under a permanently transient failure: three attempts, then the DLQ, never a fourth.

Failure 2 - an LLM call inside deterministic replay, outside an activity

Symptom (in a real Temporal system, not implemented here): the Workflow produces different commands across replays of the same Event History, and the engine rejects the replay as non-deterministic. Cause: putting the model call directly in the Workflow body instead of inside an Activity - the model's response can vary, and a Workflow has to be pure with respect to its own history. Response: the rule, straight from Temporal's own documentation, is that any side effect or non-deterministic call lives in an Activity. Since this chapter does not run Temporal, the failure is documented as a design risk rather than reproduced on a real cluster - faking a reproduction here would mean inventing evidence.

Failure 3 - dual write to DB/broker with no recovery

Symptom: the result exists in the database, but nobody was notified (or the reverse: someone was notified about a transaction that rolled back). Cause: writing to the database and publishing to the broker as two separate calls, with no shared transaction. crash-recovery.test.mjs reproduces this with a deliberately minimal naiveDualWrite function:

function naiveDualWrite(db, broker, key, payload, crashInjector) {
  db.set(key, payload); // "DB commit" - no outbox row, no durable trace an event was owed
  crashInjector('after_db_write_before_publish');
  broker.publish(payload); // never reached if the crash lands above
}

The test proves exactly what AWS describes as the pattern's motivation: db.has(key) is true, broker.published.length is zero. The event is gone, and nothing in the system knows one should exist. Response: outbox.commitResultWithEvent() writes both together; a failure in the middle leaves neither one visible to a caller trying again, and a success leaves a pending outbox row that the relay eventually publishes - even if the original process is already dead.

Failure 4 - a global idempotency key that mixes tenants

Symptom: tenant B sees (or overwrites) tenant A's document result because the two share the same documentId. Cause: an idempotency key built from documentId alone, with no tenant as part of the key. Response: buildIdempotencyKey requires a valid tenantId before it assembles anything - there is no code path that produces a key without one. The corresponding test creates two tenants with the same documentId and the same operationVersion and proves the keys, and the results stored under them, never collide.

What the tests prove and what they do not

The 15 tests (node --test examples/*.test.mjs, 100% green) prove key and payload dedupe, tenant isolation, the opening scenario end to end, relay republication absorbed by the consumer, the dual-write anti-pattern, a redacted DLQ reason, a retry cap, deadline-based cancellation as a dead end from the DLQ's perspective, and audited reprocessing. They do not prove real broker semantics (SQS, Kafka, and RabbitMQ each have their own guarantees and configuration options, some stronger than plain at-least-once under specific settings), PostgreSQL transaction atomicity under a real process crash, disk durability after power loss, or genuine concurrency between workers - the simulator is single-threaded and synchronous, so leaseNext() never actually races against itself. They also prove nothing about a real model: extract() is a fixed synthetic function in every test.

The examples/scripts/synthetic-metrics.mjs script runs 12 synthetic documents and reports three separate numbers per document: queue wait time (queueWaitMs, the only one that reflects genuine control-plane behavior under this synthetic load), inference time (inferenceMs, a fixed 40ms constant, not a measured model latency), and human review wait time (reviewWaitMs, a fixed 90ms constant, not measured reviewer behavior). Queue depth is sampled per tick and ranges from 1 to 7 across the run, peaking mid-batch across twelve documents - evidence about the shape of the synthetic load, not a production capacity claim.

Cost, security, and reversibility

Cost: every retry burns an additional call (extraction, and potentially a model call); an attempt cap is not just about stopping loops, it is about bounding how much a problematic document can spend before it becomes a human decision. Security: the reason recorded in the DLQ is always a short code, never the error text or the document itself - a queue-operations dashboard should not be a PII leak vector. Reversibility: cancellation is terminal with no automatic way back; DLQ reprocessing creates a new, versioned job and never overwrites the old one, so the decision history - including the wrong decisions - stays auditable. The real gap, recorded plainly in examples/pipeline-runbook.md: this implementation has no per-tenant rate limit on DLQ reprocessing - a misconfigured operator, or a misconfigured operator automation, could reprocess the same dead document repeatedly with nothing in the system stopping it.

Another gap worth naming instead of hiding: outbox.mjs never removes a row once it is published - it stays marked published but keeps occupying memory (or, in a database-backed version, a table) indefinitely. That is fine for this chapter's 15 tests, which run in milliseconds and never approach a volume that matters, but it is exactly the kind of decision that needs an explicit retention policy before anything close to production: how long a published row needs to survive for a retroactive audit, and who decides when to archive or delete it. Neither question has an answer in this chapter; both are recorded as future work, not as an implicit behavior of the code.

Checklist before calling a pipeline durable

• The idempotency key includes tenant, document, and operation version - never the document alone.
• A divergent payload under the same key is rejected, never silently reused, at at least two points (ingestion and persistence).
• Every job has a lease, a heartbeat, and a deadline - and retry count and deadline are independent limits, not substitutes for each other.
• Every error classification resolves to retry, dead_letter, or cancel - never an undefined state someone has to guess later.
• The result and its notification event are persisted together, never as two separate calls with no shared transaction.
• The DLQ records a classified, redacted reason; reprocessing requires an operator identity and creates a new version.
• Cancellation is terminal and distinct from the DLQ; nothing turns it back into a new job automatically.
• Queue metrics (job age, depth, time-to-review) separate queue wait, inference time, and human review time - never one number that hides all three.

Exercises

Basic - deduplicate a redelivery. Verifiable criterion: given an event that was already processed (same eventId), a second call to consumer.handle(event) returns { applied: false, reason: 'duplicate_event' }, and consumer.appliedEvents().length stays at 1. Commented solution: createDedupingConsumer() in examples/src/outbox.mjs keeps a Set of seen eventIds independent of outbox state - that is what absorbs even a relay republish (not just a queue-level redelivery) at the right point. See the 'consumer dedupes redelivered events by eventId' test in examples/idempotency.test.mjs, actually run (not just described) as part of node --test examples/*.test.mjs.

Intermediate - inject a crash after the outbox commit and prove resumption. Verifiable criterion: inject CrashSimulated at the after_commit_before_ack stage, confirm the job is still leased with exactly one pending outbox row, advance the clock past the lease, call processOnce() again, and confirm outcome === 'acked' with deduped === true. Commented solution: the 'intermediate exercise: inject a crash right after the outbox commit and prove resumption end to end' test in examples/crash-recovery.test.mjs runs exactly that sequence and finishes by publishing the event and confirming the consumer applied it exactly once - a real command, node --test examples/crash-recovery.test.mjs, not a hypothetical snippet.

Advanced - contrast the simulator with a real transactional PostgreSQL/broker integration. Verifiable criterion (not executed this session - protocol, not a result): replace the Maps in outbox.mjs with two PostgreSQL tables written inside a single transaction per commitResultWithEvent call, with idempotency_key as a UNIQUE constraint; kill the process (kill -9) between the INSERTs and the COMMIT in a real trial and confirm a full rollback; replace the direct relayPublishOnce call with a real broker client and SELECT ... FOR UPDATE SKIP LOCKED so two relay instances cannot claim the same row - exactly the concurrency case this chapter's single-threaded simulator cannot exercise. examples/pipeline-runbook.md also lists failures that integration still would not cover: a network partition during commit (the caller must not treat a timeout as "did not happen" and blindly reprocess) and message loss from broker-side misconfiguration.

Glossary

• At-least-once delivery: a guarantee that a message/job is never silently dropped, at the cost of possibly being delivered more than once.
• Idempotency: applying an operation twice produces the same observable effect as applying it once.
• Lease: temporary, renewable ownership of a job by a worker, with an explicit expiry.
• Fencing token: a monotonic number per lease that invalidates actions from a worker whose ownership already expired.
• Transactional outbox: writing a result and the event that announces it in the same transaction, so a relay can publish later without risking either losing the event or publishing one that should never have existed.
• DLQ (dead-letter queue): the destination for jobs that exhausted their attempts or failed permanently, awaiting a human decision.
• Deterministic replay: in an engine like Temporal, rebuilding an execution's state from recorded history, which only works if the re-executed code always produces the same commands.

Bridge to chapter 5

This chapter made sure an effect happens exactly once even under a crash - but it decided nothing about what an agent should remember across separate runs, or when a past model response should be allowed to influence a future decision without turning into RAG by convenience. Chapter 5 picks up exactly there: what provenance a long-term memory carries, and who decides whether it can be used once a task's budget has already grown past what was planned.