AI-native Architect: durable pipelines - queues, retries, idempotency, and outbox for agents

A RelayOps worker finishes extracting the fields from an invoice, builds the result, and dies before it acknowledges (ack) the queue. The queue, like any queue that promises not to lose work, has no way to know whether the worker died before or after finishing - it only knows it never got an acknowledgment. It redelivers the same job. A second worker (or the same one, restarted) picks it up, extracts again, and with nothing else in place creates a second review for the same document. The operator working the review queue sees two entries where there should be one. This incident is synthetic; it does not describe a customer, a production run, or real load.

The cause is not the worker dying - process crashes are expected in any system that runs long enough. The cause is treating "the extraction call finished" as the same thing as "the business effect happened exactly once." Those are different claims. A queue that redelivers only guarantees at-least-once delivery: it never silently drops a job, which means it can, and eventually will, deliver the same job more than once. Exactly-once as an end-to-end property of a distributed system with external effects does not come for free - what actually exists is at-least-once delivery combined with idempotency in the effect, so that delivering twice produces the same observable outcome as delivering once. Retrying an LLM call solves availability of the call. It does not, by itself, solve whether the review got created once or twice.

Entry state

Chapter 3 closed with a control plane for agents: policy.mjs requiring maxSteps and deadlineMs at construction, a tool registry with explicit contracts (idempotent, requiresHumanApproval, argsSchema), and a deterministic classify → extract → validate → review-handoff workflow running alongside a bounded agent runner, both writing to the same audit ledger inherited from chapter 2. This chapter does not extend that control plane - it builds the layer that sits underneath wherever either of those two paths (fixed workflow or agent runner) actually runs: what happens when the process executing any given step dies mid-step. No contract in store.mjs, state.mjs, or policy.mjs changed, because none of them is re-imported here; this chapter's queue and outbox are new modules that assume the same discipline of an explicit tenant, not an extension of the earlier ones.

The smallest useful decision: a key that refuses to exist without a tenant

Before any diagram, this chapter's core fits in one function that fails on purpose:

export function buildIdempotencyKey({ tenantId, documentId, operationVersion }) {
  assertTenantId(tenantId); // throws unless tenantId matches /^[a-z][a-z0-9-]{1,32}$/
  if (typeof documentId !== 'string' || documentId.length === 0) {
    throw new Error('documentId is required for an idempotency key');
  }
  if (!(Number.isInteger(operationVersion) && operationVersion >= 1)) {
    throw new Error('operationVersion must be an integer >= 1');
  }
  return `${tenantId}:${documentId}:v${operationVersion}`;
}

The key is tenant, document, and operation version, in that order, always. There is no code path that builds a key from documentId alone - which is the structural defense against Failure 4 further down: a documentId that collides across two tenants (a short identifier, a reused UUID, a migration that reindexed documents) never becomes the same idempotency key, because the tenant is baked into the key itself rather than checked afterward as a separate rule someone can forget to apply.

The second piece of the core is what happens when the same key shows up again with a different payload:

const existing = results.get(idempotencyKey);
if (existing) {
  if (existing.payloadHash !== payloadHash) {
    throw new IdempotencyConflictError(`${idempotencyKey} already committed with a different payload`);
  }
  return { result: existing.result, eventId: existing.eventId, deduped: true };
}

Same key, same payload: silent dedupe, return the existing result. Same key, different payload: an explicit error, never a silent reuse. That distinction matters because "same document, different operation version" is a real business event (reprocessing after a correction), while "same key, mutating payload" is almost always a bug on the caller's side - two code paths competing for the same key, or an operation version that should have been incremented and wasn't.

From control plane to pipeline: what RelayOps gains this chapter

The pipeline this chapter evolves is still ingest → extract → validate → review, the same one sketched since chapter 1, now with an execution layer that survives a crash:

Four new modules carry this layer, and each one solves exactly one problem:

• queue.mjs - an in-memory queue with lease, heartbeat, deadline, and a fencing token per lease. Every leaseNext() bumps a counter (fence); ack/nack/heartbeat only accept the current lease's token. If a "zombie" worker (one that still thinks it holds the lease after it expired and was reassigned) tries to acknowledge later, the queue rejects it - without this, two workers can both believe they own the same job at the same time.
• outbox.mjs - commitResultWithEvent() writes the result and the notification event in a single synchronous call. In this in-memory model, "atomic" means nothing else can interleave between the two writes within the same process; a real system needs a database transaction for the same guarantee across a process crash, and that was not tested here (see the limits section below).
• retry-policy.mjs - classifies every error as transient, permanent, or cancelled and decides retry, dead_letter, or cancel. There is no fourth possible outcome, which is the structural argument against infinite retry.
• worker.mjs - the lease → extract → commit → ack sequence, with named points where a test can inject a process death and then call processOnce() again to simulate a fresh worker after restart.

Job, lease, heartbeat, deadline: four clocks on one job

A job carries four timestamps with different meanings, and conflating one with another is where most queue bugs come from:

• visibleAt - when the job becomes available to be picked up again. Used both for the initial enqueue and for scheduling a retry with backoff.
• leaseUntil - how long the current worker has exclusivity over the job. Passing this does not kill the worker; it only tells the queue that if nobody acknowledged by now, the job can be reassigned.
• deadlineAt - an absolute ceiling from job creation, independent of how many attempts remain. A job can have maxAttempts left and still be cancelled if the deadline has passed; retries and deadline are two different limits acting together.
• createdAt - used only to measure job age (jobAgeMs), the metric that separates "has been waiting a long time" from "has been processing a long time."

Heartbeat exists for long jobs: a worker that is genuinely still working extends leaseUntil periodically so it is not reassigned mid-step during a legitimately slow operation. Without heartbeat, every leaseMs would have to be generous enough for the worst case, which delays detecting workers that actually died.

The atomic commit, and why simulation does not prove real broker semantics

The question outbox.mjs answers is: how do you persist "the extraction result exists" and "an event announcing it was created" in a way that never lets only one of the two happen? The classic answer, documented by AWS as the transactional outbox pattern, is to write both in the same database transaction and let a separate process (the relay) read the outbox table and publish - if the transaction fails, neither side happens; if it commits, both happened together, and the relay handles delivery of the event decoupled from whether the publish itself succeeds on the first try.

In this chapter, "same transaction" is simulated as a single synchronous JavaScript call with no await in the middle - true enough to prove the dedupe and conflict logic, but it does not prove what happens when a process dies mid-COMMIT on real disk, or what happens when two database connections race for the same row. examples/pipeline-runbook.md names this gap explicitly, along with the roadmap for closing it with real PostgreSQL (not executed in this session - see the cost/reversibility section). The pedagogical point survives the simplification: the bug the outbox prevents - losing the event, or publishing an event for a transaction that rolled back - is real and documented independent of which technology implements it.

The opening scenario, tested end to end in crash-recovery.test.mjs:

assert.throws(() => worker.processOnce(0, createCrashInjector('after_commit_before_ack')), CrashSimulated);
const resumed = worker.processOnce(5000); // a "fresh process," well past lease expiry
assert.equal(resumed.outcome, 'acked');
assert.equal(resumed.deduped, true, 'the second commit must dedupe, not create a second event');
assert.equal(extractCalls, 2, 'extraction really ran twice - at-least-once execution');
assert.equal(outbox.pendingRows().length, 1, 'but only one event was ever committed - exactly-once effect');

Extraction runs twice (the worker has no way of knowing it already finished before it died); the outbox commit is the only place that knows that key already has a result, so the second attempt returns the existing result instead of minting a second event. One review, not two.

DLQ, quarantine, and auditable reprocessing

Not every error deserves a retry. A document with an invalid schema is not going to pass on the second attempt just because it waited three hundred milliseconds - that is a permanent failure, and retry-policy.mjs sends it straight to the DLQ on the first attempt instead of burning maxAttempts on something that will not change. The reason recorded alongside the job in the DLQ is a short classified code (permanent_error, max_attempts_exceeded), never the raw error message - a parser's error message can contain a fragment of the document itself, and a queue-operations dashboard is not the place for that to leak.

Reprocessing a job from the DLQ is a command, not an automatic retry: it requires operatorId, creates a new job at operationVersion + 1 (never resurrects the dead job in place), and writes an audit entry linking the old job, the new job, and who made the call. This is deliberate: if a document genuinely needs another attempt after a manual fix, that decision has a name and a timestamp, the same way a review approval does in chapter 3.

Cancellation is a third bucket, separate from the DLQ. A job whose deadlineAt has already passed is cancelled the moment anyone tries to pick it up (leaseNext), and cancellation never enters the DLQ map - there is no reprocessFromDlq path for a cancelled job. If a cancelled document genuinely needs to be redone, that is a fresh human decision, with an explicit enqueue() call at a new version, not an automatic resurrection hiding behind a retry.

Simple queue vs. durable workflow: where Temporal actually fits

Temporal solves a broader problem than queue-plus-outbox: durability of execution, not just of a message. A Workflow Execution in Temporal can run indefinitely and, on failure, does not start over - it resumes via Replay, rebuilding its state from a recorded Event History by checking the commands it generates again against what was already recorded. That only works because Workflow code is deterministic by contract: given the same Event History, it has to generate the same commands every time it is re-executed. Anything non-deterministic - a network call, Date.now(), and especially an LLM call, whose response can vary between runs even with the same prompt - has to live inside an Activity, which Temporal treats as a re-executable black box that replay skips rather than reruns.

This chapter's queue-plus-outbox does not try to be that. It durably completes one step's effect (a result plus an event), not the entire execution of a multi-step pipeline with long-lived state between steps. If RelayOps ever needs a workflow that survives for weeks, with conditional steps that genuinely depend on document content and automatic resumption from wherever it stopped, that is exactly the use case that would justify adopting an engine like Temporal - but that adoption was not built, run, or measured in this session. What exists here is a reading of the official documentation and a conceptual comparison, not an implementation.

Four failures

Failure 1 - infinite retry

Symptom: a job never leaves the queue; the log shows the same key being attempted hundreds of times. Cause: a retry with no attempt cap, usually because someone treated "transient failure" as "always worth retrying" without ever asking how many times. Response: retry-policy.mjs has exactly three outcomes (retry, dead_letter, cancel), and retry only exists while attempt < maxAttempts. crash-recovery.test.mjs proves this under a permanently transient failure: three attempts, then the DLQ, never a fourth.

Failure 2 - an LLM call inside deterministic replay, outside an activity

Symptom (in a real Temporal system, not implemented here): the Workflow produces different commands across replays of the same Event History, and the engine rejects the replay as non-deterministic. Cause: putting the model call directly in the Workflow body instead of inside an Activity - the model's response can vary, and a Workflow has to be pure with respect to its own history. Response: the rule, straight from Temporal's own documentation, is that any side effect or non-deterministic call lives in an Activity. Since this chapter does not run Temporal, the failure is documented as a design risk rather than reproduced on a real cluster - faking a reproduction here would mean inventing evidence.

Failure 3 - dual write to DB/broker with no recovery

Symptom: the result exists in the database, but nobody was notified (or the reverse: someone was notified about a transaction that rolled back). Cause: writing to the database and publishing to the broker as two separate calls, with no shared transaction. crash-recovery.test.mjs reproduces this with a deliberately minimal naiveDualWrite function:

function naiveDualWrite(db, broker, key, payload, crashInjector) {
  db.set(key, payload); // "DB commit" - no outbox row, no durable trace an event was owed
  crashInjector('after_db_write_before_publish');
  broker.publish(payload); // never reached if the crash lands above
}

The test proves exactly what AWS describes as the pattern's motivation: db.has(key) is true, broker.published.length is zero. The event is gone, and nothing in the system knows one should exist. Response: outbox.commitResultWithEvent() writes both together; a failure in the middle leaves neither one visible to a caller trying again, and a success leaves a pending outbox row that the relay eventually publishes - even if the original process is already dead.

Failure 4 - a global idempotency key that mixes tenants

Symptom: tenant B sees (or overwrites) tenant A's document result because the two share the same documentId. Cause: an idempotency key built from documentId alone, with no tenant as part of the key. Response: buildIdempotencyKey requires a valid tenantId before it assembles anything - there is no code path that produces a key without one. The corresponding test creates two tenants with the same documentId and the same operationVersion and proves the keys, and the results stored under them, never collide.

What the tests prove and what they do not

The 15 tests (node --test examples/*.test.mjs, 100% green) prove key and payload dedupe, tenant isolation, the opening scenario end to end, relay republication absorbed by the consumer, the dual-write anti-pattern, a redacted DLQ reason, a retry cap, deadline-based cancellation as a dead end from the DLQ's perspective, and audited reprocessing. They do not prove real broker semantics (SQS, Kafka, and RabbitMQ each have their own guarantees and configuration options, some stronger than plain at-least-once under specific settings), PostgreSQL transaction atomicity under a real process crash, disk durability after power loss, or genuine concurrency between workers - the simulator is single-threaded and synchronous, so leaseNext() never actually races against itself. They also prove nothing about a real model: extract() is a fixed synthetic function in every test.

The examples/scripts/synthetic-metrics.mjs script runs 12 synthetic documents and reports three separate numbers per document: queue wait time (queueWaitMs, the only one that reflects genuine control-plane behavior under this synthetic load), inference time (inferenceMs, a fixed 40ms constant, not a measured model latency), and human review wait time (reviewWaitMs, a fixed 90ms constant, not measured reviewer behavior). Queue depth is sampled per tick and ranges from 1 to 7 across the run, peaking mid-batch across twelve documents - evidence about the shape of the synthetic load, not a production capacity claim.

Cost, security, and reversibility

Cost: every retry burns an additional call (extraction, and potentially a model call); an attempt cap is not just about stopping loops, it is about bounding how much a problematic document can spend before it becomes a human decision. Security: the reason recorded in the DLQ is always a short code, never the error text or the document itself - a queue-operations dashboard should not be a PII leak vector. Reversibility: cancellation is terminal with no automatic way back; DLQ reprocessing creates a new, versioned job and never overwrites the old one, so the decision history - including the wrong decisions - stays auditable. The real gap, recorded plainly in examples/pipeline-runbook.md: this implementation has no per-tenant rate limit on DLQ reprocessing - a misconfigured operator, or a misconfigured operator automation, could reprocess the same dead document repeatedly with nothing in the system stopping it.

Another gap worth naming instead of hiding: outbox.mjs never removes a row once it is published - it stays marked published but keeps occupying memory (or, in a database-backed version, a table) indefinitely. That is fine for this chapter's 15 tests, which run in milliseconds and never approach a volume that matters, but it is exactly the kind of decision that needs an explicit retention policy before anything close to production: how long a published row needs to survive for a retroactive audit, and who decides when to archive or delete it. Neither question has an answer in this chapter; both are recorded as future work, not as an implicit behavior of the code.

Checklist before calling a pipeline durable

• The idempotency key includes tenant, document, and operation version - never the document alone.
• A divergent payload under the same key is rejected, never silently reused, at at least two points (ingestion and persistence).
• Every job has a lease, a heartbeat, and a deadline - and retry count and deadline are independent limits, not substitutes for each other.
• Every error classification resolves to retry, dead_letter, or cancel - never an undefined state someone has to guess later.
• The result and its notification event are persisted together, never as two separate calls with no shared transaction.
• The DLQ records a classified, redacted reason; reprocessing requires an operator identity and creates a new version.
• Cancellation is terminal and distinct from the DLQ; nothing turns it back into a new job automatically.
• Queue metrics (job age, depth, time-to-review) separate queue wait, inference time, and human review time - never one number that hides all three.

Exercises

Basic - deduplicate a redelivery. Verifiable criterion: given an event that was already processed (same eventId), a second call to consumer.handle(event) returns { applied: false, reason: 'duplicate_event' }, and consumer.appliedEvents().length stays at 1. Commented solution: createDedupingConsumer() in examples/src/outbox.mjs keeps a Set of seen eventIds independent of outbox state - that is what absorbs even a relay republish (not just a queue-level redelivery) at the right point. See the 'consumer dedupes redelivered events by eventId' test in examples/idempotency.test.mjs, actually run (not just described) as part of node --test examples/*.test.mjs.

Intermediate - inject a crash after the outbox commit and prove resumption. Verifiable criterion: inject CrashSimulated at the after_commit_before_ack stage, confirm the job is still leased with exactly one pending outbox row, advance the clock past the lease, call processOnce() again, and confirm outcome === 'acked' with deduped === true. Commented solution: the 'intermediate exercise: inject a crash right after the outbox commit and prove resumption end to end' test in examples/crash-recovery.test.mjs runs exactly that sequence and finishes by publishing the event and confirming the consumer applied it exactly once - a real command, node --test examples/crash-recovery.test.mjs, not a hypothetical snippet.

Advanced - contrast the simulator with a real transactional PostgreSQL/broker integration. Verifiable criterion (not executed this session - protocol, not a result): replace the Maps in outbox.mjs with two PostgreSQL tables written inside a single transaction per commitResultWithEvent call, with idempotency_key as a UNIQUE constraint; kill the process (kill -9) between the INSERTs and the COMMIT in a real trial and confirm a full rollback; replace the direct relayPublishOnce call with a real broker client and SELECT ... FOR UPDATE SKIP LOCKED so two relay instances cannot claim the same row - exactly the concurrency case this chapter's single-threaded simulator cannot exercise. examples/pipeline-runbook.md also lists failures that integration still would not cover: a network partition during commit (the caller must not treat a timeout as "did not happen" and blindly reprocess) and message loss from broker-side misconfiguration.

Glossary

• At-least-once delivery: a guarantee that a message/job is never silently dropped, at the cost of possibly being delivered more than once.
• Idempotency: applying an operation twice produces the same observable effect as applying it once.
• Lease: temporary, renewable ownership of a job by a worker, with an explicit expiry.
• Fencing token: a monotonic number per lease that invalidates actions from a worker whose ownership already expired.
• Transactional outbox: writing a result and the event that announces it in the same transaction, so a relay can publish later without risking either losing the event or publishing one that should never have existed.
• DLQ (dead-letter queue): the destination for jobs that exhausted their attempts or failed permanently, awaiting a human decision.
• Deterministic replay: in an engine like Temporal, rebuilding an execution's state from recorded history, which only works if the re-executed code always produces the same commands.

Bridge to chapter 5

This chapter made sure an effect happens exactly once even under a crash - but it decided nothing about what an agent should remember across separate runs, or when a past model response should be allowed to influence a future decision without turning into RAG by convenience. Chapter 5 picks up exactly there: what provenance a long-term memory carries, and who decides whether it can be used once a task's budget has already grown past what was planned.