AI-native Architect: costs and capacity — routing, cache, and unit economics under budget

The RelayOps team swaps the extraction model for a cheaper one per token. The monthly API bill drops — the per-call cost dashboard shows the savings the same day. Two weeks later, someone finally computes the number that should have been the first question: cost per document actually accepted. It went up. The cheaper model misses schema more often, every miss turns into another paid call, and the document — from a critical tenant — still needs human review before it can be accepted. This incident is synthetic; it does not describe a customer, a real bill, or production load, but the shape of the mistake is common enough to open the chapter with it: one metric went down, and it was not the metric that decided whether the switch was worth it.

The cause is not an arithmetic error — it is measuring the wrong thing. Price per token answers "how much did one call cost." Cost per accepted document answers "how much did the result someone could actually use cost," and that second question sums retries, human review, storage, egress, and worker overhead into the same account, without counting anything twice. The two vary independently: a model can be cheaper per token and still more expensive per accepted document, if it misses schema often enough that retries erase the savings — and once human review enters the account, it can dominate the total so completely that model choice becomes a second-order detail next to the decision of whether to route to a human at all. This chapter builds the calculator, the cache policy, the routing policy, and the per-tenant budget that make that distinction verifiable instead of assumed.

Entry state

Chapter 5 closed out RelayOps's eval and observability layer: a dev/holdout dataset with no ID overlap, deterministic graders (schema, field, policy, task success), a release gate that fails a critical-stratum regression even when the overall mean improves, and a per-stratum report shaped { overall, n } — a measured success rate over a sample of known size. This chapter does not reopen or reimplement any file from that layer; it assumes RelayOps already knows how to measure whether a result is correct, and asks a different question: given that quality can be measured, which routing, which cache, and which budget keep that quality inside a predictable per-tenant spending limit — including when traffic is not the stable average of a spreadsheet? The only thing this chapter reuses from chapter 5 is the shape of stratum evidence ({ overall, n }), as the routing policy's input contract — no chapter 5 number is copied or cited as if it belonged here.

The smallest example that separates price from cost

Before any calculator, the distinction fits in one function:

export function tokenCallCost({ model, inputTokens, outputTokens, cacheReadTokens = 0 }, prices) {
  const rate = prices.models[model];
  const perMTok = (tokens, r) => (tokens / 1_000_000) * r;
  return perMTok(inputTokens, rate.input)
    + perMTok(outputTokens, rate.output)
    + perMTok(cacheReadTokens, rate.cacheRead);
}

That function answers "how much did this one call cost" — nothing else. It does not know whether the call needed a retry, does not know whether the document went to human review, does not know whether the tenant has budget for another call. Each of those is a separate layer: documentMarginalCost sums every call for a document (including retries) plus tool/worker/storage/egress overhead; documentAllocatedCost adds human review only when routing decided on it; and costPerAcceptedDocument divides a batch's total spend by the count of documents that actually reached an accepted state — never by the call count. A rejected document still costs money and is never counted as free; that is an automated test, not a sentence (a rejected document still costs money and is excluded from acceptedCount, never counted as free).

Intuition: the trade-off already has a name

Anthropic's own engineering guide on agents states that "agentic systems often trade latency and cost for better task performance," and explicitly recommends routing "easy/common questions to smaller, cost-efficient models... and hard/unusual questions to more capable models" as the central use case for a routing workflow. RelayOps's mistake in this chapter's opening was not using a cheaper model — it was using one cheaper model for every document type, without measuring whether that specific type tolerates the cheaper model. Conventional systems' cost/capacity intuition already solved a related problem: the Google SRE book describes overload protection based on a local utilization signal, and an adaptive throttling mechanism in which each client tracks its own recent acceptance rate and self-regulates before the shared queue is affected — the same shape this chapter uses for per-tenant budget: a local "how much budget is left" signal that slows the tenant itself down before it can consume everyone else's capacity.

RelayOps evolution: from the call to the per-tenant budget

RelayOps gains four new, independent pieces in this chapter, separate from the eval layer and the durability layer of earlier chapters:

• Cost calculator (examples/costs/calculate.mjs) over a real, dated price fixture — Anthropic API list prices, captured 2026-09-28 — plus explicitly synthetic operational assumptions (storage, egress, worker, human review), never mixed into the same table without a label.
• Cache policy (examples/src/cache-policy.mjs) with a deterministic key scoped to tenant + document checksum + schema/prompt/model/policy version, in two namespaces that never collide: exact (the same answer for the same document) and semantic (the same compiled prompt/schema/policy, reused across different documents of the same tenant and type).
• Per-tenant budget (examples/src/budget-policy.mjs) with atomic reservation at the execution boundary, reconciliation against the actual cost after the call, and a backpressure signal that grows as a tenant's budget shrinks — so one noisy tenant can never starve another tenant's share of the shared queue.
• Evidence-based routing (examples/src/routing-policy.mjs) that decides between a cheap tier, a standard tier, and a human fallback using three deterministic inputs: document-type risk, available budget, and a measured success rate — never a confidence score the model reports about itself.

None of these four pieces calls a real model, a database, or the network. All of them run offline, in plain Node, and the 24 tests covering them run in under a hundred milliseconds.

The price fixture feeding the calculator deliberately separates two categories of number that cannot live in the same unlabeled table: the prices for Claude Opus 5.5, Claude Sonnet 5, and Claude Haiku 4.5 — real, dated, copied from Anthropic's pricing documentation on 2026-09-28 — and the operational assumptions (storage, egress, worker compute, human-review cost per minute), which are entirely synthetic, chosen to keep the arithmetic auditable by hand, not to model a real cloud bill. Mixing the two categories into one plain table of numbers — without saying which row is a primary source and which is a tutorial assumption — is exactly the kind of mistake that makes a reader treat a teaching example as a benchmark. prices-fixture.json tags each block with a _type field and an explicit warning for that reason, and sources.md records the URL and access date next to every real price.

Why you reserve before spending, not spend and reconcile after

The order matters more than it looks. A system that makes the call first and only checks whether there was budget afterward will, eventually, spend past its limit — the check arrives too late to stop the very spend it is checking. BudgetLedger.reserve() inverts that order: budget is withdrawn from the tenant's available headroom before any call happens, and only converted into an actual spend (commit) or given back (release) once the call's outcome is known. That turns "budget ran out" from a fact discovered after the spend into a condition that prevents the spend from happening at all — the difference between an alarm and a brake.

Contracts and trade-offs

This chapter's most important contract is the difference between three numbers that sound alike and are not interchangeable:

• Marginal cost: the sum of everything actually spent processing one document — every LLM call (including retries), tool/worker overhead, and the slice of storage/egress attributable to that document. Excludes human review.
• Allocated cost: marginal cost plus human review, when routing decided on it. It is "allocated" because the same reviewer-hour is, in reality, shared across many documents; this chapter allocates it 1:1 to keep the arithmetic auditable, and says so explicitly in code and docs — it never pretends to be a real shared-cost allocation.
• Customer price — which this chapter does not implement — would be a third number, built on top of the allocated one, with margin and commercial policy layered in. Confusing these three is the exact mechanism behind mandatory failure #1 below: "price per token" is even narrower than marginal cost, and treating it as "cost per useful task" is comparing the wrong metric.

The second contract is the cache key: tenantId + documentChecksum + schemaVersion + promptVersion + modelId + policyVersion, hashed end to end. Changing any one of those six fields must produce a different key — a test walks all six fields, one at a time, and confirms it (mandatory failure #2). The third contract is the budget reservation: BudgetLedger.reserve() is synchronous and has no await in its critical section, on purpose — in Node, that guarantees two concurrent reservations for the same tenant cannot both pass the headroom check at once, with no explicit lock needed. That guarantee depends on a single process's event loop; a real multi-process deployment would need the same invariant enforced by a database transaction or an atomic script in a shared store — a documented gap, not solved here.

Four mandatory failures

Failure 1 — price per token became unit economics

Symptom: the API bill drops after a model switch; nobody computes cost per accepted document before declaring victory. Cause: price per token measures the input (one call); real unit economics measures the output (a document someone could actually use). A cheaper-per-token model that misses schema more often turns every miss into an extra call — still paid for — and may, on top of that, push more documents into human review, which typically costs orders of magnitude more than the LLM call itself. Response: always measure costPerAcceptedDocument, never only costPerCall. In the scenario reproduced in verification.md, the cheaper model needed three calls (two retries) to be accepted, against one call from the more expensive model for the same document type; the LLM-only cost of the "cheap" path came out 52% higher in absolute dollars than the "expensive" path — price per token lied about the direction of the savings the moment a retry entered the account. Once human review is added to the total (because the tenant is critical), it dominates the final number for both cases, which teaches a second lesson: for this document type, the real cost lever is not which model to use, it is whether human review is required at all — a routing and eval decision, not a pricing one.

Failure 2 — cache ignores tenant or policy version

Symptom: a document from tenant B receives the cached answer for a similar document from tenant A; or a cached answer under an old policy keeps being served after the policy changed. Cause: a cache key built only from document content (say, just the checksum) ignores that the same content can legitimately need different answers for different tenants, or for different versions of a tenant's own schema/prompt/policy. Response: the exact key includes the six fields above, and the read path (CacheStore.get) requires the requester's own tenantId and throws CacheLeakageError if the stored entry belongs to a different tenant — even if an upstream bug accidentally produced the same key for two tenants. That moves the isolation guarantee from key-construction time to read time, which is where a bug actually shows up. A second namespace, semantic, exists only to reuse a tenant's compiled prompt/schema/policy across different documents of the same type — never one specific document's answer — and the two namespaces never collide, even built from the same six fields (minus the checksum, which semantic does not use).

Failure 3 — fallback with no budget check

Symptom: a well-meaning retry policy keeps trying (or falls back to a "backup" model) after the tenant's budget has already run out. Cause: retry logic and budget logic live in different parts of the code, and nothing guarantees the first never runs without passing through the second. Response: withBudget() reserves budget before each attempt — including retries — and only calls the attempt function after the reservation is accepted; if the reservation fails, the attempt never runs (a test confirms this by counting how many times the attempt function actually fired). Every attempt that does run and fails validation still had a real cost — tokens were spent even though the schema came back wrong — and that cost is committed to the ledger, never released for free; only an infrastructure failure that happened before any real spend (say, a refused connection) releases the reservation at no cost. A backpressureDelayMs() grows as a tenant's remaining budget shrinks, and reaches infinity once the budget is gone — the signal exists to slow the tenant itself down before it needs to be blocked outright, and to prevent one tenant's exhausted budget from reducing another tenant's available headroom (noisy neighbor, tested explicitly).

Failure 4 — the spreadsheet uses a stable average for explosive traffic

Symptom: a capacity plan computes the average document arrival rate over some period, concludes the system sits comfortably under capacity, and is caught off guard by an unbounded queue during a real spike. Cause: Little's Law (L = λW, the average number of items in a system equals arrival rate times average time in system) and the queueing-delay formula only hold — and only make sense — for a stable system, meaning utilization under 1. The average of three demand periods can be perfectly stable even when one of them, on its own, is not: the average never "sees" the spike, because it already blended the spike with the calm periods before any queueing math runs. Response: examples/costs/capacity-notes.md reproduces this with real numbers, not a loose claim. With four workers and a 1-second service time (pooled capacity of 4 documents/second): the low-demand scenario (0.5 doc/s) sits at utilization 0.125 with a 0.036s queue delay; the baseline scenario (1.5 doc/s) sits at utilization 0.375 with a 0.15s delay; the burst scenario (4.5 doc/s) hits utilization 1.125 — unstable, no finite average delay defined, the queue growing without bound for as long as the burst lasts. The average of the three (2.1667 doc/s) works out to utilization 0.542, perfectly stable, with a 0.295s queue delay and 2.8 documents on average in the system — exactly the number a capacity spreadsheet would compute and approve, never revealing that one of the three scenarios making up that average sits outside the region where the formula itself even applies.

What eval evidence has to do with routing

routeDocument() accepts no field that looks like a model's own reported confidence (confidenceScore, probability, and variants are rejected explicitly by assertNoInventedConfidence) — because a score a model reports about its own answer is not a calibrated probability; it is a number the model produces alongside the answer, with no statistical guarantee that 90% "confidence" corresponds to 90% real accuracy. The only evidence the policy accepts is a measured success rate, in the same { overall, n } shape chapter 5's release gate already produces per stratum — and even that evidence is only used once the sample is large enough (n >= 8 in this chapter, a teaching floor, not calibrated against real regression history). A critical document type without enough sample falls back to human even with a high measured rate and an apparently low risk tier — insufficient evidence is, on its own, grounds for a safe fallback, independent of what any single number says.

Cost, security, and reversibility

Cost and security meet in the same routing decision: sending a document to the cheap tier without sufficient evidence is not only a quality risk, it is a data-leak risk if that document type's schema has sensitive fields the cheaper model misses more often (underestimating exactly where human review matters most). Tenant isolation in the cache (failure 2) is simultaneously a cost question (a wrong key produces a wrong answer, which needs a retry) and a security question (one tenant's answer leaking to another is a confidentiality incident, not just a cache bug). Every decision in this chapter is reversible by design: a budget reservation can be released (release) at no cost if the call never happened; a cache entry can be invalidated for an entire tenant (invalidateTenant) without touching other tenants; and a routing decision is recomputed per document, never fixed permanently — there is no irreversible tenant data migration in this design, and no "economy mode" that needs to be manually switched off to return to the previous state.

None of this is free of cost itself. Running the reservation, the cache lookup, and the routing check on every document adds a small, fixed amount of bookkeeping work compared to a system that simply fires a call and hopes — this chapter does not measure that bookkeeping overhead against a real production baseline, because no production baseline exists yet for RelayOps. The honest claim is narrower: the bookkeeping is cheap enough to run entirely in-process, in plain JavaScript, with no network round trip and no extra service to operate, which is the reason it is implemented as a set of pure functions and one in-memory ledger rather than a separate policy microservice. Whether that trade-off still holds once RelayOps has a real database, a real worker pool, and real concurrent load is exactly the kind of question a future chapter's evidence — not this one's arithmetic — would need to answer.

Checklist

• Every reported cost number carries a unit, a label (PROJECTION - local arithmetic...), and distinguishes marginal from allocated.
• No model comparison is made on price-per-token alone, without counting retries and human review in the same batch.
• Every exact cache key includes all six contract fields; every read requires the requester's own tenantId.
• Every call — including retries — goes through a budget reservation before running; no "fallback" bypasses that reservation.
• No field carrying a model's own reported confidence is used as a probability in a routing decision.
• Every capacity plan states utilization and stability per scenario, never only the average across scenarios.
• node --test costs/*.test.mjs passes locally before any change to cost, cache, budget, or routing policy.

Exercises

Basic — compute cost per accepted document with retries. Verifiable criterion: run node costs/calculate.mjs --case cheap-model-more-retries and node costs/calculate.mjs --case pricier-model-fewer-retries; confirm the first's costPerAcceptedDocumentUSD is higher than the second's, and that the first's marginal.breakdown.llmUSD (three calls) is higher than the second's (one call) despite the lower per-token price. Worked solution: both commands print the full breakdown; the llmUSD difference isolates exactly the retry effect, without mixing in human review (which is identical in both cases).

Intermediate — prove cache invalidation by version and tenant. Verifiable criterion: node --test costs/budget.test.mjs would fail if buildExactCacheKey stopped changing its hash when any of the six fields varies, or if CacheStore.get stopped throwing CacheLeakageError on a forced cross-tenant read. Worked solution: the mandatory failure #2 test walks the six contract fields one at a time, building an altered key and comparing its hash against the base key; the leakage test writes an entry under tenant A's key and explicitly tries to read it as tenant B, confirming that the read path — not just the write path — enforces isolation.

Advanced — simulate burst/noisy neighbor, analyze throughput and queue delay while holding quality and quota. Verifiable criterion: estimateQueueDelay() over the three demandScenarios must report stable: true for the first two and stable: false for the burst; the mean of all three must remain stable: true, demonstrating failure 4's trap. At the same time, a BudgetLedger with two tenants must show that exhausting one tenant's budget (tenant-noisy) does not reduce the other's (tenant-quiet) headroomUSD. Worked solution: the two corresponding tests (mandatory failure #4 and noisy neighbor) run independently over the same scenario file and the same BudgetLedger, and capacity-notes.md documents the full numbers, including the caveat that the pooled-queue approximation (pooled M/M/1) underestimates the real delay of an exact M/M/c with few servers — the number here is optimistic, not conservative.

Short glossary

• Marginal cost: the direct spend of processing one document, excluding human review.
• Allocated cost: marginal cost plus human review, when applicable.
• Exact cache: reuses one specific document's answer; the key includes that document's checksum.
• Semantic cache: reuses a compiled prompt/schema/policy across different documents of the same tenant and type; never one document's answer.
• Backpressure: growing delay imposed on a tenant as its budget runs low, to protect shared capacity.
• Utilization (ρ): the ratio of arrival rate to service capacity; a system only has a finite average delay when ρ < 1.
• Little's Law: L = λW — the average number of items in a system equals arrival rate times average time in system, valid only in a stable regime.

Bridge to chapter 7

This chapter assumed routing can safely choose between the cheap tier, the standard tier, and human — but it never asked what happens when a tenant deliberately tries to work around that policy, or when a partial failure of the budget system itself (not the model) needs to be handled as an incident. Chapter 7 covers security and resilience: what changes in RelayOps when the adversary is not an ambiguous document, but an agent — human or automated — actively trying to break the tenancy, budget, and cache guarantees this chapter built.