Anderson LimaSoftware Engineer & Architect
1 de outubro de 2026
19 min de leitura
26 visualizações
The cheapest model per token can be the most expensive document per accepted result
The RelayOps team swaps the extraction model for a cheaper one per token. The monthly API bill drops — the per-call cost dashboard shows the savings the same day. Two weeks later, someone finally computes the number that should have been the first question: cost per document actually accepted. It went up. The cheaper model misses schema more often, every miss turns into another paid call, and the document — from a critical tenant — still needs human review before it can be accepted. This incident is synthetic; it does not describe a customer, a real bill, or production load, but the shape of the mistake is common enough to open the chapter with it: one metric went down, and it was not the metric that decided whether the switch was worth it.
The cause is not an arithmetic error — it is measuring the wrong thing. Price per token answers "how much did one call cost." Cost per accepted document answers "how much did the result someone could actually use cost," and that second question sums retries, human review, storage, egress, and worker overhead into the same account, without counting anything twice. The two vary independently: a model can be cheaper per token and still more expensive per accepted document, if it misses schema often enough that retries erase the savings — and once human review enters the account, it can dominate the total so completely that model choice becomes a second-order detail next to the decision of whether to route to a human at all. This chapter builds the calculator, the cache policy, the routing policy, and the per-tenant budget that make that distinction verifiable instead of assumed.
Chapter 5 closed out RelayOps's eval and observability layer: a dev/holdout dataset with no ID overlap, deterministic graders (schema, field, policy, task success), a release gate that fails a critical-stratum regression even when the overall mean improves, and a per-stratum report shaped { overall, n } — a measured success rate over a sample of known size. This chapter does not reopen or reimplement any file from that layer; it assumes RelayOps already knows how to measure whether a result is correct, and asks a different question: given that quality can be measured, which routing, which cache, and which budget keep that quality inside a predictable per-tenant spending limit — including when traffic is not the stable average of a spreadsheet? The only thing this chapter reuses from chapter 5 is the shape of stratum evidence ({ overall, n }), as the routing policy's input contract — no chapter 5 number is copied or cited as if it belonged here.
Before any calculator, the distinction fits in one function:
export function tokenCallCost({ model, inputTokens, outputTokens, cacheReadTokens = 0 }, prices) {
const rate = prices.models[model];
const perMTok = (tokens, r) => (tokens / 1_000_000) * r;
return perMTok(inputTokens, rate.input)
+ perMTok(outputTokens, rate.output)
+ perMTok(cacheReadTokens, rate.cacheRead);
}
That function answers "how much did this one call cost" — nothing else. It does not know whether the call needed a retry, does not know whether the document went to human review, does not know whether the tenant has budget for another call. Each of those is a separate layer: documentMarginalCost sums every call for a document (including retries) plus tool/worker/storage/egress overhead; documentAllocatedCost adds human review only when routing decided on it; and costPerAcceptedDocument divides a batch's total spend by the count of documents that actually reached an accepted state — never by the call count. A rejected document still costs money and is never counted as free; that is an automated test, not a sentence (a rejected document still costs money and is excluded from acceptedCount, never counted as free).
Anthropic's own engineering guide on agents states that "agentic systems often trade latency and cost for better task performance," and explicitly recommends routing "easy/common questions to smaller, cost-efficient models... and hard/unusual questions to more capable models" as the central use case for a routing workflow. RelayOps's mistake in this chapter's opening was not using a cheaper model — it was using one cheaper model for every document type, without measuring whether that specific type tolerates the cheaper model. Conventional systems' cost/capacity intuition already solved a related problem: the Google SRE book describes overload protection based on a local utilization signal, and an adaptive throttling mechanism in which each client tracks its own recent acceptance rate and self-regulates before the shared queue is affected — the same shape this chapter uses for per-tenant budget: a local "how much budget is left" signal that slows the tenant itself down before it can consume everyone else's capacity.
RelayOps gains four new, independent pieces in this chapter, separate from the eval layer and the durability layer of earlier chapters:
examples/costs/calculate.mjs) over a real, dated price fixture — Anthropic API list prices, captured 2026-09-28 — plus explicitly synthetic operational assumptions (storage, egress, worker, human review), never mixed into the same table without a label.examples/src/cache-policy.mjs) with a deterministic key scoped to tenant + document checksum + schema/prompt/model/policy version, in two namespaces that never collide: exact (the same answer for the same document) and semantic (the same compiled prompt/schema/policy, reused across different documents of the same tenant and type).examples/src/budget-policy.mjs) with atomic reservation at the execution boundary, reconciliation against the actual cost after the call, and a backpressure signal that grows as a tenant's budget shrinks — so one noisy tenant can never starve another tenant's share of the shared queue.examples/src/routing-policy.mjs) that decides between a cheap tier, a standard tier, and a human fallback using three deterministic inputs: document-type risk, available budget, and a measured success rate — never a confidence score the model reports about itself.None of these four pieces calls a real model, a database, or the network. All of them run offline, in plain Node, and the 24 tests covering them run in under a hundred milliseconds.
The price fixture feeding the calculator deliberately separates two categories of number that cannot live in the same unlabeled table: the prices for Claude Opus 5.5, Claude Sonnet 5, and Claude Haiku 4.5 — real, dated, copied from Anthropic's pricing documentation on 2026-09-28 — and the operational assumptions (storage, egress, worker compute, human-review cost per minute), which are entirely synthetic, chosen to keep the arithmetic auditable by hand, not to model a real cloud bill. Mixing the two categories into one plain table of numbers — without saying which row is a primary source and which is a tutorial assumption — is exactly the kind of mistake that makes a reader treat a teaching example as a benchmark. prices-fixture.json tags each block with a _type field and an explicit warning for that reason, and sources.md records the URL and access date next to every real price.
The order matters more than it looks. A system that makes the call first and only checks whether there was budget afterward will, eventually, spend past its limit — the check arrives too late to stop the very spend it is checking. BudgetLedger.reserve() inverts that order: budget is withdrawn from the tenant's available headroom before any call happens, and only converted into an actual spend (commit) or given back (release) once the call's outcome is known. That turns "budget ran out" from a fact discovered after the spend into a condition that prevents the spend from happening at all — the difference between an alarm and a brake.
This chapter's most important contract is the difference between three numbers that sound alike and are not interchangeable:
The second contract is the cache key: tenantId + documentChecksum + schemaVersion + promptVersion + modelId + policyVersion, hashed end to end. Changing any one of those six fields must produce a different key — a test walks all six fields, one at a time, and confirms it (mandatory failure #2). The third contract is the budget reservation: BudgetLedger.reserve() is synchronous and has no await in its critical section, on purpose — in Node, that guarantees two concurrent reservations for the same tenant cannot both pass the headroom check at once, with no explicit lock needed. That guarantee depends on a single process's event loop; a real multi-process deployment would need the same invariant enforced by a database transaction or an atomic script in a shared store — a documented gap, not solved here.
Produtos gratuitos e pagos para transformar ideias em uma base que você consegue executar.
13 produtos disponíveis
nextjs · boilerplate
Boilerplate de API em Python com FastAPI, organizado em Clean Architecture (router, service, repository e adapters por módulo). Inclui autenticação JWT com hashing bcrypt, MongoDB, testes com pytest e cobertura, Dockerfile e docker-compose para subir localmente, e documentação Swagger/OpenAPI automática. Repositório de origem privado; snapshot técnico verificado antes da publicação.
Ver produto
react · boilerplate
Base React Native para apps iOS e Android com Expo, Clean Architecture, MVVM, Firebase e módulos nativos. Referência técnica do VaiDeFut, com distribuição pública no Google Play e fluxo de publicação para iOS documentado.
Ver produto
nextjs · boilerplate
Kit gratuito e open source para começar com agentes de código em Claude Code, Codex, Gemini, Copilot e OpenCode. A licença MIT está confirmada no repositório público.
Ver produto
nextjs · boilerplate
Fundação AI-first white-label para lançar produtos SaaS com Next.js, chat, RAG, memória, tenants e documentação. O snapshot verificado documenta Supabase, não MongoDB; essa limitação está explícita. MongoDB, Redis e RabbitMQ exigem integração/configuração compatível.
Ver produto
nextjs · boilerplate
Edição avançada da base white-label e multi-tenant AI-first para acelerar produtos com Next.js, chat, RAG, memória e documentação. O snapshot verificado ainda não comprova uma camada Ultra distinta; a oferta mantém essa limitação transparente e explicita as trocas de provider, cache e filas.
Ver produto
nextjs · boilerplate
Base white-label e multi-tenant para transformar uma ideia em um SaaS AI-first com Next.js, chat, RAG, memória e segurança documentados. O snapshot verificado usa Supabase; o nome identifica o repositório de origem. Provider, cache e filas ficam explícitos; Redis e RabbitMQ são integrações configuráveis.
Ver produto
react · portfolio
Uma contribuição única para apoiar conteúdo técnico gratuito, pesquisa e a continuidade de projetos relevantes publicados por Anderson Lima. Você escolhe o valor; não é uma assinatura.
Ver produto
react · boilerplate
Starter React gratuito para criar uma suíte de testes confiável desde o primeiro commit, com Vitest, Testing Library, Playwright, Vite, TypeScript, Zustand, React Query, MirageJS e PWA. O repositório não declara uma licença SPDX; revise os termos antes de uso comercial.
Ver produto
react · portfolio
Monorepo público em React, TypeScript e Vite para explorar uma biblioteca de componentes com linguagem visual inspirada em interfaces nativas. O README atual é enxuto e não declara uma licença SPDX; trate como preview gratuito.
Ver produto
nextjs · saas
Boilerplate SaaS AI-first com Next.js, Supabase, Stripe e provider routing para começar do primeiro prompt e acelerar o caminho até um deploy real. As adaptações de IA, banco, e-mail, cache e filas ficam explícitas; Redis e RabbitMQ são integrações configuráveis.
Ver produto
react · saas
Starter kit React + TypeScript para SaaS multi-tenant com Firebase, Atomic Design, temas, PWA, telas de produto, MirageJS e chat com Gemini extensível a outros providers. Valide requisitos de produção, custos e integrações antes de publicar.
Ver produto
react · boilerplate
Starter React público com TypeScript, Zustand, React Query, Tailwind CSS, PWA e integrações de IA documentadas. O repositório não declara uma licença SPDX; revise os termos antes de uso comercial.
Ver produto
nextjs · saas
Landing page SaaS para Next.js, TypeScript e Tailwind CSS com SEO, temas, i18n, PWA e pontos de integração para auth, Stripe e IA. Uma base clara para sair da tela vazia e adaptar sua proposta.
Ver produtoContinue explorando tópicos similares
Start AI-native architecture with testable constraints. Build an offline baseline, measurable quality scenarios, and reversible decisions for RelayOps.
A RelayOps deploy cuts HTTP errors and, the same week, increases the number of wrong extractions accepted as correct. Chapter 5 builds a versioned dataset, deterministic runners, graders, a release…
A document asks, in plain text, for the agent itself to export the tenant's archive and self-promote its own role. The obvious defense — instruct the model to refuse — runs at the wrong layer.…
Checklist prático para validar segurança, custos, avaliações e observabilidade antes de colocar seu app com IA em produção.
Templates testados em produção, usados por desenvolvedores. Economize semanas de setup no seu próximo projeto.
Symptom: the API bill drops after a model switch; nobody computes cost per accepted document before declaring victory.
Cause: price per token measures the input (one call); real unit economics measures the output (a document someone could actually use). A cheaper-per-token model that misses schema more often turns every miss into an extra call — still paid for — and may, on top of that, push more documents into human review, which typically costs orders of magnitude more than the LLM call itself.
Response: always measure costPerAcceptedDocument, never only costPerCall. In the scenario reproduced in verification.md, the cheaper model needed three calls (two retries) to be accepted, against one call from the more expensive model for the same document type; the LLM-only cost of the "cheap" path came out 52% higher in absolute dollars than the "expensive" path — price per token lied about the direction of the savings the moment a retry entered the account. Once human review is added to the total (because the tenant is critical), it dominates the final number for both cases, which teaches a second lesson: for this document type, the real cost lever is not which model to use, it is whether human review is required at all — a routing and eval decision, not a pricing one.
Symptom: a document from tenant B receives the cached answer for a similar document from tenant A; or a cached answer under an old policy keeps being served after the policy changed.
Cause: a cache key built only from document content (say, just the checksum) ignores that the same content can legitimately need different answers for different tenants, or for different versions of a tenant's own schema/prompt/policy.
Response: the exact key includes the six fields above, and the read path (CacheStore.get) requires the requester's own tenantId and throws CacheLeakageError if the stored entry belongs to a different tenant — even if an upstream bug accidentally produced the same key for two tenants. That moves the isolation guarantee from key-construction time to read time, which is where a bug actually shows up. A second namespace, semantic, exists only to reuse a tenant's compiled prompt/schema/policy across different documents of the same type — never one specific document's answer — and the two namespaces never collide, even built from the same six fields (minus the checksum, which semantic does not use).
Symptom: a well-meaning retry policy keeps trying (or falls back to a "backup" model) after the tenant's budget has already run out.
Cause: retry logic and budget logic live in different parts of the code, and nothing guarantees the first never runs without passing through the second.
Response: withBudget() reserves budget before each attempt — including retries — and only calls the attempt function after the reservation is accepted; if the reservation fails, the attempt never runs (a test confirms this by counting how many times the attempt function actually fired). Every attempt that does run and fails validation still had a real cost — tokens were spent even though the schema came back wrong — and that cost is committed to the ledger, never released for free; only an infrastructure failure that happened before any real spend (say, a refused connection) releases the reservation at no cost. A backpressureDelayMs() grows as a tenant's remaining budget shrinks, and reaches infinity once the budget is gone — the signal exists to slow the tenant itself down before it needs to be blocked outright, and to prevent one tenant's exhausted budget from reducing another tenant's available headroom (noisy neighbor, tested explicitly).
Symptom: a capacity plan computes the average document arrival rate over some period, concludes the system sits comfortably under capacity, and is caught off guard by an unbounded queue during a real spike.
Cause: Little's Law (L = λW, the average number of items in a system equals arrival rate times average time in system) and the queueing-delay formula only hold — and only make sense — for a stable system, meaning utilization under 1. The average of three demand periods can be perfectly stable even when one of them, on its own, is not: the average never "sees" the spike, because it already blended the spike with the calm periods before any queueing math runs.
Response: examples/costs/capacity-notes.md reproduces this with real numbers, not a loose claim. With four workers and a 1-second service time (pooled capacity of 4 documents/second): the low-demand scenario (0.5 doc/s) sits at utilization 0.125 with a 0.036s queue delay; the baseline scenario (1.5 doc/s) sits at utilization 0.375 with a 0.15s delay; the burst scenario (4.5 doc/s) hits utilization 1.125 — unstable, no finite average delay defined, the queue growing without bound for as long as the burst lasts. The average of the three (2.1667 doc/s) works out to utilization 0.542, perfectly stable, with a 0.295s queue delay and 2.8 documents on average in the system — exactly the number a capacity spreadsheet would compute and approve, never revealing that one of the three scenarios making up that average sits outside the region where the formula itself even applies.
routeDocument() accepts no field that looks like a model's own reported confidence (confidenceScore, probability, and variants are rejected explicitly by assertNoInventedConfidence) — because a score a model reports about its own answer is not a calibrated probability; it is a number the model produces alongside the answer, with no statistical guarantee that 90% "confidence" corresponds to 90% real accuracy. The only evidence the policy accepts is a measured success rate, in the same { overall, n } shape chapter 5's release gate already produces per stratum — and even that evidence is only used once the sample is large enough (n >= 8 in this chapter, a teaching floor, not calibrated against real regression history). A critical document type without enough sample falls back to human even with a high measured rate and an apparently low risk tier — insufficient evidence is, on its own, grounds for a safe fallback, independent of what any single number says.
Cost and security meet in the same routing decision: sending a document to the cheap tier without sufficient evidence is not only a quality risk, it is a data-leak risk if that document type's schema has sensitive fields the cheaper model misses more often (underestimating exactly where human review matters most). Tenant isolation in the cache (failure 2) is simultaneously a cost question (a wrong key produces a wrong answer, which needs a retry) and a security question (one tenant's answer leaking to another is a confidentiality incident, not just a cache bug). Every decision in this chapter is reversible by design: a budget reservation can be released (release) at no cost if the call never happened; a cache entry can be invalidated for an entire tenant (invalidateTenant) without touching other tenants; and a routing decision is recomputed per document, never fixed permanently — there is no irreversible tenant data migration in this design, and no "economy mode" that needs to be manually switched off to return to the previous state.
None of this is free of cost itself. Running the reservation, the cache lookup, and the routing check on every document adds a small, fixed amount of bookkeeping work compared to a system that simply fires a call and hopes — this chapter does not measure that bookkeeping overhead against a real production baseline, because no production baseline exists yet for RelayOps. The honest claim is narrower: the bookkeeping is cheap enough to run entirely in-process, in plain JavaScript, with no network round trip and no extra service to operate, which is the reason it is implemented as a set of pure functions and one in-memory ledger rather than a separate policy microservice. Whether that trade-off still holds once RelayOps has a real database, a real worker pool, and real concurrent load is exactly the kind of question a future chapter's evidence — not this one's arithmetic — would need to answer.
PROJECTION - local arithmetic...), and distinguishes marginal from allocated.tenantId.node --test costs/*.test.mjs passes locally before any change to cost, cache, budget, or routing policy.Basic — compute cost per accepted document with retries. Verifiable criterion: run node costs/calculate.mjs --case cheap-model-more-retries and node costs/calculate.mjs --case pricier-model-fewer-retries; confirm the first's costPerAcceptedDocumentUSD is higher than the second's, and that the first's marginal.breakdown.llmUSD (three calls) is higher than the second's (one call) despite the lower per-token price. Worked solution: both commands print the full breakdown; the llmUSD difference isolates exactly the retry effect, without mixing in human review (which is identical in both cases).
Intermediate — prove cache invalidation by version and tenant. Verifiable criterion: node --test costs/budget.test.mjs would fail if buildExactCacheKey stopped changing its hash when any of the six fields varies, or if CacheStore.get stopped throwing CacheLeakageError on a forced cross-tenant read. Worked solution: the mandatory failure #2 test walks the six contract fields one at a time, building an altered key and comparing its hash against the base key; the leakage test writes an entry under tenant A's key and explicitly tries to read it as tenant B, confirming that the read path — not just the write path — enforces isolation.
Advanced — simulate burst/noisy neighbor, analyze throughput and queue delay while holding quality and quota. Verifiable criterion: estimateQueueDelay() over the three demandScenarios must report stable: true for the first two and stable: false for the burst; the mean of all three must remain stable: true, demonstrating failure 4's trap. At the same time, a BudgetLedger with two tenants must show that exhausting one tenant's budget (tenant-noisy) does not reduce the other's (tenant-quiet) headroomUSD. Worked solution: the two corresponding tests (mandatory failure #4 and noisy neighbor) run independently over the same scenario file and the same BudgetLedger, and capacity-notes.md documents the full numbers, including the caveat that the pooled-queue approximation (pooled M/M/1) underestimates the real delay of an exact M/M/c with few servers — the number here is optimistic, not conservative.
ρ < 1.L = λW — the average number of items in a system equals arrival rate times average time in system, valid only in a stable regime.This chapter assumed routing can safely choose between the cheap tier, the standard tier, and human — but it never asked what happens when a tenant deliberately tries to work around that policy, or when a partial failure of the budget system itself (not the model) needs to be handled as an incident. Chapter 7 covers security and resilience: what changes in RelayOps when the adversary is not an ambiguous document, but an agent — human or automated — actively trying to break the tenancy, budget, and cache guarantees this chapter built.