Contracts and trade-offs

This chapter separates capacity (workers and throughput) from serving order (who gets the next slot). With finite workers and FIFO, an earlier burst can delay later tenants. Adding workers changes absolute wait but not the selection rule; with enough workers for every job, queue wait would vanish. The relative effect depends on workload and needs measurement. The fair scheduler conserves jobs under leaseNext(), pinned by does not lose or duplicate a single job across a full drain. weight also has its own test (Weight is proportional, not cosmetic): an early bug made different weights alternate 1:1; the current implementation grants slots roughly in proportion to weight.

The four mandatory failures

Failure 1 — microservices as a synonym for scale

Symptom: a diagram with new services shows up before any measured bottleneck - the justification is "this is how serious systems scale," never "we measured X and Y would fix it." Cause: conflating an architectural shape (how many services, how many agents) with a measure of maturity. Neither is the same thing as solved capacity or reduced risk - and the shape is easy to draw before any evidence exists, because it requires measuring nothing. Response: examples/adrs/006-evolution.md compares alternatives and selects fair scheduling within the current process because it is reversible and improves serving order in the simulation. Service separation remains deferred because operational measurements are missing.

Failure 2 — extrapolating a stub to the cloud

Symptom: a number generated from invented stage costs gets cited as real-provider latency or the trial gets called a "cloud benchmark." Cause: the stub exists to measure the calling code's CONTRACT and BEHAVIOR - the order jobs get served in, whether any get lost, whether fairness works - not to measure how long a real model takes to respond. The two numbers look comparable because both are called "latency," but they measure completely different things. Response: DOC_TYPE_PROFILES feeds INVENTED costs to a virtual clock. "Invoice" costs less than "contract" only in this fixture. examples/portfolio/pilot-plan.md requires per-stage percentiles under real providers and traffic before identifying a RelayOps bottleneck. Queue dominance here follows from the model's instant burst.

Failure 3 — a benchmark with no environment and no denominator

Symptom: a simulation number is cited without its seed, sample window, denominator, or virtual-millisecond unit. Cause: "13x faster" without a unit or context suggests empirical improvement that this model cannot establish. Response: examples/scale/scenarios.json fixes seed, excluded initial input IDs (legacy key warmupDocs), and window. verification.md records paired results for the same jobs: acme-small (n=11) 976.003→72.433 virtual ms (13.47x); initech-small (n=11) 1027.072→77.703 (13.22x); globex-dominant (n=198) 469.164→569.398 (+21.36%). sampleSizeLimitation rules out a stable p99 claim for either small tenant. Host hardware neither explains these values nor validates real capacity.

Failure 4 — a portfolio that erases failures and trade-offs

Symptom: an architecture portfolio shows only the approved final result - the scheduler that worked, the ADR that got accepted - without recording what was tried and rejected, the bug found along the way, or the cost the accepted decision actually imposes on someone. Cause: a portfolio that only shows success optimizes for looking successful, not for being auditable - and a reader who only sees the final version cannot tell whether the decision was made well or just presented well. Response: examples/portfolio/evidence-index.md separates test-demonstrated contracts, simulated results, a local synthetic measurement, and missing pilot evidence. ADR-006 records the dominant tenant's +21.36% virtual-wait regression; the recorded local trial did not repeat it. allowedRegressionTenantIds names the fixture gate's exception. The earlier weight bug remains documented in scheduler code.

Evaluation: what 22 tests prove, and what they do not

The 22 tests in examples/scale/fairness.test.mjs check that FIFO delays a small tenant until lease grant 51 after 50 dominant jobs; fair scheduling grants it on call 2; neither scheduler loses or duplicates jobs; 2:1 weights produce roughly proportional service; and the fitness functions catch injected violations. New tests check paired jobs in the virtual model and conservation in the local synthetic CPU trial.

What these tests do NOT prove: the virtual simulation uses one process; real-trial.mjs uses worker threads for synthetic CPU work, without multi-node isolation or a real pipeline. No test measures representative contention between tenants or restores in-flight leases after restart. checkRecoveryFitness checks only counts after JSON serialization. The ADR's operational trigger remains pending.

Cost, security, and reversibility

This chapter's decision is cheap to adopt and cheap to reverse: createFairScheduler is a new data structure inside the same process, with no new deploy, no new network boundary, no production configuration key to migrate - reverting to createNaiveFifoScheduler is swapping a function call, not an infrastructure rollback. That cheap reversibility is exactly why the ADR accepts the change now and defers service separation: separation, if it ever happens, is expensive to undo (a new network boundary, a versioned contract between two deploys, in-flight jobs crossing a boundary), and that is why the ADR already writes the expand/contract plan and the rollback BEFORE separation becomes necessary, not after - the same "document before you need it under incident pressure" discipline chapter 7's rollback runbook already applied to a code revert.

On security, the scheduler decides serving order, never authorization. checkTenancyFitness checks IDs in synthetic state; authorization stays in chapter 7's authorizeToolCall. In the paired simulation, the dominant tenant pays another 100.234 virtual ms per job on average. This is a model trade-off accepted for testing fairness; real operational cost still needs a pilot.

Checklist

• No proposal to separate a service, add an agent, or create a microservice enters an ADR without a locally measured bottleneck that justifies it.
• Every performance number cited declares environment, seed, excluded initial input IDs, and sample size - never a loose percentage.
• No number measured against a stub is cited as if it were real-provider latency or production capacity.
• Every measured improvement is paired with the question "who pays for this improvement, and how much" - and the answer, if there is a regression, is recorded, not hidden.
• Every new scheduler or queue preserves throughput conservation: no job lost or duplicated, explicitly tested.
• Every architecture fitness function states, in its own comment, what passing it does NOT prove about production.
• node --test examples/scale/fairness.test.mjs passes locally before any change to the scheduler, the trial, or the fitness functions.
• Every evolution ADR includes alternatives compared with explicit operational cost, not just the chosen option.

Exercises

Basic — identify the bottleneck from the per-stage report. Verifiable criterion: run node scale/run.mjs --scenario higher-volume and confirm queueMs (p50, p95, or p99) is at least 10 times larger than any of the other four stages (parseMs, extractMs, validateMs, reviewMs) in the same sample window. Worked solution: the test 'under the higher-volume scenario, queue wait dominates...' in fairness.test.mjs already pins this comparison with a 10x threshold, over the same scenario configuration - running the command should not produce a different result than what the test already proves, since both use the same seed and the same simulation logic.

Intermediate — apply fairness and prove the small tenant makes progress. Verifiable criterion: using createFairScheduler with a dominant tenant (50 jobs enqueued first) and a small tenant (1 job enqueued after), the small tenant's job must be granted by, at most, the 2nd call to leaseNext() - never after the 51st, which is what the naive queue would produce on the identical scenario. Worked solution: the two corresponding tests in fairness.test.mjs ('a dominant tenant burst of 50 jobs delays a small tenant's 1 job until lease call #51' for the naive queue and 'the same 50-dominant-then-1-small burst gives the small tenant its lease on call #2...' for the fair one) run exactly this scenario side by side, proving both halves of the comparison against the identical job set.

Advanced — write a separation ADR with a trigger, a migration/rollback, and the additional evidence required before production. Verifiable criterion: the ADR must name at least three measurable (not subjective) conditions that, together, would justify reopening the decision to separate the service, an expand/contract plan that runs both paths (single process and new service) in parallel before either is turned off, and a rollback that preserves reconciliation of in-flight jobs. Worked solution: examples/adrs/006-evolution.md already contains this full structure - the three conditions in "Review trigger" (a defined and exceeded operational floor for queueMs, CPU/memory contention that is measured rather than assumed, more than one consented tenant with a contractual isolation requirement), the six-step expand/contract migration, and a rollback that reuses chapter 7's reconciliation discipline (findJobsNeedingReconciliation, cited as a conceptual reference, never re-imported) instead of inventing a new one.

Short glossary

• Measured bottleneck: the pipeline stage that provably, not by assumption, consumes most of the total time under a declared scenario - the only thing that authorizes an architectural change in this chapter.
• Deficit Round Robin (DRR): a fair-scheduling algorithm that grants each queue (here, each tenant) a turn in rotation, using a local credit counter so a heavier-weighted queue gets proportionally more turns without any other queue being fully starved.
• Fairness (serving order): the guarantee that the ORDER in which tenants get served does not depend only on who enqueued more or first - distinct from capacity (how much work the system processes in total).
• Architecture fitness function: a small, fast, deterministic check that runs in CI against synthetic state - never proof of production behavior, a boundary this chapter states explicitly for each of the five functions.
• Expand/contract: a migration pattern where a new path is added without removing the old one (expand), run in parallel until validated, and only then the old path is removed (contract) - which preserves rollback at any point before removal.
• Auditable portfolio: a set of artifacts where every claim points to specific evidence, classified as demonstrated, simulated, or pending pilot - never a claim with no trail back to the command that produced it.

Closing the series

This is the last chapter of the series. There is no scoped chapter 9. The next step is a consolidated review of the portfolio — six recorded ADRs, eight example sets, and the gaps declared by each chapter — followed by the still-pending pilot in examples/portfolio/pilot-plan.md. The examples demonstrate local contracts for human approval, isolation, idempotency, evaluation, budgets, and authorization. This chapter adds a comparison of tenant serving order under synthetic load. None of these artifacts demonstrates RelayOps performance with real tenants, providers, or traffic; the pilot must produce that evidence.