Versioned contracts: expand before contract

examples/contracts/document.schema.json is version 2 of the document contract. Version 1, frozen in examples/contracts/document.schema.v1.json, has no idempotency_key field; version 2 adds that field as optional. That is the "expand" half of an expand/contract migration: any v1 record becomes a valid v2 record just by filling in idempotency_key: null, without requiring any v1 producer to change first. The migrateDocumentV1toV2 function in examples/src/migrations.mjs does exactly that, and examples/compatibility.test.mjs proves three things with a real v1 fixture (examples/fixtures/document-v1.json):

1. The v1 fixture still validates against the frozen v1 schema - freezing a schema does not mean deleting it.
2. The v1 fixture, unmigrated, does not validate against v2 (the const on schema_version diverges) - there is no automatic compatibility just because most fields happen to match.
3. After migrateDocumentV1toV2, the same document validates against v2, keeps tenant_id and source_checksum identical, and gains idempotency_key: null without losing any prior information.

The "contract" half - dropping tolerance for v1, making idempotency_key required - is deliberately not implemented. Contracting before every producer emits v2 would break any caller that has not migrated yet. That is recorded as future work in examples/src/migrations.mjs and in ADR-004, not as a hidden gap.

The most important pedagogical point here is what compatibility.test.mjs tests as an expected failure: a v2 document with review_status: "closed" - a new enum value introduced without versioning the schema - is rejected by the validator. That is the scenario the prompt describes as "a mutable schema without migration": someone tweaks an extraction prompt, or a handler, to start emitting a new status, without touching the contract. If the validator silently accepted it, the system would have an undocumented behavior fork. Rejecting it is the correct behavior; documenting why is this chapter's job.

The four required failures

1. Tenant controlled by model output

Symptom: an extractor (rule-based or model-based) processes a document and the field proposal includes, among other things, a tenant_id - either because the text mentioned another organization, or because a tool call decided "this document looks like it's from Globex." If the system uses that field to decide where to write or who can see the result, authorization just became a model opinion.

Cause: confusing "the model may describe something about tenancy" with "the model may decide tenancy." An LLM granted permission to propose an action with an authorization effect is exactly the risk the OWASP GenAI Security Project describes as LLM06:2025, Excessive Agency - an LLM-based system granted a degree of autonomy greater than the task requires (OWASP, LLM Top 10, 2025 edition).

Response: extraction.mjs accepts an optional options.modelProposal (designed for when a model-based extractor exists), and if that object contains tenant_id or tenant, the field is ignored and logged in the audit trail with reason proposal_declared_tenant - never merged into the record. The corresponding test injects an adversarial proposal with tenant_id: 'globex' while the real actorContext is acme, and confirms the resulting document still has tenant_id: 'acme'.

2. A mutable schema without migration

Symptom: a new field, a new enum value, or a type change appears in the data without schema_version changing. Old and new consumers silently disagree about the format.

Cause: treating the schema as documentation rather than as an executable contract - especially easy when the change is born from a prompt tweak ("now the extractor also returns confidence") rather than a deliberate schema decision.

Response: every document writer (ingest.mjs, extraction.mjs) validates against schema-lite.mjs before writing, and any property outside the declared list is rejected (additionalProperties: false). compatibility.test.mjs proves this by injecting model_suggested_tenant as an extra property and confirming the validator rejects it - the same defense that keeps failure 1 from entering through the schema door instead of the parameter door.

3. Human review overwrites the source

Symptom: a reviewer corrects a wrong field, and the correction replaces the original text or the previous revision. After an incident, there is no way to know what the document said before the correction, or who decided what.

Cause: treating "review" as editing a single mutable record, instead of creating a new auditable revision on top of an immutable previous one.

Response: store.putDocument always appends to history (docs.get(documentId).push(...)); it never overwrites a previous entry. review.reprocess creates a new revision with larger revision and extraction_version, but inherits source_checksum from the original document - the source text is written once, at ingest, and never recomputed. The "reprocessing preserves checksum and prior revision" test confirms all three properties at once: reprocessed.source_checksum === uploaded.source_checksum, and store.history('acme', 'INV-2001') still contains the review_status: 'rejected' entry from the previous revision.

4. A monolith called "modular" despite circular dependencies

Symptom: the team describes the system as "modular architecture" because the code lives in separate files with domain names (ingest.js, review.js...), but extraction imports a function from review to "signal that it's done," and review imports something from extraction to "reprocess automatically." Neither can be understood, tested, or replaced in isolation, because loading one loads the other.

Cause: adding a direct call between domain modules the first time it "would be convenient" for one to call the other, without checking whether that closes a cycle in the dependency graph.

Response: this chapter solved the same coordination problem - extraction needs to advance state, review needs to send it back for reprocessing - without a cycle, by extracting the transition table into state.mjs, a leaf module both import. Neither extraction.mjs nor review.mjs needs to know the other exists. The proof that this holds is not a design claim; it is the static grep reproduced above, which any reader can run again.

What the tests prove and what they do not

The 16 local tests (node --test examples/*.test.mjs, see verification.md for full output) cover: the full positive path (ingest → extract → approve); the opening synthetic incident, with two tenants using the same document_id; cross-tenant read, extraction, and approve/reject attempts denied and audited; a forged tenant_id injected into an extraction proposal; rejecting approved -> extracted without a new version (the basic exercise); preserving checksum and prior revision on reprocessing; a checksum conflict when re-ingesting the same document_id with different content; validating the v1 fixture against its own v1 schema; an expected failure validating v1 directly against v2; a lossless v1-to-v2 migration; rejecting a migration attempt on a document that already claims v2; accepting an explicit idempotency_key; and the two schema failures described above (an unversioned enum, an unknown property).

What this does not prove: no line of PostgreSQL ran in this session. store.mjs correctly partitions within a single Node process using a single Map instance - that is not the same as proving that a row-level security policy secures a shared table under concurrent connections, with a connection pool that could, by mistake, run as the table owner. The next section and the advanced exercise spell out exactly what would be missing. We also did not measure extraction quality: parseInvoiceText is the same class of rule-based parser as chapter 1's baseline, tested here only for contract behavior (does it produce a valid record, does it record provenance), not for accuracy against a real distribution of invoices.

Cost, security, and reversibility

The AWS Well-Architected Framework frames this as questions, not a certification checklist: what security decision does this design assume, and what does it cost to operate? In this chapter, the answer is recorded in ADR-004: keep tenancy pooled (one logical partition per tenant_id, not one schema or database per tenant) and enforce isolation in application code, deferring proof on shared infrastructure.

That carries an explicit reversibility cost. Migrating a specific tenant to its own schema or database later is a data extract-and-reload problem, not a redesign, because tenant_id is already a first-class key from the start. The reverse path - merging tenants that have already diverged back into shared silos - is harder. That favors deferring the silo decision until a real trigger exists: a consented pilot with a contractual data-residency requirement, or a load measurement (not done yet) showing one tenant degrading another's latency in a shared table.

On database security specifically, PostgreSQL's Row Security Policies documentation - reopened in this session, reflecting the "current" docs (PostgreSQL 18 at the time of access) - records three details that change what "proof of isolation" means in practice: ENABLE ROW LEVEL SECURITY must be switched on per table, or the policy simply does not run; superusers and the table owner bypass RLS by default, so a connection pool running as the owner would pass any cross-tenant test while proving nothing; and referential-integrity checks (unique keys, foreign keys) bypass RLS by design, which the documentation itself calls a potential secondary information-leak channel. None of these conditions were tested here - they are documented in ADR-004 as what is missing before any production isolation claim.

Checklist before moving on

• Every operation's tenant_id comes from a caller-authenticated context, never from a field inside the document or a model proposal.
• A cross-tenant read, write, or transition returns the same result as a nonexistent resource - no existence leak.
• Every state transition goes through a single, independently testable table that no domain module reimplements on its own.
• Reprocessing a document always creates a new revision with the original checksum preserved; it never overwrites the previous revision.
• A schema has an explicit version; changing its shape without bumping the version is treated as a bug, not as flexibility.
• The import graph between domain modules is acyclic - and that is verified, not assumed.
• Every tenancy decision (pooled vs. silo) is recorded as an ADR with an explicit review trigger, not as an implicit choice baked into the first implementation.

Exercises

Basic. Verifiable criterion: calling review.reprocess(store, auditLog, reviewer, 'INV-2001', { nextRevision: currentRevision, reason: '...' }) from the approved state, using the same current revision (without incrementing it), must throw a TransitionError whose message contains "new revision," and the audit log must contain an entry with result: 'denied' and reason: 'missing_versioning'. Commented solution: this is already implemented and tested in examples/tenancy.test.mjs, in the test "basic exercise: approved -> extracted is rejected without a new revision/extraction_version" - the check lives in assertReprocessVersioning (examples/src/state.mjs), called by review.reprocess before any write to the store.

Intermediate. Verifiable criterion: add a v2 schema for the document contract without invalidating the existing v1 fixture. Commented solution: already solved by examples/contracts/document.schema.v1.json (frozen) + examples/contracts/document.schema.json (v2, with optional idempotency_key) + migrateDocumentV1toV2 in examples/src/migrations.mjs. The "intermediate exercise" test in examples/compatibility.test.mjs runs exactly that migration over examples/fixtures/document-v1.json and confirms nothing is lost. Demonstrative snippet (not a new command - the one already run in this session): node --test examples/compatibility.test.mjs.

Advanced. Verifiable criterion: write a test that attempts to read, by a known document_id, a document belonging to another tenant, and describe - without implementing - the additional requirements to prove the same property against a real PostgreSQL instance. Commented solution: the test already exists ('cross-tenant read of a known document_id returns nothing, same as a nonexistent id', in examples/tenancy.test.mjs) and proves the property inside a single Node process using one Map instance. The additional requirements, described in examples/adrs/004-tenancy.md and not executed in this session, are: (1) ALTER TABLE ... ENABLE ROW LEVEL SECURITY explicitly turned on for the table; (2) a policy such as USING (tenant_id = current_setting('app.tenant_id')::text) - demonstrative snippet, not executed:

-- Demonstrative. Not executed in this session; requires a real Postgres instance and a dedicated role.
ALTER TABLE documents ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON documents
  USING (tenant_id = current_setting('app.tenant_id')::text);

(3) a test role that is neither the table owner nor has BYPASSRLS, since the owner and superuser bypass RLS by default; (4) two separate transactions with different SET LOCAL app.tenant_id = '...' values, testing cross-tenant SELECT/UPDATE/DELETE and expecting zero rows or a permission error; (5) attention to the secondary channel PostgreSQL itself documents: unique/foreign-key violations bypass RLS and could, in principle, reveal that a row exists in another partition.

Short glossary

• Pooled tenancy: multiple organizations share the same tables/structures, distinguished by a tenant column/key.
• Silo tenancy: each organization has its own schema or database.
• Expand/contract: a two-phase schema migration pattern - first add fields in a backward-compatible way (expand), then remove tolerance for the old version only once every producer has migrated (contract).
• RLS (row-level security): a PostgreSQL mechanism that filters visible/writable rows by policy, evaluated per row, and that does not apply to superusers or table owners by default.
• Excessive agency: OWASP GenAI Security Project item LLM06:2025 - an LLM-based system granted a degree of autonomy greater than the task requires, especially dangerous when that autonomy can affect authorization.

Bridge to chapter 3

This chapter proves isolation and contract behavior inside a single Node process. What remains explicitly open for the rest of the series: a real integration with a transactional database (ADR-004 already lists the RLS requirements that need to be satisfied, not just stated); a retention and PII-redaction policy more complete than the current redactFields, which today only masks string length; and the first point where a real model-based extractor - not the parseInvoiceText rule - enters the system without inheriting any authority over tenancy or state transitions. None of those three items is implemented here; they are recorded as the next experiment, not as a result.