The fixture: dirty data on purpose

examples/analytics/events.synthetic.jsonl has 93 lines, 14 tickets, 3 deliberately unbalanced tenants (t_apex with 9 tickets, t_beta with 3, t_gamma with 2), and four real data problems built in on purpose:

• Duplicate: T-1008's ticket.created appears twice, same event_id, simulating an at-least-once producer redelivery. SELECT DISTINCT ON (event_id) ... ORDER BY event_id, ingested_at resolves it — verified in verification.md (2 raw rows, 1 after dedup).
• Out of order: T-1007's ticket.first_response occurred at 09:30 but only reached the pipeline at 15:35 — after that same ticket's own ticket.resolved, which occurred at 15:00 and arrived at 15:05. Every query in this package sorts by occurred_at, never by ingested_at and never by file order; the computed time-to-first-response is 90 minutes (09:30 − 08:00), not a number inflated by the late arrival.
• Reopened ticket: T-1002 was assigned, resolved, reopened by the customer, assigned again, and resolved again. The primary metric uses only the first assignment (45 minutes, within high's SLA); the second assignment belongs to rework analysis, not initial attribution — counting both would double one attribution opportunity into two.
• Unbalanced tenants: t_gamma, with 2 tickets over 2 days, does not have enough traffic for any trustworthy quantitative read — and is treated that way, never forced onto the same statistical ruler as t_apex.

examples/analytics/metrics.sql (11 Postgres views, run against postgres:17.6 in a series-ai-pe06-pg container, removed afterward) and examples/analytics/check-metrics.ts (13 native Node tests, an independent implementation that never touches a database) agree number for number on this fixture — see verification.md for the full output of both.

Contracts and trade-offs

• Accepted: two independent implementations (SQL and TypeScript) of the same dedup/ordering/denominator logic, instead of one implementation plus a test that re-runs that same implementation — a bug shared between the query and its check would go uncaught otherwise. That is exactly how a real bug was caught this session: the first version of suggestion_downstream_quality classified a ticket with no CSAT as clean_close (through a stray OR csat_score IS NULL), and the sum of the three buckets (10) did not match the total accepted-or-edited suggestions (9) — the error surfaced because the two implementations disagreed, not because someone reviewed the logic by hand.
• Accepted: eligibility correction as an additive event (ticket.eligibility_corrected), never an UPDATE on the original event — the same append-only principle chapter 03 already applies to the audit log, extended to the analytics stream.
• Accepted: a guardrail with veto power over the primary metric, not a second decorative number sitting next to it — see the evaluation section.
• Rejected: computing an MDE (minimum detectable effect) or statistical power for the t_apex/t_beta pair in this fixture. With one tenant per arm, there is no between-arm variance to estimate at all; any number here would be a textbook formula applied to a place where it does not hold.
• Rejected: a week-over-week retention dashboard. The fixture covers one full week plus two days of a second week for a single new tenant — fewer than three cohorts, not enough to say anything about retention without inventing a number.

The four failures this chapter has to resolve

Failure 1 — treating acceptance as the final success metric

Symptom: a product slide says "80% acceptance" and proposes expanding rollout based on that number alone. Cause: acceptance measures a one-second decision, before any consequence of the ticket has appeared — it never sees a reopen, a customer score, or rework. Fix: suggestion_downstream_quality classifies every accepted/edited suggestion into three exhaustive, non-overlapping buckets — clean_close (7), rework (1), quality_unknown (1) — and the dashboard shows both rates side by side: 81.8% acceptance, 63.6% clean close (measured against every suggestion shown, not just the accepted ones). The gap between the two numbers is the size of the problem "acceptance" alone was hiding.

Failure 2 — excluding failures from the denominator

Symptom: a query reports 100% suggestion acceptance. Cause: the query's denominator is "accepted or edited suggestions" — the same set as the numerator. By construction that fraction is always 100%, no matter how many suggestions agents actually rejected. Fix: suggestion_acceptance_WRONG in metrics.sql exists only to show this trap next to suggestion_funnel, which uses the correct denominator — every suggestion shown, including the 2 rejected ones. 9/9 = 100% (wrong) against 9/11 = 81.8% (correct, and still not sufficient on its own — see Failure 1).

Failure 3 — mixing cohort with cumulative total

Symptom: a single number, "61.5% since inception," on a dashboard that should show trend. Cause: summing tickets from weeks with wildly different volume (11 eligible in week 1, 2 in week 2) into one pool hides both the real difference between weeks and the fact that week 2 is too small to trust on its own. Fix: primary_metric_by_week reports 63.6% (week 1, 11 tickets) and 50.0% (week 2, only 2 tickets) separately; primary_metric_cumulative_WRONG exists only to show the blended 61.5% next to them, named as the pattern to avoid, not an acceptable alternative.

Failure 4 — treating a before/after comparison as causal

Symptom: "the rate dropped from 63.6% to 50% between weeks — the AI got worse." Cause: week 2 has a single new tenant (t_gamma), an agent who had never worked before (agent_5, activated in that same week), and only 2 tickets — any one of those, alone, explains a 13.6-point drop without the AI having changed at all. There is no randomization between the two weeks: they are different periods, different tenants, no control. Fix: no causal claim appears anywhere in this package. examples/experiment-plan.md designs the correct experiment — tenant-level randomization, with the unit justified by interference risk between agents sharing a queue — and states plainly, in its own power section, that two tenants (one per arm) support no p-value at all. "We don't have enough tenants yet" is the correct answer this session, not a placeholder for a number to be filled in later without more tenants.

Evaluation: what the numbers prove, and what they do not

This session's primary metric — 63.6% of week-1 eligible tickets correctly attributed within SLA — proves the assignment queue works for a little over 6 in 10 eligible tickets in this synthetic fixture. It does not prove the AI suggestion caused that number: T-1009, correctly assigned within SLA, never received a suggestion at all (low priority, outside the rollout gate), and T-1003, which received an accepted suggestion, was assigned to the wrong agent and outside SLA anyway — a severe-error case (accepted suggestion, incorrect assignment) that no aggregate rate would surface on its own without a query built specifically to find it.

The reopen-after-accept guardrail — 0% on this fixture (0 of 7 accepted suggestions led to a reopen) — is clean, but on a small sample: a single additional case would move that number disproportionately. The dataset's one reopened ticket (T-1002) followed an edited suggestion, not an accepted one, so it does not enter this specific guardrail's numerator — a documented design decision, not a way of hiding the reopen from the count: it still shows up in the overall per-tenant reopen rate (time_to_stage_by_tenant, 12.5% for t_apex).

Agent activation — 83.3%, 5 of 6 agents who ever owned a ticket reviewed at least one suggestion — has a structural explanation, not a mysterious one: agent_6 only ever worked the dataset's single low-priority ticket, which the rollout itself excludes from receiving a suggestion. That is not a tool-resistant agent; it is an agent the tool does not reach yet.

What this chapter does not measure, and says so instead of estimating: real business impact, tenant retention, or any causal effect of showing the suggestion versus not showing it. examples/experiment-plan.md is the design for measuring that once enough tenants exist — not a substitute for the measurement itself.

Cost, security, and reversibility

Cost: examples/unit-economics.csv assigns each of the 12 tasks (11 shown suggestions plus one fallback attempt) an inference cost (Claude Haiku 4.5, $1/$5 per million input/output tokens, re-verified at claude.com/pricing on 2026-09-28), an estimated infrastructure cost, human review minutes at a scenario rate ($35/hour, fully loaded, a declared teaching assumption) and rework minutes for the one reopened ticket. The result: an average of $1.24 per task, but $2.13 per useful task (clean close) — because the single reworked ticket (T-1002, $10.21, dominated by 15 extra minutes on a second handling cycle) alone cost more than the seven clean-close tasks combined ($3.22). Time saved on the first pass does not mean lower total spend when a single instance of rework concentrates most of the cost — the opposite of what "80% acceptance" would suggest to anyone reading only that number.

Security: no analytics event carries ticket text, customer name, or any PII — payloads are closed to category, action, numeric score, and identifiers. The eligibility correction (ticket.eligibility_corrected) preserves the original event instead of overwriting it, keeping the audit trail intact even when an initial decision was wrong. Tenant isolation was not re-evaluated in this chapter — inherited unchanged from chapters 03–05, where cross_tenant_denied is already tested at the domain layer.

Reversibility: the reopen-after-accept guardrail, by design, can revert a suggestion from "shown" back to "shadow only" without deleting any data — events keep being generated and recorded, they simply stop reaching an agent's screen. No decision in this chapter is automatic: the weekly reading routine (examples/experiment-plan.md, section 7) always ends in "continue," "limit rollout," or "retire the AI," recorded with a reason, never silently.

Exit checklist for this chapter

• Every dashboard metric has its denominator and window written next to it, never implied.
• An acceptance rate never appears alone — always next to a downstream quality rate (reopen, satisfaction).
• No query uses the same set as both numerator and denominator.
• Cohort (by creation week) and cumulative total never appear as a single number — always side by side.
• No before/after comparison is presented as causal proof without randomization.
• The severe-error/reopen guardrail can veto the primary metric, documented in writing.
• Unit cost includes human review and rework, not just inference.
• "Inconclusive" is recorded as a valid outcome when traffic does not support a real read.

Exercises

Basic. Using time_to_stage_by_tenant, compute the p50 time to first response for t_beta only. Acceptance criterion: your query uses WHERE tenant_id = 't_beta' against the same view, without duplicating ticket_facts's logic. Commented solution: SELECT p50_minutes_to_first_response FROM time_to_stage_by_tenant WHERE tenant_id = 't_beta'; returns 107.5 — the view already groups by tenant, so filtering is enough; rewriting the logic from scratch would risk drifting from the version already tested in check-metrics.ts.

Intermediate. Write a contamination check: no assignee_id should ever appear under more than one tenant_id in ticket.assigned. Acceptance criterion: the query runs against events_dedup and returns zero rows on this fixture. Commented solution:

SELECT payload->>'assignee_id' AS agent_id, count(DISTINCT tenant_id) AS tenants
FROM events_dedup
WHERE type = 'ticket.assigned'
GROUP BY 1
HAVING count(DISTINCT tenant_id) > 1;

Zero rows — every agent in this fixture is scoped to a single tenant, the same check examples/experiment-plan.md (section 3) describes as a precondition for a contamination-free tenant-level experiment.

Advanced. Write a query or a check-metrics.ts test that finds a "severe error": an accepted (not edited) suggestion whose ticket was assigned incorrectly (correct_assignment = false). Acceptance criterion: the query identifies exactly T-1003 in this fixture, and you explain why an aggregate acceptance rate alone would not have surfaced it. Commented solution:

SELECT ticket_id FROM ticket_facts
WHERE suggestion_category IS NOT NULL
  AND review_action = 'accepted'
  AND correct_assignment = false;
-- T-1003

Acceptance rate treats T-1003 as a success (the suggestion was accepted); clean-close rate excludes it too, but for the wrong reason (missing CSAT, not the actual incorrect assignment). Only a query built specifically for the accepted-plus-wrong-assignment pair reveals the severe error — which is why examples/experiment-plan.md treats that case individually, never diluted into an average.

Glossary

• Denominator: the total a rate is computed against; changing the denominator without saying so changes what the metric means without changing the number attached to it.
• Guardrail: a secondary metric with veto power over the primary one — a gain on the primary metric authorizes nothing if the guardrail got worse.
• MDE (minimum detectable effect): the smallest effect an experiment can reliably detect given its sample size; only computable once there is enough between-unit variance to estimate.
• SRM (sample ratio mismatch): when the real proportion of units landing in each experiment arm diverges from the intended split, invalidating the read even if the metric looks good.
• Clean close: in this chapter, a ticket resolved, never reopened, with a real customer satisfaction score of 3 or higher — the bar "accepted suggestion" alone does not guarantee.

Bridge to chapter 07

This chapter hands chapter 07 exactly three things, with no dependence on session memory: the metric tree with a denominator and window declared at every level; the dedup/ordering/correction pipeline any future event (including suggestion.reviewed, defined in the tracking plan but not yet emitted by the code) needs to respect; and an experiment design ready to run once enough tenants exist to produce a real MDE, not a formula applied to a one-tenant-per-arm sample. See ../product-designer-serie-07/prompt.md for what comes next.