From document to experiment: toy and production

The first increment need not be a chatbot. To learn whether the brief captures a decision, synthetic request fixtures, a described manual flow, optional suggestion examples, and a deterministic checker are enough. This avoids integration work before anyone knows the problem exists and keeps the non-AI alternative eligible to win.

In the offline toy, examples/check-brief.mjs reads the document and fails if it lacks stakeholder roles, a falsifiable hypothesis, an owner, a non-AI alternative, missing evidence, or rollback. An intentionally invalid fixture gives the checker a meaningful negative case. It does not measure service time. That gap is explicit. Later chapters may use TypeScript, a pinned Node.js LTS release, PostgreSQL, and a simple web UI as a teaching stack. Picking those tools now would not validate the opportunity.

An authorized pilot would need a stronger contract. Deterministic backend rules would control tenant boundaries, authorization, SLA, states, and assignment. AI would only suggest a category and summary from permitted text. An agent would see the suggestion, correct or reject it, and confirm an action. The backend would validate the resulting command regardless of what the UI or model says. There would be no automatic external reply, real billing, or deletion of external data. Personal data would be minimized, logs redacted, and retention and access approved before ingestion. A prompt cannot enforce tenant isolation.

Every effect needs an owner, permission, evidence, and a reversal path. A category change requires the agent's permission, an auditable event without sensitive request text, and a way to restore the previous value where feasible. A real pilot requires organizational consent, an operational owner, and an off switch. Even a “suggestions only” interface has an effect: it can bias an agent toward a wrong decision. The agent must be able to spot uncertainty and keep using the manual route.

We still have no evidence that a model-based architecture is needed. Compare improved fields and a simple taxonomy against the observed current process first. If most delay comes from unavailable staff, better classification will not clear the queue. If unclear routing policy is the cause, a rule update may beat a model at lower cost and risk. The team must keep that possibility open until it sees the work being done.

Negotiate autonomy in the organization you have

A ticket team often receives a specified solution; a product team is more likely to receive a problem and negotiate one. Real organizations mix both modes. You can do useful discovery in a ticket team without claiming permission to change policy, contact customers, or spend budget.

In a ticket team, propose a bounded conversation with the requester and a support lead: “Can I use a defined window to map the workflow and compare a non-AI option before estimating the chatbot? I need supervised access to agents, no sensitive data, and feedback on Friday. I will bring a brief and open questions; priority remains your decision.” Record investigation time, authorized people, permitted data, spending limit, decisions you can make alone, and who approves a pilot. If user access is denied, use documented workflow and mark the missing evidence. Imagination does not become an interview.

In a product team, PM, designer, engineer, and support lead can shape the hypothesis and guardrails together. Engineering can own feasibility and reversible implementation. Design can study how an agent detects and fixes mistakes. PM can maintain priority and strategic fit. Support can describe operational load and escalation. A named pilot owner makes a go/no-go call after hearing those views. Collaboration should not spread responsibility so thinly that no one can stop a harmful experiment. In either setting, agree on a feedback cadence: review the brief before building, observe each round, and record a decision to continue, change, or stop.

A useful autonomy agreement answers seven questions: what problem may we investigate; which people and data may we access; what may we decide; what requires approval; which effects are forbidden; who resolves conflict; when do we revisit the agreement? It protects the engineer and the team. “Act like an owner” without these answers often increases accountability without increasing the ability to act.

Four failure modes to recognize early

1. A new title with no decision rights

Symptom: someone is called a Product Engineer but sees work only after prioritization and cannot talk with support. Cause: the label changed while access, decision rights, and responsibility did not. Response: ask for a small discovery window, an operational counterpart, and an explicit decision boundary. Record any refusal and the available alternatives. If the organization does not authorize contact, work within the real boundary and state what remains unverified. The user otherwise gets an interface built on assumptions no one can correct.

2. Choosing AI before locating the problem

Symptom: the team compares models and token prices before seeing why requests wait. Cause: a technology demo has become the problem statement. Response: map the flow, establish a baseline, try a form or rules, and state the condition under which AI would outperform them. The delay might come from staffing or queue policy; a classifier cannot solve either by itself. For the end customer, irrelevant automation may make service less legible.

3. Treating a feature as impact

Symptom: “100% of tickets classified” becomes a success claim while reassignments rise. Cause: output is easy to count; correctness and harm take observation. Response: report technical validity, adoption, and outcome separately; define denominator and guardrails before the pilot; review an error sample. A prototype can work technically and still fail to improve service. That is a useful finding and may prevent an unnecessary recurring cost.

4. Making decisions without the people who bear them

Symptom: engineering changes categories or workflow without consulting agents, a support manager, design, privacy, or the people handling complaints. Cause: “ownership” has been mistaken for the right to take over another person's decision. Response: map affected people, seek review before creating effects, name the final owner, and define escalation. A bad suggestion reaches agent and customer; excessive retention raises organizational risk; a new fee affects the buyer. Consultation will not remove disagreement, but it will reveal the trade-off.

Evaluate without manufacturing a success story

Before measuring benefit, define the unit (eligible ticket), denominator (every eligible request arriving in the channel and window), observation cutoff (a fixed interval after each arrival), method (arrival, first assignment, sample-checked confirmation or correction), segments (type and complexity), and limits (seasonality, agent differences, missing records). Count tickets without a correct assignment at the cutoff and report their share of all eligible tickets. The median of completed times includes only tickets with an observed event and states how many entered that calculation. Record agent review effort and ignored suggestions. Without both views, a lower median may only indicate easier requests.

Where feasible, compare the observed current flow, a non-AI alternative, and AI suggestions with human review. A small sequential comparison can teach a team something, but it cannot control every difference. Do not call correlation causation. Decide in advance what would stop the trial: significant assignment errors, data exposure, loss of agent trust, unacceptable total cost, or no improvement after a suitable window. The brief's 20% over two weeks is a provisional teaching target, not an approved decision threshold; operational limits require a baseline and local agreement.

For the offline brief, the success criterion is narrower: required files exist, the valid fixture passes, the incomplete fixture fails, and the document does not present a hypothesis as observed evidence. That verifies decision discipline, not commercial value. The next meaningful step is to observe a few authorized interactions and revise the brief, including if the original problem proves wrong.

Cost, safety, and reversibility belong in the hypothesis

A suggestion costs more than a model call. It creates integration, processing, human review, monitoring, incident response, training, and opportunity costs. The non-AI alternative also costs maintenance, but may be cheaper and more predictable. Estimate from local data; do not publish return on investment derived from a fixture. A continue decision should account for the customer's service, not only a price per request.

Support tickets can contain names, contracts, passwords, or sensitive business details. Even a proof of concept should use synthetic data until there is an authorized basis for handling real requests. A real pilot needs data minimization, backend tenant and role enforcement, redacted logs, retention, consent, and a deletion procedure owned by the appropriate team. This chapter claims no certification or legal compliance. Legal and privacy review belongs to qualified owners. If an external provider is considered, document data flow and terms before sending anything.

Reversibility is a feature: keep the manual flow, enable suggestions for a limited group, disable them without breaking request states, retain only the minimum evidence needed to investigate errors, and avoid a migration that forces every agent through a model. The cost of going back belongs in the same discussion that approves moving forward. A feature that can only be disabled through a risky deployment weakens operational control when it is most needed.

Starting checklist

• Did I describe the problem as behavior and consequence before naming a solution?
• Did I separate user, potential buyer, and affected person with distinct interests and risks?
• Is the current alternative observed or explicitly marked as an assumption?
• Have I compared a non-AI option and stated when AI might earn its place?
• Are outcome, unit, denominator, window, method, and guardrails defined?
• Does the missing-evidence list keep fixtures separate from interviews?
• Are owner, autonomy limits, authorized access, and feedback cadence recorded?
• Does each possible effect have permission, evidence, and a rollback path?
• Have I checked the actual valid and invalid checker results in verification.md?

Three exercises for your first week

Basic — rewrite a ticket. Turn “add a chatbot” into one observable problem statement and one falsifiable hypothesis. Include a user, a measure, and a guardrail. Pass condition: a support colleague can describe an observation that would refute the proposal. Worked response: “Assignment may take too long because agents must interpret free text; compare median time to correct assignment and reassignments with the current flow.” The baseline is still missing. That is an honest open question.

Intermediate — negotiate an investigation. Draft a request for 30 minutes of authorized observation, a brief-review meeting, and a list of decisions reserved for the owner. Pass condition: the recipient knows which people, data, and time are involved and can approve discovery without approving a pilot. Worked response: ask for supervised access to a queue without sensitive data, report observations by Friday, and leave SLA changes, spending, and deployment with existing owners. If access is refused, record the gap and review documentation.

Advanced — design a comparison. Specify events, denominator, three conditions, and a stop rule for DeskPilot. Pass condition: unassigned requests remain in the denominator, error severity is captured, complexity differences are reported, and no fictional percentage appears. Worked response: record arrival, assignment, confirmation, and reassignment; compare observed baseline, improved forms/rules, and reviewed suggestions in agreed windows; segment by request type; stop on data exposure or a severe error. Suggestion acceptance is diagnostic, not the outcome.

Glossary and the next step

An outcome is an observed change for users or the business, including harmful side effects. An output is what the team shipped. A baseline is a measured description of the existing flow, with a method and window. A guardrail prevents improvement in one measure from hiding harm elsewhere. A tenant is one organization's data boundary in a shared application. A rollback is a procedure to restore a safe state, not a hope that nothing goes wrong.

The first useful result here is a decision someone can reject with evidence. DeskPilot has moved from zero to a candidate problem, three distinct stakeholders, a possible comparison, and written boundaries. Customers, interviews, and a baseline remain missing. In the planned chapter 02 of this series, we will investigate who experiences the delay, how they work now, and whether this scope warrants a product. Until then, the brief remains v0.

Sources and their limits

• PostHog, What is a product engineer?. Living handbook, accessed September 27, 2026. Describes one company's practice; does not establish a universal standard or measured productivity gain.
• Marty Cagan / SVPG, Product vs Feature Teams, August 29, 2019; accessed September 27, 2026. Author's framework; terminology expressly described as nonstandard.
• DORA, State of AI-assisted Software Development 2025, 2025; accessed September 27, 2026. Research context, without causal inference about individual careers or DeskPilot.