Testing without charging: concierge and willingness to pay

The pilot plan defines two tests that happen before any production code. In Opportunity A's concierge test, a senior agent manually reviews a sample of ambiguous tickets for one week and writes down, by hand, which category they would have suggested — without building the automation. This measures whether the idea would work in hindsight, without the cost or risk of exposing a model to live traffic.

The willingness-to-pay test charges nothing. It is a structured conversation with the support manager, presenting the concierge test's result and asking, in this order: what budget exists today for this kind of delay; what would need to be true to approve a paid pilot; who else needs to sign off. A specific answer about budget and approver counts as a willingness-to-pay signal. A line like INT-006's — "sounds like that would solve a lot of my problems" — with no mention of budget, approver, or condition, counts as praise, not demand. The plan records that distinction explicitly, because it is the easiest mistake to make right after a good conversation.

The final gate weighs three dimensions, drawing on the risk split used by outcome-oriented teams as described by Marty Cagan/SVPG, accessed 2026-09-28: viability (does the manager have real budget and authority, or does this depend on an unmapped external approval), usability (did the senior agent agree with the retrospective suggestion for most of the sample), and data access (can the team authorize anonymized historical tickets and a CRM with reliable SLA data). Advance requires all three favorable. Change segment applies when viability or data access fails on this team but the same problem shows up with comparable force on another queue available for discovery. Abandon applies when usability fails and no simpler solution resolves the ambiguity — in that case, chapter 01's decision log and this chapter's method remain valid as a case study, even if the specific product changes direction. None of these three paths is a process failure; they are the process working.

Contracts between roles and AI

The buyer is not the user, and confusing the two is the most expensive mistake in this chapter. The support manager (INT-005, INT-006) decides budget and approval; the agent (INT-001 through INT-004) decides, day to day, whether to accept or reject a suggestion. A team that only interviews managers "because they're the ones who buy" will never discover that the biggest source of delay is an extra tab the agent opens to check the customer's plan — and a team that only interviews agents "because they're the ones who use it" will never discover that the manager would not approve budget without a measured reduction against a baseline. Both interviews are necessary, and they answer different questions.

AI-assisted synthesis comes in after the six interviews, not instead of them. An automated summary generated from the interview JSON might write something like "agents report occasional reclassification." That sentence is technically true and editorially dangerous: it erases the difference between "1 in 30" and "4-5 in 20," and INT-003's link between reclassification and trainee onboarding, which is the most actionable piece of the interview. The contract adopted here is simple: any AI-generated synthesis must point back to the source interview's id, and a disagreement between two sources (like INT-003/INT-004, marked with a mutual contradicts field in interviews.synthetic.json) must remain visible in the final document, not resolved by averaging or automatic consensus. The checker in evidence-check.mjs rejects the opportunity tree if no contradiction pair survives — a mechanical way to stop editorial convenience from erasing the research's most interesting signal.

Four failures worth naming

1. Leading interviews

Symptom: the question already describes the solution — "would you use an AI suggestion to categorize tickets faster?" — before the agent describes the episode itself. Cause: the interviewer wants validation, not discovery; the question asks for a polite opinion about an already-formed idea. Response: ask about the episode (trigger, flow, workaround, frequency, consequence) before any mention of AI; if the interviewee brings up AI on their own, that is data, but it should never be prompted. This chapter's guide explicitly bans questions like "would that be useful?" for the same reason: almost anyone polite will say yes.

2. Buyer mistaken for user

Symptom: the brief lists "customer: support manager" and treats the manager's answers as evidence that agents will adopt the suggestion. Cause: it is easier to book a meeting with whoever approves budget than with whoever does the work every day; commercial language around "the customer" hides the fact that at least two roles have distinct interests. Response: keep the user, buyer, and affected-person columns separate (inherited from chapter 01), and check, for every claim in the v1 brief, which role it actually comes from. A sentence like "the customer would love this feature" should answer: which customer — the one who uses it, the one who pays, or the one affected by the outcome?

3. AI synthesis erasing contradictions

Symptom: an automatically generated summary reconciles divergent reports into a single consensus sentence, and nobody notices the information that got lost. Cause: language models are optimized to produce coherent text; coherence and fidelity to the source don't always coincide, especially when two sources disagree. Response: require every synthesis to point back to the source excerpt and id, manually review any summary that combines more than one interview, and treat preserved disagreement as valid data — often the most valuable finding of the round, because it reveals that the problem is not uniform across agents.

4. A purchase promise treated as payment

Symptom: a verbal expression of enthusiasm ("that would solve a lot of my problems") becomes, in the next meeting's notes, "customer confirmed interest in buying." Cause: ending a conversation on a positive signal feels good, and the pressure to show progress pushes toward the most optimistic possible reading. Response: record the literal answer, require specific mention of budget, approver, and success condition before counting it as a willingness-to-pay signal, and never book a verbal promise as revenue, a contract, or pipeline. This chapter's checker mechanically rejects any line in the pilot plan that claims a payment as already completed outside an explicit restrictions section — a second layer of protection against the same language habit.

Evaluating discovery without manufacturing success

This chapter's success criterion is not "DeskPilot validated demand" — that would manufacture a result no synthetic interview can produce. The criterion is: the v1 brief distinguishes observation from hypothesis in every opportunity; every opportunity compares at least three alternatives; every riskiest-assumption test has an explicit owner, deadline, criterion, and decision; and at least one real disagreement between sources remains visible rather than averaged out. node examples/evidence-check.mjs all verifies these four conditions structurally and fails if any is missing (see verification.md). That is process discipline, not a product measurement: the checker has no idea whether "Opportunity A" is real, only whether the document describing it follows the minimum contract to be taken seriously.

Once the team finally interviews real people, the bar changes: chapter 01's outcome (median time to correct assignment, with pending tickets visible in the denominator) is still what needs to be measured after any pilot, not the number of interviews conducted or the elegance of the opportunity tree. A well-built tree that points at the wrong segment is still a failed discovery effort; a modest tree that avoids an expensive pilot with no real demand is a successful discovery effort, even without producing a single line of code.

Cost, safety, and reversibility of the research itself

Interviewing has real cost even when nothing is charged: agent and manager time pulled away from work, research fatigue if the team keeps coming back every week for another conversation, and expectation risk — an interviewee may assume the company has already decided to build something just because someone asked. This chapter's guide caps the round at four to six agents and two managers, avoids hypothetical AI questions, and sets short deadlines (5 business days for the concierge test, one 30-minute meeting for willingness to pay) so discovery does not turn into an endless process.

Data safety starts at the interview, not at the pilot: no identifiable end-customer data should be collected at this stage; if an agent mentions a specific ticket, the customer name and ticket number are stripped before the record is stored. The synthetic interview JSON contains no real data at all — but the protocol it documents (interview-guide.md) is the same one that would apply to real data, including recorded consent, a retention period for the notes, and the interviewee's right to pause or end the conversation.

Reversibility here means two things. First, each test (concierge, willingness to pay) is designed to leave no operational trace if it fails: the concierge test never touches a real ticket, and the willingness-to-pay conversation generates no charge or contract. Second, the gate itself is reversible by design: "change segment" and "abandon" are valid outcomes, recorded in the decision log, not failures to hide. A team that only knows how to record "advance" is, in practice, forbidden from discovering that the right opportunity is somewhere else.

Starting checklist

• Did every interview record trigger, flow, workaround, frequency, consequence, and purchase decision (where applicable), without asking "would you use AI?"
• Were consent, minimization, and anonymization applied even in a synthetic exercise?
• Does every opportunity in the tree point to a labeled observation or hypothesis, not a team opinion?
• Does every opportunity compare at least three solutions: manual, deterministic rule, and AI-assisted?
• Does every riskiest-assumption test have an owner, deadline, criterion, and decision?
• Was the priority score treated as an ordering heuristic, not proof of value?
• Does the willingness-to-pay test distinguish praise from a specific budget and approver signal?
• Does the gate weigh viability, usability, and data access, with explicit advance, change-segment, and abandon paths?
• Does at least one real disagreement between interviews remain visible in the synthesis rather than averaged out?
• Did I run the checker on a valid and an invalid case and read the real result in verification.md?

Three exercises for a discovery week

Basic — rewrite a leading question. Take "would you use an AI suggestion for this?" and rewrite it as a sequence of questions about a past episode (trigger, flow, workaround, frequency, consequence). Criterion: the new sequence does not mention AI until the interviewee brings it up first. Worked solution: replace it with "think about the last ticket that gave you the most trouble to classify; what made it feel different? what did you do, step by step? is there anything you do on your own that the system doesn't offer? how often does this happen? what happened next?" — the same structure used in interview-guide.md.

Intermediate — separate buyer from user in a real sentence. Take the sentence "the customer would love this feature" (or an equivalent you've heard in a meeting) and rewrite it identifying which specific role supports it: user, buyer, or affected person. Criterion: the rewritten sentence names the role and the type of evidence still missing for the other two roles. Worked solution: "the support manager (buyer) said they would approve budget if there were a measured reduction; we still don't know whether agents (user) would accept the suggestion day to day, or whether end customers (affected) would notice a difference in the response."

Advanced — design a willingness-to-pay test without charging. For one of DeskPilot's three opportunities (or a real opportunity in your own context), write the three questions for the willingness-to-pay conversation and the criterion that separates praise from a buying signal. Criterion: the criterion requires specific mention of budget, approver, and success condition; no real charge occurs; the "advance" versus "keep validating" decision is written before the conversation, not after hearing the answer. Worked solution: see examples/pilot-plan.md, "Teste de disposição a pagar" section — INT-006's line is the documented counterexample of praise without commitment.

Glossary and the bridge to chapter 03

An opportunity solution tree is a structure linking a business outcome to observed opportunities, candidate solutions, and assumption tests, avoiding the jump straight from a problem to a favorite solution. A concierge test is a manual simulation of an automated solution, performed by a person, before building any system. Willingness to pay is a specific signal of budget, approver, and success condition — not verbal enthusiasm. A gate is the explicit decision point between advancing, changing segment, or abandoning an opportunity, with a criterion defined before seeing the result. Synthetic evidence is data generated for teaching purposes, labeled as such, that should never be reused as proof of real demand.

This chapter ends with one prioritized opportunity (billing vs. product ambiguity), a better-understood triage flow, and a set of gates not yet exercised against real data. No real interview happened; no customer confirmed payment; chapter 01's illustrative 20% target still has no real baseline behind it. In the planned chapter 03 of this series, the task turns technical: translating the prioritized opportunity into a verifiable domain model — a state machine, a permission matrix, and an event contract — so AI can suggest category and summary inside rules it cannot invent on its own.

Sources and limits

• Teresa Torres, Opportunity Solution Trees, Product Talk, updated 2023-12-06; accessed 2026-09-28. Tree structure and the recommendation of three to four story-based interviews; a qualitative method, not a market estimate.
• Jakob Nielsen, Thinking Aloud: The #1 Usability Tool, Nielsen Norman Group, 2012-01-15; accessed 2026-09-28. Justifies not treating six synthetic interviews as a statistical sample.
• Marty Cagan / SVPG, Product vs Feature Teams, 2019-08-29; accessed 2026-09-28. Author's framework on value, usability, feasibility, and business risk; used here only to name the gate's three dimensions.