Failure mode 2: passing off a synthetic demo as a customer

What it sounds like: “Customers accepted 81.8% of suggestions.” That percentage came from PE06's synthetic fixture, not customers. Replacing one noun changes the evidence class. A demo can test presentation and a fixture can test a parser. Neither establishes demand in the middle of a real support shift.

Why it happens: numbers migrate into slides without their denominator and population. Pressure to show traction makes “synthetic” look disposable. It is not. The remedy is to keep method, window, and source in the sentence where the number appears. If evidence is absent, say “not measured” and state how it could be measured with authorization.

The guard: the claim registry keeps external_outcome empty. The checker rejects a fixture claiming “real customers became 30% more productive” under local_measurement, including because its denominator and window are absent. The checker could still be fooled by a fabricated file or URL. That is why provenance review, consent, and skeptical reading belong in the publishing gate, not in a marketing footnote.

Failure mode 3: reducing a career to prompts

What it looks like: a portfolio catalogs prompts and model versions as if their count establishes product judgment. Prompts can speed up writing and implementation. They cannot decide which ticket exception matters, who owns a dead letter, whether a buyer accepts the cost, or when to stop a pilot.

Why it happens: tools produce visible output faster than discovery, negotiation, and operation. PostHog's product engineer handbook describes that company's expectation that engineers take part in support, roadmap, and experience decisions. It is a company's published practice, not a universal job definition or a promise of employment. The 2025 DORA report page describes AI as an amplifier of organizational strengths and weaknesses. Its summary is not a basis for predicting an individual salary, promotion, or productivity gain from using more prompts.

A better demonstration: show a decision the prompt could not settle. DeskPilot's RLS tests, restore as the application role, and blocked release matrix say more about the engineer's judgment than the first draft instruction. AI skill still matters. The evidence of product engineering is asking for proof, correcting overclaims, and owning limits in the system the tool helped build.

Failure mode 4: collecting every role without boundaries

What it sounds like: “A product engineer does PM, design, support, security, legal, and engineering.” It can sound autonomous. In practice it can mean unlimited workload and nobody clearly accountable for value, usability, data use, or spending. One person without access to participants cannot fix the absence of user evidence through effort alone.

Why it happens: understanding the whole product gets mistaken for having authority over all of it. SVPG's Product vs Feature Teams essay contrasts output-led work with teams empowered to solve problems and pursue outcomes. It is an author's framework, not a universal organizational taxonomy. In its account, product, design, and engineering collaborate on different risks. It does not imply that an engineer should replace every specialist.

The working agreement: PM owns the value question, prioritization, and access to participants. Design leads usability observation and the language of the review UI. Engineering owns feasibility, instrumentation, tests, and rollback. Support rehearses incidents. Security and legal review data purpose and contract. Leadership sets spending limits and names a risk owner. A small team may share several of these roles, but the agreement should still name decision rights, review time, and load. Autonomy means the engineer can propose, test, and stop with evidence, not carry every responsibility alone.

A skills matrix that avoids a magic score

The CSV matrix covers discovery, domain, UX, engineering, AI evaluation, analytics, economics, operations, communication, and influence. Each row gives an entry behavior, a behavior in practice, DeskPilot evidence, and the next feedback gap. It does not rate a person from one to ten. A single score would suggest precision the case cannot support.

Consider UX. A web client exists, but it does not render the suggestion. Marking that row “advanced UX” because there is a UI would be a category mistake. The evidence cell instead points to the blocked gate and the needed design and agent review. Operations similarly pairs a local restore with the missing durable alerting and real operator. The matrix allows an early-career engineer to show learning without claiming seniority by decree.

Behavior changes with context. A junior can frame one hypothesis and test one failure with mentoring; a mid-level or senior engineer can close a pilot gate end to end and negotiate support ownership; a staff engineer can make guards reusable without becoming a central approval queue; a consultant must distinguish buyer, operator, and end customer and obtain a data agreement before proposing an AI trial. The same repository artifact is not equivalent evidence for every job. Say who made the decision and with whose review.

The 30/60/90-day plan

In days 0–30, audit the story. Does each claim have a path? Does each measurement carry a denominator, window, population, and environment? Was discovery conducted with an authorized person or only designed on paper? Ask PM to challenge the value hypothesis, design to challenge the task, and support to list exceptions. The observable deliverable is a problem statement with alternatives and an honest account of available access. If access is unavailable, deliver a pending protocol rather than fabricated interviews.

In days 31–60, close the gates needed before a pilot: a provider-enforced billed-cost limit and reconciliation; data retention, deletion, and export operations; a UI that shows the suggestion, evidence, and degradation; and reviewed consent. Engineering implements and measures; security reviews the threat model; design observes comprehension; support performs the runbook. If a gate fails, pause with an owner and a new verification step. The milestone is not “launch because the calendar says day 60.”

In days 61–90, only after the gates are closed and access is authorized, PM recruits a small opt-in group. Engineering and design observe the workflow, support receives complaints, and the team compares against manual triage while honoring stop conditions. If this cannot happen, record “pilot not executed” and choose another obtainable piece of evidence. The full plan gives owners, deliverables, and feedback for each window, plus adaptations for junior, senior, staff, and consulting work. No time box promises a promotion or a job.

The next product might need less AI

DeskPilot's lesson changes with user and risk. For an internal product, an employee wants less triage rework; a reviewable suggestion and manual fallback may be appropriate, but data and process still have owners. In growth, clicks and acceptance are dangerous proxies; the outcome is useful activation with retention guardrails, and unconsented automatic messaging is not a shortcut. For a developer platform, the user is an integrator; stable contracts, idempotency, and spending limits matter more than a polished UI demo. In regulated B2B, access, data purpose, auditability, and reversibility may call for less AI autonomy and more human review.

Each setting changes the user, outcome, dominant risk, and permitted automation. If support agents do not want suggestions or the cost exceeds the value, the next iteration might be better-instrumented manual triage, deterministic rules, or the end of the AI hypothesis. If “hero” is a useful career word, it should mean someone who learns and operates responsibly, including when stopping is the right choice. Profit and product-market fit still require external evidence.

What evidence could change the case?

A useful case names an experiment that could disprove its hypothesis. Before inviting agents, the team would observe manual triage, estimate task volume, and agree with a tenant manager which tickets may enter a pilot. That gives the AI feature a baseline. The testable hypothesis would be that reviewable suggestions reduce rework without increasing reopening, data exposure, or cost per useful task. “Less rework” needs an event definition and window. “Better quality” needs case review, not just an acceptance button.

Even after technical gates close, a first small pilot would mainly test usability and operations. Can agents see the suggestion and recognize degradation? Can they reject a poor answer without pressure? Does the manager understand spending and the stop rule? Which errors reach support? A small sample can uncover serious faults while remaining far too weak to estimate an average effect or PMF. Reporting that “three agents struggled” would require three actual agents, consent, and a record; nothing in the current fixture supports it.

The experiment must allow different outcomes. If manual triage is already fast, clearer queue and SLA rules may be a better product than AI. If suggestions save time but increase reopening, restrict categories or strengthen review. If billed cost escapes the reservation, pause regardless of stated satisfaction. If agents distrust the summary, change the experience before assuming a new model will solve it. This is a backlog shaped by evidence, not a forecast of success.

A five-minute interview story, with questions in return

Start with the proposed support problem and the agent's job. Show a technical decision linked to experience: manual ticket triage remains possible when the AI provider fails. Open the injected-failure test and say what it can and cannot prove. Show the rejected alternative, automatic customer replies. Explain the unexpected restore finding, the corrected procedure, and its local-only scope. End with the no-go and the experiment that could change it. Every step gives the interviewer something concrete to challenge.

Ask about the workplace too. Who gets access to users? Who can cut a feature after feedback? Are PM, design, and support available, or is one engineer expected to cover them without time? Is there an incident owner and spending limit? Does the team measure useful work or only deployments? These are practical questions, not an ideological test of whether a company calls itself a “product team.” A warning sign is a demand for revenue impact without data access, research claims built on opinion, or on-call and consent assigned to someone without authority to pause the system.

Evaluate the portfolio before sharing it

Run the checker and open the paths it cites. Confirm that every number carries its denominator and synthetic or local status; that the pilot is marked unexecuted; that the UI and billed-cost limit remain blocked; and that a CMS draft is never described as a live page. Ask PM, design, and support to review interpretations that fall within their expertise. A green script is a consistency check on the registry, not proof of product impact.

• Basic exercise: rewrite “I built AI and increased productivity” with problem, decision, test, and limit. Pass when no real person or gain appears without a source. One answer: “I added a shadow-mode suggestion; local fixture evaluation and fallback tests passed; agent productivity was not measured.”
• Intermediate exercise: pick an attractive number in the index and write the complete claim. Include denominator, window, population, and method. “76/76 offline tests in PE07's local environment” describes regression checks in that snapshot, not adoption or an uptime SLO. For 81.8%, open PE06's synthetic fixture and metric cuts.
• Advanced exercise: propose a pilot decision for a regulated B2B tenant. Name a data and consent owner, billed-cost limit, review UI, rollback, feedback channel, and no-go condition. A defensible answer keeps no-go while PE07 gates 6–8 and 10 are open, then invites only under opt-in with an identified operator. Do not write the proposed invitation as though it happened.

Glossary and closing decision

Output is work delivered; outcome is an observed change in a user's task or business. A claim is a statement in the case that needs an evidence class. A gate is a verifiable condition for moving forward or pausing. Shadow mode produces a suggestion without automating a decision. PMF is a product-market-fit hypothesis this teaching project did not measure.

The DeskPilot portfolio does not end with “product launched.” It ends with a case a colleague can audit and a next experiment that still requires people, authorization, and real operations. That is more useful career evidence than changing a title in a profile: when someone asks “How do you know?”, there is an artifact, a test, a denominator, and an honest account of what remains unknown.

Sources

• PostHog, What is a product engineer? — company practice, accessed September 29, 2026.
• SVPG, Product vs Feature Teams — Marty Cagan, 2019; an authored framework for collaboration and outcomes.
• DORA, State of AI-assisted Software Development 2025 — report overview, without extrapolating to individual career effects.
• Internal evidence and limitations: sources.md, case-study.md, and portfolio-index.md.