A product engineer portfolio built on decisions and evidence

Two candidates present the same project. One says, “I built authentication, multi-tenancy, PostgreSQL, AI, analytics, an outbox, and a dashboard.” The other says, “We kept AI suggestions in shadow mode because tenant boundaries could not depend on a model. A local fault test showed that the backend could fall back, but the UI never told an agent. We decided not to invite users.” The feature list is accurate. The second account is useful: a reviewer can inspect the decision, its test, and the remaining gap.

This is the final chapter of the AI-native Product Engineer series. DeskPilot is a runnable teaching product for assisted B2B support triage. A support agent would use it, a tenant manager would buy it, and an end customer would be affected by the result. The repository contains a working example and local tests. It does not contain an executed pilot, a customer, revenue, or evidence of product-market fit. A credible portfolio starts with that distinction.

The short version: the case study links each decision to an artifact, a test, and a limit. The portfolio index separates built work, local measurement, user validation, and plans. A claim checker rejects unsupported statements and requires denominators for local measurements. The next product step is to close the pilot gates and seek authorized feedback, or to simplify the AI hypothesis if it fails to earn its place.

Start with one claim someone can challenge

Write a sentence that can survive “How do you know?” “The PE07 offline suite passed 76 of 76 tests in the recorded local environment” has a denominator and a verification record. “Customers became more productive” would need participants, a baseline, a defined task, a time window, and consent to collect the relevant data. It is not a more confident version of the first sentence. It is a different claim with missing evidence.

The file examples/claims.json makes state explicit. Every claim has an evidenceClass: built, local_measurement, external_outcome, or planned. Every claim links to evidence files. Local measurements also state a denominator and window. An external outcome would require a source URL and verification date. This series has no external outcome to enter. The checker validates structure and local paths; it does not authenticate an external document or decide whether an experiment was well designed. A human still has to read the source.

node examples/check-portfolio.mjs  # PASS: 7 claims with structured evidence
node examples/check-portfolio.mjs examples/claims.invalid.json  # expected FAIL: real-customer gain labelled local measurement

The second input is an intentionally false fixture used to test the guard. No customer gained anything in that test. The point is to make a portfolio capable of failing for a visible reason, before the claim reaches an interviewer or reader.

Why a feature list is an incomplete story

Technology names describe output. They seldom explain who needed it, what risk it addressed, or whether it worked outside a demo. “Outbox” says nothing about duplicate delivery, dead-letter ownership, or who may authorize a replay. “Dashboard” says nothing about which signal pages which operator. “AI” says nothing about whether an agent can see or contest a suggestion, or whether it makes support outcomes better.

DeskPilot's local restore drill illustrates the missing half. A database dump restored tables into a fresh PostgreSQL container, but nine grants failed because the destination cluster lacked the deskpilot_app role. A screenshot of the rows queried as a superuser would have made a tidy but misleading success story. The procedure changed: restore the database, run the migration that recreates the role and grants, then read the ticket and audit event as the application role under row-level security. That shows recovery for this fixture and environment. It does not establish production backup policy, recovery time, or long-term durability. The skill shown is not simply “knows Postgres”; it is noticing when a test uses the wrong actor.

Each decision in the full case therefore has five parts: the problem, a plausible alternative, the choice, the test, and its limit. The index gives a path to each chapter's handoff. If someone asks “Which feature do customers use most?”, the honest answer is that there are no customers in this exercise. You can still explain the experiment that would answer the question. Inventing usage to smooth out the story would weaken precisely the judgment the portfolio should demonstrate.

Four evidence states, in plain language

Built means there is an inspectable artifact. DeskPilot v0.3 has domain rules, a basic web client, typed AI suggestions in shadow mode, local metrics, and operational examples. A file's existence does not show that anybody used it.

Measured locally adds environment, method, window, and denominator. PE07's npm test run passed 76/76 offline cases. Its record includes 16/16 PostgreSQL flow tests and a local restore drill. PE06 reported 81.8% suggestion acceptance and 63.6% clean close in a synthetic fixture, with the relevant cuts and denominators in its report. None of these numbers describes customers or supports a causal claim about AI.

Validated with a user would require an actual support agent or manager, authorized access, an observed task, feedback, and a success criterion set before looking at the outcome. This series has no such validation. Showing a demo to a colleague would not by itself convert a fixture into customer research.

Planned is a protocol: who recruits, how people opt in, why data is used, where feedback goes, and which observation stops the pilot. PE07's pilot protocol says “pilot not executed” at the top. An executable plan can be good work while still being a plan. These states stop an attractive but invalid leap from “tests pass” to “the market wants it.”

The chain of decisions behind DeskPilot's no-go

The portfolio index walks through the series without relying on chat history. PE01 framed the brief and autonomy; PE02 designed discovery; PE03 fixed tenant, role, and ticket invariants; PE04 built manual triage; PE05 added typed suggestions and synthetic evaluation; PE06 separated acceptance from useful completion; PE07 examined operations and pilot readiness; this chapter makes the chain presentable and testable. It is a chronology of artifacts and decisions, not commercial milestones.

The first lasting choice was not to let the model issue commands. A model can propose a category and summary. Deterministic rules govern tenant boundaries, authorization, SLA, state transitions, and assignment. A human reviews the proposal and the backend validates the resulting command. We retained this boundary as the example grew. The absence of automatic customer messages is a deliberate risk decision, not a missing box on a roadmap.

The second choice was not to report the nicest metric as value. PE06's synthetic acceptance rate exceeded its stricter clean-close rate. That difference exposed a metric interpretation problem. It did not prove that AI helped or hurt support agents. The response was to design a reopen guardrail and a tenant-level experiment, while refusing to calculate a meaningful p-value from one tenant per arm. A credible portfolio leaves the denominator next to the headline, even when it makes the headline less exciting.

The third choice was to narrow automation when the dependency fails. PE07 injects provider unavailability. After five consecutive fallbacks for a tenant, a sixth suggestion request is suspended; manual ticket triage stays available. The backend records the fallback, but the UI does not display a suggestion or a degradation notice to the agent. Calling that a proven user-facing fallback would overstate the result. The UX gate remains blocked, alongside provider-billed spending limits, real-data operations, and consent review. The go/no-go matrix says do not invite real users yet.

That chain includes both a decision not to build and a decision to take automation away: no automatic reply to an end customer, and a tenant-scoped stop on suggestions during repeated provider failure. A conventional feature list tends to hide both because neither yields an impressive new screen. In this case, they carry the argument: product engineering includes preserving a safe path and acknowledging when the person using it still lacks enough information.

Failure mode 1: calling a feature list impact

What it sounds like: “I delivered eight chapters and an AI-native platform, so I raised productivity.” The first clause can be checked as output. The second needs observed task time, rework, quality, cost, baseline, and participants. The bridge is missing.

Why it happens: building is visible and measurable in commits; product effects are slower and shared across roles. A list of technologies can impress without forcing the author to define a user or outcome. In an interview it invites questions the project cannot answer: How many customers? How much time saved? Did reopening get worse?

A better account: “We proposed reducing triage rework; chose a reviewable suggestion over automatic action; tested tenant isolation and fallback locally; found UI and final billing gates still open; and specified the opt-in study needed next.” Every clause can be inspected or clearly marked as a plan. Do not turn “impact unmeasured” into “impact positive” just to fill a résumé line.

Failure mode 2: passing off a synthetic demo as a customer

What it sounds like: “Customers accepted 81.8% of suggestions.” That percentage came from PE06's synthetic fixture, not customers. Replacing one noun changes the evidence class. A demo can test presentation and a fixture can test a parser. Neither establishes demand in the middle of a real support shift.

Why it happens: numbers migrate into slides without their denominator and population. Pressure to show traction makes “synthetic” look disposable. It is not. The remedy is to keep method, window, and source in the sentence where the number appears. If evidence is absent, say “not measured” and state how it could be measured with authorization.

The guard: the claim registry keeps external_outcome empty. The checker rejects a fixture claiming “real customers became 30% more productive” under local_measurement, including because its denominator and window are absent. The checker could still be fooled by a fabricated file or URL. That is why provenance review, consent, and skeptical reading belong in the publishing gate, not in a marketing footnote.

Failure mode 3: reducing a career to prompts

What it looks like: a portfolio catalogs prompts and model versions as if their count establishes product judgment. Prompts can speed up writing and implementation. They cannot decide which ticket exception matters, who owns a dead letter, whether a buyer accepts the cost, or when to stop a pilot.

Why it happens: tools produce visible output faster than discovery, negotiation, and operation. PostHog's product engineer handbook describes that company's expectation that engineers take part in support, roadmap, and experience decisions. It is a company's published practice, not a universal job definition or a promise of employment. The 2025 DORA report page describes AI as an amplifier of organizational strengths and weaknesses. Its summary is not a basis for predicting an individual salary, promotion, or productivity gain from using more prompts.

A better demonstration: show a decision the prompt could not settle. DeskPilot's RLS tests, restore as the application role, and blocked release matrix say more about the engineer's judgment than the first draft instruction. AI skill still matters. The evidence of product engineering is asking for proof, correcting overclaims, and owning limits in the system the tool helped build.

Failure mode 4: collecting every role without boundaries

What it sounds like: “A product engineer does PM, design, support, security, legal, and engineering.” It can sound autonomous. In practice it can mean unlimited workload and nobody clearly accountable for value, usability, data use, or spending. One person without access to participants cannot fix the absence of user evidence through effort alone.

Why it happens: understanding the whole product gets mistaken for having authority over all of it. SVPG's Product vs Feature Teams essay contrasts output-led work with teams empowered to solve problems and pursue outcomes. It is an author's framework, not a universal organizational taxonomy. In its account, product, design, and engineering collaborate on different risks. It does not imply that an engineer should replace every specialist.

The working agreement: PM owns the value question, prioritization, and access to participants. Design leads usability observation and the language of the review UI. Engineering owns feasibility, instrumentation, tests, and rollback. Support rehearses incidents. Security and legal review data purpose and contract. Leadership sets spending limits and names a risk owner. A small team may share several of these roles, but the agreement should still name decision rights, review time, and load. Autonomy means the engineer can propose, test, and stop with evidence, not carry every responsibility alone.

A skills matrix that avoids a magic score

The CSV matrix covers discovery, domain, UX, engineering, AI evaluation, analytics, economics, operations, communication, and influence. Each row gives an entry behavior, a behavior in practice, DeskPilot evidence, and the next feedback gap. It does not rate a person from one to ten. A single score would suggest precision the case cannot support.

Consider UX. A web client exists, but it does not render the suggestion. Marking that row “advanced UX” because there is a UI would be a category mistake. The evidence cell instead points to the blocked gate and the needed design and agent review. Operations similarly pairs a local restore with the missing durable alerting and real operator. The matrix allows an early-career engineer to show learning without claiming seniority by decree.

Behavior changes with context. A junior can frame one hypothesis and test one failure with mentoring; a mid-level or senior engineer can close a pilot gate end to end and negotiate support ownership; a staff engineer can make guards reusable without becoming a central approval queue; a consultant must distinguish buyer, operator, and end customer and obtain a data agreement before proposing an AI trial. The same repository artifact is not equivalent evidence for every job. Say who made the decision and with whose review.

The 30/60/90-day plan

In days 0–30, audit the story. Does each claim have a path? Does each measurement carry a denominator, window, population, and environment? Was discovery conducted with an authorized person or only designed on paper? Ask PM to challenge the value hypothesis, design to challenge the task, and support to list exceptions. The observable deliverable is a problem statement with alternatives and an honest account of available access. If access is unavailable, deliver a pending protocol rather than fabricated interviews.

In days 31–60, close the gates needed before a pilot: a provider-enforced billed-cost limit and reconciliation; data retention, deletion, and export operations; a UI that shows the suggestion, evidence, and degradation; and reviewed consent. Engineering implements and measures; security reviews the threat model; design observes comprehension; support performs the runbook. If a gate fails, pause with an owner and a new verification step. The milestone is not “launch because the calendar says day 60.”

In days 61–90, only after the gates are closed and access is authorized, PM recruits a small opt-in group. Engineering and design observe the workflow, support receives complaints, and the team compares against manual triage while honoring stop conditions. If this cannot happen, record “pilot not executed” and choose another obtainable piece of evidence. The full plan gives owners, deliverables, and feedback for each window, plus adaptations for junior, senior, staff, and consulting work. No time box promises a promotion or a job.

The next product might need less AI

DeskPilot's lesson changes with user and risk. For an internal product, an employee wants less triage rework; a reviewable suggestion and manual fallback may be appropriate, but data and process still have owners. In growth, clicks and acceptance are dangerous proxies; the outcome is useful activation with retention guardrails, and unconsented automatic messaging is not a shortcut. For a developer platform, the user is an integrator; stable contracts, idempotency, and spending limits matter more than a polished UI demo. In regulated B2B, access, data purpose, auditability, and reversibility may call for less AI autonomy and more human review.

Each setting changes the user, outcome, dominant risk, and permitted automation. If support agents do not want suggestions or the cost exceeds the value, the next iteration might be better-instrumented manual triage, deterministic rules, or the end of the AI hypothesis. If “hero” is a useful career word, it should mean someone who learns and operates responsibly, including when stopping is the right choice. Profit and product-market fit still require external evidence.

What evidence could change the case?

A useful case names an experiment that could disprove its hypothesis. Before inviting agents, the team would observe manual triage, estimate task volume, and agree with a tenant manager which tickets may enter a pilot. That gives the AI feature a baseline. The testable hypothesis would be that reviewable suggestions reduce rework without increasing reopening, data exposure, or cost per useful task. “Less rework” needs an event definition and window. “Better quality” needs case review, not just an acceptance button.

Even after technical gates close, a first small pilot would mainly test usability and operations. Can agents see the suggestion and recognize degradation? Can they reject a poor answer without pressure? Does the manager understand spending and the stop rule? Which errors reach support? A small sample can uncover serious faults while remaining far too weak to estimate an average effect or PMF. Reporting that “three agents struggled” would require three actual agents, consent, and a record; nothing in the current fixture supports it.

The experiment must allow different outcomes. If manual triage is already fast, clearer queue and SLA rules may be a better product than AI. If suggestions save time but increase reopening, restrict categories or strengthen review. If billed cost escapes the reservation, pause regardless of stated satisfaction. If agents distrust the summary, change the experience before assuming a new model will solve it. This is a backlog shaped by evidence, not a forecast of success.

A five-minute interview story, with questions in return

Start with the proposed support problem and the agent's job. Show a technical decision linked to experience: manual ticket triage remains possible when the AI provider fails. Open the injected-failure test and say what it can and cannot prove. Show the rejected alternative, automatic customer replies. Explain the unexpected restore finding, the corrected procedure, and its local-only scope. End with the no-go and the experiment that could change it. Every step gives the interviewer something concrete to challenge.

Ask about the workplace too. Who gets access to users? Who can cut a feature after feedback? Are PM, design, and support available, or is one engineer expected to cover them without time? Is there an incident owner and spending limit? Does the team measure useful work or only deployments? These are practical questions, not an ideological test of whether a company calls itself a “product team.” A warning sign is a demand for revenue impact without data access, research claims built on opinion, or on-call and consent assigned to someone without authority to pause the system.

Evaluate the portfolio before sharing it

Run the checker and open the paths it cites. Confirm that every number carries its denominator and synthetic or local status; that the pilot is marked unexecuted; that the UI and billed-cost limit remain blocked; and that a CMS draft is never described as a live page. Ask PM, design, and support to review interpretations that fall within their expertise. A green script is a consistency check on the registry, not proof of product impact.

• Basic exercise: rewrite “I built AI and increased productivity” with problem, decision, test, and limit. Pass when no real person or gain appears without a source. One answer: “I added a shadow-mode suggestion; local fixture evaluation and fallback tests passed; agent productivity was not measured.”
• Intermediate exercise: pick an attractive number in the index and write the complete claim. Include denominator, window, population, and method. “76/76 offline tests in PE07's local environment” describes regression checks in that snapshot, not adoption or an uptime SLO. For 81.8%, open PE06's synthetic fixture and metric cuts.
• Advanced exercise: propose a pilot decision for a regulated B2B tenant. Name a data and consent owner, billed-cost limit, review UI, rollback, feedback channel, and no-go condition. A defensible answer keeps no-go while PE07 gates 6–8 and 10 are open, then invites only under opt-in with an identified operator. Do not write the proposed invitation as though it happened.

Glossary and closing decision

Output is work delivered; outcome is an observed change in a user's task or business. A claim is a statement in the case that needs an evidence class. A gate is a verifiable condition for moving forward or pausing. Shadow mode produces a suggestion without automating a decision. PMF is a product-market-fit hypothesis this teaching project did not measure.

The DeskPilot portfolio does not end with “product launched.” It ends with a case a colleague can audit and a next experiment that still requires people, authorization, and real operations. That is more useful career evidence than changing a title in a profile: when someone asks “How do you know?”, there is an artifact, a test, a denominator, and an honest account of what remains unknown.

Sources

• PostHog, What is a product engineer? — company practice, accessed September 29, 2026.
• SVPG, Product vs Feature Teams — Marty Cagan, 2019; an authored framework for collaboration and outcomes.
• DORA, State of AI-assisted Software Development 2025 — report overview, without extrapolating to individual career effects.
• Internal evidence and limitations: sources.md, case-study.md, and portfolio-index.md.