I spent seven years on the discovery end of product — research, positioning and MVP definition for banks and fintechs at EY and Fifty Thousand Feet, then product strategy and design operations for Capital One Travel while it grew 84% in a year: intake from product, the weekly cross-company design review, and work carried from research through legal, risk and compliance review to delivery. For nearly two years I have built Prox end to end, including the release governance and evaluation program that decide what ships.
Looking for a product team that needs AI features customers will trust. Product strategy, design strategy, design operations, AI product, or research-led roles — Chicago or remote.
Sesami. Value proposition and messaging framework for an AI-driven cash-management platform, grounded in quantitative research with 550+ participants; aligned six to ten stakeholders on go-to-market.
Also · EY for Morgan Stanley, 2019-21
Wealth management. Owned research across six product lines, synthesizing findings into prioritized feature recommendations for executive stakeholders.
About
I've spent my career on the discovery half of product: finding the problem worth solving, defining the bet, and checking the result. At EY and Fifty Thousand Feet that meant discovery and positioning for banks and fintechs. At Capital One it meant product strategy and the operating cadence for Capital One Travel during its fastest growth: a weekly design review across two companies that decided what came in, what shipped, and in what order. Since 2025 it has meant doing all of it at once on Prox — and learning that shipping an AI feature is the easy part; knowing whether it works is the job. I built Prox to learn that loop. I'm not founding again.
I design as well as define — the Prox interface is mine — and I'm most useful where the two meet: deciding what to build, making it usable, and finding out honestly whether it works. Chicago is home.
Education
MBA + Master of Design, Illinois Institute of Technology (Institute of Design), 2019 · BS, Indiana University
Certification
Microsoft AI Product Manager Professional Certificate, 2026
A shopping assistant for furnishing a home, built and shipped by one person, and the measurement program underneath it that says, with evidence, where it is wrong and what is allowed to ship.
What it is
Describe the room → one question → a small set with a recommendation and the reason → saved to a project that remembers what you chose.
Stack
Next.js on Vercel · FastAPI on Fly.io · Neon Postgres + pgvector · Voyage AI embeddings · Claude · Braintrust · Stripe
Status
Live at proxdesign.co with a paid tier on Stripe since July 2026; the guided recommendations below shipped October 12, 2026. Pre-traction. Built to learn the whole loop; not founding again.
My role
Everything: strategy, research, interface, release governance, and the evaluation program.
Prox is a shopping assistant for furnishing a home. You describe the room in your own words, it asks one or two short questions, searches more than 300,000 pieces of furniture and decor, and comes back with a small set, a recommendation, and the reason for it. It remembers what you have already chosen as the room comes together.
I built the whole thing myself, from the interface to the search system to the release process. The part I care most about is the last one: a measurement program that tells me, with evidence, where the product is wrong, and a gate that decides what ships.
300K+furniture and decor items it can search
71%of 163 hand-labeled failures traced to one step: reading the request
9 of 9recommendations backed by the card's own facts in the post-release check, Oct 12, 2026
The live product, October 12, 2026: a sofa request, a recommendation with its reason, notes on the alternatives, and the off-spec cards dimmed with a reason each. Unedited apart from cropping.
How it works
One request, start to finish
Follow a single request through the product. Each step shows what you see on screen, what happens underneath, and how I know whether it worked. Evaluation runs beneath all five. Click a step, or a check, to open it.
1 Query
One box. You type what you would say to a friend: “Refreshing my living room. It feels dated and cluttered, I want it warmer and more put-together.” No category to pick, no filters to set.
That sentence is the hardest input in the product. It names no piece, no budget and no style, and the whole experience depends on reading it well.
The brief, as typed on the live product. One box, no filters.
Before anything is searched, a language model turns the sentence into a small structured record: the room, the budget if one was stated, the style if one was named, what the person is trying to do, and how confident the system is that it understood. Everything downstream reads that record, not the sentence.
The record also carries a confidence level. Low confidence is what sends the request to step 2 instead of straight to search.
Sentence“dated and cluttered… warmer”
Roomliving room
Budgetnone stated
Stylenone stated
Intentredesign, exploring
Confidencelow → ask first
This is where most of the product's mistakes begin. When I hand-labeled 163 failures across 60 test conversations and traced each one back to the earliest step that made it inevitable, 71% landed here, on reading the request. Not on the search, not on the writing, and not on the question rule I had suspected.
That one number redirected the work. The redesign that followed targeted this step, with a measurable goal: cut the failures that start here by at least half.
Reading the request116
Combining search results33
Writing the reply12
Final filters2
163 failure instances by the step they originated in. 60 conversations, single rater.
2 Context gathering
Instead of guessing, Prox asks, one question at a time. First, which style you lean toward, shown as picture tiles because style is easier to point at than to name. Then where you are starting from: fresh, or keeping most of what you have.
It never asks more than two questions in a conversation. The panel is a place to think for a moment, not a form to fill in, and the answers ride along with the request.
Question 1 of 2
Question 2 of 2
Both questions on the live product, October 13, 2026: style first, then the starting point. Two answers, then it searches.
The panel fires on one condition: the record from step 1 says confidence is low. A specific request such as “a walnut console at least 60 inches wide” skips it entirely. The answers are folded back into the record and the request goes to search with the gaps filled.
When to ask and when to just search turned out to be the central quality question of the product, which is why it has its own measurement below.
Low confidenceask, then search
High confidencesearch now
Limittwo questions, ever
Answersfolded into the record
Five nearly identical requests from people closing on a house got three different decisions about whether to ask. My hypothesis was that the rule was reacting to help-seeking phrasing like “where do I begin?”
Rather than write that into the findings, I tested it: a 60-row cross-check of seven candidate causes. Phrasing had no signal. The panel rule was reading the confidence level correctly; the inconsistency came from step 1, which was assigning confidence unevenly to whole groups of people. I rewrote the finding to match the evidence and moved the fix upstream.
Suspected
help-seeking phrasing drives the question
Found
no signal; the confidence level drives it, and confidence was being set unevenly one step earlier
3 Product search
Nothing, for about ten seconds. This step is invisible on purpose. What you get at the end is a small set, eight at most, rather than a wall of results, and the set is already filtered to the color, price and style you asked for.
The thing to notice is what is not there: no page of ten thousand sofas, no “sort by relevance” to argue with.
The record becomes three or four short searches, and they run through a chain ordered by cost. First, in parallel: a meaning-based search over the catalog Prox indexes itself, and live lookups against two retailer networks. If that comes back thin, a plain keyword search. Only if that also fails, a paid search service, last.
Color, budget and style are applied inside the search, not after it, so the set that comes back is already the right color rather than a nearest-match set trimmed down to one item. Results are then scored for relevance and handed on.
1meaning search + two retailer networks, in parallel
2keyword search if thin
3paid search, last resort
Filterscolor, price, style applied inside the search
Scoreranked for relevance
An advisor set a target for search quality that could not be reached by any ranking, and I could show why. With no real traffic to learn from, I built 150 questions with certified answer keys: a written rulebook, two independent model graders required to cite it, my own written rulings on their disagreements, and a third-model audit that agreed on 97.9% of a blind sample.
The keys showed that a typical question has about 17 right answers, so a top-10 list can never hold more than roughly half of them, however good the search. I proposed restating the target against measures that account for that ceiling instead of shipping against a number that cannot be hit.
Answer keys150 questions, two disjoint sets
Agreementscores within 0.02 across sets
Audit97.9% on a blind sample
Findingthe target was unreachable by construction
4 Products shown
A short reply that takes a position: “The Homary 78.7-inch rust boucle is your pick: three seats, rust color, boucle fabric, $699.99.” The recommended card is marked. The others carry a one-line note on what makes them different. Cards that miss what you asked for, a sectional when you wanted a three-seat sofa, are dimmed with the reason.
At the bottom, one next step: the rug, with a search already written, or a single question that would change the pick. Never both, never a list.
The first set for the brief above, on the live product: a pick with its reason, and a note on each alternative.
The first reply is drafted before the search finishes, so it can only announce what it is about to look for. This step is a second, separate pass that sees the actual results and is allowed to have an opinion. It is handed each product's listed facts: the name, the price, the brand, and the materials, colors and style the catalog records.
Two rules keep it honest. The model may only claim what those facts support. And the arithmetic is done in code before the model sees anything: each price is already marked under or over the stated budget, “around 60 inches” is already spelled out as 54 to 66, and whether a piece has closed storage is read from its name. A validator then drops any note the facts contradict.
Second passpick, reason, notes, skips, one next step
Validatordrops anything the facts do not support
I wrote the pass/hold bar before running the read: no recommendation may rest on a claim the card does not support, no on-spec item may be dimmed, no invented prices. The first production read, the morning this shipped, failed it: two recommendations rested on false comparisons, six on-spec items were dimmed, and the reply was quoting a budget the shopper had given for a different piece.
Three rounds of fixes later, each re-read on the same 31 conversations: the arithmetic moved out of the model, the memory rule was rewritten, and the validator grew. The post-release check on production found every recommendation supported by its card's facts, 9 of 9, with no invented prices and no misquoted budgets. I still call it a hold on the strict bar, because a few borderline items were dimmed for reasons I would argue with, and I would rather say that than round it up.
After the fixes, prod9 of 9 picks supported · 0 misquoted budgets · 0 invented prices
Costabout $4 of model calls for five full reads and three probes
5 Project memory
Ask for the rug next and the reply talks about the sofa: a thin washable rug “will look undersized and cheap next to a rust or camel sofa.” Ask for the media console and it says the walnut will sit warmly alongside the sofa you chose. Nothing had to be repeated.
It also declines when it should. Nothing in the rug set was actually 8x10, so instead of picking one it says so and writes the search to run next.
Anything you save goes to a project board with a budget, where your top choices count against the total and the board can be shared with someone who does not have an account.
The rug turn: the reply refers back to the sofa from the previous turn, and picks nothing because nothing fits.
The second pass is given a small ledger: what you asked for earlier in the conversation, the style you chose if you chose one, and, when a project board is open, its budget and what is already committed. A rule I learned the hard way: the ledger carries only what you actually said. The morning version let a budget stated for the rug ride into the console turn as “your $600 ceiling,” and let a style the system had merely inferred become “the direction you've set.”
Earlier asks“fabric in rust or camel”
Chosen styleonly if you chose it
Budgetonly for this piece, only if you said it
Boardcommitted spend, what is still needed
The test conversation has three follow-ups after the sofa. The bar: at least two of them should use the sofa, the style or the budget left, and none should state something the shopper never said. The morning read had zero correct uses and two false ones. After the fix, production used the memory on all three turns with none false.
Before
0 of 3 turns used the memory · 2 of 3 stated something the shopper never said
After, on production
3 of 3 used it · 0 false
6 Evaluation
Nothing, and that is the point. Every change to how Prox reads, searches, or recommends is measured before it reaches you, and the ones that fail are held back. Two of the three biggest changes I built this year did not ship, because the measurement said they were worse.
Three instruments. A failure catalog built from reading real conversations, every failure tagged and traced to the step where it started, with a written definition for each tag. A scoring rubric on four qualities, applied by a model acting as judge, calibrated against 30 conversations I scored by hand. And a gate: a pass/hold rule written down before the test runs, with what would reverse the decision stated in advance.
The judge is locked to a model and setting that give the same answer twice. The newer, more capable model changed its own mind on 17% of cases between identical runs, so I kept the older one. A baseline you cannot re-measure is not a baseline.
Judgelocked model, temperature zero, 30 hand-scored calibration cases
Gatepass/hold written before the run
Publicthe catalog, rubric and judge prompt are on GitHub
I ran the gate on my own work. A redesigned prompt for the project board, tested against the shipped one in 15 head-to-heads under the locked judge, lost 10 to 4 with one tie. I retired it. The same run exposed a bug in the live version, a board's stated needs never reached search, which I fixed in code, taking off-target recommendations from 4 of 7 to 0 of 7.
For the room-render feature, a quality gate built on pixel changes rejected 89% of good renders while rating the real failures best. I replaced it with a check that inventories the furniture before and after and flags anything that vanished without being asked. It caught every unrequested removal in the labeled set, plus two the hand labels had missed, for about two cents a render.
Limits, stated plainly: one rater, so the categories are consistent but not independently validated. Mostly single-turn test conversations, so what a follow-up would recover still counts as a failure. And Prox is pre-traction. This is a case study about how a product was built and how I know what is wrong with it, not about growth.
Board redesignheld lost 10-4-1 under the locked judge
Live bug foundfixed off-target 4 of 7 → 0 of 7
Render gatereplaced rejected 89% of good renders
Removal checkcaught all plus two the labels missed
Recorded from the live product on October 13, 2026 (2:10, no narration): a dated living room, the two context-gathering questions (a style from the tiles, then what to tackle), then a sofa under $1,500, a rug under $600, a walnut media console and a brass floor lamp, each with a pick, its reason, and a note on the alternatives. The waits while it searches are sped up 4×; nothing else is edited. Captured by a scripted recorder that refuses any take where a result ignores the stated budget.
Context
Furnishing a home is a dozen linked purchases over three to nine months, made by someone who comparison-shops by default and does not want an AI to decide for them. Search engines answer one query at a time; marketplaces answer with ten thousand results. Prox sits in between: you describe the room, it asks the one or two questions that matter, returns a small set with reasons, and remembers the project.
I built it solo — the product decisions, the interface, the retrieval system, and the release process. The part I'm proudest of is not the build. It's that I can tell you, with evidence, what's wrong with it.
The product
The retrieval layer is a RAG pipeline: a user turn is extracted into structured context (room, budget, style, intent, confidence), turned into search terms, and run through a cost-ordered cascade — vector search over the indexed catalog alongside two affiliate-network lookups, a keyword fallback when the top relevance score falls below a pinned threshold, and a paid search below a second one. What comes back is scored, filtered, and handed to the model to explain. Semantic retrieval and relevance scoring together took the typical session from six search turns to two on the flows I tracked.
The interface decision that mattered most was the clarifying panel: when extraction is uncertain, the system asks a short structured question instead of guessing — starting point, style, budget — and only then retrieves. The panel is the deliberation surface, not a modal interruption. Whether it fires at the right moment turned out to be the central quality question of the whole product. Saved items go to a project board, shipped as a gallery where status is a dropdown on the card and the top tier counts against the project budget — a four-column kanban I had specced was cut once building it showed items never left the first column.
Here is one session, run anonymously against the live product on September 24, 2026, before the guided recommendation shipped, unedited apart from cropping: a vague brief, two clarifying answers, and four follow-ups that build a room piece by piece.
Turn 1The brief
“Refreshing my living room. It feels dated and cluttered — I want it warmer and more put-together.”
The extraction is uncertain — no style, no budget, no piece named — so the system asks instead of guessing: keep what you have, replace the major pieces, or a mix?
Turn 2One more question
Style, with image tiles doing what words can’t.
Mid-Century Modern. The panel closes after two questions; it is context gathering, not a form.
Turn 3The first set
The brief plus two answers become one query: walnut, tapered legs, a range of prices.
Wide on purpose. The follow-ups are where it narrows.
Turn 4The sofa
“Three seats, fabric in rust or camel, under $1,500.”
Rust Hollyann sofas, in budget, rust acknowledged in the reply. The faux-leather sectional in the middle is off-spec — exactly the kind of card the failure taxonomy below counts.
Turn 5The rug
“An 8×10 rug in a warm neutral to ground it, under $600.”
8×10s in beige and taupe, and a reply that quotes the price range it actually found rather than the one I asked for.
Turn 6The media console
“Walnut, closed storage, at least 60 inches wide.”
It answers in widths — 63", 72", 80" — because that is the constraint that mattered. The 55" stand shouldn’t be there; also counted.
Turn 7The floor lamp
“For the reading corner, in warm brass.”
Arc lamps from Adesso and Living District, named, with wattages, because wattage is what the product descriptions specify and dimming is not.
This trace predates the guided recommendation that shipped October 12, 2026; the same session on today's product is shown in the flow above. Two cards in this trace are off-spec and I left them in. The evaluation program below exists because of cards like these: the point is not that the product is finished, it is that I can say precisely where it fails.
Release governance
Taking the paid tier live was designed as a single reversible switch: allowlist-gated on real Stripe billing, checkout-to-entitlement validated at $0, then public GA on July 12, 2026 — a flag flip, not a deploy, with rollback on the same switch. Every production change is a separately authorized step with a named rollback. Sharing a board outside the account is the riskiest access path in the product, so it ships behind a ten-assertion gate: guest tokens bound to one board, ownership-scoped reads and writes, revocation re-checked per request, and an off switch that restores the pre-feature response byte for byte.
New surfaces went live one at a time, per a launch memo written before exposure rather than reconstructed after it. Before each flip I ran a dogfood pass against a production build with production flags: automated boot and read-path checks on every surface, and a written manual checklist for the write flows that would have created junk rows in the live database if automated.
I also audited my own system for privacy compliance and treated the result as a defect list rather than a memo: the policy made retention and deletion promises the code did not keep. Global Privacy Control and Do-Not-Track became binding opt-outs the following month; the retention work is still open, and I would rather say that than imply it is finished.
The evaluation program
My first instrument was an overall quality score. It told me the system got worse; it never told me which component to open. So I rebuilt the program around a different question: when it fails, where did the failure originate?
Persona and dataset. I wrote a versioned persona — the trigger-window homeowner — with operational definitions, explicit exclusions, and ten falsification triggers. From it I generated a 60-query iteration set stratified across eight persona cells plus an edge cell, each query carrying the claims it tests and the behavior expected.
Labeling. I hand-labeled all 60 rows myself — every failure tagged, with the note that justified it. Categories emerged bottom-up from what I saw, not from a textbook list: each tag carries an operational definition and a boundary rule, and each failure is attributed to the most upstream layer at which it became inevitable. Scoring is a separate instrument — a four-dimension rubric (Relevance, Specificity, Factuality, Actionability) applied by a locked judge to the same conversations, with 30 cases I scored by hand as its calibration set.
The finding. Taxonomy v1.1 locked at 25 tags and 163 failure instances across 47 of the 60 rows. 71% of instances attributed to one layer — context extraction — up from 47% in the eight-row discovery sample. The fix target was not the generator, not retrieval, and not the panel-firing rule I'd suspected.
L2 · context extraction116
L7 · retrieval fusion33
L8 · generation12
L9 · post-filters2
Failure instances by originating layer, Foundation baseline run (n = 163 instances, 60 queries, single rater). Layers 1, 3-6, 10-11 recorded no attributable instances in this single-turn harness.
The distribution only concentrates like this because of the attribution rule. Naively, most of these would have landed on the generator.
Falsifying my own hypothesis
Five near-identical queries in the same cell — homeowners closing on a house, same need, slightly different phrasing, identical extracted confidence and intent — got three different decisions about whether to fire the clarifying panel. My hypothesis: the rule keyed on help-seeking phrasing ("where do I begin?") rather than genuine vagueness.
Instead of writing that into the taxonomy, I ran a 60-row cross-tab over seven candidate features. Help-seeking phrasing had no signal. What actually drove panel firing was the extracted confidence and intent — which meant the panel rule was calibrated correctly and the bug was one layer upstream, in how the extraction prompt assigns confidence to entire persona cells. I rewrote the findings to match the evidence and moved the fix.
0.93×lift from help-seeking phrasing — my hypothesis. No signal.
4.0×lift from extracted confidence = low
2.5×lift from extracted intent = exploratory
Panel-firing lift by feature, 60-row cross-tab. Re-run after the full sweep produced identical values. Directional, not powered: the confidence signal rests on 5 low-confidence rows, 4 of which fired the panel.
Judges and gates
Choosing a judge. Before trusting an LLM-as-judge, I measured self-disagreement on identical re-runs. The newer, more capable model flipped its own pass/fail factuality call on 17% of cases between runs, and its head-to-head winner on 13%. A baseline you cannot re-measure is not a baseline, so I locked the older model at temperature zero. Reproducibility beat capability.
Running the gate on my own work. I pre-committed a ship/hold rule, then tested a board-context prompt I'd redesigned against the shipped version: 15 head-to-heads under the locked judge plus a deterministic zero-products tripwire. It lost 10-4-1. I retired it. The same run exposed a retrieval bug in the live version — a board's stated needs never reached retrieval — which I fixed in code, taking off-target recommendations from 4 of 7 to 0 of 7.
Restating an unreachable bar. An advisor set recall@10 ≥ 0.80 as the retrieval acceptance criterion. With no production traffic to learn from, I built 150 ground-truth queries in two disjoint sets and certified the answer keys: a written rulebook, a blind two-vendor panel required to cite it, my own written policy on the disagreements, and a third-model audit that agreed on 97.9% of a blind sample. Scores landed within 0.02 across the two sets. They also showed the bar was unreachable by construction — with a median of 17 right answers per query, recall@10 tops out near 0.5 however good the ranking is — so I proposed restating the criterion against nDCG@10 and a ceiling-aware recall instead of shipping against a number that can't be hit. What keeps the answer key honest is that 31% of the candidates it was built from were reachable only by drawing straight from the catalog, never by the product's own search.
Killing a bad gate. For the room-render feature, a pixel-drift quality gate rejected 89% of good renders and scored the true failures best. The real failure mode was collateral object removal. So I built the guard the gate was assumed to be: an object-level check that inventories furnishings before and after and flags what vanished outside the request. It caught every unrequested removal in the labeled set, plus two the hand labels had missed, at roughly two cents a render. It runs in observe-only mode while the pixel gate's retirement is decided.
CandidateBoard-context prompt v2 (mine)VerdictHOLD · lost 10-4-1
Side effectRetrieval bug found in shipped v1After fixoff-target 4/7 → 0/7
JudgeOlder model, temperature 0, 17% flip rate avoidedTripwiredeterministic zero-products check
What it changed
The taxonomy alone is a backend-prioritization artifact. Read together with the persona brief, it became the case for a redesign: several failures that looked minor in the counts were load-bearing interface gaps, because the single-turn harness couldn't measure surfaces that didn't exist yet — project memory, room-segmented results, plans as objects the user can edit. That mapping drove the next epic, with a quantified target of cutting extraction-attributed instances by at least half before the retrieval-augmentation experiments resume. The first pass against it, re-judged by the locked judge on the worst-affected queries, moved Factuality from 39% to 68% and Relevance from 1.91 to 2.66 — offline, behind flags. The first of those changes to go live was the guided recommendation (step 4 above): a second model pass that picks, explains and flags off-spec cards, shipped October 12, 2026 after three rounds of a pre-committed read, with the arithmetic moved out of the model and a validator that drops any claim the card's facts do not support.
The taxonomy is public, with the 163 labeled failures behind it, the scoring rubrics, the judge prompt, and a dependency-free script that recounts the distribution above — no API key, no database access. github.com/D-Pollack/llm-failure-taxonomy ↗
Limits
Single rater, so the categories are internally consistent and not independently validated. Single-turn synthetic queries, so anything a follow-up would recover still counts as a failure, and turn-two behavior is under-measured. The extraction and board fixes the program produced are measured offline; the guided-recommendation pass is the first of them to reach production (October 12, 2026), and its post-release check on the live product is reported in step 4 above. The rest of the numbers here are evidence about the method, not a claim about what live users get today. And Prox is pre-traction: this is a case study about how a product was built and how I know what's wrong with it — not about growth.
Capital One Travel · 2022-2024 · Design Strategy Manager
From 30 competitor screens to a travel hub vision
Competitive analysis usually produces a screenshot deck nobody opens twice. I ran it like qualitative research instead — coded, counted, then translated into a structure the team could build toward.
Setting
Capital One Travel, growing 84% year over year, with a mobile hub that had accreted features faster than it had been designed
Method
30 competitive and comparative home screens · thematic coding of every content block · prevalence by type · directional framework · vision prototype
My role
Initiated and ran the analysis; built the framework and the vision; presented to product leadership
Outcome
Fed the team's product-vision work; designers used the framework to back proposals and it added a component to the Capital One design system; the methodology earned formal company recognition and became a repeatable process
Context
Travel is a category where people start on one platform and book on another, so the hub is judged against everyone else's. Our market analysis was ad hoc — someone would pull up a competitor when a question came up and the answer would live in that one conversation. I wanted a way to learn from the market that was systematic enough to point a roadmap at.
Coding the screens
I collected the home screens of 30 platforms — online travel agencies, airlines, card-issuer travel portals, and a few comparative apps outside travel — and coded them the way you'd code interview transcripts. Every content block on every screen got a tag: welcome, service or offering type, search, promotional content, brand content, important information, account and rewards information. Open coding first, then consolidation into a stable set, then a count.
Counting is the step most competitive analyses skip, and it's the one that turns opinion into a finding. Seventy-eight percent of platforms led with their offering types; roughly half put search on the home screen at all; only a third spent hub real estate on brand content. Account and rewards information was rare in the set — and it was the one thing a card issuer could do that an OTA couldn't.
Service / offering type78%
Promotional content59%
Important info59%
Welcome56%
Search56%
Brand content37%
Share of the 30 home screens carrying each content type. Account/rewards information was coded separately as the issuer-specific differentiator.
The directional framework
Prevalence plus position became a framework: a stacked structure of what belongs on a travel hub, in what order, and which blocks are table stakes versus where Capital One could differentiate. It's deliberately not a wireframe — it's the argument for one, with a number behind every row. That made it usable by product and business partners who'd never open a Figma file, and it made the conversation about what to leave out as concrete as the one about what to add.
The vision
I translated the framework into a vision for the mobile hub: quick actions for the four things people come to do (book, lounges, trips, price alerts), search up top, a resume-where-you-left-off module, upcoming trips, inspiration, then the issuer-only layer — rewards and credits, card benefits — followed by offers, support, and editorial. Each section maps back to a row of the framework, so the vision could be defended block by block rather than as a matter of taste.
The prototype was presented as a walkthrough to product leadership and fed the team's product-vision work. The coding-and-counting method was written up as a repeatable process and earned formal company recognition.
Left: the three steps on one slide — a coded competitor screen, the directional framework with prevalence, the vision. Right: the full hub vision.
What I'd tell you in the room
The transferable part isn't the travel hub. It's that competitive analysis becomes evidence the moment you code and count it, and evidence is what lets a strategist move a roadmap without a title that says "product." I've used the same move since — on Prox, the failure taxonomy is the same discipline pointed at model output instead of competitor screens.
A vision is a direction, not a spec. Parts of it fed the roadmap and the framework outlived the deck; I'm careful not to claim more than that.
Capital One Travel · 2022-2024 · Design Strategy Manager
Keeping the experience honest through 84% growth
Hyper-growth is when quality quietly slips. Four pieces of work that kept the user's voice in the roadmap while revenue nearly doubled.
Setting
$45M → $83M in a year; NPS 47-51 against a travel-industry norm of 18-32
Cadence I ran
Weekly design review for the travel portal — design quality, intake and roadmap planning — with product, engineering and partners including Hopper, Legal, Risk and Compliance; 1:1 roadmap planning with two product teams
Outcome
38 insights → 17 roadmap-ready proposals, several adopted into the 2024 roadmap; traveler-profile MVP shipped; itinerary research became a year-long build
1 · The usability study
Usability showed up as a theme among NPS detractors, but survey verbatims didn't say what was hard. I co-led an open-ended usability study in Q3 2023 — 13 observed 30-minute sessions with Capital One associates across the flight and hotel funnels — to establish an experience baseline and find what the surveys couldn't.
Five things came out of it that shaped how the team talked about the product: people explore elsewhere and book with us for the earn rates, so we had to make exploring worth doing here; they need to trust the value of a booking on both price and quality; they need to trust the platform broadly enough to find and use the features they value; they expect loyalty to be rewarded; and they expect personalization and are willing to answer questions to get it.
Underneath the themes were 38 specific insights, synthesized into 17 prioritized opportunities with a set of design heuristics — explore, compare, book — that gave the team evaluation criteria rather than a list of complaints. Several made the 2024 roadmap with full stakeholder consensus.
13observed 30-min sessions with associates, flight + hotel funnels
5experience themes
38actionable insights
17prioritized roadmap opportunities
2 · Unblocking the traveler profile
A traveler-profile feature had stalled on prioritization: too many candidate features, partners across two companies, and no shared way to decide. I ran the experience planning: a feature audit from existing research, then a 2×2 that separated what we were confident about from what needed more discovery, then a Now / Next / Beyond sequence tied to quarters.
The mapping did the political work. Features with strong evidence and high importance went to Now; things we liked but couldn't yet defend went to discovery. Cross-company stakeholders aligned on the sequence, and the MVP shipped.
High confidence · high convenienceReady for final design — into Now
High confidence · low convenienceDesign it, sequence it — Next
Low confidence · high convenienceNeeds discovery before commitment
Low confidence · low convenienceParked with a stated reason
NowQ4 / Q1 · portal-focused intent
evidenced, high-importance profile fields
saved-traveler basics
NextQ2
preferences that need one more round of testing
BeyondQ3 / Q4 · portal, offers, experiences
personalization-driven ideas pending discovery
Reconstructed from the experience-planning deck; feature-level detail withheld.
3 · The weekly design review
Under all of this was a standing cadence: a weekly design review for the Capital One Travel portal that I ran, covering design quality, intake and roadmap planning in one room. The people in it were PMs, designers and engineers from Capital One and from Hopper — a 50-plus-person org split across two companies. It was where a request became a scoped piece of work, where designs were reviewed before engineering built them, and where roadmap planning happened.
4 · Itinerary research
Post-booking was the least-examined part of the journey. I led itinerary validation research — a 122-person survey followed by eight in-depth interviews with concept testing — in partnership with product and design. The findings separated MVP updates from post-launch exploration and led to a year-long development initiative: three new screens and a data integration with Hopper.
What it added up to
None of these were dramatic. Together they meant that through the fastest growth the platform had seen, roadmap decisions stayed tied to evidence and NPS held at 47-51. I learned to use research less as validation and more as the thing that builds consensus across two product teams and a partner company.
Capital One materials are confidential; frameworks on this page are redrawn and figures are as reported internally.
EY for Barclays · 2020-2021 · UX Research & Strategy Manager
Discovery for a freelancer banking platform
A Fortune 500 bank saw 57 million US gig workers with no one serving their money, their work and their community together. We found out what they'd actually want.
Question
Is there a unified banking-and-jobs platform freelancers would trust a bank to build — and what goes in the MVP?
Method
29 moderated interviews across 3 cities · storyboarded concepts · card sorting and points-to-features · 51-question survey, 450+ responses
My role
Led the research plan end to end and co-authored the platform strategy point of view
Outcome
Three platform pillars, user modes, partnership models and a prioritized MVP feature set
Context
The gig economy was growing at roughly 10% a year, freelancers were carrying more financial risk than employees, and the tools they used — for finding work, managing it, and managing money — were scattered across a dozen apps. The bank had an opening, a lot of established competitors, and no evidence yet about what a differentiated product would be.
Approach
We ran qualitative discovery into quantitative validation. Interviews and storyboarding refined five candidate concepts down to three — Find Your Gig, Coaching & Insights, and Finances — which 29 freelancers reacted to across four activities, including card sorting and a points-to-features exercise that forced trade-offs. The survey then tested the concepts and feature interest at scale, with willingness-to-buy questions to separate "nice" from "would pay."
The research journey: kick-off and sense-making, interview prep, concept testing and validation, quantitative survey, analysis and recommendation.
What we found
The insight that defined the MVP was simple to say and hard to see without the research: freelancers can't connect what they earn to how they get paid — job data and financial data live in different apps, and the anxiety lives in the gap. That was the thing a bank could uniquely close. Three more themes carried the rest of the strategy. Trust: a bank behind the platform was itself the differentiator — freelancers valued security and trustworthiness competitors couldn't easily replicate. Community: coaching and mentorship mattered for professional and financial growth, not just networking. Guaranteed payments: a specific, monetizable pain for freelancers waiting 30+ days to get paid.
The participant analysis produced a segmentation that shaped feature priority: four user modes on axes of experience and network dependence — Nest, Maintain, Wander, Build — each wanting a different first feature.
User modes and the survey's feature-interest read by mode, which let us see how interest shifted between segments.
The strategy
The point of view landed on three platform pillars — finance management, work management, and a freelancer community with cross-platform insights — and named the competencies a platform needs to be worth building: facilitating multiple sides, generating network effects, and the ability to scale. We evaluated four partnership models (marketplace, extended core, strategic alliance, software-as-a-service) with the partners each would require, so leadership could choose a path rather than a feature list.
The MVP surface and the ecosystem it would need. Client anonymized in the deck as "a Fortune 500 bank."
What I took from it
This is where I learned to move from qualitative discovery to quantitative validation without losing the thread — and that a strategy document earns its keep by making the trade-offs visible, not by hiding them. Both habits show up directly in how I built Prox's persona and iteration set five years later.
The honest limit of consulting work: the strategy earned a three-month extension and shaped the MVP definition, and then we handed it over. I can tell you what we recommended and why; I can't tell you how much of it Barclays built.
AI evaluation is discovery research pointed at a system that talks back. I did the research half for seven years — interviews, observed sessions, coding, synthesis, decision frameworks — before I built the system. This is the loop I run, where each move was learned, and where design research methods become the tools that define an AI use case well enough to test it.
Marked terms open a short note on where the idea comes from — the design research canon I trained in at the Institute of Design, and the people setting the standard for AI evals.
A design process
Most of what makes an AI feature good or bad never shows up in a metric. It shows up in a transcript: someone asks for a small desk under a stated budget and the budget quietly disappears between the question and the results. Finding those moments, naming them, counting them, and tracing each one to the component that caused it is the whole job — the same job a researcher does with interview transcripts and usability recordings. The material changed. The discipline didn't.
The loop below is a design process with a system standing where the participant used to be. It runs from discovery to evaluation, and the hinge in the middle — synthesis — is where design research earns its place in AI work: it is how you define the use case tightly enough that a rubric, a test set and a gate can be built from it.
DiscoverDefineEvaluate
01Collect dataLook before you measure
02AnalyzeName what you saw
03SynthesizeDecide what it means
04CalibrateTrust the instrument
05DeployDecide, then keep looking
In design research
Activity-focused research (POEMS, AEIOU)
Contextual inquiry
Ethnographic interviewing
Constant comparison; open → axial coding
Affinity diagramming
Insight ≠ observation ≠ finding
Insight matrix: score, sort, cluster
Criteria & principles from insights
Value statement: who · unmet need · benefit · reason to choose
Problem statement & use case definition
Personas · service blueprint · experience map
Usability testing to find problems, not confirm design
Heuristic evaluation, 3–5 evaluators aggregated
Second coder, agreement check
Concept evaluation against criteria
Hypotheses and criteria set before concepts
Evaluative-phase research on the live product
NPS & analytics
→ back to Collect
In LLM evaluation
Error analysis, first half: trace collection and logging
Error analysis, second half: open coding of failures
Axial coding → failure taxonomy with boundary rules
Failure counts & first-failure attribution
Theoretical saturation as the stopping rule
Use case definition — who, what job, what inputs, what "good" means, what's out of scope
Surface map: which layer owns which failure
Rubric v0 as binary pass/fail checks
Dimensions for the eval set
Labeled calibration set, train/dev/test split
Judge self-consistency, then judge–human agreement (TPR/TNR)
Code assertions before LLM-as-judge
Lock the judge: model, temperature, prompt
Ship/hold gate on a pass-all rate
CI evals on the regression set
Online monitoring & drift
→ re-run error analysis on fresh traces
Produces
Observations and annotated traces
Insights; a taxonomy with counts
A use case defined tightly enough to test
An instrument you can trust
A decision — and fresh traces
Left to right is one pass. Deploy hands its traces back to Collect, so the shipped system becomes the next study's field site. The hinge is Synthesize: the same methods that turn research into criteria and an opportunity statement turn it into an AI use case definition.
Where it was learned
MBA + MDes, 2019 · IIT Institute of Design
Where the methods come from
Activity-focused research, analysis and synthesis as separate disciplines, problem framing, and structured methods that turn observations into criteria. Collect · Analyze · Synthesize
2019–21 · EY
Barclays, Morgan Stanley
Moderated interviews, transcript coding, concept testing, and the move from 29 conversations to a 450-response survey without losing the thread. Collect · Analyze · Synthesize
2021–22 · Fifty Thousand Feet
Sesami
Quantitative discovery with 550+ participants, and aligning six to ten stakeholders on what the numbers meant. Synthesize · Deploy
2022–24 · Capital One
Capital One Travel
Observed usability sessions, thematic coding of 30 competitor screens, evidence-versus-importance decision frameworks, NPS as the metric that outlives any study. Collect · Analyze · Synthesize · Deploy
2025– · Prox
The system itself
The same loop, pointed at a RAG pipeline I built: a hand-labeled taxonomy, a locked judge, pre-committed gates, and an honest list of limits. All five
01 · Collect data
The minimum viable evaluation is a person reading twenty to fifty outputs by hand — the first half of error analysis, run on traces. A researcher would call it contextual inquiry with the participant swapped for a system, and the discipline is the one I was taught: study the activity people are trying to accomplish, not the product; ask for use, not meaning; and touch all of the data informally before any formal analysis begins. At Capital One, NPS detractors said the product was hard to use and the surveys couldn't say what; thirteen observed sessions could. At Barclays, twenty-nine moderated interviews found the insight a national survey later confirmed but could never have found on its own.
At Prox I read all sixty rows of the iteration set myself and wrote a rationale beside each judgment. You read a trace whole; you don't skim for the score. And you note everything that made the product less useful, not only what the model got wrong — a clarifying question at the wrong moment is a design failure, and it goes in the notes even though no prompt fixes it. The gap between you and your data closes only by looking.
02 · Analyze
Don't write the rubric first. Your definition of quality changes as you grade — criteria drift — so a rubric written in advance stops you noticing the failures you didn't anticipate. This is the second half of error analysis, and researchers have run it for sixty years as constant comparison: open codes first, free text, one observation at a time; then axial coding, grouping the notes into categories and counting them; then keep going until new transcripts stop producing new categories. The travel hub work was this applied to screens — thirty platforms' home screens coded like transcripts, the content types and their prevalence falling out of the coding rather than being imposed on it. The usability study went from 38 observations to 5 themes to 17 opportunities the same way, on an affinity wall.
At Prox the codes became a failure taxonomy: twenty-five tags, 163 labeled instances, each tag with an operational definition, a positive example and a boundary rule. A category isn't real until it can say what it is not — the same test that separates an insight from an observation. A model helps cluster notes and propose look-alikes; the first thirty rows are always mine, because that is where the categories come from.
03 · Synthesize
Synthesis is where research becomes a decision. In design, insights are turned into criteria and principles — forward-looking statements that ideas can be judged against — and then into a value statement: who the customer is, what need is unmet, what new benefit is possible, and why they would choose it. Pointed at an AI feature, the same move produces a use case definition: who the user is, the job they are trying to get done, the inputs the system will actually see, what a good answer looks like, and what is explicitly out of scope. It has to be stated solution-free — in terms of the outcome the person wants, not the feature you plan to build — because nothing downstream works without it: the rubric is the use case's definition of good, the test set is its population, the surface map is where it lives in the system. Prox's was written before a single trace was labeled: a homeowner in a trigger window, furnishing a room through a dozen linked purchases, who comparison-shops by default and does not want the AI to decide for them.
A service blueprint asks the same question of every pain point: where in the flow did this become inevitable, and was it onstage or backstage? An AI pipeline is a blueprint with more backstage than front, so every failure is attributed to the most upstream layer at which it became unavoidable — the first upstream failure, because errors compound. That rule is what makes a distribution mean something: without it every failure lands on the generator, where you see it; with it, failures concentrate. At Prox, 71% traced to context extraction, when my theory had pointed at the panel-firing rule two layers downstream. The extraction prompt never said what I meant by "confidence" — a specification gap — so it assigned it to whole persona cells at once.
Mapping the experience also shows what isn't there. Failures that looked minor in the counts were load-bearing interface gaps — project memory, room-segmented results — that a single-turn harness couldn't measure because the surface didn't exist yet. Read next to the persona brief, the taxonomy became the case for a redesign.
Share of 163 failure instances by originating layer, Prox baseline run. The panel-firing rule I'd suspected (L3) recorded none: its inputs were wrong, not its logic.
The test set is built the way a study is recruited: decide the dimensions along which real queries vary, hand-write twenty combinations, then scale up and turn each into a natural-language query. It's a screener with quota cells — and the participants being generated makes it more important, not less, to say where they came from. At Prox the dimensions came from the persona: operational definitions, explicit exclusions, and ten triggers that would invalidate it; sixty queries stratified across eight persona cells plus an edge cell, each carrying the claims it tests. The Barclays user modes — Nest, Maintain, Wander, Build — did the same job for a survey five years earlier. A stratified set exists to expose the generalization gap: a pipeline that works on the inputs you wrote and fails on the ones you didn't.
Sampling also decides what you can conclude. A single row is a symptom; a pair of near-identical rows is a hypothesis; a cross-tab over the whole set is a cause. Five homeowners with the same need and different phrasing got three different panel decisions, and I was sure the rule keyed on help-seeking language. The sixty-row cross-tab said no — no signal from phrasing, four-times lift from extracted confidence — so the finding was rewritten and the fix moved upstream. Interviews generate the hypothesis; the survey is allowed to kill it.
04 · Calibrate
Two rules a research methodologist would recognize. Binary over Likert: a five-point scale invites the middle, hides disagreement between adjacent points, and needs more samples to detect a change — the Barclays survey used willingness-to-pay for that reason; "would you use it" gets a four, "would you pay" gets an answer. And code before judges: if an assertion can check it, an assertion checks it, and the LLM-as-judge is saved for calls that need judgment. A deterministic zero-products tripwire sits in front of the Prox judge. When a pixel-drift gate on room renders rejected 89% of good renders and scored the true failures best, the fix wasn't a better model but an object-level check — inventory the furnishings before and after, flag what vanished outside the request — at two cents a render. Every failure tag is binary by construction; the four-dimension rubric is the older instrument and still uses a scale, and the version I'd build today splits each dimension into pass/fail checks. The evaluation exists to find problems, not to confirm the design, and its criteria have to fit the method that produced the data rather than be borrowed from a benchmark.
A judge is a second rater and gets treated like one: agreement with itself first, then with humans. The newer model flipped its own factuality call on 17% of identical re-runs, so I locked the older one at temperature zero — a rater who disagrees with themselves can't be calibrated against anyone. Then thirty hand-scored cases as calibration; and for retrieval ground truth, a written rulebook, a blind two-vendor panel required to cite it, and a third-model audit that agreed with the key on 97.9% of a blind sample. Who evaluates changes what gets found, which is why the judge's criteria get written alongside the human labels, never ahead of them — someone has to validate the validator. The bar I'd hold a team to is higher: a hundred-plus labels per failure mode, a held-out test set, and true-positive and true-negative rates reported separately. Thirty cases can disqualify a judge; they can't certify one.
judge says failjudge says passhuman says fail
CaughtTrue-positive rate — the share of real failures the judge finds. Too low and the gate is decorative.
MissedThe failure ships. Inspect these first: they usually mean the judge lacks context, not intelligence.
human says pass
False alarmGood work held back. The pixel-drift gate lived here — 89% of good renders rejected.
PassedTrue-negative rate — the share of good outputs the judge lets through.
The two questions to ask any judge, human or model, before believing it. The human labels come first; the judge is graded against them, never the reverse.
05 · Deploy
Say what you expect before you measure. In design synthesis the concept space is framed by stating hypotheses and criteria before any concept exists, and a point of view is treated as a hypothesis — a strong opinion the work is allowed to overturn. At Capital One the traveler-profile feature was stuck because a dozen stakeholders across two companies had no shared rule for deciding; an evidence-versus-importance map with a Now / Next / Beyond sequence, agreed before any feature was placed, did the political work and the MVP shipped.
A ship/hold gate is the same instrument with a system on the other side. The rule is written before the results exist — what passes, what is release-blocking versus tolerable, what the tripwire is — and run on my own work first. A board-context prompt I'd redesigned lost 10–4–1 over fifteen head-to-heads under the locked judge; I retired it. The same run exposed a retrieval bug in the shipped version, fixed in code, taking off-target recommendations from four of seven to none. Gates can be wrong too: an advisor set recall@10 ≥ 0.80, and 150 certified ground-truth queries showed a ceiling near 0.5 by construction, because most queries have more right answers than slots. I proposed restating the criterion — concepts scored against the criteria, as a pass-all rate you can drill into, not a composite that hides what moved — rather than shipping against a number that can't be hit.
Written first
Fifteen head-to-heads. Locked judge, temperature zero. Zero-products tripwire is an automatic hold. Candidate ships only on a clear win.
Measured second
Candidate (mine) lost 10–4–1 → HOLD. Side finding: retrieval bug in production, fixed → off-target 4/7 → 0/7
Error analysis isn't a phase. Research has always come in explorative and evaluative kinds, and the shipped product is where the second kind runs — under its current name, analyze, measure, improve. Re-read a hundred fresh traces after any significant change, a handful a week in between, always after an incident; retire checks that pass every time, because they've stopped testing anything. Over longer horizons eval scores stop being comparable as the set changes, so you watch the product metric. At Capital One that was NPS, held at 47–51 through 84% growth, each study feeding the roadmap and the next one picking up where the last one's numbers ran out.
And don't sell "evals" — show what you found: the top failure modes, how often, the bugs caught before users hit them. That is how research earns its place in a roadmap too; at Capital One it was less validation than the thing that built consensus across two product teams and a partner company. A failure taxonomy read aloud to a room is a research readout.
With a team
Prox had one rater, which made me the benevolent dictator by default — the single domain expert whose call on quality is final. On a team the role still exists, usually the PM or researcher closest to the user; what changes is everything around it. A second annotator on the first hundred traces, agreement measured with Cohen's kappa, disagreements argued out and written into the codebook as boundary rules before anyone relabels. Engineers in the first pass, because they see retrieval and tool failures a PM reads past; then one owner, so the taxonomy doesn't drift by committee.
And a review interface worth using: outputs rendered the way users see them, the context a reviewer needs on one screen, keyboard shortcuts, a progress count. That is a design problem before it's an infrastructure one — the usability heuristics apply to the labeling tool as much as to the product — and I design as well as define, so it's a tool I'd build rather than wait for. Labeling isn't a task to outsource; it's where product judgment comes from.
The limits
Single rater, so the taxonomy is internally consistent and not independently validated. Single-turn synthetic queries, so turn-two behavior is under-measured and anything a follow-up would recover still counts as a failure. Thirty calibration cases for the judge, not the hundred-plus the field's standard asks for. Fixes measured offline, behind flags. Naming these plainly is what makes the rest believable; an unqualified result reads as marketing, and a researcher who won't state the limits of a study shouldn't be trusted with its findings.
The artifact from an actual pass — the twenty-five categories, the 163 labeled failures behind them, the rubric, the judge prompt and the label schema — is public at github.com/D-Pollack/llm-failure-taxonomy ↗. A dependency-free script recounts every headline number on this page from the data.
Product strategist in Chicago. Discovery, prioritization and delivery for seven years, including the design review and roadmap cadence for Capital One Travel; nearly two years building, governing and evaluating an AI product on my own.
Path
Prox (2025-) — Founder, Product & Design. Strategy, interface and evaluation for an AI discovery product; live with a paid tier, pre-traction.
Prudential Financial (2024) — Contract UX research lead for enterprise product discovery.
Capital One (2022-24) — Design Strategy Manager on Capital One Travel: ran the weekly cross-company design review; discovery, MVP prioritization and delivery through 84% growth.
Fifty Thousand Feet (2021-22) — Lead Brand Strategist: positioning and go-to-market for WEX and Sesami.
EY (2019-21) — UX Research & Strategy Manager: product discovery for Barclays and Morgan Stanley.
IIT Institute of Design (2019) — MBA + Master of Design.
The honest version
I've spent my career on the discovery half of product — find the problem, define the bet, check the result — and I design as well as define; the Prox interface is mine. If your team needs someone who can figure out what an AI feature should do, make it usable, and tell you honestly whether it's working, that's the job I'm good at.
I built Prox to learn the whole loop, and it did that. I'm not founding again.
Chicago is home. Outside work I build furniture; the modular shelf I designed and prototyped in my Lincoln Park workshop is where Prox started — as a physical product, before it became a discovery problem.
