Using LLMs to Synthesize Customer Interviews

Synthesize customer interviews with an LLM in four separate passes: code each transcript into atomic observations, cluster those codes into themes, quantify how many distinct people raised each theme, then verify every quote against the source. Run the passes independently — one mega-prompt that "just summarizes" is exactly where hallucinated themes and paraphrased quotes sneak in.

Quick Answer: Don't ask an LLM to summarize interviews. Make it code, cluster, count distinct speakers, and cite sources — as four separate steps — so every theme is traceable to something a real person actually said.

If you came from product management into founding, you already know the failure mode. You ran eight discovery calls, the notes are sitting in a folder, and the temptation is to paste them into a chat window and ask "what are the big themes?" That shortcut feels like synthesis. It is closer to a well-written horoscope. This guide walks through a repeatable method that keeps the machine doing mechanical work and keeps you doing the judgment.

Why raw summarization loses the signal you need

Raw summarization loses signal because an LLM optimizes for a smooth, readable paragraph — not a faithful map of what each participant actually said. Those two goals pull in opposite directions, and the paragraph almost always wins.

When you paste five transcripts in and ask "what are the main themes," the model does what it is built to do: produce the most probable, most fluent summary. Fluency rewards the common and the average. But early-stage insight usually lives in the outlier — the one participant whose ugly workaround reveals the real job to be done. A tidy summary smooths that person out of existence.

There are three specific ways this goes wrong, and naming them helps you catch them:

• Frequency bias dressed up as importance. The model surfaces whatever repeats across transcripts and quietly treats repetition as significance. A pain that one person described in vivid, specific detail often matters more than a vague complaint five people echoed to be polite.
• Lost verbatim language. In Rob Fitzpatrick's The Mom Test, the entire discipline is separating what people literally said from what you wish they meant. Summaries paraphrase by design, and paraphrase is where your interpretation silently overwrites their words.
• Confirmation bias on tap. Prompt the model with "what do customers love about our onboarding" and it will cheerfully assemble a flattering answer. You primed it. It complied. That is not a finding; it is an echo.

That last failure is the quiet killer for founders, because you are emotionally invested in the idea and your prompts leak that investment. The model is agreeable. It will meet your framing.

The stakes are not academic. CB Insights' widely-cited analysis of startup postmortems found that building something with no market need was the single most common reason startups failed — named in over 40% of cases in its referenced report, and later reframed as a lack of product-market fit. A synthesis process that flatters your assumptions is a direct on-ramp to exactly that outcome. You will have documentation, charts, and quotes, all pointing confidently at a need that a more honest reading would have flagged as thin.

Consider what averaging does to a real dataset. Suppose eight of your ten participants describe a mild scheduling annoyance, and two describe a workflow so broken they built a shadow tool in a spreadsheet to survive it. A summary leads with the scheduling annoyance because it is the majority note. But the two desperate participants are the ones telling you where a business might be — intense pain in a narrow group is often a far better wedge than mild pain spread wide.

If you are still upstream of synthesis — designing the interview guide, recruiting the right participants, asking non-leading questions — fix that first, because synthesis can only ever be as good as the conversations feeding it. Our complete guide to customer research for founders covers the interviewing craft that makes everything below worth doing. Good synthesis on bad interviews just launders noise into a confident-looking chart.

The four-pass method: code, cluster, quantify, verify

The four-pass method works because it separates extraction from interpretation, so each step can be checked before the next one builds on it. A single prompt collapses all four jobs into one opaque leap; four passes give you four places to catch an error.

Think of it as an assembly line with inspection points. Each pass takes a clean, checkable input and produces a clean, checkable output: transcripts become codes, codes become themes, themes become counts, counts become verified findings. Because every handoff is inspectable, an error introduced in one stage gets caught before it compounds into the next — which is precisely what a single all-in-one prompt makes impossible, since the only thing you can inspect there is the final, confident-sounding answer.

This mirrors how qualitative researchers have coded interviews for decades — open coding, then axial coding, then counting prevalence — long before LLMs existed. The model does not replace that method. It accelerates the tedious parts and leaves the judgment calls to you.

Pass 1: Code each transcript into atomic observations

Coding means breaking each transcript into small, single-idea observations before you interpret anything. You are not looking for themes yet; you are inventorying raw material, one participant at a time.

Feed the model one transcript per prompt, never the whole stack at once. Ask it to return structured output — a table or JSON — where each row is a single observation with four fields: the participant ID, the verbatim quote, a short code label, and a flag for whether the code is stated by the participant or inferred by the model. That last field is your early-warning system for interpretation creeping in too soon.

A worked example makes the granularity concrete. If a participant says, "Honestly I just export everything to a spreadsheet on Friday because I don't trust the dashboard," that becomes one code for the Friday export ritual and a second, separate code for distrust of the dashboard — two atomic observations, not one blurry "reporting" bucket. Splitting them now is what lets Pass 2 discover that distrust, not exporting, is the actual theme.

Keep the codes descriptive, not evaluative. "Exports data to spreadsheet every Friday to reconcile numbers" is a code. "Wants better reporting" is a conclusion you have not earned yet. The discipline echoes grounded theory's open coding: let the labels stay close to the data and resist jumping to your solution. Every time you write a code that already implies a feature, you have skipped ahead of the evidence. For copy-paste prompt templates and a starter coding rubric, see our deep dive on coding interview transcripts with an LLM.

Pass 2: Cluster codes into themes

Clustering groups the individual codes into themes across all participants — this is where patterns emerge, but only from material you already extracted and verified. Crucially, you feed the model the codes from Pass 1, not the raw transcripts again.

Why the distinction matters: if you hand it raw transcripts, it re-reads, re-interprets, and can invent connections that were never in your coded data. Feeding it the structured codes constrains it to the evidence. Ask it to propose candidate themes, list which codes roll up into each, and name the codes it could not place. The orphans are often where a second, quieter insight hides.

Then review every merge yourself. The model will happily fold "too expensive" and "hard to justify to my boss" into one theme, but those are different problems — one is pricing, the other is internal selling. In Teresa Torres's Continuous Discovery Habits, this is the move from scattered observations toward an opportunity solution tree: distinct needs kept distinct, not averaged into mush. You are the arbiter of what counts as one theme versus two.

Watch especially for the model collapsing a symptom into its cause. "Report takes too long to build" and "I don't trust the numbers" can look adjacent, but one is a speed problem and the other is a trust problem, and they point at completely different products. When you are unsure whether two codes belong together, keep them apart — over-splitting is cheap to fix later, while a premature merge quietly buries a distinct opportunity you will never think to look for again.

Pass 3: Quantify by distinct participants, not mentions

Quantifying means counting how many different people raised each theme, not how many times it was said in total. This single reframe prevents more bad roadmap decisions than any other step in the method.

Here is the trap. One articulate, talkative participant might mention the same frustration nine times across a call. A naive count logs nine hits and promotes it to your top theme — when in reality it is the strongly-held view of exactly one person. Prevalence by distinct participant tells you whether a pattern is shared or whether one loud voice is dominating your data.

Ask the model to produce a table: theme, number of distinct participants who raised it, and the participant IDs. The IDs are non-negotiable, because they make the count auditable — you can spot-check any number by tracing it back. Resist the urge to treat these figures as statistics. With a dozen interviews you are measuring direction and salience, not market share. "Six of nine participants" is a signal worth chasing; it is not "67% of the market."

It also helps to record the opposite figure: how many participants had a natural opening to raise a theme and did not. A pain that six of nine people volunteered unprompted is stronger evidence than one you had to fish for with a leading follow-up. Keeping both numbers in view — who raised it freely, and who stayed silent — protects you from reading enthusiasm into a room that was merely being polite.

Pass 4: Verify quotes and hunt disconfirming evidence

Verifying means checking that every quote is real and every theme survives a deliberate search for evidence against it. This is the pass founders skip, and skipping it is how a hallucinated quote ends up in a board deck.

Run verification as its own prompt. For each theme and each quote, ask the model to point to the exact participant ID and confirm the quote appears verbatim in that transcript. Any quote it cannot locate gets deleted, no exceptions. Then flip the lens: explicitly ask "which participants contradicted this theme, or showed evidence against it?" Disconfirming evidence is the fastest route to an honest read of your idea, and models will not volunteer it unless you demand it.

One more safeguard is worth the extra call: run verification in a fresh context, treating the model as a skeptical reviewer rather than the author of the themes. A model asked to defend its own output in the same thread tends to rationalize; the same model handed the themes cold and asked "which of these is weakly supported?" will flag the shakier claims it would otherwise have protected. Once themes survive verification, the real work begins — turning validated pain points into prioritized, sized opportunities and product bets. That translation from evidence to decision is its own discipline, covered in turning interview data into product insights.

Manual coding vs LLM-assisted coding: the real tradeoffs

LLM-assisted coding wins on speed, consistency, and scale, while manual coding wins on nuance and judgment — so the right answer is almost always a blend, not a winner. Understanding exactly where each approach is strong tells you which parts of the work to hand off and which to keep.

Here is how the two approaches compare on the dimensions founders actually weigh when deciding how much to automate:

The pattern is clear: let the machine do the mechanical, repetitive coding at speed, and reserve your human attention for interpretation, merges, and the sensitive calls. The four-pass method is essentially a division of labor along exactly this line — machine extracts, human decides.

Two caveats keep this honest. First, LLM consistency is only as good as your rubric; an ambiguous coding instruction produces confidently consistent nonsense. Second, the model's nuance weakness is not fixable by better prompting alone. If a hesitation, a burst of sarcasm, or a long meaningful pause carried the real signal, that lived in the room and rarely survives into a transcript at all. Some things you simply had to have heard yourself.

Preventing hallucinated quotes and invented themes

You prevent hallucinated quotes by forcing every claim to carry a traceable source, and you prevent invented themes by making the model prove prevalence rather than assert it. Hallucination is not random; it thrives in exactly the conditions loose prompting creates.

The mechanism is worth understanding. A model generates plausible text, and a plausible-sounding quote is easy to fabricate when nothing anchors it to a specific transcript. Remove the anchor and you are asking for invention. Add the anchor and you make invention structurally hard. A few practical controls do most of the work:

• Demand verbatim quotes with participant IDs. Every quote must map to a specific transcript and appear there word for word. No ID, no quote.
• Make "not found in the data" a valid answer. Explicitly tell the model that returning nothing is better than filling a gap. Models pad because we implicitly reward completeness; remove that pressure.
• Keep chunks inside the context window. One transcript per pass, so nothing gets truncated or half-remembered. Overflowing the window is a leading cause of invented detail.
• Use low temperature and a fixed schema. Deterministic settings and a rigid output structure leave less room for creative embellishment.
• Verify in a separate pass. The same model that generated a theme is a weak checker of its own work in the same breath; a fresh prompt scoped only to verification catches more.
• Spot-check a human sample. Read two or three transcripts fully and confirm the coded output matches. If it drifts, your rubric or your prompt needs tightening.

None of these controls are exotic, and that is the point. Hallucination in synthesis is rarely a mysterious model failure; it is the predictable result of asking an unconstrained question and rewarding a complete-looking answer. Constrain the question, make gaps acceptable, and demand a source for every claim, and the fabrication rate drops sharply — not because the model got smarter, but because you stopped creating the conditions it needs to invent.

For a deeper treatment of the failure modes and prompt patterns specific to this problem, see how to reduce hallucination in interview summaries. Treat every unverified quote as a liability until it is traced — this is the same disciplined, source-anchored synthesis workflow Edmired builds around, and it is the difference between evidence and confident fiction.

Key Takeaways

• Never ask an LLM to "summarize" interviews. Summarization optimizes for a readable paragraph, which averages away the outliers and verbatim language where your sharpest insights actually live.
• Separate extraction from interpretation. Running code, cluster, quantify, and verify as four independent passes gives you four checkpoints instead of one opaque leap from transcript to conclusion.
• Count distinct people, not total mentions. One talkative participant repeating a point nine times is not nine data points; prevalence by unique participant keeps a single loud voice from hijacking your roadmap.
• Every quote needs a participant ID. A quote the model cannot trace to a specific transcript is a hallucination until proven otherwise, and it should be deleted on sight.
• Ask for disconfirming evidence on purpose. Models will not volunteer the participants who contradicted a theme; you have to demand the counter-evidence, and it is the fastest path to an honest read.
• Keep a human on interpretation and merges. The machine codes and counts reliably; deciding whether "too expensive" and "hard to justify" are one theme or two is judgment only you can own.
• Great synthesis cannot rescue weak interviews. Non-leading questions and the right participants upstream determine the ceiling on everything the LLM can extract downstream.

Frequently Asked Questions

Can I just paste all my transcripts into ChatGPT and ask for themes?

You can, but you should not trust the result. A single mega-prompt collapses coding, clustering, counting, and verification into one opaque step, so themes arrive with no traceable source and paraphrased quotes look identical to real ones. Split the work into separate passes and force participant-ID citations. The extra fifteen minutes is what separates evidence you can act on from a confident-sounding guess.

How many interviews do I need before LLM synthesis is useful?

Synthesis becomes useful once themes start repeating across participants, which for early discovery often begins to show around five to a dozen conversations. Fewer than that and you are reading anecdotes, not patterns. More importantly, treat the counts as direction and salience, not statistics — "six of nine participants" signals where to dig next, not a market-share percentage you can put in a forecast.

How do I stop the LLM from inventing quotes?

Force every quote to carry a participant ID and appear verbatim in that transcript, then run a separate verification pass whose only job is to confirm each quote exists. Keep one transcript per prompt so nothing overflows the context window, use low temperature, and explicitly permit "not found in the data" as an answer. Any quote the model cannot locate gets deleted, no exceptions.

Should I count how many times something was mentioned or how many people mentioned it?

Count how many distinct people mentioned it. Total mentions reward whoever talked most, so one articulate participant repeating a frustration can masquerade as your top theme. Prevalence by unique participant tells you whether a pattern is genuinely shared or driven by a single loud voice. Ask for the participant IDs alongside each count so any number stays auditable and you can trace it back.

Does LLM synthesis replace manual coding entirely?

No. LLM-assisted coding wins on speed, consistency, and scale, but manual reading still wins on nuance, subtext, and sensitive judgment calls. The reliable approach is a blend: let the model do the mechanical first-pass coding and counting across many transcripts, then keep a human on interpretation, theme merges, and spot-checking a sample. For small, high-stakes studies, close manual reading may still be the safer primary method.