Most market research lives inside surveys. Surveys are the worst place to learn what buyers actually feel. People answer surveys the way they want to be perceived, not the way they live. They round their pain into something defensible and translate frustration into politeness. Then a research firm aggregates the polite answers, hands you a deck, and you ship a campaign that sounds exactly like every competitor's campaign because everyone is reading from the same sanitized script.
The buyer's real language lives somewhere else. It lives in the Reddit thread at 2am where they vent to a username instead of a person. It lives in the YouTube comments under a tutorial video for the tool they're frustrated with. It lives in the support ticket they typed in caps lock, then softened, then sent. It lives in the 1-star Amazon review where they explain in 700 words why a competitor's product wasted six months of their work. It lives in the sales call transcript where they say honestly, the real reason and then say something nobody put on the survey.
This guide lays out how to extract that signal at scale. It's the field-tested method behind the Market Research service I ship under Business Intelligence. The output is what I call the Lexicon of Pain: a structured corpus of the actual words your best buyers use to describe the problem you solve, the alternatives they reject, and the future they want.
Why surveys lie and behavioral corpora don't
The survey problem is structural, not methodological. The moment a buyer knows they're being measured, they switch into the register they reserve for being measured. They guess what answer the researcher wants and smooth over the embarrassing parts. They rate satisfaction at a 7 because giving a 6 feels like complaining and giving an 8 feels dishonest. The result is a corpus of socially-acceptable opinions that nobody acts on, including the buyer who wrote them.
Behavioral corpora work the opposite way. A Reddit thread captures the buyer when they're venting to other buyers, not performing for a clipboard. Nobody on r/Entrepreneur edits their post for the brand mentioned in paragraph three. A YouTube comment under a tutorial about a CRM feature catches the buyer at the exact moment they ran into the friction the tutorial is trying to explain. A 1-star review is the buyer's last word on a transaction they regret, written when they have nothing left to lose. These signals are noisy and unsolicited, and they tell you the truth.
Andy's portfolio carries the verbatim instruction: scrape social media, listen to their comments, and analyze competitors' reviews to find the exact words your customers use where they think no one cares to listen. That's the operating thesis, and the work is structured listening at the places where the buyer thinks nobody is listening.
The five channels where buyers tell the truth
Five channels carry most of the load, and each one catches the same buyer talking in a different way.
Reddit threads are the deepest qualitative channel for B2B and prosumer categories. The buyer goes to a subreddit when they want to be treated as a person instead of a lead. They ask candid questions, post screenshots of broken dashboards, vent about their boss, and compare two tools their company is forcing them to choose between. The subreddit thread is the best single source for understanding the world model the buyer is living in. Search for the category, the competitor names, and the specific friction verbs (the verbs matter; I cannot get our CRM to turns up better than CRM problems).
YouTube comments sit underneath the tutorial videos buyers watch when they are mid-friction. When a buyer is in enough pain to type a comment under a tutorial, that comment is exactly where you want to be. The comments are short, urgent, written in real frustration syntax. I followed every step and it still does X is the kind of comment you mine. Comments under category-defining channels (Ahrefs for SEO, Charlie Marie for ecommerce, Ben Meer for productivity) are denser than comments under brand-owned content.
Support tickets are gold if you have access. Your own tickets tell you where the people you already won are still hurting. Public ticket archives at competitors' help-center forums tell you where their customers are losing trust. The pattern in support tickets is unique because the buyer types fast, types twice, and corrects themselves mid-sentence. The truth slips out in the corrections.
1-star reviews on Amazon, G2, Trustpilot, App Store, and Google Reviews are the buyer's exit interview. They are the cleanest source for understanding what specifically broke trust and what the buyer expected. Filter for reviews under 200 words and reviews over 700 words and ignore the middle. The short ones are usually pure rage. The long ones are usually structured grievance with verbatim feature mentions and direct competitor comparisons. Both surface different signal.
Sales call transcripts are the highest-resolution channel if your team runs Gong, Fathom, or Fireflies. The phrase to search for is honestly. Buyers say honestly immediately before saying something they didn't put on the inbound form. Build a query that surfaces every honestly in your last hundred call transcripts and read the next two sentences. That's your Lexicon of Pain at the highest resolution available.
The extraction pipeline, end to end
The pipeline has five stages, and each stage produces an artifact the next stage consumes.
Stage 1. Buyer hypothesis. Before you scrape anything, write down who you think the buyer is in twenty words. Solo accounting practice owner, post-pandemic, $300K to $1M revenue, sells time, hates QuickBooks but cannot leave it. The hypothesis is what you're pressure-testing, so it can be wrong, as long as it's specific enough that a Reddit search query falls out of it.
Stage 2. Channel pull. Pull the five channels listed above with queries derived from the hypothesis. Reddit's old.reddit.com search is more useful than the redesign for keyword crawling. YouTube comments require either the YouTube API or a scraper like youtube-comment-downloader. App store reviews come from Sensor Tower, Apptopia, or direct scraping. G2 and Trustpilot expose review feeds. Save raw output as plain text or JSON, one row per comment, with the source URL, timestamp, and (if available) the username preserved.
Stage 3. Normalize. Collapse all five channels into a single schema. The minimum fields are source_channel, source_url, verbatim_text, captured_at, polarity_signal (1-star vs 5-star, complaint vs question vs vent), and category_tag (which sub-problem the entry hits). The schema lives in one Pydantic V2 model on the Python side, which makes it the single source of truth. The normalize step is where 80% of the value compounds because subsequent steps depend on the corpus being uniformly queryable.
Stage 4. Cluster. Cluster the normalized rows by jobs-to-be-done, not by topic. The buyer is trying to [verb] [noun] [under what constraint]. The constraint is load-bearing. Reconcile QuickBooks with their bank feed without breaking historical reports is one job. Show the partner-track CPA a clean trial balance before review is a different job, even if both are about QuickBooks. Embeddings (Gemini Embedding 2 with the CLUSTERING task type, in my stack) handle the heavy lifting; you read the cluster centroids and label them by hand. Aim for 6-12 clusters at the top level, because more than that is noise and fewer is over-aggregation.
Stage 5. Surface. The output is a structured artifact, not a slide deck: a table of jobs, ranked by frequency and emotional intensity, each row linking to the verbatim quotes that defined it. Whoever reads the artifact should be able to write copy from it without translating, because every row already contains the buyer's actual phrasing. Intelligence Engineering covers the broader framework for moving raw signal into decision-grade artifacts; this essay is the market-research-specific application.
The Lexicon of Pain as deliverable
The Lexicon of Pain has three sections, and each one is a category of buyer language that downstream functions consume differently.
Problem language is the verbs and nouns the buyer uses to name what's broken. Instead of inefficiency, the actual phrase is usually closer to I keep losing the thread when or nobody on my team can answer when the CFO asks. Problem language is the input to Sales & Growth headline writing. The headline that mirrors back the problem in the buyer's words is the headline that converts. Anything else is a translation cost the buyer pays.
Rejection language covers the alternatives the buyer has tried and abandoned. I tried [competitor] but it did X. I had a contractor do this and they Y. We bought [tool] for the team and Z. Rejection language is the input to positioning. You position against what the buyer has already rejected, not against the category abstract. The buyer who rejected three competitors before finding you is skeptical of category-shaped solutions, not of you, and primed for whatever doesn't look like the previous three.
Future language is the before-and-after the buyer imagines. Instead of I want to scale, the actual phrase is usually I want to walk into the Monday meeting and have one number I can defend or I want to stop being the only person who knows where this lives. Future language is the input to offer design. The offer that names the future state in the buyer's words is the offer that closes. Andy's portfolio puts it directly: clients don't buy deliverables. They buy a better version of themselves. The Lexicon of Pain is how you find out which version.
Case evidence: where this method has paid
Three cases anchor the method.
The home-decor metal-art brand was stuck at $10K monthly while a competitor scaled past them on a personality angle. The team had been copying the competitor's male-founder positioning. Behavioral analysis of who was actually buying, pulled from the Facebook ad data exports, surfaced the real buyer: women 45 and over, on iPhone, purchasing for emotional gift moments. The phrasing in their reviews and comments was nothing like the male-founder positioning. It was about family permanence, about home as identity, about the wall behind the dining table. The Lexicon shifted the messaging to Family is Forever and Make Your House a Home, and the campaign moved from $10K to $150K monthly inside ninety days. See Home Decor E-Com. The signal was already in the data; surveying nobody would have surfaced it.
The forensic ad audit pattern carries the same logic at the operational layer. Audits regularly recover 10 to 30 percent of ad spend by reading the structural pattern in account history rather than running a survey of which campaigns feel important. The buyer's actual behavior, captured in clicks, add-to-carts, and refund rates, is the corpus. The Lexicon of misallocated intent is in the data. See Forensic Ad Audits for the full pattern.
The garden metal-art brand is the negative case. It had a beautiful product, real engagement, click-through rates of 3 to 8 percent, and three rounds of paid testing at roughly $3K each. The Lexicon work surfaced the truth fast: high curiosity, low purchase intent, no path to repeatable cold-traffic conversion at the available budget and site quality. The founder got a clear answer in weeks instead of a slow bleed over a year. See Garden Metal Art.
Anti-patterns to avoid
Reading the executive summary of someone else's research. An industry report tells you what an analyst thinks. The Lexicon tells you what your buyer actually says. The two aren't interchangeable, and the industry report is upstream of the Lexicon at best and irrelevant at worst.
Asking the buyer what they want. Henry Ford's apocryphal quote about faster horses is overused but the underlying error is real. The buyer can't describe a future state they haven't seen yet, but they can describe the friction they hit yesterday. Mine the friction, then build the future.
Sample sizes too small to cluster. Twelve Reddit threads is anecdote. Two hundred is qualitative data. Two thousand is a corpus. The signal density at small sample sizes will mislead you because the loudest comment will dominate. Get the volume up before the synthesis runs.
Treating the Lexicon as a one-time project. Buyer language drifts. New competitors enter. Macro events change what the buyer worries about. Refresh the corpus quarterly at minimum and event-trigger refreshes when something specific shifts (a competitor exits, a new platform launches, an industry incident). The Lexicon is a living artifact; the dashboard around it (frequency over time, sentiment by cluster, emerging cluster detection) is its own deliverable.
Where to start
There are three starting points, in order of difficulty.
Easiest, do today. Read the last fifty 1-star reviews of your closest competitor on G2 or Amazon or Trustpilot. Tag the verbs. Underline the constraint phrases (without, cannot, still). One hour, one document, and you'll know more about your buyer than most product teams know after a quarter of customer-development calls.
Medium, this week. Pull two hundred Reddit threads in your category using the search verbs you derived from the buyer hypothesis. Normalize them into a single sheet. Cluster by hand into six to twelve jobs-to-be-done. Write one paragraph per cluster citing two verbatim quotes. That paragraph set IS your messaging brief for the next quarter.
Hardest, this month. Build the persistent pipeline, with five channels, one normalized schema, embedded clustering, and a dashboard that surfaces emerging clusters and tracks frequency over time. The pipeline is what compounds. The first run is research. The tenth run is institutional memory. By month six, you'll know your buyer better than they know themselves, which is the only durable advantage in any market that has more than three competitors.
Surveys ask the buyer to perform. Behavioral corpora catch the buyer telling the truth. The Lexicon of Pain is what you build out of the second one. Every downstream decision (copy, offer, positioning, pricing, retention) gets sharper when the operator stops translating their assumptions and starts mirroring their buyer's actual language. For the broader frame, see Intelligence Engineering. For the qualification gate that uses this Lexicon to filter inbound, see The Velvet Rope.