Most market research lives inside surveys. Surveys are the worst place to learn what buyers actually feel. People answer surveys the way they want to be perceived, not the way they live. They round their pain into something defensible. They translate frustration into politeness. Then a research firm aggregates the polite answers, hands you a deck, and you ship a campaign that sounds exactly like every competitor's campaign because everyone is reading from the same sanitized script.
The buyer's real language lives somewhere else. It lives in the Reddit thread at 2am where they vent to a username instead of a person. It lives in the YouTube comments under a tutorial video for the tool they're frustrated with. It lives in the support ticket they typed in caps lock, then softened, then sent. It lives in the 1-star Amazon review where they explain in 700 words why a competitor's product wasted six months of their work. It lives in the sales call transcript where they say honestly, the real reason and then say something nobody put on the survey.
This essay is the practitioner playbook for extracting that signal at scale. It is the field-tested method behind the Market Research sub-service Andy ships under Business Intelligence. The output is what I call the Lexicon of Pain: a structured corpus of the actual words your best buyers use to describe the problem you solve, the alternatives they reject, and the future they want.
Why surveys lie and behavioral corpora don't
The survey problem is structural, not methodological. The moment a buyer knows they are being measured, they switch into the register they reserve for being measured. They guess what answer the researcher wants. They smooth over the embarrassing parts. They rate satisfaction at a 7 because giving a 6 feels like complaining and giving an 8 feels dishonest. The result is a corpus of socially-acceptable opinions that nobody acts on, including the buyer who wrote them.
Behavioral corpora work the opposite way. A Reddit thread captures the buyer when they are venting to other buyers, not performing for a clipboard. Nobody on r/Entrepreneur edits their post for the brand mentioned in paragraph three. A YouTube comment under a tutorial about a CRM feature catches the buyer at the exact moment they ran into the friction the tutorial is trying to explain. A 1-star review is the buyer's last word on a transaction they regret, written when they have nothing left to lose. These signals are noisy. They are unsolicited. And they tell you the truth.
Andy's portfolio carries the verbatim instruction: scrape social media, listen to their comments, and analyze competitors' reviews to find the exact words your customers use where they think no one cares to listen. That is the operating thesis. The work is structured listening at the places where the buyer thinks nobody is listening.
The five channels where buyers tell the truth
Five channels carry most of the load. Each one captures a different register of the same buyer.
Reddit threads are the deepest qualitative channel for B2B and prosumer categories. The buyer goes to a subreddit when they want to be treated as a person instead of a lead. They ask honest questions, post screenshots of broken dashboards, vent about their boss, compare two tools their company is forcing them to choose between. The subreddit thread is the best single source for understanding the world model the buyer is living in. Search for the category, the competitor names, and the specific friction verbs (the verbs matter; I cannot get our CRM to turns up better than CRM problems).
YouTube comments sit underneath the tutorial videos buyers watch when they are mid-friction. A buyer who is in pain enough to type a comment under a tutorial is exactly where you want to be. The comments are short, urgent, written in real frustration syntax. I followed every step and it still does X is the kind of comment you mine. Comments under category-defining channels (Ahrefs for SEO, Charlie Marie for ecommerce, Ben Meer for productivity) are denser than comments under brand-owned content.
Support tickets are gold if you have access. Your own tickets tell you where the people you already won are still hurting. Public ticket archives at competitors' help-center forums tell you where their customers are losing trust. The pattern in support tickets is unique because the buyer types fast, types twice, and corrects themselves mid-sentence. The truth slips out in the corrections.
1-star reviews on Amazon, G2, Trustpilot, App Store, and Google Reviews are the buyer's exit interview. They are the cleanest source for understanding what specifically broke trust and what the buyer expected. Filter for reviews under 200 words and reviews over 700 words and ignore the middle. The short ones are usually pure rage. The long ones are usually structured grievance with verbatim feature mentions and direct competitor comparisons. Both surface different signal.
Sales call transcripts are the highest-resolution channel if your team runs Gong, Fathom, or Fireflies. The phrase to search for is honestly. Buyers say honestly immediately before saying something they did not put on the inbound form. Build a query that surfaces every honestly in your last hundred call transcripts and read the next two sentences. That is your Lexicon of Pain at the highest resolution available.
The extraction pipeline, end to end
The pipeline has five stages. Each stage produces an artifact the next stage consumes.
Stage 1. Buyer hypothesis. Before you scrape anything, write down who you think the buyer is in twenty words. Solo accounting practice owner, post-pandemic, $300K to $1M revenue, sells time, hates QuickBooks but cannot leave it. The hypothesis is what you are pressure-testing. It does not need to be right. It needs to be specific enough that a Reddit search query falls out of it.
Stage 2. Channel pull. Pull the five channels listed above with queries derived from the hypothesis. Reddit's old.reddit.com search is more useful than the redesign for keyword crawling. YouTube comments require either the YouTube API or a scraper like youtube-comment-downloader. App store reviews come from Sensor Tower, Apptopia, or direct scraping. G2 and Trustpilot expose review feeds. Save raw output as plain text or JSON, one row per comment, with the source URL, timestamp, and (if available) the username preserved.
Stage 3. Normalize. Collapse all five channels into a single schema. Minimum fields: source_channel, source_url, verbatim_text, captured_at, polarity_signal (1-star vs 5-star, complaint vs question vs vent), category_tag (which sub-problem the entry hits). Pydantic V2 model on the python side, single source of truth. The normalize step is where 80% of the value compounds because subsequent steps depend on the corpus being uniformly queryable.
Stage 4. Cluster. Cluster the normalized rows by jobs-to-be-done, not by topic. The buyer is trying to [verb] [noun] [under what constraint]. The constraint is load-bearing. Reconcile QuickBooks with their bank feed without breaking historical reports is one job. Show the partner-track CPA a clean trial balance before review is a different job, even if both are about QuickBooks. Embeddings (Gemini Embedding 2 with the CLUSTERING task type per the project stack) handle the heavy lifting; you read the cluster centroids and label them by hand. Aim for 6-12 clusters at the top level. More than that is noise. Fewer is over-aggregation.
Stage 5. Surface. The output is not a slide deck. It is a structured artifact: a table of jobs, ranked by frequency and emotional intensity, each row linking to the verbatim quotes that defined it. The operator who reads the artifact should be able to write copy from it without translating, because every row already contains the buyer's actual phrasing. Intelligence Engineering covers the broader framework for moving raw signal into decision-grade artifacts; this essay is the market-research-specific application.
The Lexicon of Pain as deliverable
The Lexicon of Pain has three sections. Each one is a category of buyer language that downstream functions consume differently.
Problem language. The verbs and nouns the buyer uses to name what is broken. Not inefficiency; the actual phrase, which is usually closer to I keep losing the thread when or nobody on my team can answer when the CFO asks. Problem language is the input to Sales & Growth headline writing. The headline that mirrors back the problem in the buyer's own words is the headline that converts. Anything else is a translation cost the buyer pays.
Rejection language. The alternatives the buyer has tried and abandoned. I tried [competitor] but it did X. I had a contractor do this and they Y. We bought [tool] for the team and Z. Rejection language is the input to positioning. You position against what the buyer has already rejected, not against the category abstract. The buyer who rejected three competitors before finding you is not skeptical of you; they are skeptical of category-shaped solutions and primed for whatever does not look like the previous three.
Future language. The before-and-after the buyer imagines. Not I want to scale; the actual phrase, which is usually I want to walk into the Monday meeting and have one number I can defend or I want to stop being the only person who knows where this lives. Future language is the input to offer design. The offer that names the future state in the buyer's own words is the offer that closes. Andy's portfolio puts it directly: clients don't buy deliverables. They buy a better version of themselves. The Lexicon of Pain is how you find out which version.
Case evidence: where this method has paid
Three cases anchor the method.
The home-decor metal-art brand was stuck at $10K monthly while a competitor scaled past them on a personality angle. The team had been copying the competitor's male-founder positioning. Behavioral analysis of who was actually buying, pulled from the Facebook ad data exports, surfaced the real buyer: women 45 and over, on iPhone, purchasing for emotional gift moments. The phrasing in their reviews and comments was nothing like the male-founder positioning. It was about family permanence, about home as identity, about the wall behind the dining table. The Lexicon shifted the messaging to Family is Forever and Make Your House a Home, and the campaign moved from $10K to $150K monthly inside ninety days. See Home Decor E-Com. The signal was already in the data; surveying nobody would have surfaced it.
The forensic ad audit pattern carries the same logic at the operational layer. Audits regularly recover 10 to 30 percent of ad spend by reading the structural pattern in account history rather than running a survey of which campaigns feel important. The buyer's actual behavior, captured in clicks, add-to-carts, and refund rates, is the corpus. The Lexicon of misallocated intent is in the data. See Forensic Ad Audits for the full pattern.
The negative case matters too. The garden metal-art brand had a beautiful product, real engagement, click-through rates of 3 to 8 percent, and three rounds of paid testing at roughly $3K each. The Lexicon work surfaced the truth fast: high curiosity, low purchase intent, no path to repeatable cold-traffic conversion at the available budget and site quality. The founder got a clear answer in weeks instead of a slow bleed over a year. Honest research kills the wrong bet quickly. See Garden Metal Art.
Anti-patterns to avoid
Reading the executive summary of someone else's research. An industry report tells you what an analyst thinks. The Lexicon tells you what your buyer actually says. The two are not interchangeable. The industry report is upstream of the Lexicon at best, irrelevant at worst.
Asking the buyer what they want. Henry Ford's apocryphal quote about faster horses is overused but the underlying error is real. The buyer cannot describe a future state they have not seen yet. They can describe the friction they hit yesterday. Mine the friction. Build the future.
Sample sizes too small to cluster. Twelve Reddit threads is anecdote. Two hundred is qualitative data. Two thousand is a corpus. The signal density at small sample sizes will mislead you because the loudest comment will dominate. Get the volume up before the synthesis runs.
Treating the Lexicon as a one-time project. Buyer language drifts. New competitors enter. Macro events change what the buyer worries about. Refresh the corpus quarterly at minimum and event-trigger refreshes when something specific shifts (a competitor exits, a new platform launches, an industry incident). The Lexicon is a living artifact; the dashboard around it (frequency over time, sentiment by cluster, emerging cluster detection) is its own deliverable.
Where to start
Three starting points, in order of difficulty.
Easiest, do today. Read the last fifty 1-star reviews of your closest competitor on G2 or Amazon or Trustpilot. Tag the verbs. Underline the constraint phrases (without, cannot, still). One hour, one document, and you will know more about your buyer than most product teams know after a quarter of customer-development calls.
Medium, this week. Pull two hundred Reddit threads in your category using the search verbs you derived from the buyer hypothesis. Normalize them into a single sheet. Cluster by hand into six to twelve jobs-to-be-done. Write one paragraph per cluster citing two verbatim quotes. That paragraph set IS your messaging brief for the next quarter.
Hardest, this month. Build the persistent pipeline. Five channels, one normalized schema, embedded clustering, a dashboard that surfaces emerging clusters and tracks frequency over time. The pipeline is what compounds. The first run is research. The tenth run is institutional memory. By month six, you will know your buyer better than they know themselves, which is the only durable advantage in any market that has more than three competitors.
Surveys ask the buyer to perform. Behavioral corpora catch the buyer being honest. The Lexicon of Pain is what you build out of the second one. Every downstream decision (copy, offer, positioning, pricing, retention) gets sharper when the operator stops translating their assumptions and starts mirroring their buyer's actual language. For the broader frame, see Intelligence Engineering. For the qualification gate that uses this Lexicon to filter inbound, see The Velvet Rope.