From Big Data to Better Explanation
Rethinking Consumer Insight in the Age of LLMs
The All Data Illusion
A beauty brand once came to me with a request:
“We want to mine all the data from social media—X, TikTok, Instagram, Rednote… everything. Then we’ll analyze it to guide new product development, benchmark competitors, and extract selling points.”
It sounded like the holy grail: billions of posts revealing the truth about what customers want.
But then the questions began:
Is it even legal? (Most platforms forbid full scraping.)
How much would it cost? (Millions per platform.)
How fresh would it be? (Real-time? Daily? Weekly?)
How do we clean the noise—bots, ads, fake reviews?
Most importantly: how do we turn all that noise into real insight?
That week, I also heard about a 40-person start-up that didn’t scrape anything. They interviewed just 30 customers, spotted an emerging niche, and dominated it within six months.
Why?
The Orange Juice Test
Imagine two labs studying orange juice.
Lab A has cutting-edge spectrometers. They can tell you the exact chemical breakdown: 85.97% water, 4.23% sucrose… every number to two decimal places.
Lab B has one goal: make something that tastes like fresh-squeezed orange juice. They tweak and test until nine out of ten tasters say, “This is it.”
Lab A gets what is “real” — the objective composition of orange juice.
Lab B gets what is “true” — the subjective experience of orange juice.
In business, accuracy tells you the boundaries (what’s safe, legal, feasible). Truth tells you the direction (what people will pay for, love, and remember).
We measure every click, yet miss the desire behind it.
We track conversion rates, but overlook the motive.
We count keywords, but ignore the emotions they hide.
The Limits of Big Data
Back to the beauty brand’s story. Their data analysis was impeccable:
“Natural ingredients” mentions up 892%
“Sensitive-skin friendly” engagement up 34%
Competitors priced $35–45
Target audience active on Instagram from 8–10 p.m.
They launched a $39.99, natural, sensitive-skin line, with 70% of ad spend in that time slot.
What they didn’t see:
When 25-year-old Emma said “natural skincare,” she meant “a simpler life.” She missed the ease of her university days.
When 28-year-old Jessica said “sensitive skin,” she meant “I’m done being disappointed.” Her vanity held 17 half-used bottles—each a failed experiment.
The winning start-up asked different questions: not “What ingredients do you like?” but “What’s on your mind when you look in the mirror in the morning?”
Their copy didn’t say “98% natural ingredients.” It said “Skincare in three simple steps.” Not “For sensitive skin,” but “We tested 1,000 times so you don’t have to.”
The Problem of Induction
Physicist David Deutsch tells the story of the Thanksgiving turkey in his book The Beginning of Infinity:
Day 1–364: Food arrives at 9 a.m.
Conclusion: “Food always arrives at 9 a.m.”
Confidence: 99.7%.
Day 365: Thanksgiving. No food.
No matter how many data points you have, you can’t guarantee the next one. This is the flaw of induction.
Data can mislead in three ways:
Logic — You can’t generalize from all white swans to all swans being white.
Practice — Correlation isn’t causation. Ice cream sales and drownings rise together because of summer, not because ice cream kills.
Epistemology — Knowledge isn’t extracted from data; it’s created through conjectures.
Science advances not by gathering more facts, but by making bold guesses—Einstein imagining constant light speed, Darwin imagining natural selection, Wegener imagining drifting continents.
What Makes a Good Explanation?
Deutsch’s rules, in business language:
Hard to Vary — “Users didn’t buy because the price was too high” is too easy to change. “Millennials reject anti-aging because it conflicts with self-identity” is harder to fudge.
Testable — “Women buy skincare for a sense of control” can be tested by comparing ad copy that emphasizes control vs. care.
Explanatory Depth — “Users like minimal design” is shallow. “Information overload makes minimalism signal trust” explains why and predicts other behaviors.
Big Data vs. Good Explanations
Good explanations are risky; data feels safe. “The numbers told us…” sounds better in a boardroom than “I have a theory…”
Data scales; human insight doesn’t. One good researcher might produce ten strong hypotheses a year; a dashboard spits out 10,000 charts a day.
And we’ve lacked tools to scale good explanations—until now.
LLM Scaling the Conjecture
Large language models (LLMs) can simulate not just data patterns, but the mental models behind behavior.
Our project, Atypica, builds “real-person agents” from interviews or social media content. These agents think, feel, and decide with 85% accuracy compared to real consumers.
Instead of just asking what people say, we can ask: “What psychological mechanism would produce that statement or choice?”
Example:
Social listening might show “creative packaging” is popular.
Atypica reveals that some buyers are “gift explorers” seeking items that signal taste, and that mini sets reduce the risk of gift failure—not just “offer variety.”
In other words: LLMs give us a way to mass-produce better conjectures—and then test them.
The Infinite Beginning
As Deutsch says, each good explanation sparks a new question, and each new question demands a better explanation.
For the first time, we have the computing power to test at scale and the cognitive tools to guess at scale.
Big data can check our theories; LLMs can generate them.
It’s a new era—not of more numbers, but of better questions. Atypica is still an early attempt…