Lessons from Prof. Chase Kusterer on why most analytics work fails before it starts
Data has become one of the most overused words in startups, but in most companies the problem is not a lack of data — it is a lack of clarity on what the data is actually supposed to answer.
That is the core message from Prof. Chase Kusterer’s Growth Summit session at Orbit Ventures. A reminder that most analytics failures do not begin with bad models or weak tools. They begin much earlier, with the wrong question.
And once the question is wrong, everything built on top of it becomes expensive noise.
The Real Failure Point in Analytics
Most teams assume analytics breaks because of technical limitations. Chase argues the opposite. Analytics usually fails because no one agreed on the question in the first place.
A model gets built, a dashboard gets designed, and a pipeline gets deployed. But none of it connects back to a real decision. In many cases, the answer was never hidden in advanced machine learning or complex systems. It was already visible in something much simpler, sometimes even a basic Excel pivot table or summary view.
The issue is not capability. The issue is overengineering before understanding.
When More Data Makes Things Worse
The instinct in most teams is simple. More data will create more clarity. Chase challenges that assumption directly.
More data rarely fixes a weak question. It usually amplifies confusion. When the underlying decision is unclear, additional signals do not converge — they diverge.
Teams end up optimizing metrics that do not map to real decisions. Dashboards become reports, reports become noise, and noise starts to feel like insight.
At that point, the problem is no longer analytical. It is strategic.
Analytics Is Not Magic
There is a tendency in startups to treat analytics as something close to intelligence. Chase breaks that illusion early.
Even core techniques like regression are not mysterious systems. They are simple frameworks for improving a baseline estimate using additional signals.
A returning customer behaves differently from a first-time visitor. A larger house tends to cost more than a smaller one. Prior behavior improves prediction, but it does not guarantee certainty.
The math is not the hard part. The hard part is understanding what decision the math is actually supporting.
Domain Knowledge Beats Technical Sophistication
One of the strongest themes in the session was that technical depth alone is not enough. Chase frames it simply.
Who performs better in the real world? A statistician with no sailing experience, or an experienced sailor with no statistics background?
The answer is obvious, and intentional. Analytics is not just about modeling systems. It is about understanding reality.
Without domain context, even the best models become disconnected from how decisions are actually made. This is where many teams fail — not because they lack skill, but because they lack proximity to the problem.
The most useful insight often does not come from the dataset. It comes from being close enough to reality to understand what the dataset is missing.
The Cost of Solving the Wrong Problem
One of the most expensive mistakes in analytics is not technical failure. It is building the right solution for the wrong question.
By the time teams realize this, the cost has already compounded. Months of work have been spent, systems have been built, and stakeholders have aligned. Reversing direction is no longer incremental — it requires restarting.
This is why early alignment on the question matters more than optimization later in the process. Not after the model is built, and not after the dashboard is shipped, but before any of it begins.
Frontline Context Changes Everything
Chase’s experience at EF highlights this clearly. His analytics team was building lead scoring models, but progress was slow and inconsistent.
The shift happened when analysts stopped working only with data and started working with the people behind the data. They spent time in call centers, observed sales conversations, and saw how “bad leads” were actually handled in real life.
What changed was not the model. What changed was the definition of the problem.
A good lead was no longer an abstract label in a dataset. It became a behavior observed in real interactions. Once that shift happened, execution accelerated significantly, and projects that previously took close to a year were completed in a fraction of the time.
AI Does Not Remove the Need for Judgment
On AI, Chase is direct. Tools like ChatGPT and modern automation systems will reduce repetitive analytical work, but they do not remove the need for judgment.
What gets automated is execution. What remains is interpretation.
The ability to define the right question, interpret outputs, and connect insights to real decisions becomes more important, not less. The constraint does not disappear — it moves upstream.
Final Thought: The Question Is the Strategy
Across the entire session, one idea consistently came back. Analytics is not the strategy — the question is.
Data, models, and tools are only as useful as the clarity of the problem they are trying to solve.
If the question is wrong, everything downstream becomes optimization of the wrong outcome. If the question is right, even simple tools are enough to create meaningful decisions.
The difference is not technical. It is conceptual. And in most startups, that is the real edge.
