The Research Industry Faces its Innovator's Dilemma

In early August, a UN report went viral revealing that most UN reports go unread. Fewer than 5% reach 5,500 downloads. Ignoring the irony of this revelation appearing in yet another report, the diagnosis is useful. And it affirms what many people have whispered for years. 

However, this is not only a UN problem. It’s the trend across the research industrial-complex which has been optimized for the wrong metrics. 

In Why you should be active on social media, I wrote:

It is a sobering fact that some 90% of papers that have been published in academic journals are never cited. Indeed, as many as 50% of papers are never read by anyone other than their authors, referees and journal editors.

While this is hugely problematic, it also presents a massive opportunity. 

In a 2022 newsletter titled "The AI Unbundling", tech analyst Ben Thompson argued that the history of human communication can be understood as a long process of unbundling the idea propagation value chain - creation, substantiation, duplication, distribution, and consumption, by removing bottlenecks at each step.

The printing press removed the duplication barrier, the internet shattered the cost of distribution. Newspapers, music, movies, books and publishing, retail and commerce have forever been transformed with category incumbents losing, and new winners emerging.

And now AI is dissolving the final bottleneck: the act of creation and substantiation itself. By automating substantiation (turning loose ideas into structured outputs) AI is transforming not just content creation but the core value engine of entire industries.

Every industry will face the same pressures of zero marginal cost, commoditization, and unbundling. Including the research industrial complex.

Traditional knowledge centres like research institutes, universities, think tanks, and UN agencies invest heavily via budgets, staff time, and career incentives into the first two stages, Creation and Substantiation. 

• Creation: research proposals, data collection, rigorous analysis. 

• Substantiation: report writing and publication.

Over the years, these institutions have built world-class capabilities and reputations in Creation and Substantiation of evidence but everything downstream remains underfunded and under-resourced. 

Now with AI in the equation, the moat these institutions built around Creation and Substantiation is eroding. Tools that once required teams of experts and months of work, such as literature reviews, draft reports, and data visualizations, can now be replicated (at least passably) in minutes. This doesn’t mean quality research is obsolete. But it does mean its competitive advantage is shrinking. Writing, analysis, and synthesis, once considered scarce, are becoming cheap and abundant.

The implications of this are enormous and presents a fundamental shift in supply-demand dynamics within the industry. 

Key Insight: The research industrial complex is experiencing the same unbundling that hit media, retail, and finance. The question is who will capture the value in the new stack?

Obviously, the major opportunities today are in the areas that have been underfunded and the industry has least innovated:

If knowledge generating institutions are to remain relevant, we have to tip more resources down the value chain.

1) Packaging: turn results into decisions people can use. Decisions are made with tools and shared narratives, not just reports. Packaging converts one-off insights into repeatable units that the right people can carry into rooms you will never enter.

2) Distribution: research organisations will not outspend large corporations on media. But they can out-earn attention by publishing a consistent, searchable trail of knowledge products designed specifically for the internet + AI era. 

3) Adoption: engineer the “yes”. Barriers to adoption are often operational friction and risk. This is something I’m still thinking about. 

4) Consumption: I’d argue that the value of news is less than the value of research-backed insight as insight has a longer shelf life. But insight without reinforcement fades. Strategic repetition through newsletters, convenings, internal training, or embedded partnerships creates recall, reuse, and eventually routinization.

Essentially, knowledge creation centers must evolve into knowledge sharing and decision-support entities. Their future relevance depends on whether they can become high-leverage distribution and decision infrastructure. This means treating Packaging, Distribution, Adoption, and Consumption not as afterthoughts, but as design problems, each requiring its own logic, incentives, and skill sets. Often this won’t look elegant. It will look like stitching pieces together until ideas actually move and decisions shift. But that is how change happens in the real world.

Take agriculture. For decades, weather and soil research was locked up in technical bulletins few farmers ever read. Then came a shift downstream. Insights were packaged into SMS-based weather alerts and WhatsApp advisory bots, delivering guidance in plain language. Distribution piggybacked on existing infrastructure i.e. partnerships with telcos and community radio, which let the message reach millions. Adoption rose because the advice was specific and low-friction (“plant next week, fertilizer prices rising in 10 days”). And because messages arrived predictably with the planting cycle, consumption became routine, not one-off.

This illustrates how research becomes infinitely useful when it is embedded in existing systems and tied to the decision rhythms of end users.

So the big money question now is how to translate this into the research space: how to make evidence not just available to policymakers, but woven into their routines, so it shapes decisions the way weather alerts shape planting.

It sounds like more work and it is. AI + Internet is a new continent. And the opportunities are for those willing to reinvent themselves.

There are several signals that this is the right time for reinvention:

Emerging Institutional Realities

Funding mechanisms like Horizon Europe now place explicit emphasis on societal impact, not just number of citations. Applicants are expected to articulate value beyond academic outputs, how a project addresses EU priorities, contributes to SDGs, and leads to societal benefits. In the UK, the Research Excellence Framework (REF) now requires narrative case studies to determine how much money a university gets. If your work can’t show impact in the real world, the system won’t pay you.

Even the ongoing UN reforms (and the cited report at the beginning of this piece) mention reducing the number of reports published and being more strategic about knowledge creation. Basically, you'd be forced to increase the shelf life of your reports by telling long term stories.

The winners in this transition will be knowledge institutions that understand they're not in the report generation business but the decision-support business. They'll build direct relationships with the people who need their insights, create repeatable systems for packaging and distributing knowledge, and measure success by behavior change rather than number of reports published.

Emerging Individual Researcher Realities

Two major headwinds I see blowing across academia and the broader research community.

1. The career ladder is collapsing sideways for early to mid career researchers.

In the United States, roughly only one-third of faculty are tenured or tenure track. In medical schools it is closer to one in five. Several states have moved to weaken tenure further. For Global South scholars, the squeeze is harsher: there are fewer protected roles, more precarious contracts and promotion pipelines are narrowing. What’s the translation for early-career scholars, especially outside the Global North? Do not bank your future on a single internal promotion narrative.

2. Paper reviewer overload is intensifying.

Publish-or-perish incentives have created a surge in paper submissions to journals. As the manuscripts multiply, journals keep putting out calls for competent eyes. The labor to read has become a kind of invisible tax on the labor to write. And the trend will get worse. 

AI tools now make it easier for researchers to research, analyse and submit manuscripts faster. To keep up, publishers are experimenting with automation in peer review. We’ve seen this before in HR: CVs generated by bots, screened by bots. The losers were the early career job seekers without connections. In research and academia, it will be early-career scholars and Global South researchers who are squeezed out.

So yes, publishing your way to the top will get harder. 

Researchers and evidence-producing institutions must instead create new pathways: building demand for evidence, cultivating audiences beyond journals, and positioning expertise within broader public and policy debates. This is not a retreat from scholarship, but a return to its core purpose ie to make knowledge public in times of heightened complexity and noise.

A world that desperately needs it.

I have said several things in this newsletter but if there's one thing I'd love you to leave with, it's that it's no longer business as usual. The knowledge economy is experiencing its "Innovator's Dilemma". The landscape has shifted and over the next few years, we will witness dramatic shifts in academia and research, especially in the development sector. 

For individual researchers, you need to update your playbook on building a research career and driving impact as well. For institutions, you need to update your playbook so your work remains relevant as the new world emerges.

The Philosopher-Builder

I love the way this essay explores the ends vs means question. It critiques the way many modern inventors and builders pursue scale, efficiency, or market capture without asking the deeper ethical question: What is this for? It argues that technology is never neutral, it encodes values. So to build without a guiding philosophy risks embedding exploitation or control into our systems by default.