Part 4: Urgency — The High Cost Enemy of Data & AI Initiatives

See Part 1 here; Part 2 here; and Part 3 here.

Acknowledging Niklas von Maltzahn for his review of this post.

“Why was the rushed AI project always talking about "technical debt"? Because it kept taking out loans of strategic planning just to meet urgent deadlines”

In the first chapter of my favourite business strategy book, Playing to Win: How business strategy really works, Alan George Lafley (CEO of Procter & Gamble from 2000–2010) and Roger L. Martin (Dean of the Rotman School of Management) define strategy as:

an integrated set of choices that uniquely position a firm in its industry so as to create sustainable advantage and superior value relative to the competition.

These choices determine how much success or failure a company will have in the market relative to its competition. In the book, they outline the Strategy Choice Cascade — a framework to help guide the strategic decision making process, which I have already discussed in Part 3.

They then describe a challenge often faced by companies:

Making choices is hard work, and it doesn’t always fit with all the other work to be done. In our view, far too few companies have a clear, choiceful, and compelling winning strategy in place. Too often, CEOs in particular will allow what is urgent to crowd out what is really important. When an organisational bias for action drives doing, often thinking falls by the wayside.

I believe this trade-off between doing and thinking is applicable to the role of almost any employee: the fewer short term deadlines, the more space one has to think strategically; consider what business problems should be prioritised; and find innovative ways to adapt existing solutions and processes so that they are more efficient.

Ever heard of a team of savvy data ninjas and skilled product managers, working on a data & AI project for months to years that hardly added business value (indeed according to a RAND corporation study, 80% of Data & AI projects fail)? Probably… and what’s worse is that often these teams consist of skilled, hard working, and well intentioned individuals.

The root cause can be multi-faceted: inefficient dependency chains; misaligned priorities across the company; culture and change management struggles; poorly designed data systems; and too few employees with cross-disciplinary expertise (e.g. business & AI understanding). However in every case, a high delivery, high pressure work environment does not solve these problems, and can sometimes amplify them.

The trade-off in a simple conceptual diagram:

In the fast-paced world of business, “urgency” is often seen as a virtue. It signifies momentum, a drive to deliver quickly, a response to market demands. And in many areas, a bias for action is indeed critical. However, when it comes to Data and AI initiatives, urgency is often an insidious enemy, silently sabotaging long-term success and racking up hidden costs.

We live in an era where AI products clearly have great potential. Companies are eager to find hidden insights, automate tasks, create smart products, and get an edge over their competitors. This excitement, while positive, frequently translates into pressure to deliver results now. Stakeholders want a dashboard yesterday, a predictive model next week, and a fully autonomous system by the end of three months.

You can’t have your cake and eat it too. This intense pressure forces teams to cut corners. Data cleaning is rushed or skipped; testing is minimal — only testing what happens in common situations rather than edge cases; sound architectural design patterns are neglected; code creeps into towards spaghettification; and documentation is too often deprioritised.

When urgency dictates the pace, teams achieve an illusion of speed. Something might get delivered quickly, but it’s often fragile, unreliable, and built on shaky foundations. This is where the second, equally critical problem emerges: the accumulation of complexity.

Everything is a design choice. When you take shortcuts under pressure, you build up technical debt. For example, a data pipeline built in a hurry might be fragile and break easily when the data changes slightly. A model deployed quickly might not have proper monitoring, leading it to become less accurate over time without anyone noticing. An urgent, one-off connection between systems creates a messy link that’s difficult to manage later.

Each “quick fix” or “temporary solution” adds to this debt. What starts as a straightforward script can become a tangled mess of code. Data sources are cobbled together imperfectly. Different parts of the system are built using inconsistent methods or tools because the focus was on getting it done fast.

This isn’t just messy; it’s profoundly costly:

The price of this accumulated complexity isn’t always obvious upfront, but it becomes very impactful over time:

• Maintenance Burden: A large portion of time shifts from developing new capabilities to fixing existing issues. Even small updates become difficult because changing one element can have unintended consequences elsewhere.

• Reduced Flexibility: Adapting to new requirements or adding new features becomes slower and riskier. The systems intended to enable business agility instead become obstacles.

• Scalability Issues: Systems optimised for quick initial delivery often struggle to efficiently handle increasing data volumes or user loads, eventually requiring expensive and disruptive re-architecture.

• Impact on Talent: Engineers and data scientists can become frustrated working with unstable, complex systems, potentially leading to lower morale and challenges in attracting and keeping skilled team members.

• Onboarding Time: The more complex data systems are, the longer it will take new employees to be productive.

• Security Vulnerabilities: Patchwork systems can inadvertently introduce security gaps that are hard to identify and address comprehensively.

• Stifled Innovation: The team becomes so preoccupied with managing the existing technical debt that they have limited capacity or time to pursue new, innovative data and AI initiatives. This relates to the Doing-Thinking trade-off I mentioned earlier.

In essence, the perceived “speed” gained by rushing projects incurs a significant cost in the form of crippling complexity and ongoing expenses. While the initial project might be considered a success for being delivered, its long-term value is often substantially reduced compared to an approach that prioritised sustainable development from the start.

The difficulty here is the Prisoner’s Dilemma (also discussed in Part 1) which is often present in the workplace. In a nutshell this tempts employees in the direction of short-term personal deliverables and disinterest in cooperation across teams, resulting in suboptimal outcomes for the company as a whole. Another way to think of it: how many employees care about the success of a company or the teams they were once part of once they leave the company?

Because Data & AI initiatives need to generate positive ROI longer than the average employee turnover rate, systems should be designed in a way that accommodate that.

Here are some solutions to move away from the trap of urgency which inevitably results in costly complexity:

• Culture: “Culture eats strategy for breakfast”. Although not a technological factor, culture is too important to ignore. No matter how well-designed your strategic plan is, it will fall flat unless your team shares the appropriate culture. Foster a positive and collaborative culture, that balances both short and long-term wins for the company.

• Prioritise ruthlessly: Not everything is genuinely urgent. And more crucially, if you work on the wrong thing you’re never going to have an positive business impact. Focus on initiatives with the highest potential impact and allocate appropriate time and resources.

• Favour simple solutions over complex ones: In my experience, technically-skilled employees tend to have a bias toward the latest and greatest tools, coding solutions, algorithms, and architectural design patterns. Often less is more and sometimes that means questioning the status quo of the past decades. For example, do you need separate Development (DEV); User Acceptance Testing (UAT); and Production (PRD) environments? Not always…

• Unified Data Platforms: Although the software costs are more expensive than building data systems up from scratch, the hidden costs (e.g. maintenance, onboarding time) can be reduced due to the guardrails that constrain complexity, as well as the user-friendly UIs. These platforms (e.g. Foundry, Databricks, Fabric) can facilitate continuity beyond employee turnover.

• Invest in foundations: Prioritise robust data infrastructure; clean data pipelines; standardised naming conventions; thorough unit testing; coding & MLOps best practices, etc, before rushing to deploy models.

• Proof of Concepts: You’ll never stop hearing me emphasise the importance of POCs. POCs are an approach to gauge the potential of an idea, without racking up costs over months. During this short phase, forget all the best practices discussed above. Only once the value-add becomes clear, build it properly.

Ultimately, the choice between the urgent sprint and the strategic ultramarathon defines the trajectory of our Data & AI initiatives. While the pressure for immediate results is constant, yielding to the illusion of speed inevitably leads to the crushing burden of technical debt and complexity, undermining the very value we seek to create. By embracing this patient, deliberate approach, we move beyond the cycle of rushed projects and costly fixes, building Data & AI capabilities that are not only delivered, but truly endure and drive meaningful business impact.