Lessons from Deloitte's AI disaster

I don’t know if you read the Financial Times today, but recently the Australian government hired Deloitte for what they call an “independent assurance review.” You know what governments do - they hire one of the Big Four because they need somebody credible, trustworthy, with a reputation. They pay them $440,000 Australian dollars for a report that’s supposed to be what? Accurate, independently assured, full of quality and rigor and trust, right?

Then the report comes out and it’s not just wrong, it’s not just sloppy - it’s full of AI hallucinations.

References to reports that didn’t even exist from Sydney University. Incorrect citations. Misquoted statements from someone called Justice Davis but their name is written wrong. And the problem? This was NOT discovered by Deloitte. Not by Deloitte’s own quality control. It was found by a researcher, a deputy director of Sydney University’s Health Law Research.

Now they have to repay the final installment. The whole thing is a huge disaster.

The Financial Times is being kind, but other headlines? “Deloitte to refund government after admitting using AI” and “Full refund: Senator slams Deloitte’s human intelligence problem.”

This isn’t just one mistake. The question today is: Is this the death rattle of the old consulting model?

Let’s talk about how consulting has worked for the last 50 years so we understand why I’m saying they’re on an ejector seat position.

The traditional consulting model - the Big Four, McKinsey, Deloitte, BCG, PwC - it’s essentially a pyramid model. Not a pyramid scheme, a pyramid model.

At the top you have a handful of partners. What do they do? They sell the work. They take clients out for golf, spend time with them, schmooze with the client. They’re making the serious money - a million per year at the top firms.

Below them are managers or senior consultants, engagement managers. Mid-level people who’ve been around 5, 6, 7, maybe 10 years. They know the methodologies, they know the clients, they manage the projects.

At the bottom? An army of junior consultants. Analysts. Fresh out of college kids who are really, really smart and ready to work 80 hours a week. They’re the ones who actually do the work - research, data collection, slide formatting, Excel modeling.

But the pyramid is very, very wide at the bottom. The ratio is 1 partner to 10-15 juniors. You charge the customer $500-1000 an hour for the partner, maybe $300-400 for a manager, but only $150-250 for a junior analyst. But you’re not paying that junior analyst $150 per hour - you’re only paying them maybe $35-40 an hour when you break down their salary.

The margin is not on the top. The margin is at the bottom. That’s where the profit is.

But what happens when AI can do the work of 10 junior analysts? When you open Perplexity and it can handle deep research better than any junior consultant? When Claude can do better financial analysis? When you build a graph RAG that can analyze thousands of inputs from your customer better than any junior consultant?

This is when the bottom of the pyramid collapses. Boom.

You don’t need 20 juniors anymore. You need maybe two or three people who know how to prompt AI correctly, who can verify outputs and think critically. And suddenly that margin is gone because the scaling model is gone.

That’s exactly what happened to Deloitte. They used Azure OpenAI GPT-4o - that works very, very well. It helped them produce this $440,000 assurance review report. But it had hallucinations. References that were completely non-existent. It misjudged a fake judge’s name.

But hallucinations are not the problem. Every single AI in the world hallucinates. The problem is that the human oversight was missing.

If you’re using AI when it’s customer facing, you cannot send out something to a customer without spending 40% of the time of your AI usage to check if it’s correct.

But Deloitte? They shipped this to the government without checking the report.

What does this say about the quality control processes? I’ll tell you what it says: They’re treating AI like a junior analyst that they could exploit for margin instead of teaching the people who are using AI how to use AI correctly.

That, my friends, I call organizational malpractice.

Already in June 2024 - one year and a couple of months ago - the UK regulator said: “Look guys, Big Four accounting and consulting firms, you’re not tracking how AI is affecting your audit quality. You’re using AI, but when we ask you for the logs or how your employees use AI, you’re not able to tell us.”

This is Deloitte with an annual revenue of $70 billion. They announced they’re investing another $4.5 billion in Gen AI until 2030. That’s billions with a B.

We talk with management consultants all the time - I studied an MBA, I have lots of buddies, we coach some of them - they’re all telling me exactly the same thing:

“Malcolm, the problem is if we use AI and we tell our customers, then they will come back and ask for a discount.”

And it’s totally normal! If you’ve been getting a certain output from McKinsey and they charged you half a million a year ago, and now they come back for the same project and want to charge half a million again - you’re like wait, bro, you’re bragging all the time that you use your own AI tool 500,000 times a month. So you’re more efficient and you’re charging me the same?

This is PwC’s Chief AI Officer: “Clients hear us talking about AI all the time and they want a fair share of these efficiencies.” They want them to reduce the pricing.

PwC started lowering prices on certain services because artificial intelligence has improved their internal operations.

But what are the Big Four doing? They’re taking that pyramid principle, which is already broken, and they’re sprinkling - I’m going to say sprinkle - a bit of AI everywhere. But they’re not asking: What if AI is making this pyramid obsolete?

That’s why I’m saying they’re sitting on an ejector seat position.

The old model is dying. What’s the new model? Let’s call it an obelisk - not Obelix from Asterix and Obelix - an obelisk. It’s like a pyramid but very, very small and goes up very high.

You don’t have an army of junior analysts anymore. You really only have two or three roles left.

At the bottom: Juniors, but juniors are AI agents. And anybody on this podcast who says “oh AI agents, Malcolm is living on another planet” - guys, sorry, come to the Academy. We teach you AI agents. If you’re not running 10 AI agents a day, you’re living behind the planet.

In the middle: The orchestrator. This person designs and refines AI-driven workflows. For this one I need an agent. For this one I can’t use an agent, I need my own brain. For this one I have to have a meeting with colleagues. You look at the output of the AI and you interpret it and then you translate it into strategy.

On top: The partners or the sales guy. I just call myself a sales guy. I’m responsible for sales in this company. That person talks to the clients, to the leaders, builds these deep and trusted relationships.

How are we different from McKinsey or Deloitte? We’re shit scared all the time. Even our business model - we’re afraid of being on an ejector seat position. We’re hyper-vigilant.

Claude Sonnet 4.5 came out last week - that’s the latest model. We tried it, we tested it. We have an enterprise API and we roll it out. Within 24-48 hours everybody in the company uses the latest model.

Every Friday we have lunch and learns. Every single person in the company has their 10-minute slot to pitch what they did with AI and how they’re using AI. And in the super fast-moving AI world, everybody can speak for hours, but they only have 10 minutes because they discovered 20 things last week.

When I talk to traditional firms: “How do you guys do lessons learned?” “Man, we’re too busy. Every consultant is working 60-80 hours, doing slides until 2 in the morning. We have no time to learn about AI.”

I’m like, chief, isn’t this crazy? This is the greatest leverage you will ever have and you’re squeezing your people for billable hours.

Multi-LLM verification is everything. If they don’t have a multi-LLM approach, they’re not able to even verify the output. It’s difficult to take output from ChatGPT and put it back into ChatGPT and say “can you check it?”

Every AI output needs to be checked. Every citation needs to be verified. Every statistic needs a source.

Even in preparation of this episode - I had Perplexity open and it gave me reports but there were no sources. So I prompted: “I don’t trust you. Where are the sources?” That’s the difference between an AI-first consulting company and a consulting company that uses AI.

We build solutions live with customers. When I run workshops for organizations, we build AI solutions live in real time while the customer is watching. I’m not going to go home and sit down and do a slide deck. We build these things live with them.

We have AI-powered workshops where we sit in the meeting, record with Copilot, and every 20-30 minutes we take it and build a dashboard. You go into Copilot, press the researcher agent, then you press “try Claude” and say “build me an HTML CSS dashboard based on this meeting. Be brutal and criticize me.”

You’re doing a C-level workshop and they’ve been talking for two hours. Then you prompt: “Be critical.” You know what comes out? “You’re overestimating the talent of your employees and you’re underestimating how much manual work you have. This is organizational malpractice.”

Mr. BCG or Mr. McKinsey can’t go in front of customers and say “what you’re doing is organizational malpractice.” They’ll throw him out. But when you’re an AI-native firm, you say “Look, it’s not me. I just prompted the AI to be brutally honest to you guys. No bias.”

This is the future of consulting - services that are born from AI capabilities and not limited by traditional billing models.

We don’t bill by the hour. We charge by outcome. You pay for a workshop and in that workshop we build the tool together. You see how it works, you test it, you give feedback, maybe we iterate. At the end of the day you get a working solution, not a slide deck.

Democratize intelligence. We record every meeting - every single internal and external meeting. Every conversation, every brainstorm session. Everybody in the company has access to every meeting. Even the partners meeting where we discuss strategy and pricing and who’s going to get what salary - even the junior guy who started last month has access.

For sure, sometimes we bitch about them and they bitch about us. We’re not using these meetings to spy on people. I don’t care. We’re using them because when someone comes to me and says “I need to prepare for that company, what did you do?” I say “Don’t talk to me. Go and check the transcripts.”

The transcripts are what the customer said. It’s not diluted. It has the strategy meetings on why we explained the importance of the customer. It has my daily meetings at 6 in the morning with Purni where we discussed the use cases for the customer. Everything is there.

Which means everybody in the company has access to all the pain points of our customers. If you don’t democratize information, you’re not giving your employees the possibility of mining this information.

Traditional firms? They hold information in hierarchical structures. Only the partners have access to this, only the account managers have access to that. That strategic intelligence is not available to everybody who needs it.

The money flows a lot into AI. In average, the expenditure per employee is anywhere between $800-1500 per month - LLMs, video editing, automation platforms.

If an employee comes and says “Malcolm, I need Claude Max, that costs $200 per month,” I say yeah, take it. Don’t worry. Let’s not discuss it. You think it’ll save you time? You think you’ll get better output? Take it.

But it’s not just the LLMs. It’s also access to intelligence.

Let me show you something. I’m opening my phone, screen time, last week. I have a total app usage on TikTok of 21 hours.

Let me break it down by day: Monday 4 hours, Tuesday 57 minutes, Wednesday 2 hours, Thursday 1 hour 24 minutes, Friday 3 hours, Saturday 7 hours, Sunday close to zero.

Bro, you spent 7 hours on TikTok on a Saturday?

Yeah. Because TikTok is my #1 platform for understanding AI.

You have such cool content creators - they’re always like “this workflow came out, this MCP server, you can automate this, this is the problem.” I download these videos and I send them to my team. If I look at the communication between me and the team on Saturday, there were 40 videos that I sent them. We bombard each other with TikToks.

Why is TikTok so amazing? The algorithm is so good. It knows Malcolm is in love with Claude Code - it’s going to feed me one Claude Code video after another.

Which management consultant have you spoken to that says “last week I spent 21 hours on research for AI”? Which consultant could say that unless his job is AI consultant?

That’s the difference between consulting for clients and consulting for your shareholders. If you’re trapped in that old business model, you have an employee who spends 20 hours a week on TikTok threatening your cost structure because you’re not able to free up people for 20 hours a week because you have to charge them their billable hours.

In the AI-first model, you flip this - you charge by outcome.

How Neanderthal are consulting companies hiring right now? They’re still living in the past.

First question: “Where did you get your MBA?”

OK, I need to be careful because our three partners - two of them have an MBA and the other one has a doctorate in mathematics. So I can’t trash people with degrees too much. But I really couldn’t care less where people studied. I couldn’t care less where they come from.

The first questions - probably the first 20 questions: What have you built with AI? Share your screen. Show me your projects. How would you build this workflow?

When people join our firm and they have real enthusiasm for technology first, they’re not AI-tolerant. They’re AI enthusiasts.

We know from the Accenture case that Accenture is throwing out 10,000 people - I think 13,000 people - who are not able to be trained on AI. You can imagine most of the time these are people with grey hair because Accenture is saying “I’m not going to wait until I train you in AI, I will get you out.”

But as a company, you need to not get these people in. You cannot bring somebody in today who’s not an AI enthusiast from top to bottom.

Age, by the way, is not a factor. Our most senior advisor is older than I am and he’s a total AI freak.

That’s the mindset shift. If your firm isn’t teaching you and training you how to use AI, you need to bring in people who are able to accelerate this.

Learn AI tools. Don’t only learn ChatGPT. You need to get familiar with all of them. Each of them is prompted differently. Different models are good at different things. Build yourself a personal toolkit, a personal army.

Use 40% of your time to check outputs. Especially when you’re on customer projects, spot hallucinations, double-check the output of the AIs, develop a checklist for quality control. Build AI agents that do that.

Record every single meeting. Every customer call, every strategic meeting, every company brainstorm. Don’t only record it - transcribe it, put it in a database, run a few LLMs behind it, make it searchable, democratize access to information. That helps break down silos and increases LLM adoption.

Free up your employees. I beg you. Free them up for rapid experimentation. Let them say “this week I’m taking a day off, I’m learning about AI.” Go for it.

If my employees after a certain amount of time leave my company, I know they will be one of the most sought-after employees because they will not sit in a one-month AI Academy. Their entire job is an AI Academy.

I know what you’re thinking: “My God, I have a certain way of working, this doesn’t work, I can’t do this.”

But it’s super good for you because it will not replace the human element in consulting. It will amplify it. You’re freeing up people from all the shit that they don’t like to do.

The Deloitte scandal - that $440,000 AI hallucination disaster - is not just one firm’s mistake. It’s a symptom of a bigger problem: the old consulting model is fundamentally broken. That pyramid they’re trying to still charge customers for does not work anymore.

Live from Bregenz - in a couple of hours I’m getting in my car and I’m driving 6 hours to go and work with a construction company for the next two days.

In the age of AI, learners will be the biggest winners.