This is Alex from CyberInsights and today I am going to talk about a growing cybersecurity concern.
AI scams in an increasingly AI driven world.
Scams aren’t new, but their increasing complexity is.
To simulate this, I asked ChatGPT to generate a convincing phishing email to gain access to my Substack account. This is what it gave me:
A clearly quite convincing email.
Of course it does have some clear hallmarks of a phishing email.
Sender address: looks official, but isn’t from substack.com
Urgency: “within 24 hours” is meant to scare you.
Fake link: Although you cannot see it the link points to substack-login-verify.com which is misdirect.
Threat of loss: attackers use fear of losing subscribers/revenue to push action.
Whilst you may be able to decipher this scam quite easily, the real danger comes from the ease of creation combined with the ability to send this en masse.
It takes one broken endpoint to compromise a network.
One unsuspecting employee or one thoughtless moment could open yourself to compromise.
And this is ChatGPT, albeit a strong LLM, but with more guardrails built in than other newer generative AIs.
What about LLMs with less regulation that focus on privacy.
Whilst I believe strongly in the value of privacy, ultimate freedom in new tools opens the pathway to abuse.
By this I mean that tools such as venice.ai, which focus on a privacy first solution and lack of regulation, make phishing even easier for adversaries. The usage of these fringe tools can allow them to craft strong social engineering campaigns (learning lots about individuals/companies) to spear phish (craft convincing targeted phishing attacks).
If you are interested about venice.ai you can check out my post about it here.
The rise of generative AI in the last few years, particularly the advances in AI videos in the last few months, also leads onto a new scarier form of phishing.
Imagine your boss has uploaded videos with their voice in it onto the internet and hackers get hold of it.
They input the voice into an LLM and manipulate your boss’s voice. Now they can call you up with an adaptable script and convince you to enter valuable details. Convince you that the suspicious phishing email is actually real, etcetera.
This is Vishing.
It’s hard to detect, difficult to teach about and is something that was unthinkable not long ago.
It sounds implausible but it does actually happen.
Even back before the widespread usage of generative AI.
In 2019, a U.K.-based energy company lost $243,000 when scammers cloned the voice of its CEO and tricked a subsidiary into making a transfer.
By 2024, the stakes had escalated dramatically. Engineering giant Arup was scammed out of $25 million after an employee joined what they thought was a legitimate video call with senior management. Every voice, every face on the call was fake.
And these are not isolated incidents.
At Ferrari, a deepfake CEO call was foiled with a personal verification question.
Wiz was targeted by a fabricated voice message from its CEO, though staff noticed inconsistencies in tone and avoided the scam.
WPP was nearly duped by a sophisticated impersonation using WhatsApp and video conferencing.
As of Q1 2025, the U.S. logged over 105,000 deepfake attacks, resulting in more than $200 million in losses. This surge spans fast-moving industries and often targets staff with privileged access. Regulatory agencies like FinCEN and the American Bankers Association are now urging organisations to adopt verification protocols, training, and detection tools because basic technical safeguards are no longer enough.
These attacks are multiplying because they work. One employee, one rushed decision, one moment of misplaced trust can cost millions, or bring down an entire business.
This isn’t a theoretical risk, its a current reality. If you run a company, manage people, or simply work somewhere that handles valuable information you need to understand how to defend yourself.
Fortunately, there are concrete steps you can take to reduce your risk.
That’s what I will show you next.
The next section includes the exact playbook companies are using right now to train staff against AI-powered scams.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.