Phishing has long been a major threat in cybersecurity, with one of the first known instances emerging in the mid 1990s, on America Online (AOL). As described in Jigsaw’s The History of Phishing.
In 1995, hackers using the handle “AOHell” created tools to trick users into giving up their passwords and credit card details by posing as AOL employees. This is widely considered the birth of phishing.
Phishing increased further with the rise of the internet and emails, with spoofed messages and fake websites used to steal credentials en-masse.
How does Phishing Work?
Phishing is effective because it exploits human psychology rather than technical flaws.
“Only amateurs attack machines; professionals target people”
Bruce Schneier
A typical phishing attack follows a pattern designed to manipulate the victim into taking an action they normally wouldn’t consider. Such as clicking a malicious link, entering sensitive information, or downloading a harmful attachment.
Historically, one of the key challenges for hackers was that effective, targeted phishing required extensive information gathering. To deceive someone successfully, attackers had to research individuals or organisations in detail and then craft highly specific, convincing messages.
Another common weakness in phishing campaigns was the lack of proper grammar and spelling, which often gave the scam away. Below is an example of a phishing attempt where poor language use reveals its malicious intent:
Anatomy of an Effective, Pre-GenAI Phishing Attack
Whilst I could carry on talking about sloppy scam email phishing attacks sent out en-masse, I want to talk about targeted phishing attack prior to the GenAI revolution of the past few years before examining how this new technology has changed the sophisticated phishing market.
Unlike the attacks that I discussed before, a well executed phishing attack is strategic, tailored, and psychologically manipulative. Here’s how attackers typically carry out an effective campaign:
Reconnaissance (Researching the Target)
The attacker begins gathering information on their target. This can include:
Public social media profiles (LinkedIn, Instagram, Facebook)
Company websites (employee directories, press releases)
Data from previous breaches (email addresses, passwords)
Dark web marketplaces (for stolen credentials or corporate data)
The goal is to build a realistic context around the victim, such as knowing who they report to, recent meetings, or services they use.
Choosing the Attack Vector
The attacker selects the method that they’ll use to reach the target. Common vectors include:
Email
SMS (“smishing”)
Phone calls (“vishing”)
Social Media DM’s
Crafting a Convincing Message
Using what they’ve learned, the attacker writes a message that appears legitimate and urgent. Tactics often include:
Posing as someone the victim trusts (e.g. their manager, IT support, a client)
Using company branding, tone, and typical language
Creating urgency: “Your account will be locked in 24 hours”, or “Urgent invoice issue, respond immediately”
Creating a Payload
The message includes a malicious element designed to capture information or compromise the system. This could be:
A phishing link that leads to a fake login page
A malicious attachment (e.g. disguised malware in a PDF or Excel file)
A QR code that redirects to a credential-harvesting site
A form asking for personal or payment information
Delivery and Timing
Sophisticated attackers pay attention to when they send the message. They may make it:
Just before a weekend or holiday (when IT is slower to respond)
During busy hours (to catch the victim off-guard)
In-line with real events (e.g. just after a software update or internal announcement)
Exploitation
Once the victim interacts with the message (clicking a link, entering credentials, or opening a file) the attacker gains access. This could result in:
Credential theft and account compromise
Ransomware deployment
Business email compromise (BEC) or internal impersonation
Data exfiltration or lateral movement within the network
How Does GenAI Change The Phishing Threat Landscape?
AI-Enhanced Spear Phishing
Spear phishing is no longer a manual task. GenAI allows attackers to:
Generate hyper-personalised emails using public and leaked data (e.g. LinkedIn roles, email breaches, or social media activity)
Mimic tone and writing style of a known colleague or superior
Automate hundreds of unique messages at scale
The result? Emails that are more believable, timely, and likely to succeed, even against experienced users.
At Black Hat USA 2021, researchers from Singapore’s GovTech agency tested GPT-3 generated phishing emails against human-written ones. The AI emails had a higher click rate, showing that well crafted language from an AI can be more effective than traditional methods.
Another Real-World Study: IEEE Access (2025)
In a 2025 peer-reviewed study published in IEEE Access, “Lateral Phishing With Large Language Models: A large Organisation Comparative Study”, Bethany et al. tested GPT-generated spear phishing emails against human-written ones inside a large organisation of over 8,800 users.
The results were striking:
LLM-generated emails performed as well or better than human-crafted emails
Around 10% of recipients entered credentials into fake login pages
Existing security filters failed to reliably detect the AI-generated emails
The authors developed a custom detection model that achieved a 98% F1 score, highlighting how traditional defences are no longer sufficient.
This study confirms that GenAI doesn’t just improve phishing, it actively undermines existing email defences and user awareness training.
The Threat Doesn’t End with Email
What makes this trend even more disturbing is that the email-based phishing examples above are only part of the picture. Today’s generative AI models can also create realistic-sounding voices and synthetic videos, which are far less known and even more likely to deceive users.
New Research: Vishing Classifiers Under Fire
A fresh arXiv preprint, “Talking like a Phisher: LLM-Based Attacks on Voice Phishing Classifiers” from July 2025, reveals how advanced language models can game AI detection systems, not just craft realistic sounding messages.
In this work, researchers used large language models to generate voice phishing transcripts, written prompts resembling phone scam scripts, that were designed to evade automated vishing classifiers while preserving deceptive intent. The study shows:
LLMs can craft adversarial transcripts that fool detection systems (even those trained to recognise phishing language patterns).
Classifiers trained on human-written phishing scripts struggle to flag cleverly LLM-generated vishing content.
This exposes a growing blind spot: vishing detection pipelines are vulnerable to adversarial manipulation by AI
What this means
GenAI isn’t just improving phishing execution, its also evolving around detection systems, making them less effective.
Defences based on detecting known patterns are insufficient against AI-generated vishing, which can vary phrasing, tone, and scouting approaches.
As LLM-powered phishing grows more evasive, humans become the final layer of defence. Unfortunately, human detection of phishing remains inconsistent and error-prone.
In the context of vishing and synthetic media, users can no longer rely on gut instinct alone to determine whether a message or call is real.
What Can be Done?
Phishing has always adapted, from AOL chatrooms in the 1990s to today’s hyper-personalised AI attacks. But the acceleration driven by GenAI requires a broader, more strategic response across three fronts: organisations, individuals, and governments.
For Organisations: Don’t Rely on Legacy Systems
Organisations must accept that traditional phishing defences (keyword filters, static training, and rule-based email gateways) are no longer enough.
Key actions include:
Invest in AI-powered threat detection
Use, regularly updated, anomaly-based detection models and adversarial-aware email filters trained to recognise synthetic language patterns.
Implements multi-layered phishing simulations
Move beyond basic awareness campaigns. Run internal red team exercises using LLM-generated messages across email, voice, and chat channels.
Strengthen identity verification policies
Use out-of-band authentication for sensitive requests. Always verify via a second channel before actioning finance or access requests.
Monitor behavioural signals, not just message content
Detection should extend beyond message wording, look for behavioural anomalies like logins at odd times, location mismatches, or unusual file transfers.
Limit publicly exposed data
Personal and organisational details fuel AI-enhanced social engineering. Audit your digital footprint, from employee bios to supplier relationships.
For Individuals: Training Must Evolve
Most people are still being trained to spot bad grammar and generic links, which is no longer relevant when facing AI-generated phishing.
Individuals need to:
Adopt a mindset of healthy scepticism
Just because a message looks right doesn’t mean that it is. Pause and verify (especially for urgent, sensitive, or unusual requests).
Question context, not just content
Ask: Does this make sense in my workflow? Is this request typical of this person or department?
Verify through trusted channels
Don’t respond directly to emails or calls with suspicious requests. Reach out through known, official contact methods.
Understand common manipulation techniques
Training should include examples of urgency, authority pressure, spoofing, and false familiarity.
Stay informed
AI-enhanced threats evolve fast. Regular security updates and real-world examples should be part of every organisations culture.
For Governments: Regulation, Research and Resilience
Governments have a crucial role to play in mitigating systemic risk and building national cyber resilience.
Policy areas for action:
Set standards for AI content verification
Promote watermarking and provenance-tracking of AI-generated content, especially in communications used by public and critical sectors.
Fund adversarial AI research
Support development of detection models that can keep pace with LLM threats (especially for voice, video, and multi-type phishing content.
Mandate minimum cyber hygiene standards
For sectors handling sensitive data (e.g. finance, healthcare, infrastructure), set clearer mandates around employee training, MFA, incident response, and data minimisation.
Establish public awareness campaigns
Just like anti-smoking or drunk driving campaigns, phishing education needs to go mainstream (clear, relatable, and accessible).
Encourage responsible LLM deployment
Promote best practices in open-source LLM release, API rate-limiting, and misuse detection (especially for tools that can generate persuasive scam content).
Unfortunately, many governments are framing generative AI primarily as a geopolitical competition, often positioning it as a race (most notably between China and the West) rather than viewing it as a shared global risk requiring collaboration.
In the United States, President Donald Trump’s administration has leaned toward a national-first rhetoric in AI policy. This includes executive actions that emphasize technological supremacy over coordinated safety and regulatory frameworks, rather than an approach grounded in international risk mitigation.
This mindset:
Encourages companies to release high-impact systems without adequate safeguards or oversight
Weakens momentum for multilateral agreements on AI transparency, watermarking, or misuse detection
Undermines global efforts to build shared threat intelligence and coordinated response strategies
If governments continue to prioritise winning the AI race over securing the AI landscape, they risk empowering cybercriminals more than protecting their citizens.
A Better Path Forward: Cooperation over Competition
To truly address the AI-enabled phishing threat (across email, voice, video, and beyond) governments must:
Treat AI as a shared global challenge, not just a national asset
Facilitate international collaboration on AI safety, transparency, and ethical deployment
Prioritise human-centric cybersecurity policies, over aggressive tech-first strategies
Only by changing the narrative (from “who develops AI fastest” to “who uses AI most responsibly) can we build a world where innovation and security go hand in hand.
A Shared Responsibility
The phishing landscape has changed, and the response must change with it. This isn’t just an IT problem. It’s a human, technical, and policy-level challenge.
The tools used to deceive us are getting smarter. So we must get smarter too (not just individually, but together).
Authors Note
This was a fascinating topic to write about, and I feel that I’ve barely scratched the surface. If you found it interesting, let me know. I’d be happy to write follow-up pieces exploring LLM-related security challenges in more depth.
If you enjoyed this article, consider subscribing or sharing it. Your support means a lot, thank you for reading.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.