Reading · Artificial Impact

Hello everyone, in this newsletter we are talking about AI and cybersecurity! This is a rapidly evolving subjects, with challenges as AI models and systems can be used for defence, or to attacks, and also AI systems can be themselves be threats or hacked etc. Let’s have an overlook of the subject, and if you want me to go into more details on one specific point in another newsletter, tell me!

Artificial Intelligence (AI) has become more and more important in cybersecurity! It can spot threats, automate defences and respond to incidents quickly and accurately. This has completely changed the field. However, as AI can be used for good and for evil, it can also be used by attackers.

Real-time threat detection and mitigation
- AI systems excel at analysing vast volumes of data instantaneously, spotting anomalies and patterns that could indicate a security threat.
- These systems enable swift, automated responses — isolating compromised systems or blocking malicious activity in real-time.
Behavioural analysis and adaptive access management
- By establishing a baseline of normal user behaviour, AI can flag unusual activity, identifying insider threats or unauthorised access attempts.
- It simplifies access for legitimate users while curbing fraud and enhancing security, striking a balance between usability and safety.
Proactive defence with predictive analytics
- AI models analyse historical data and detect patterns to anticipate potential threats before they materialise.
- Tools such as machine learning and clustering algorithms uncover vulnerabilities, allowing organisations to patch weak spots proactively.
Incident analysis and vulnerability management
- AI aids in post-incident investigations by reconstructing timelines, determining root causes, and identifying areas requiring improvement.
- Automated vulnerability scans prioritise risks, ensuring resources are focused where they are most needed.
Enhanced defences against phishing and malware
- AI systems are adept at detecting sophisticated phishing attempts, evaluating email content and sender behaviour to thwart attacks.
- They evolve alongside threats, adapting to counter novel malware strategies with dynamic, tailored responses.

Speed and scalability: AI dramatically reduces response times, delivering near-instant insights across large datasets.
Accuracy and precision: by continuously learning, AI minimises false positives compared to rules-based systems and ensures critical threats are prioritised.
Proactive security: it shifts the paradigm from reactive to pre-emptive defence, keeping organisations one step ahead of attackers.

While AI strengthens defences, its capabilities can also be weaponised:

Automated reconnaissance: attackers use AI to scrape vast amounts of public data to identify potential targets and weaknesses.
Tailored phishing and social engineering: generative AI enables highly convincing, personalised phishing campaigns that evade traditional detection methods.
Evasive malware: AI-powered malware can adapt in real time, mimicking legitimate activity to bypass conventional safeguards.

Generative AI, like Copilot, Claude or ChatGPT, is reshaping cybersecurity by providing actionable insights, automating reporting, and accelerating decision-making. For example, companies are leveraging GenAI to analyse legacy software containers, identifying vulnerabilities, with high accuracy, up to four times faster than human analysts.

However, these same tools can amplify attackers’ capabilities, enabling them to scale operations and execute sophisticated attacks with unprecedented ease. For example, cybercriminals used ChatGPT to create polymorphic malware capable of evading security products by continuously changing their codes, making their detection more challenging.

AI is powerful, so when hacked or attacked, the impact and damage can be huge. As artificial intelligence continues to evolve, its integration into critical systems comes with significant cybersecurity challenges. AI systems are uniquely vulnerable due to their reliance on vast datasets, complex models, and interconnected pipelines. Protecting these systems is crucial to ensure their reliability, trustworthiness, and resistance to malicious exploitation.

Data pipeline attacks
- AI relies on large data pipelines for training and operation. Attackers exploit these pipelines by introducing malicious inputs, modifying datasets, or gaining unauthorised access.
- Such breaches can corrupt models, violate privacy, or disrupt operations.
Data poisoning
- Inserting harmful or misleading data into training datasets can manipulate AI behaviour.
- For example, tampered datasets might make a fraud-detection model fail to recognise suspicious transactions.
Model exploitation
- Model evasion: Attackers manipulate inputs to deceive AI systems into producing incorrect outputs, such as bypassing facial recognition.
- Model inversion: Reverse-engineering model outputs can reveal sensitive training data, like personal or proprietary information.
- Model control: Malware can take over a model, leading to dangerous outcomes, such as tampering with automated vehicle systems.
Supply chain and third-party risks
- Open-source libraries and external APIs used in AI development present significant risks. Malicious code introduced through compromised components can alter AI behaviour.
- A notable example includes leaked HuggingFace API tokens that allowed hackers access to AI pipelines.
Denial-of-Service (DoS) attacks
- Overloading AI systems with excessive requests can degrade performance or cause system failures, disrupting operations and incurring costly downtime.
Prompt attacks on generative AI
- Exploiting weaknesses in large language models (LLMs), attackers manipulate chatbots or virtual assistants to extract confidential information or bypass safeguards.
Bias and ethical concerns
- Poorly designed training datasets can embed biases into AI systems, leading to unfair outcomes and reputational damage. For instance, biased facial recognition systems may fail to recognise underrepresented groups accurately.

Here is a (non-exhaustive) list of best practices to secure AI systems:

Data governance and privacy ⇒ Implement robust policies to manage data access and usage while protecting sensitive information.
Adversarial resilience ⇒ Use adversarial training techniques to prepare models for real-world attacks.
Supply chain security ⇒ Vet third-party libraries and APIs thoroughly, and maintain an “AI Bill of Materials” (AI-BOM) to track dependencies.
Continuous monitoring and auditing ⇒ Employ real-time monitoring to detect anomalies and conduct regular audits to evaluate vulnerabilities.
Transparency and explainability ⇒ Ensure models are interpretable, enabling stakeholders to understand their decisions and mitigate risks effectively.
Incident response planning ⇒ Develop and test response strategies for AI-related incidents, including DoS attacks and model exploitation.
Cross-disciplinary collaboration ⇒ Engage experts in cybersecurity, data science, and ethics to address AI’s unique challenges holistically.

To tackle these challenges, the Framework for Artificial Intelligence Cybersecurity Practices (FAICP), developed by the European Union Agency for Cybersecurity (ENISA), offers a structured approach to securing AI systems.

Layer 1: cybersecurity foundations

Focuses on securing ICT infrastructure hosting AI systems, ensuring confidentiality, integrity, and availability.
Key components include:
- Security management: conducting risk analyses to identify and mitigate threats.
- Threat analysis: using resources like the ENISA Threat Landscape report to understand risks, including adversarial, accidental, and environmental threats.
- Compliance: aligning with cybersecurity regulations — like the NIS 2 Directive, the EU Cybersecurity Act, and the Digital Operational Resilience Act (DORA).

Layer 2: AI-specific cybersecurity practices

Addresses risks inherent to AI components, such as data poisoning and evasion attacks.
Recommends measures like adversarial training, secure data handling, and access controls to mitigate threats.
Emphasises trustworthiness through characteristics like accountability, explainability, fairness, and robustness.

Layer 3: sector-specific best practices

Tailored recommendations for industries using AI, including energy, healthcare, and automotive sectors.
Examples include securing IoT-enabled medical devices and mitigating adversarial attacks on autonomous vehicles.

AI is both a shield and a weapon in the evolving landscape of cybersecurity. To counteract increasingly sophisticated AI-driven attacks, organisations must adopt equally advanced defences. These systems need to be self-learning, adaptive, and capable of responding to threats in real-time while ensuring robust security measures are embedded throughout their lifecycle.

At the same time, human oversight remains critical to maintaining fairness, transparency, and ethical operation. Frameworks like FAICP offer essential guidance for safeguarding AI systems and mitigating risks, but they must be complemented by comprehensive strategies and continuous vigilance.

The stakes are high: with a 690% increase in AI-related incidents from 2017 to 2023, securing AI is no longer optional — it’s an urgent imperative! Whether defending against AI-driven threats or protecting AI systems themselves, the future of cybersecurity depends on our ability to harness AI responsibly and securely.

AI & Cybersecurity: a two-way defence revolution!

Comments