Last week, Luca Bertuzzi published an investigative piece examining whether effective altruism’s risk agenda is shaping how the EU AI Act gets enforced. Bertuzzi is MLex’s Chief AI Correspondent in Brussels, and his reporting is standard reading for Commission officials, MEPs, and regulatory lawyers across the EU.
The article focuses on the AI Office’s Safety Unit, the intellectual backgrounds of its staff, and the organisations that won contracts to support systemic risk evaluation for general-purpose AI models.
The article surfaces a real and important tension:
Civil society organisations working on fundamental rights (discrimination, surveillance, electoral interference, online abuse) feel that their concerns are being deprioritised in how AI governance is operationalised.
As many in this field know: there should probably be more active collaboration between AI safety research orgs and these groups. Among other reasons, because the technical expertise that safety orgs bring is directly relevant to the harms civil society cares about.
However, I think the article leaves some significant gaps that are actually prejudicial to the (likely) intention behind this article: it reinforces a dangerous “us vs them” narrative that will NOT contribute to mitigation of fundamental rights and, instead, further amplifies the current “Responsible AI divides”.
The AI Office’s €9 million technical assistance tender was split into six lots, each addressing a different dimension of systemic risk. The article names the EA-aligned organisations that won contracts: EquiStamp, METR, Epoch AI, FAR.AI, SaferAI, SecureBio, Clarity AI Research, and Centre pour la Sécurité de l’IA. These organisations won Lot 1 (CBRN), Lot 3 (Loss of Control), and Lot 4 (Harmful Manipulation). This is all publicly available information.
What the article does not mention is who won the other three lots:
Lot 2 (Cyber Offence): Capgemini Consulting, Wavestone, and Laboratoire National de Métrologie et d’Essais; major French consulting and national laboratory institutions.
Lot 5 (Sociotechnical Risks): Technopolis Consulting Group Belgium, WIK-Consult (Germany), and TNO (Netherlands); established European policy research organisations.
Lot 6 (Agentic Evaluation): The French Ministry of Budget and Laboratoire National de Métrologie et d’Essais; a sovereign government entity.
Roughly half the total contract value went to mainstream European institutions with no EA affiliation. This is publicly available information on the EU’s Tenders Electronic Daily portal.
A more complete picture of how the money was allocated would have given the article’s thesis more credibility, not less; the genuine concerns it raises don’t need selective evidence to stand on their own.
The AI Act draws a clear line between the AI Office and Member State Authorities (MSAs). The AI Office oversees general-purpose AI model providers: it monitors compliance, performs model-level evaluations, and assesses systemic risks at Union level under Articles 89, 92, and 93 of the AI Act.
MSAs, acting as market surveillance authorities, are the bodies that handle complaints brought by individuals about specific AI system deployments that affect them directly.
The article suggests that the AI Office is deprioritising fundamental rights in its work, but the AI Office’s mandate is upstream. It operates at the GPAI model level, working with providers on systemic risk assessment.
In reality, individual harms materialise downstream: because natural persons will report indicents to their country’s MSA as starting point.
These are two distinct enforcement tracks, and conflating them misrepresents how the regulatory architecture actually works.
There is a valid conversation to be had about the coordination / mutual assistance that the AI Act envisions, or how the evaluation work being done at the model level feeds into enforcement at the deployment level, and whether MSAs will have what they need when complaints arrive.
That’s a more productive critique than suggesting the AI Office is ignoring fundamental rights.
The article frames risk categories like CBRN, harmful manipulation, and cyber offence as speculative EA preoccupations, disconnected from the real-world harms that civil society cares about. But this framing doesn’t accurately reflect how AI Safety work supports fundamental rights.
Harmful manipulation deals precisely with the targeting of vulnerable individuals and collectives: political manipulation, deceptive techniques that exploit cognitive vulnerabilities, large-scale fraud through multi-turn interaction, malicious actors using AI to blackmail or coerce, including the targeting of minors. These are not abstract concerns. They directly relate to the AI Act’s prohibited practices under Article 5 and the Commission’s Guidelines on prohibited AI practices.
The people working on harmful manipulation evaluation are working on mitigation of risks that affect fundamental rights.
CBRN is not hypothetical. The article specifically mentions “AI supporting nuclear proliferation” and AI “going rogue” as concerns that “don’t exist yet.” But current frontier models can already be elicited to provide meaningful assistance with biosynthesis pathways and dangerous capabilities when safety mitigations are circumvented.
The risk isn’t limited to the most dramatic scenarios the article highlights. Lower-barrier biological and chemical risks enabled by current SOTA models can harm people now, not in a few years.
Cyber offence capabilities threaten the critical infrastructure that societies depend on, and the people who depend on that infrastructure. For example, in November 2025, Anthropic reported disrupting the first documented large-scale AI-orchestrated cyber espionage campaign, where the AI executed 80-90% of the operation independently.
Framing these as disconnected from fundamental rights misses that they are vectors through which fundamental rights violations occur.
The AI safety organisations working on these evaluations, the same ones the article presents as indifferent to real-world harms, are actively developing tools to measure and mitigate exactly these risks.
The article’s underlying concern is that the AI safety community and the fundamental rights community are talking past each other, and that concern is legitimate.
The technical expertise that AI safety orgs bring to evaluation and risk modelling is directly relevant to the harms that civil society organisations are fighting. And civil society’s grounding in rights frameworks, enforcement precedents, and democratic accountability is something the technical safety community needs.
Rather than treating this as a zero-sum competition over whose risks matter more, there is an opportunity to bridge the gap.
AI safety orgs can do more to demonstrate how their work connects to the fundamental rights framework that underpins the AI Act. And civil society can engage with the technical evaluation work that will ultimately determine how effectively those rights are protected in practice.
The article raises questions about the ideological composition of the AI Office’s staff. That’s beyond my capacity to assess, and I don’t think it’s my place to try. I’ve attended EA-related events and been involved in EA-adjacent projects, but I’ve also been equally known to criticise certain beliefs where I think they’re not beneficial.
My commitment is to AI safety as a field, not to any particular intellectual tradition within it.
But I do think the article conflates AI safety with EA in a way that is unhelpful.
Caring about AI safety is not the same as subscribing to effective altruism. The field includes people from many different backgrounds, motivations, and intellectual traditions.
By treating them as interchangeable, Mlex risks looking like they treat technical AI safety expertise itself as part of the problem- simply by not defining what technical AI safety work should do, and instead choosing to focus on what it’s perceived to be by ideological association.
This is a missed opportunity.
If the concern is that EU AI enforcement needs more people with diverse backgrounds working on these problems, that’s a call worth making.
People working in algorithmic auditing, in industry compliance, in civil society, in fundamental rights law: their expertise is needed in this space, including for research at GPAI model risk level.
But the article doesn’t make that call. Instead, it presents AI safety research as “suspect” by association with EA, which risks discouraging exactly the kind of broader engagement that would address the diversity concern.
The technical expertise that AI safety orgs bring is crucial to advancing fundamental rights protections. We need more of it from more directions, not less of it.
The conversation this article started is worth having, but it’s more productive if it’s based on the full picture.
Yutong Liu / https://betterimagesofai.org / https://creativecommons.org/licenses/by/4.0/
Disclosure: I have an ongoing relationship with Equistamp, one of the named contractor organisations, for whom I have produced legal research. This post reflects my personal opinion only, acknowledging my own stance on AI Safety work while trying to provide a fair assessment.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.