Responsible AI is not "compliance" — Part II: The Skill Mismatch

In Part I, I argued that Responsible AI is not compliance.

Compliance is passing your MOT. RAI is having brakes and a seatbelt.

This post is about who is actually holding the extinguisher in most companies right now. And why, in many cases, it’s the wrong person. Not through any fault of their own, but because the right person may not exist yet. At least not at scale, not deployment-ready, and not sitting in your org chart.

Here’s what happens in most organisations when “AI governance” lands on the executive agenda:

First, it goes to Privacy, Legal and Compliance. This makes sense on paper. The EU AI Act is a regulation. Regulations are what Legal does. So the privacy team, fresh from years of GDPR implementation, absorbs the AI Act the way they absorbed every other acronym: DSA, DMA, NIS2.

Then someone realises that AI risk is “technical”. So the second delegation happens: to Cybersecurity. The CISO’s office inherits model risk because, in the corporate taxonomy, anything involving threat surfaces, adversarial actors, and technical controls belongs to security.

Both delegations are rational. Both are wrong. Or rather: both are incomplete in ways that compound each other.

I say this as a lawyer, and I say it with genuine respect for my own profession: there is a ceiling on what legal and compliance professionals can realistically upskill into, and most corporate training budgets are in denial about where that ceiling sits.

A privacy professional can learn what a transformer is. They can learn the vocabulary: weights, fine-tuning, RLHF, system prompts. Plenty of excellent courses now exist for exactly this, and they’re valuable.

But vocabulary is not capability. And to be clear about the setting: these professionals are not sitting in frontier labs. They sit in normal companies that build products on top of foundation models. The thing they have to assess is the product. Consider what that actually requires someone to do:

• Design and interpret evaluations of the product: If your team build on foundation models provided by AI companies, the logical thing (although it does not happen as much) is that you inform your Risk assessment with the model card. But the model card tells you how the foundation model scored on someone else’s benchmarks. It tells you almost nothing about how your product behaves: your system prompt, your retrieval pipeline, your tool integrations, your users’ actual queries. Someone has to test the assembled system. Decide what failure looks like in your domain, build test sets from your own data, and know why upstream benchmark scores don’t transfer to your use case.¹ This is empirical work. It has more in common with experimental psychology and ML engineering than with anything in a legal practice course.

• Assess whether a mitigation actually mitigates. When a vendor tells you their guardrails block harmful outputs, someone needs to be able to ask: blocked at what layer? Classifier on inputs, classifier on outputs, or trained into the model? What’s the bypass rate under adversarial pressure? Has anyone red-teamed the combination of your retrieval pipeline and their filter? A compliance professional can ask whether the vendor has documentation. They cannot evaluate whether the documentation describes something that works.

• Reason about distribution shift at the product level. Your product behaved one way in the proof of concept. Production will feed it inputs nobody tested: different languages, adversarial users, edge cases from your actual customer base. And the ground can move under you from the other direction too. The vendor updates or swaps the underlying model, and your product’s behaviour changes overnight without a single line of your code changing. You’ve inherited the unpredictability of internals you cannot inspect. The only defence is testing against your own input distribution and monitoring after deployment, continuously. Your DPO cannot stand up that capability after a two-day workshop.

None of this is an insult to compliance professionals. It’s really a category observation. We don’t ask cybersecurity engineers to draft data processing agreements, and nobody calls that a skills gap. We recognise it as a different profession. The mistake is that with AI governance, we haven’t yet admitted we’re looking at a different profession.

This is the delegation I want to spend the most time on, because it’s the one that feels most defensible and is quietly doing the most damage.

Cybersecurity, as a discipline, is built around a specific threat model: an external (or insider) adversary attacks your systems to compromise confidentiality, integrity, or availability. The system itself is assumed to be deterministic and on your side. The attacker is the variable. Decades of excellent practice (threat modelling, pen testing, SOC operations, incident response) are built on that assumption.

Frontier AI breaks the assumption in two distinct ways, and each break corresponds to a discipline that is not cybersecurity:

AI Security (adversarial machine learning) deals with attacks that exploit the statistical nature of the model itself: prompt injection, jailbreaks, data poisoning, model extraction, membership inference. Some of this is starting to enter the cybersecurity mainstream (the OWASP Top 10 for LLM Applications is a genuine step forward²). But notice what these frameworks cover: the application layer. Injection via retrieved documents, insecure plugin design, excessive agency. Important, real, and still fundamentally framed as “attacker manipulates system”.

AI Safety deals with something cybersecurity has no native frame for at all: the system itself as the source of risk, with no attacker required. Misalignment. Reward hacking. Deceptive behaviour under evaluation. Emergent capabilities that appear at scale and propagate into every agent and product built on top of the model.³

There is no “cybersecurity patch” for a model that has learned to behave differently when it infers it’s being monitored. There is no patch for a learned representation. Your SOC cannot triage an alignment failure, because nothing was breached. The model did exactly what its training produced, and what its training produced was not what anyone specified. And as I argued in Part I: you didn’t train it, you can’t audit it, but if you’ve built your product on it, you own the failure anyway.

When companies hand AI risk to the CISO, they get coverage of the first category, partially, and coverage of the second category approximately never. Worse: the organisation believes the risk is covered, because it sits in a box labelled “security” with serious people in it. The delegation produces false assurance, which is more dangerous than acknowledged exposure.

And here’s the uncomfortable symmetry: cybersecurity professionals face the same upskilling ceiling as compliance professionals, just from the other side.

A penetration tester can learn to jailbreak an LLM in a weekend. Understanding why the jailbreak works, whether the fix generalises, and what the behaviour implies about the system you’ve shipped is a different knowledge stack entirely.

Red-teaming a network and red-teaming a product built around a mind-shaped artefact produced by gradient descent are not the same job.

So if it’s not Legal, and it’s not Cybersec, who is it?

This is where I think we have to be honest about something structural: there is an entire professional layer that the market needs and has not yet produced at scale.

Let’s call it “technical AI governance”, but I do hope that the label “Responsible AI” eventually merges on to this.

The profile looks something like: enough ML depth to read an evals paper critically, design test protocols for deployed products, and interrogate a model card or vendor claim; enough governance fluency to translate findings into risk registers, contractual allocations, and regulatory mappings; enough organisational skill to operate inside an enterprise rather than a research lab.

Now ask where these people currently are. The honest answer: mostly in frontier labs, a handful of specialised evaluation organisations, government AI safety institutes, and academic groups. The talent pipeline that exists flows toward the model developers, not toward the thousands of enterprises deploying on top of them. The companies carrying the deployment risk I described in Part I (you don’t control the weights, you can’t audit the internals, you own the failure anyway) are precisely the companies least able to hire for it.

Even multinationals. I’ve seen the inside of large-enterprise Responsible AI functions, and the pattern repeats: governance frameworks drafted by people who can’t test the systems they govern, technical questions routed to engineering teams whose incentives are shipping, and a cybersecurity function asked to vouch for risks outside its discipline.

Everyone is competent. The configuration is incompetent.

The wrong conclusion from all this would be: “fire the lawyers, hire ML PhDs.” That fails for the opposite reason. A brilliant evals researcher who cannot map findings to Article 9 risk management obligations, contractual liability allocation, or board-level risk appetite produces research, not governance.

The right model, I’d argue, is the one the field itself uses as an architecture metaphor: a mixture of experts.⁴

Legal and compliance professionals remain critical. Someone has to translate system behaviour into obligations, allocate liability across the value chain, and tell the board what the actual exposure is.

Cybersecurity remains critical. Application-layer AI security genuinely is their fight, and the infrastructure around models (access controls, logging, supply chain integrity) is classic security work.

But both need to sit alongside a third function that most companies haven’t built: people who can run, commission, or at minimum critically evaluate technical assessments of the products the company actually ships.

Product-level evals. Red-team design. Monitoring for drift and behaviour change after vendor model updates. Interrogating vendor claims at a technical level.

And the connective tissue matters as much as the nodes. The lawyer needs to know enough to ask the evals person the right question. The evals person needs to know enough to flag which behaviours create legal exposure.

Neither needs to be the other. The unicorn job spec (”JD plus ML engineering experience plus CISSP”) is a sign that an organisation has confused a team design problem for a hiring problem.

If you’re building or fixing an RAI function, three things follow:

• Stop treating technical AI governance as a training module. Budget for it as a capability you acquire: a hire, a retained external evaluator, a standing engagement with an organisation that does this work professionally. The two-day “AI literacy for compliance teams” course is fine for Article 4 purposes. It does not produce someone who can evaluate a frontier-model-based product.

• Audit your delegation map. Write down, explicitly, which function in your organisation is accountable for: application-layer AI security, product behavioural risk, and regulatory compliance. If the same name appears in all three boxes, you don’t have coverage. You have a single point of failure with a broad job title.

• Treat the gap as a market signal, not just a problem. For individuals: the intersection of technical evaluation skills and governance fluency is, right now, one of the most undersupplied skill sets in the entire economy. For organisations: whoever builds this layer first gets something their competitors are faking.

Responsible AI is a mixture of experts because the risk itself is layered: regulatory, adversarial, and emergent. One profession per layer. Pretending one function can hold all three isn’t lean staffing.

It’s just the org-chart version of the category error from Part I: filing the fire extinguisher under paperwork, and then handing it to whoever happened to be standing closest.

¹ Two reasons among several: benchmark contamination (the benchmark leaked into training data, so the score measures memorisation, not capability; see Sainz et al., “NLP Evaluation in trouble”) and evaluation-aware behaviour (frontier lab system cards now discuss models detecting test conditions; see the Claude Sonnet 4.5 system card, which devotes substantial analysis to exactly this). Both mean upstream scores cannot be taken at face value, let alone extrapolated to your product.

² OWASP Top 10 for Large Language Model Applications. Genuinely useful, and notice its scope: application-layer vulnerabilities. It is a floor for AI Security, not a framework for AI Safety.

³ See Part I on emergence and Wei et al., “Emergent Abilities of Large Language Models”.

⁴ Yes, the pun is intentional. In ML, a mixture-of-experts architecture routes each input to the specialised sub-networks best equipped to handle it, rather than forcing everything through one monolith. Apply to your org chart accordingly.