Banned AI, à la EU (Part 1)

Disclaimer: personal observations. not professional advice of any kind.

Later this year, the EU’s world-first AI Act will come into force.

It’s important because it will undoubtedly inform AI laws in various jurisdictions. Countries regularly trade legal frameworks for their merits, but also do so to avoid disparate laws that impede international trade.

The questions at the forefront of my mind include:

How will this impact current practices?

How will regulators enforce the law?

Any platform that uses AI and AI-adjacent software now has to scramble to figure out if they’re going to be compliant with what looks to be very broadly defined laws. Tech lawyers are salivating.

It will roll out in phases over the next three years, aiming to regulate the:

1. development,

2. offering,

3. supply, and

4. use

of AI.1

This follows a landslide majority in EU Parliament of 523-46, and the decision of the European Council to formally adopt the AI Act, in March and May 2024 respectively.

Central to this seminal law is the classification of different types of AI by risk. In turn, it applies varying levels of restriction that correspondingly decrease with lower levels of risk.

How does it define risk and AI?2

Risk is defined in the Act as the combination of:

1. the probability of an occurrence of harm, and

2. the severity of that harm.

Here’s how the same Act defines AI system:

a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments.

Tl;dr: Risk is the likelihood and level of harm; AI is an adaptive machine that takes inputs and generates influential content.

The AI definition seems to capture almost anything remotely imaginable to be an ‘AI system’.

The law divides AI risks into four broad categories, being:

1. unacceptable,

2. high,

3. limited, and

4. minimal.

The risk level determines the level of restriction imposed by the law.

*Provisions from the Act are in italics.

The Act enumerates prohibited AI practices.3

Manipulating informed decision-making causing detriment

You don’t have to imagine AI-generated deepfake videos promoting investment scams because you’ve probably seen them. Think Joe Biden extolling this stock that has gained 8,000% in the last 5 minutes on your grandma’s Facebook feed.

This is an example of an AI system that uses a purposefully manipulative technique (fake POTUS) to materially distort the behaviour of a person (your grandma) by appreciably impairing their ability to make an informed decision (POTUS did say it was a sure-fire investment!), thereby causing them to take a decision that they would not have otherwise taken (sending money to scammers) in a manner that causes that person significant harm (there goes granny’s life savings).

And that is the first prohibition on the AI Act’s no-fly list. It doesn’t specifically say no video deepfakes used to scam seniors, but it does prohibit the:

1. placing on the market,4

2. putting into service,5 or

3. the use of

Let’s just abbreviate these three activities as ‘MSU’ for ease of use later on.

an AI system that deploys:

1. subliminal techniques beyond a person’s consciousness, or

2. purposefully manipulative or deceptive techniques,

with the objective, or the effect of:

1. materially distorting the behaviour of a person by appreciably impairing their ability to make an informed decision,

causing them to take a decision that they would not have otherwise taken,

1. in a manner that causes, or is reasonably likely to cause that person, another person or a group of persons significant harm.

Exploiting vulnerabilities causing detriment

It is probably unethical to collect data to identify low-income earners and then inundate them with ads for exploitative payday loans.

Also unethical would be identifying likely alcoholics and then overwhelming them with ads for cheap beer.

However, are these practices likely to become illegal under the new laws?

*Notwithstanding the discussion of whether or not it should or shouldn’t be legal.

The ambiguity lies in the method. What does using an AI system mean? Can current methods of information-gathering fall into the definition of AI system under the Act? Can harm be directly linked to the platforms that target users using AI systems?

The Act prohibits the MSU of AI systems that exploits any of the vulnerabilities of a person/group of people due to their:

1. age,

2. disability, or

3. specific social or economic situation

with the objective or effect of:

materially distorting the behaviour of that person or a person belonging to that group in a manner that causes or is reasonably likely to cause that person or another person significant harm.

Social scoring causing detriment

Contrary to popular belief, China does not have a unified, national social scoring system straight out of a Black Mirror episode. From what I could gather online (granted, I haven’t written a whole thesis on this), China’s social credit system is rather fragmented and only executed in varying forms at the local council levels. National-level data aggregation is chiefly used for financial credit scoring (like in Western countries). Centralised databases on legal compliance mainly cover companies rather than individuals.

Anyway, what I’m trying to get at here is that China doesn’t have a national scoring system that uses AI to scan behaviour and arbitrarily adjust assigned social scores.

What’s interesting, however, is that it seems that such a system wouldn’t be banned outright under the new Act. Such a system would only be illegal in the EU if certain conditions on harm are met.

There are two independent, qualifying conditions for AI-based social scoring to be illegal under article 5(1)(c). Both require some form of detriment to a person or a group of people to occur. In addition to harm:

The first requires the data used by the AI system to have been generated or collected in an unrelated context to the social context it is used in.

An example is a job screening system that uses AI to score applicants against certain traits like leadership and teamwork based on their social media pages (e.g. posts, likes, etc). If it infers negative traits and lowers the relative score of an applicant, that would be using data collected in an unrelated context that causes a detriment.

The second requires the detriment to be disproportionate to the social behaviour.

An example of this could be the use of AI to infer someone’s health trends based on biometric/physical activity data collected by their personal devices to determine their health insurance premiums or deny access to certain types of coverage. If you’re asking why anyone would ever give insurers access to their health data—I’ll tell you—insurers like Medibank provide financial incentives through programs like ‘Live Better’ to have access to this data (e.g. hit 10,000 steps per day to get 5 points and if you get 1,000 points you can redeem an Apple Watch).

So you end up with a scenario where all you wanted was to get an Apple Watch, but your insurer has decided that because you excessively bungee jump and sleep poorly, your premiums are going up by $50 a month.

Conversely, this begs the question of whether there is scope for legal social scoring systems that don’t fulfil these two harm requirements (Hint: there may be, but probably with heavy burdens to manage risk and safety).

The new laws prohibit the MSU of AI systems for the evaluation or classification of a person/group of people over a certain period of time based on their:

1. social behaviour, or

2. known/inferred/predicted personal or personality characteristics

with the social score leading to either or both of:

detrimental/unfavourable treatment of a person/group of people

1. in social contexts unrelated to contexts in which the data was generated/collected

2. that is unjustified/disproportionate to their social behaviour/gravity.

I think we’ll end Part 1 there to accommodate this day and age’s short attention spans.

See you in the next one where we’ll explore the other five prohibitions under the Act. And then the following piece where we’ll dive into high-risk AI systems.

If you’re interested, you can read the full final draft of the EU AI Act here and the high-level summary of the AI Act here.