Banned AI, à la EU (Part 2)

Disclaimer: Personal observations. Not professional advice of any kind.

This article is the second in a series. Read Banned AI, à la EU (Part 1) here:

In the previous piece, I discussed three activities involving AI that are prohibited in the EU with examples.

Upon further reflection on the banned activities, I can’t shake the feeling that the government has granted itself an unrestrained latitude in clamping down on any AI activities it deems ‘harmful’. The combination of an all-encompassing ‘AI systems’ definition and the generous scope of the rules are salient in this regard.

This is not necessarily a negative outcome per se; it is the regulator’s enforcement posture that will determine the scale of caution warranted.

Put simply, focus on the regulator, not the laws.

*Provisions from the Act are in italics.

Assessing tendency to commit crimes from profiles or personality traits // ‘Minority Report’

Here’s a screenshot from Spielberg’s sci-fi flick, Minority Report. The AI Act essentially bans this practice, with one exception.

You can’t place on the market, put into service, or use (MSU)1 AI that assesses people’s tendencies to commit crimes based solely on profiling them or assessing their personality traits/characteristics.

Here’s an example: Software that collects data on Barry online and outputs a percentage score indicating the likelihood of a liquor store robbery being committed by Barry.

The exception to this rule is when AI that does this is used to assist human analysis which already reasonably suspect an individual’s (likely) involvement in a crime. In other words, it’s likely that law enforcement will be allowed to use it during criminal investigations that have progressed.

The Act prohibits the MSU of AI systems for making risk assessments of people in order to:

1. assess or predict the risk of a person committing a criminal offence,

based solely on the:

1. profiling of a person or on assessing their personality traits and characteristics;

this prohibition shall not apply to AI systems used to support the human assessment of the involvement of a person in a criminal activity, which is already based on objective and verifiable facts directly linked to a criminal activity.

Compiling facial recognition databases through the scraping of the internet/CCTV footage

Social media has empowered stalkers. For example, say you’re going on a Tinder date. Before even meeting, you’ve probably already uncovered where they work and what their second cousin looks like from their LinkedIn and Instagram. What you found on these platforms probably even factored into your decision to go on said date.

Now, imagine smart glasses that give you information on people you see on the street in real-time – a snapshot of where they work and the picture they took of their lunch 3 hours ago. All of this powered by an AI system that has, in a split second, matched the face captured through the camera with a face in a database that’s linked to a list of social media profiles to provide you key information on a person.

Eerie and an incisive invasion of privacy.

Preventing the ‘feeling’ of being constantly surveilled2 is one of the reasons why the Act has banned the MSU of AI that compiles facial recognition databases from the internet or CCTV footage.

The Act prohibits the MSU of AI systems that:

1. create, or

2. expand

facial recognition databases through the untargeted scraping of facial images from:

1. the internet,

2. or CCTV footage.

Inferring emotions at work or school

When you really think about this prohibition, it makes sense. There’s something quite unsettling about using a semi-autonomous system to decipher body language, tone of voice, choice of words, etc. to conclude what human emotion is being displayed. Now imagine your boss at work or teacher at school having a pretty graph of the emotions you displayed during the week and how often. Weird.

And we haven’t even discussed how Big Brother it is to have your every move analysed to provide emotional insights to some decisionmaker. Also probably a violation of a bajillion privacy laws. Can AI systems even detect schadenfreude?

There are exceptions however, where used for medical or safety reasons.

An example of a medical reason could be the use of AI to use emotional analysis to treat mentally ill patients. This could aid the health practitioner in further understanding changes in behaviour over time. Granted, the AI system would also need to be sophisticated enough with a properly-built user interface for this purpose to be of any practical use to a psychiatrist.

An example of a safety reason could be the use of AI in a high-risk workplace, such as construction, to detect fatigue.

These are theoretical examples of course, and will probably have to comply with high-burden privacy and consent regulations in the real world if manifested.

The Act prohibits the MSU of AI systems to:

1. infer emotions of a natural person in the areas of workplace and education institutions,

except where:

1. the use of the AI system is intended to be put in place or into the market for medical or safety reasons.

Going off on a bit of a tangent here with how the EU AI regulator has been/will be set up.

The EU has set up the AI Office, which will ensure coherent implementation in each Member State. Member States will also set up or appoint their own regulatory authorities to administer the new AI laws.

This dual-layered regulation is common in the EU, with the most prominent example probably being the administration of the General Data Protection Regulation (GDPR). Member State data protection authorities (DPA) include:

• France’s Commission Nationale de l'Informatique et des Libertés (CNIL), and

• Italy’s Garante per la protezione dei dati personali (GPDP),

with The European Data Protection Board (EDPB) coordinating the DPAs and ensuring the consistent application of GDPR across the Member States.

See you in the next one where we’ll explore the final prohibitions that relate to the use of biometrics to categorise and identify people, that EU lawmakers spent five pages detailing (each of the others had a paragraph or two).

If you’re interested, you can read the full final draft of the EU AI Act here and the high-level summary of the AI Act here.