RSS Amplifier

Artificial Impact · Jul 28, 2026

Bias in AI systems: beyond "the training data was biased"

Read inOriginalenesdefrptjaittr
0
Sign in to vote or save

Alexandre du Sordet · Artificial Impact

AI generated image

Hello everyone,

In 2023, a man named Derek Mobley sued Workday after being rejected from more than 100 jobs, alleging the AI screening tools did the rejecting. By March 2026, a US federal court had ruled that age-discrimination protections cover job applicants, certified a collective action, and put a number on the table that should make every buyer of AI tooling sit up: roughly 1.1 billion rejected applications potentially implicated. Here is the detail that matters for us: nobody in that courtroom is arguing about a dataset. In this article we are going to deep dive into where bias actually enters an AI system. Spoiler: there are seven doors, and “the training data was biased” covers barely two of them. By the end, you will have the full map, and a 7-checkpoint review your team can actually run. Let’s go.

Why does everyone reach for “the training data was biased”? Because it is sometimes true, because it blames nobody in the room, and because it sounds fixable: collect better data, done. It also quietly assigns the entire problem to one team (data science) while the highest-leverage failures happen upstream, in how the problem was framed, and downstream, in how the system is deployed and used.

Think of a restaurant kitchen. “The ingredients were bad” explains some ruined dinners. But a dish can fail at shopping, at deciding what to cook and for whom, at prepping, at cooking one big pot for every table, at tasting only your own portion, and at serving. If you only ever audit the groceries, you will keep buying better ingredients and keep serving the same bad meals. MIT’s Harini Suresh and John Guttag mapped this formally: seven distinct sources of harm across the machine-learning lifecycle. NIST’s SP 1270 goes further: statistical bias is only one of three categories, next to systemic and human-cognitive bias, and zero bias risk is not achievable. The goal is management, not eradication.

And here is a reframing worth holding onto before we go hunting bias down: bias is not the enemy, it is a survival tool that sometimes misfires. Human brains run on bias by design. The snap judgment that a rustle in the grass is a predator rather than the wind is exactly what kept our ancestors alive; evolutionary psychologists David Buss and Martie Haselton call this error management, the smoke-detector principle, better a thousand false alarms than one missed fire. The same mental shortcuts, what Gerd Gigerenzer calls “fast and frugal” heuristics, let a doctor triage in seconds and let you cross a busy street without solving a physics problem. Bias is, quite literally, what makes fast decisions possible at all. The trouble is that a shortcut tuned for the savannah, or for a company’s own past, quietly bakes yesterday’s world into tomorrow’s automated decision, at a scale and speed no single brain could ever reach. So we are not chasing a bias-free system, which is neither achievable nor even desirable. We are making sure the shortcuts baked into ours are the ones we actually chose, and that we know exactly where they live.

ℹ️ Small glossary

  • Proxy = a measurable stand-in for the thing you actually care about (cost instead of health, clicks instead of quality).

  • Disaggregated evaluation = reporting performance per subgroup instead of one aggregate score.

  • RLHF = reinforcement learning from human feedback; the tuning stage where human raters shape an LLM’s behavior.

  • Feedback loop = when a model’s outputs shape its own future inputs.

  • Fairness metric = a formal definition of “fair” (equal error rates, calibration...); there are many, and they conflict.

So... where exactly does it get in? Walk the lifecycle with us.

The data faithfully records a world that was already unfair, and the model treats that record as the target to reproduce. Amazon’s experimental recruiting model learned from ten years of résumés sent to a male-dominated tech industry, decided the winning pattern was essentially “looks like the men we already hired”, and marked down anything that signalled otherwise: it penalized the word “women’s” (as in “women’s chess club captain”) and downgraded graduates of two all-women’s colleges. Now the part that surprises people. The team deleted gender as an input, and it barely helped. That is what “proxies leak” means: gender was still written all over the résumé in a hundred indirect ways, a women’s college here, a differently-coded verb or sport there, so the model simply reconstructed the attribute it was no longer allowed to see. Removing the label does not remove the information. Kitchen stage: the market you shop in is already stocked with yesterday’s inequalities.

Some group is under-represented in the data, so the model is simply less practised on them, and quietly worse where it matters. Gender Shades found commercial face-analysis systems erring on at most 0.8% of lighter-skinned men and up to 34.7% of darker-skinned women, because the data they learned from was mostly lighter-skinned and male. There is a pre-AI ancestor worth knowing: for decades Kodak calibrated colour film against a photo of a white model, the “Shirley card”, so darker skin tones rendered badly for a whole generation, not out of malice but because the calibration target quietly assumed who the default user was. Every evaluation set is a Shirley card. Whoever it depicts is who your system will quietly work for; whoever it forgets is who it will quietly fail.

This is the centerpiece, because it is invisible to any dataset audit: the data can be spotless and the model still wrong, because you measured the wrong thing. A health algorithm used on roughly 200 million Americans a year predicted healthcare cost and quietly used it as a stand-in for health need, which sounds sensible until you remember that less money is historically spent on Black patients at the same level of illness. So the model learned to score equally-sick Black patients as lower-risk. Obermeyer and colleagues found in Science that at the same risk score, Black patients actually had 26.3% more chronic conditions; fixing the target would nearly triple the share of Black patients flagged for extra care, from 17.7% to 46.5%. The data was clean. The question, cost instead of need, was wrong. It is like scouting basketball players purely by height: real, measurable, genuinely correlated with skill, and you will still miss Steph Curry (one of the greatest shooter in basketball history) every single time.

One model is forced to serve groups that genuinely behave differently, so it settles on a compromise that fits none of them well. The textbook case is medical: HbA1c, the blood marker used to diagnose diabetes, reads differently across ethnic groups, so a single one-size-fits-all model can systematically under or over-diagnose a subgroup while looking perfectly accurate on the population as a whole. The tell is a model that scores well overall but has one segment it reliably mishandles. Ask the blunt question: would two or three separate, per-segment models beat our single pot?

Every model is trained to chase a single number, and it chases it with a literal-mindedness no human would. That number has a name: the objective function (you will also hear “loss function”, its mirror image). Think of it as the one score the model is being graded on, the exam it is trying to ace. Training is just the model making millions of tiny adjustments and keeping whatever nudges that score in the right direction. So whatever you put into the score becomes the model’s entire definition of “good”, full stop, and it will optimise for exactly that, including the parts you never meant. Tell a hiring model that a good candidate is one who “resembles the people we hired before”, and it will faithfully, tirelessly rebuild the past, Amazon again. It is like paying an employee on one KPI and one KPI only: they will hit that number, sometimes in ways you deeply regret.

For LLMs there is a second, subtler pass. After the raw model is trained, it is fine-tuned with RLHF (reinforcement learning from human feedback): people are shown two possible answers and pick the better one, and those human preferences become the new score the model optimises. Which means the model quietly drifts toward the opinions, style and blind spots of the specific people who did the rating, and because raters tend to prefer answers that sound confident and agreeable, the model learns to tell you what you want to hear. Researchers call this sycophancy, and once you know it, you see it everywhere: the model agrees with a premise even when it is wrong, caves the instant you push back on an answer that was actually correct, praises a mediocre idea because praise tests well, and mirrors your politics back at you. It is not lying exactly; it is optimising for your approval instead of the truth, because approval was the thing it got graded on. Flattery dressed up as helpfulness, and a quiet bias bent toward whoever is in the chair. The lesson holds either way: an objective function is never neutral. It is a values statement written in maths, and the model will honour it to the letter, not to the spirit.

You can only catch the bias you actually test for, and a single headline accuracy number hides everything underneath it. Remember Gender Shades: a system can look “93% accurate” overall while failing one subgroup a third of the time (that 0.8% versus 34.7% again). Testing broken out per subgroup and per language is the whole game, which is why a serious team writes an internal model card reporting the group-by-group numbers before anyone claps at the average. If you have not disaggregated, you have not evaluated. You have tasted your own portion and pronounced the kitchen clean.

Finally the system meets the real world, real institutions and real humans, and fresh bias enters through how it is actually used. The Netherlands’ childcare-benefits model used dual nationality as a fraud-risk factor; about 26,000 parents were falsely accused, more than a thousand children were placed in foster care, and the government resigned in January 2021. Then there are feedback loops, the sneakiest kind of all: predictive policing sends patrols where past arrests happened, those patrols make fresh arrests in the same neighbourhoods, and that new data circles back to “prove” the model was right all along. It is a thermostat bolted next to the oven, cranking the air conditioning forever and completely confident, because it keeps reading its own heat.

If you build on foundation models, the map gains doors after the dataset. A Nature study showed models whose overt stereotypes had been trained away still assigned speakers of African American English less prestigious jobs and harsher criminal judgments: alignment polished the surface, not the depth. A University of Washington resume study across ~40,000 comparisons found LLM screening preferred white-associated names in 85.1% of cases, and preferred Black-male-associated names over white-male ones in exactly 0%. Add multilingual gaps (quality drops hard in low-resource languages) and the conclusion for your team: the system prompt, the rater pool and the eval language are now bias surfaces, owned by whoever writes and picks them.

Now the nuance that earns your trust in the vendor meeting, because “just make the AI fair” turns out to be, in a precise mathematical sense, impossible to fully satisfy. The famous case is COMPAS, a risk-scoring tool once used in US courtrooms to estimate how likely a defendant was to reoffend. Two camps looked at the exact same tool and both were right. The investigative newsroom ProPublica pointed out that among people who did not go on to reoffend, Black defendants were flagged “high risk” almost twice as often as white ones, an unfair share of false positives (a false positive is simply the system raising the alarm on someone who turned out to do nothing wrong). The vendor, Northpointe, replied that its scores were equally calibrated across races, meaning a given score carried the same real-world reoffense rate for everyone: among all the people it rated “7 out of 10”, the same proportion actually reoffended, Black or white. So the number itself was honest. Here is the uncomfortable part, as a tiny worked example:

So how can both sides be right? Because the two groups had different underlying reoffense rates in the historical data, and Kleinberg, Mullainathan and Raghavan proved something almost cruel: when those base rates differ, no test can be both “calibrated” (a score means the same thing for everyone) and “equal in its mistakes” (the false alarms fall evenly across groups) at the same time, unless the test is a perfect crystal ball. It is a see-saw. Push one kind of fairness down and the other pops up; you cannot flatten both. Which is why “fair” is not a single switch you flip. There are several competing definitions of fairness, they genuinely conflict, and picking one is really choosing whom to protect from which kind of mistake. That is a values decision wearing a maths costume, and the only truly wrong move is to make it by accident. Choose your definition on purpose, and write down why you chose it.

Two more honest notes, because part of this newsletter’s job is to stop you over-reacting as well as under-reacting. First, not every disparity is discrimination. When the Apple Card (the credit card run by Goldman Sachs) seemed in 2019 to hand men far higher limits than their wives, the outrage was instant, but New York regulators investigated and found no unlawful bias; the gaps traced back to lawful factors like income and separate credit histories. The real black eye was that Goldman could not clearly explain its own decisions to furious customers, which is its own kind of failure. A disparity is a reason to open the hood, not a guilty verdict on its own. Second, clumsy debiasing can be worse than the bias it fixes. When Google’s Gemini image generator tried in early 2024 to force diversity into every image, it cheerfully produced racially diverse 1940s German soldiers and other historical nonsense, because it was slapping a correction onto the output instead of fixing the pipeline underneath. Bolting a patch onto the end is not the same as understanding which of the seven doors the bias walked through, which is the whole point of this article.

Here is the payoff, one checkpoint per door, each runnable as a half-day workshop with a named owner:

  1. Proxy check (product owner + domain expert) ⇒ what do we want to predict, what are we measuring instead, and what falls in the gap? One page, before any data work. This is where Obermeyer-sized mistakes live.

  2. Data provenance (data engineering) ⇒ a datasheet per dataset: who is in it, who is missing, which world and which year it reflects. Check subgroup sample sizes, not totals.

  3. Labels (data science + domain expert) ⇒ who labeled, with what instructions, what agreement rates across subgroups? For LLM pipelines: who were the raters, in which language?

  4. Aggregation (ML lead) ⇒ does one model average a subgroup away? Test per-segment alternatives.

  5. Objective (ML lead) ⇒ what does the loss reward? Run Fairlearn or AIF360, pick your fairness metric explicitly, document why.

  6. Disaggregated evaluation (someone who did NOT build it) ⇒ per-subgroup, per-language error rates in an internal model card. An aggregate score is a Shirley card.

  7. Deployment watch (MLOps + legal) ⇒ monitor disparity in production, hunt feedback loops, design against rubber-stamping, and map your legal exposure: NYC’s Local Law 144 already mandates annual bias audits for hiring tools (though a December 2025 state audit found enforcement toothless), the EU AI Act’s Article 10 makes bias detection a legal duty for high-risk systems (timing now shifting under the 2026 digital omnibus; check the current date before you relax), and Mobley means “the vendor did it” is a weak shield. One little-known detail worth its own meeting: Article 10(5) explicitly allows processing special-category data for bias detection. You often need demographic data to measure disparity; the law anticipated it.

For startups: checkpoint 1 is nearly free and a disaggregated eval is a weekend that becomes a sales asset in enterprise procurement. For enterprises: align the seven checkpoints with your NIST AI RMF or EU AI Act conformity paperwork so ethics and compliance are one artifact, produced once.

Yes. The recap:

  • “Biased training data” explains two doors out of seven ⇒ auditing datasets while ignoring problem formulation and deployment is inspecting one door of a seven-door kitchen.

  • The worst bias survives clean data ⇒ Obermeyer’s algorithm was accurate; the proxy was wrong. Ask the proxy question first, it costs one meeting.

  • Aggregate metrics hide subgroup failures ⇒ 0.8% vs 34.7% lived under one flattering average. Disaggregate or you have not evaluated.

  • Fairness metrics mathematically conflict ⇒ pick one explicitly, document the choice, and stop promising “unbiased AI” in decks.

  • The liability perimeter now includes you and your vendors ⇒ Mobley, LL144 and the EU AI Act turn bias review from virtue into paperwork with fines attached.

The one-line takeaway: great kitchens do not just buy better ingredients; they taste at every station. Your Monday action: pick your riskiest AI system and run checkpoint 1, the proxy question, this week.

Which of the seven doors have you actually audited on your current system? Comment and tell me which checkpoint scares you most; the answers shape future deep dives.

Thanks for reading Artificial Impact! This post is public so feel free to share it.

Share

Research

News / investigations / regulation

Read the original on artificialimpact.substack.com ↗

Comments

Nothing yet. Say the first thing.

    Sign in to join the conversation.