If the algorithm didn't introduce the bias — where is it? And who made the decision that put it there?
MIT researcher Joy Buolamwini was working with facial recognition systems and found they couldn't recognise her face reliably. When she wore a white mask, recognition improved. She turned this observation into a systematic study. The Gender Shades project tested three major commercial facial recognition systems — IBM, Microsoft, and Face++ — across a dataset of faces balanced by gender and skin tone. The results were not ambiguous. The systems were nearly perfect for lighter-skinned men and dramatically worse for darker-skinned women. The gap was not a flaw in the algorithm. It was the consequence of a human decision made before the algorithm ran: whose faces went into the training data.
Google's People and AI Research team describes three types of bias that appear in AI systems. Each has a different origin, a different mechanism, and requires a different governance intervention.
The model learns from the humans who interact with it. If early users are a particular demographic — young, English-speaking, urban, technically literate — the model learns to serve them well and learns less well how to serve people who weren't in the room during training. This isn't malice. It's arithmetic. But the effect is the same: certain people get worse outcomes. Microsoft's Tay absorbed racist language from the users who interacted with it on Twitter within hours of launch.
Bias that comes not from what people do, but from what historical data reflects. If you ask an AI image model to generate a picture of a surgeon, it draws on decades of photographs, films, and textbooks that associated surgery predominantly with men. The model isn't making a judgment — it's showing you a statistical average of history. The past is encoded in the present. Latent bias is hardest to detect because no individual decision introduced it: it entered with the corpus.
Bias from underrepresentation in the training data. If certain groups are not present in sufficient numbers, the system performs worse for them. This is what Gender Shades found. IBM, Microsoft, and Face++ had all trained their systems on datasets that did not adequately represent darker-skinned faces, particularly darker-skinned women. The result was a 34.7% error rate — not a small discrepancy. The difference between a system that works and one that doesn't, depending on who's using it.
Buolamwini's 2018 study tested IBM, Microsoft, and Face++ facial recognition systems on a dataset of 1,270 faces, balanced across gender and the Fitzpatrick skin tone scale. The results exposed a systematic pattern across all three providers.
| Group | Error rate range (IBM) | What this means |
|---|---|---|
| Lighter-skinned males | Below 1% | Effectively works — recognition is reliable |
| Lighter-skinned females | 7–12% | Noticeably worse but still functional |
| Darker-skinned males | 12–20% | Significant error rate — not suitable for high-stakes use |
| Darker-skinned females | Up to 34.7% | Functionally unreliable — more than one in three faces misidentified |
In 2019, NIST tested 189 face recognition algorithms and found similar patterns across the industry. The finding was not specific to one provider — it was a property of the development process across the sector. IBM subsequently overhauled its training datasets and auditing processes. The overhaul came after the public finding.
Google's Responsible AI framework identifies where bias is actually introduced. The algorithm does what it is optimised to do. The problem sits upstream, in four human decisions that precede any calculation.
| Human decision | How it introduces bias | Gender Shades example |
|---|---|---|
| Who chose the data | Underrepresented groups produce systems that underperform for those groups | Training datasets lacked sufficient diversity in skin tone, particularly darker-skinned women |
| Who labelled it | Labellers' own biases and gaps are encoded in what counts as correct | Ground truth labels may have reflected labellers' own recognition patterns |
| What the system was optimised for | The objective function determines what "good" means — and who benefits from it | Optimising for overall accuracy masked poor performance on minority subgroups |
| How success was defined | Aggregate metrics hide distributional failures | A system with 95% overall accuracy can still fail 34% of the time for specific groups |
Bias doesn't enter AI systems through the algorithm — it enters through human decisions about data, labels, objectives, and success metrics. Governance that only audits outputs after deployment catches consequences, not causes. Accountability for bias must be attached to the people who made those four upstream decisions — before a single line of code ran.
Name and define the three types of AI bias identified by Google's PAIR team. Give a real example of each.
What did the Gender Shades study find — and what does the finding reveal about where bias is introduced in AI systems?
Google's framework identifies four human decisions that introduce bias before the algorithm runs. Name them — and explain why bias governance must address these upstream decisions, not just audit outputs.