The hiring model taught itself that being a man was a qualification
Amazon · 2014–2017
What they did
Amazon built an experimental ML tool to score résumés 1–5 stars for technical roles, training it on a decade of the company's own historical hiring data — a resume pool that skewed heavily male, as most of the tech industry's did at the time.
What happened
By 2015, Amazon's own team found the model had taught itself that male candidates were preferable: it downgraded résumés containing the word "women's" (as in "women's chess club captain") and penalized graduates of two all-women's colleges. Engineers patched those specific signals, but couldn't rule out the model finding other proxies for gender, and Amazon scrapped the project entirely in 2017 rather than ship it.
The so-what
A model trained on "who we hired before" doesn't learn who's qualified — it learns who got hired before, bias included. This is now the canonical example cited whenever a company says its historical data will keep an AI hiring tool fair.