Supervised Learning

In 2014 a team at Amazon began building a program to rank job applicants from one to five stars. They trained it the standard way, on examples: the patterns in ten years of résumés submitted to the company. By 2015 the team saw that the system was not rating candidates for software-developer and other technical jobs in a gender-neutral way. Most of those résumés had come from men, and in Reuters' words the system "taught itself that male candidates were preferable": it penalized résumés containing the word "women's", as in "women's chess club captain", and downgraded graduates of two all-women's colleges. Amazon edited the program to be neutral to those particular terms, but could not be sure it would not find other ways to sort candidates that were just as discriminatory, and the team was disbanded by the start of 2017. Reuters reported the story in October 2018; Amazon said its recruiters had looked at the tool's recommendations but never relied on them alone.