When I looked at the Rorschach inkblot, I saw a giant, as seen from below, as if through a glass ceiling. A normal, well-adjusted artificial intelligence (AI) bot interpreted it as a black and white photo of a small bird.
A psycho bot who’s been trained on Reddit images saw a guy getting pulled into a dough machine. That’s what a bit too much exposure to the darkest subreddits will do to a bot, evidently – there’s nothing quite like an r/ dedicated to watching people die to mangle your wetware.
This is what Norman sees when he looks at inkblots. It’s not his fault that he sees a man electrocuted when “normal” AIs see a group of birds sitting on a tree branch. (I see Siamese twin bats connected at the torso/head. Nobody has asked me to train AI, so any people who don’t like bats can relax.)
Rather than the non-gruesome images that most AI is trained on, Norman – named after Norman Bates, the homicidal hotel owner-manager in Alfred Hitchcock’s unforgettable psychological horror Psycho – “suffered from extended exposure to the darkest corners of Reddit,” MIT says.
The point of the Norman project is to present a case study on the dangers of AI gone bad when machine-learning algorithms are fed biased data. MIT says the Norman team trained the AI on image captions from an infamous subreddit whose name it redacted “due to its graphic content,” dedicated as it is to documenting and observing “the disturbing reality of death.”
From the project page:
Norman was inspired by the fact that the data used to teach a machine learning algorithm can significantly influence its behavior. So when people say that AI algorithms can be biased and unfair, the culprit is often not the algorithm itself, but the biased data that was fed to it. The same method can see very different things in an image, even “sick” things, if trained on the wrong (or, the right!) data set. Norman suffered from extended exposure to the darkest corners of Reddit, and represents a case study on the dangers of artificial intelligence gone wrong when biased data is used in machine learning algorithms.
MIT made a point of staying away from training Norman on actual images of a real person dying, due to ethical concerns. The team only used image captions that it matched with randomly generated inkblots.
Media Lab then compared Norman’s responses with those from a standard image captioning neural network (one that had been trained on images from the Microsoft Common Objects in Context [MS COCO] dataset). As you can see in this YouTube video showing 80,000 MS COCO images in five minutes, there’s a whole lot of sports-related images in there. The photos go by quickly, but it’s safe to say that gore is under-represented, if not completely missing.
The Rorschach test is used by some psychologists to detect underlying thought disorders, though many have questioned its validity, raising issues such as “illusory and invisible correlations” or the testing psychologists’ projections – for example, the response “bra” has been considered a “sex” response by male psychologists, but a “clothing” response by females.
Be that as it may, Norman interprets a boatload of what’s inarguably nasty: a man shot dead instead of a closeup of a vase full of flowers, say, or a man murdered by machine gun in broad daylight rather than a black and white baseball glove.
Prof. Iyad Rahwan, part of the three-person team that developed Norman, told the BBC that “Data matters more than the algorithm.”
It highlights the idea that the data we use to train AI is reflected in the way the AI perceives the world and how it behaves.
If nothing else, the Norman experiment demonstrates that AI trained on bad data can itself turn bad. That has real consequence outside of the lab: In 2016, Pro Publica released a study that found that algorithms used across the US to predict future criminals – algorithms that come up with “risk assessments” by crunching answers to questions such as whether a defendant’s parents ever did jail time, how many people they know who take illegal drugs, how often they’ve missed bond hearings, or if they believe that hungry people have a right to steal – are biased against black people.
Pro Publica came up with that conclusion after analyzing what it called “remarkably unreliable” risk assessments assigned to defendants:
Only 20% of the people predicted to commit violent crimes actually went on to do so.
What Pro Publica’s data editors couldn’t do: inspect the algorithms that are used to come up with such scores. That’s because they’re proprietary.
The algorithms that produce the risk assessment scores that are widely used throughout the country’s criminal justice systems aren’t the only ones that have been found to be discriminatory: studies have also found that black faces are disproportionately targeted by facial recognition technology. The algorithms themselves have been found to be less accurate at identifying black faces – particularly those of black women. The algorithms are, after all, typically created by white people, so again we can see the data being tilted toward particular outcomes right from the algorithm design stage.
In December 2017, New York City actually passed a bill to study biases in the algorithms used by the city. New York’s efforts are thought to be the first attempt in the country to push for open sourcing of the algorithms used by courts, police and city agencies and to thereby hopefully root out bias in the computer algorithms that are used in public service.
It’s not just racial discrimination or the depths of subreddit darkness that can introduce bias to AI. As the “bra=sex/bra=clothing” bias shows, machine learning can also lead to sexist bots. The BBC cited one study that showed that Google News data taught sexism when used to train software: When asked to complete the statement, “Man is to computer programmer as woman is to X”, the software replied ‘homemaker.”
The BBC quotes Dr. Joanna Bryson, from the University of Bath’s department of computer science, who said that the issue of sexist AI, just like the issues with racist facial recognition databases, could be accounted for by the fact that a lot of machines are programmed by “white, single guys from California” and can be addressed, at least partially, by diversifying the workforce.
Diversify the work force, and you’ll have taken at least one step toward addressing AI bias, she said.
When we train machines by choosing our culture, we necessarily transfer our own biases.
There is no mathematical way to create fairness. Bias is not a bad word in machine learning. It just means that the machine is picking up regularities.