Sophos Cloud Optix
When I looked at the Rorschach inkblot, I saw a giant, as seen from below, as if through a glass ceiling. A normal, well-adjusted artificial intelligence (AI) bot interpreted it as a black and white photo of a small bird.
A psycho bot who’s been trained on Reddit images saw a guy getting pulled into a dough machine. That’s what a bit too much exposure to the darkest subreddits will do to a bot, evidently – there’s nothing quite like an r/ dedicated to watching people die to mangle your wetware.
At any rate, say hello to Norman, a bot that MIT’s Media Lab claims is the “world’s first psychopath AI [artificial intelligence].”
This is what Norman sees when he looks at inkblots. It’s not his fault that he sees a man electrocuted when “normal” AIs see a group of birds sitting on a tree branch. (I see Siamese twin bats connected at the torso/head. Nobody has asked me to train AI, so any people who don’t like bats can relax.)
Rather than the non-gruesome images that most AI is trained on, Norman – named after Norman Bates, the homicidal hotel owner-manager in Alfred Hitchcock’s unforgettable psychological horror Psycho – “suffered from extended exposure to the darkest corners of Reddit,” MIT says.
The point of the Norman project is to present a case study on the dangers of AI gone bad when machine-learning algorithms are fed biased data. MIT says the Norman team trained the AI on image captions from an infamous subreddit whose name it redacted “due to its graphic content,” dedicated as it is to documenting and observing “the disturbing reality of death.”
From the project page:
Norman was inspired by the fact that the data used to teach a machine learning algorithm can significantly influence its behavior. So when people say that AI algorithms can be biased and unfair, the culprit is often not the algorithm itself, but the biased data that was fed to it. The same method can see very different things in an image, even “sick” things, if trained on the wrong (or, the right!) data set. Norman suffered from extended exposure to the darkest corners of Reddit, and represents a case study on the dangers of artificial intelligence gone wrong when biased data is used in machine learning algorithms.
MIT made a point of staying away from training Norman on actual images of a real person dying, due to ethical concerns. The team only used image captions that it matched with randomly generated inkblots.
Media Lab then compared Norman’s responses with those from a standard image captioning neural network (one that had been trained on images from the Microsoft Common Objects in Context [MS COCO] dataset). As you can see in this YouTube video showing 80,000 MS COCO images in five minutes, there’s a whole lot of sports-related images in there. The photos go by quickly, but it’s safe to say that gore is under-represented, if not completely missing.
The Rorschach test is used by some psychologists to detect underlying thought disorders, though many have questioned its validity, raising issues such as “illusory and invisible correlations” or the testing psychologists’ projections – for example, the response “bra” has been considered a “sex” response by male psychologists, but a “clothing” response by females.
Be that as it may, Norman interprets a boatload of what’s inarguably nasty: a man shot dead instead of a closeup of a vase full of flowers, say, or a man murdered by machine gun in broad daylight rather than a black and white baseball glove.
Prof. Iyad Rahwan, part of the three-person team that developed Norman, told the BBC that “Data matters more than the algorithm.”
It highlights the idea that the data we use to train AI is reflected in the way the AI perceives the world and how it behaves.
If nothing else, the Norman experiment demonstrates that AI trained on bad data can itself turn bad. That has real consequence outside of the lab: In 2016, Pro Publica released a study that found that algorithms used across the US to predict future criminals – algorithms that come up with “risk assessments” by crunching answers to questions such as whether a defendant’s parents ever did jail time, how many people they know who take illegal drugs, how often they’ve missed bond hearings, or if they believe that hungry people have a right to steal – are biased against black people.
Pro Publica came up with that conclusion after analyzing what it called “remarkably unreliable” risk assessments assigned to defendants:
Only 20% of the people predicted to commit violent crimes actually went on to do so.
What Pro Publica’s data editors couldn’t do: inspect the algorithms that are used to come up with such scores. That’s because they’re proprietary.
The algorithms that produce the risk assessment scores that are widely used throughout the country’s criminal justice systems aren’t the only ones that have been found to be discriminatory: studies have also found that black faces are disproportionately targeted by facial recognition technology. The algorithms themselves have been found to be less accurate at identifying black faces – particularly those of black women. The algorithms are, after all, typically created by white people, so again we can see the data being tilted toward particular outcomes right from the algorithm design stage.
In December 2017, New York City actually passed a bill to study biases in the algorithms used by the city. New York’s efforts are thought to be the first attempt in the country to push for open sourcing of the algorithms used by courts, police and city agencies and to thereby hopefully root out bias in the computer algorithms that are used in public service.
It’s not just racial discrimination or the depths of subreddit darkness that can introduce bias to AI. As the “bra=sex/bra=clothing” bias shows, machine learning can also lead to sexist bots. The BBC cited one study that showed that Google News data taught sexism when used to train software: When asked to complete the statement, “Man is to computer programmer as woman is to X”, the software replied ‘homemaker.”
The BBC quotes Dr. Joanna Bryson, from the University of Bath’s department of computer science, who said that the issue of sexist AI, just like the issues with racist facial recognition databases, could be accounted for by the fact that a lot of machines are programmed by “white, single guys from California” and can be addressed, at least partially, by diversifying the workforce.
Diversify the work force, and you’ll have taken at least one step toward addressing AI bias, she said.
When we train machines by choosing our culture, we necessarily transfer our own biases.
There is no mathematical way to create fairness. Bias is not a bad word in machine learning. It just means that the machine is picking up regularities.
Creepy photo!
If the online communities we are creating and participating in are creating psychos out of AI, who’s to say it isn’t doing the same thing to us?
I remember reading a story about an AI with a learning input made public. Anyone could interact with the AI and teach it new information connections. 4Chan found the input and taught the AI to be rather racist and offensive.
I’m not sure if it was the same AI robot, named Tay that was turned into a racist, sexist nazi sympathizer on Twitter in less than 24 hours.
There was another AI bot, Mitsuku, where two attempts were made from 4Chan to teach it to be offensive, but since its learning algorithm was curated, the attempts failed.
Your post reminds me that I forgot to include MIT’s invitation to add your own inkblot interpretations and thus help to retrain psycho Norman. Adding that now!
Great article.
“The algorithms themselves have been found to be less accurate at identifying black faces – particularly those of black women. The algorithms are, after all, typically created by white people”
Not sure this is the direct cause of that problem for facial recognition. Would rather we rule out physical challenges from lighting and sensors (to name two possible areas of problems) before jumping to the conclusion that it must be the with the people who programmed the algorithms.
Has anyone done that? If so, would like to see that added to statements like these to let us know that was at least considered.
Notwithstanding that, when referring to skewing results from machine learning by feeding it skewed input – e.g. if it were to predict criminality based on photos or narrow demographic background of U.S. convicted persons – agree fully that is a problem, as it probably isn’t capturing the full story, thus prone to bias inherent in the communities and how law is enforced.
I agree with BM – one major factor is probably the significant lack of detail in dark/black images of any kind. Dark/black images are caused by lack of light reflection from the surface of an object which limits the available dynamic range – think black cat in a coal mine. Small details that are more clearly seen in lighter toned images are hard to discern in dark images. Racism is not always the correct answer.
Hmmm… almost as if you could remove the “A” from “AI” and it would still be true… 😉
How many attempts did they make that didn’t produce headline-worthy results?
At least one (documented in this article :-)
To be fair, we did put the word “psycho” in double quotes.
Wait, so they didn’t actually train the AI on violent graphic images and that led it to see different things in inkblots? Of course if you show the AI pictures of kittens and label them “vampires” it’s going to assume kittens are vampires. When it later then sees a kitten it will say “vampire” but still meaning a kitten. If you teach a kid the wrong words for things it doesn’t mean it has psychopathic tendencies, it just made the wrong word-to-object connections that you taught it. I think the inkblot thing is very misleading. Norman thinks a man electrocuted looks like an inkblot because you showed it a certain inkblot that said “man getting electrocuted”, therefor when it sees a similar inkblot it thinks its a man electrocuted. I don’t see the point or meaning in that if it has no concept of what a man electrocuted actually is.
They don’t call it *artificial* intelligence for nothing :-)
Interesting 🙂
Not unlike the average human. The environment you live in is often your perception of normal/ expected. If you grow up in a place where most people xxxxx, then you will expect that of most people. It would be more fair to the experiment if they exposed it to seemingly positive images also, then ask what it sees in the blot test. This would help clear up if Norman is simply a product of it’s environment, or is capable of deciding how it wants to see the world.
“The team only used image captions that it matched with randomly generated inkblots.”
So basically they could have thrown together a “crude headline generator” and trained it using that alongside their inkblots and been just as scientific? Why bother using Reddit at all?
On one level, this reads a lot like a “lets bash reddit” attempt. I understand that their purpose was “bad data leads to bad results” but… we’ve had one of those sayings for a long time “trash in, trash out” and it’s proven to be true across the rest of our technology as well as populace (of Earth), in a broad sense, so why should AIs be any different?
I guess I’m just questioning the worth of this exercise…