A Viral Art Project Exposed Biases in Facial Recognition Technology—and Spurred the Largest AI Database to Remove Hundreds of Thousands of Images

If you have been on social media
at all in the past week, chances are you might have seen people
sharing photos of themselves tagged #ImageNetRoulette, accompanied
by amusing, sometimes less than flattering, annotations. Indeed,
you may have been perplexed or even angered by these viral images,
as the captions tipped over from amusing to offensive.

As with other recent viral art
initiatives like FaceApp and Google Arts & Culture’s art
Doppelgänger-finder, people were uploading images of themselves
to a website where an AI, trained on the most widely used image
recognition database, analyzed what it saw. At one point last week
it was spitting out as many as 100,000 labels an hour, according to
the New York
Times.

When I run my own image through
the website, I get labeled “mediatrix: a woman who is a mediator,”
which is humorous enough. But scroll through the hashtag on
Twitter, and you can see where the amusing gaffes of the algorithm
dip into the deeply problematic. I see, for instance, people of
color sharing their own labels: a dark-skinned man is labeled as
“wrongdoer, offender,” an Asian woman as a “Jihadist.”

As it turns out, this is all part
of an art project initiated by the artist Trevor Paglen and an AI
researcher, Kate Crawford, aimed at exposing how systemic biases
have been passed onto machines through the humans who trained their
algorithms. 

“I’ve been really surprised by
the attention it’s gotten online, and heartened by how many people
‘get it’ in terms of seeing the bigger point I’m trying to make
with the piece about how dangerous it is for machine learning
systems to be in the business of ‘classifying’ humans and how
easily those efforts can—and do—go horribly wrong,” Paglen tells
artnet News.

He adds that the images uploaded
to the site are deleted instantly, and that data of the people
using it is not collected.

Training Humans

Paglen and Crawford’s project is
on view in an exhibition called “Training Humans,” which
opened at the Fondazione Prada’s Osservatorio space in Milan last
week, and is on view through February 24. As artificial
intelligence and facial recognition technologies have been creeping
more and more into our daily lives, the pair wanted to conduct a
sort of archaeology of the images used to “recognize” humans in
computer vision and AI systems.

“Understanding the politics within AI systems
matters more than ever, as they are quickly moving into the
architecture of social institutions: deciding whom to interview for
a job, which students are paying attention in class, which suspects
to arrest, and much else,” Paglen and Crawford write in
an essay accompanying the
exhibition.

Surveillance cameras equipped
with facial recognition technology are increasingly being used
around the world. Indeed, some have even recently been approved for
use in London’s cultural institutions, including the Barbican
Center, where Paglen’s work on the subject is about to be shown in
an upcoming
exhibition.

Paglen and Crawford’s project
hones in on an expansive database of photographs called ImageNet,
used by researchers to train artificial intelligence systems in how
to understand the world. The database was first compiled by
researchers at Stanford University in 2009 to develop the
algorithms applied in deep learning, the process by which machines
are trained to recognize images. It is one of the most widely used
training sets for machine learning.

To simplify how it works, an
algorithm is trained to recognize images of say, dogs or flowers
based on being fed a great number of images labeled “dog” or
“flower.” But among the
more than 14 million photographs ImageNet was trained on were
thousands of photographs of people sorted into descriptive
categories from “cheerleaders” to more loaded terms like
“slattern, slut, slovenly woman,
trollop.”

These labels were assigned
manually by humans in labs, or strangers paid to label the images
through crowdsourced tools like Amazon’s Mechanical Turk. Humans
categorized what they saw in terms of race, gender, age, emotion,
and sometimes personal character. In so doing, they injected their
own conscious and unconscious opinions and biases into the tissue
of the algorithm.

Warning Signs

The outcomes of ImageNet
Roulette expose the biases and politics behind these
problematic datasets, which are the bedrock of
the AI systems used to
classify humans today. While
AI seems to offer an objective look at what something is, the
process of classification at its root is a continuation of embedded
systemic biases such as racism and misogyny.

“The project is meant to call
attention to the real harms that machine learning systems can
perpetuate,” Paglen says. Confronting people with photographs next to the
labels given by the AI to identify them shows just how the
boundaries between science, history, politics, prejudice, and
ideology can be blurred in artificial intelligence, and how the
outcomes tilt in favor of those who have the power to build these
systems. 

The artwork has already had an
impact. Last week, in the wake of the attention received
by Crawford and Paglen’s project, the researchers behind ImageNet
announced that they would scrub more than half of the 1.2 million
pictures in the dataset’s “people” category.

“Science progresses through trial
and error, through understanding the limitations and flaws of past
results,” ImageNet said in a statement. “We believe that ImageNet, as
an influential research dataset, deserves to be critically
examined, in order for the research community to design better
collection methods and build better datasets.”

Still, Paglen and Crawford stress
that their project calls attention to the problem of categorizing
people in this way at all, given the rapid uptake of these
algorithmic systems within institutions from education to
healthcare to law enforcement.

“There is no easy technical ‘fix’
by shifting demographics, deleting offensive terms, or seeking
equal representation by skin tone,” they write. “The whole endeavor
of collecting images, categorizing them, and labeling them is
itself a form of politics, filled with questions about who gets to
decide what images mean and what kinds of social and political work
those representations perform.”

The post A Viral Art Project Exposed Biases in Facial
Recognition Technology—and Spurred the Largest AI Database to
Remove Hundreds of Thousands of Images appeared first on
artnet News.