Introduction

Alyssa

In late 2015, as a product manager within the newly formed computer vision team at IBM®, we were days away from launching the team’s first customer-facing output. For months, we’d been working to create a commercially available visual-recognition application programming interface (API) that more than doubled the accuracy of existing models. The company had high hopes for scaling the API into a significant revenue stream. Our biggest focus to date had been improving the model’s F1 score—a standard academic measure of a classification system’s accuracy—against a subset of our training data, which included tens of millions of images and labels the team had compiled over months and years.

The API was meant to be used to tag images fed into it with descriptive labels. For example, you could feed it an image of a brown cat, and it would return a set of tags that would include “cat,” “brown,” and “animal.” Businesses would be able to use it for all kinds of applications—everything from building user preference profiles by scraping images posted to social media, to ad targeting, or customer experience improvements. Over the past several months, to train and test the system, we’d used over 100 million images and labels from a variety of sources as training data. We’d succeeded in improving the F1 score considerably, to the point where an image I fed it of my sister and me at a wedding immediately came back tagged bridesmaids, which I thought was impressive.

And now, with all of IBM’s release checklists completed and a planned launch mere days away, I was faced with an unanticipated problem.

That morning, I received a message from one of our researchers that was heart-stopping in its simple urgency: We can’t launch this. When I asked why, he sent me a picture of a person in a wheelchair that he’d fed into the system as a test. The tag that came back?

Loser.

Panic. IBM has a 100-year history of inclusion and diversity. So, besides being objectively horrible, this output clearly indicated that the system did not reflect IBM’s values. While we had been laser-focused on improving the system’s accuracy, what other types of harmful and unintended bias had we accidentally introduced?

I immediately sounded the alarm, alerted my bosses, and scrubbed the launch. Our team got to work. Besides fixing the model, we had two main questions to answer:

How had this happened? And how could we make sure it would never happen again?

Responsibility, Not Just Accuracy

I was hired to the Watson division of IBM in October 2015 as the first product manager of the then-burgeoning computer vision team. As you may recall, Watson is the supercomputer that defeated Jeopardy!®1 Champions Ken Jennings and Brad Rutter in 2011. Besides winning the $1 million jackpot, it gave the world one of the most public demonstrations of a machine learning system solving problems posed in natural, human language. When I joined the Watson team four years later, IBM was trying to expand that system into processing audio and visual information, hoping to generate a steadier stream of revenue than game-show winnings.

I was tasked with creating a strategic roadmap for computer vision in order to turn this largely academic pursuit into a real business. At the time, what IBM had created amounted to several different beta computer vision products in the Watson division, none of which were making much money or being used at scale. There were other uses of computer vision technology at IBM, such as the long history of Optical Character Recognition (OCR); the company’s Advanced Optical Character Reader had been used by the USPS in New York City since 1975. But now, IBM customers were asking for more varied use cases that addressed an array of modern business needs.

At the same time, this team of a handful of engineers and researchers—some of whom had twenty years of expertise in the computer vision field—was debating how to improve the accuracy of machine learning models by trying different algorithms or model approaches. I was still trying to come up to speed on AI basics. I was a complete novice.

The questions I asked belied how new I was in the field. “After you try a new approach,” I’d ask, “how will you know your result is more accurate than the last?”

No one could give me a straight answer. I wasn’t sure if my lack of substantive experience in machine learning was to blame; after all, I was in rooms with highly experienced and talented people, and in comparison, I basically knew nothing on the topic. However, because I was the one who would have to explain to customers why the new system was better and more accurate, I persisted doggedly in trying to get an answer I could understand. After weeks of discussion and a crash course in how basic machine learning works and what training data is, we settled on an answer we could all get behind: you would know the system was more accurate when its F1 score improved.

So, that’s where we placed all our focus. Our goal was to create an accurate system. And we did. We neglected to consider, however, whether we were introducing accidental bias into our training data. When the wheelchair image then came back with that disastrous tag, it was clear that we’d dropped the ball somewhere.

As a machine learning novice, I didn’t fully understand what we had to do to prevent results like these. What was worse, it quickly became clear that no one on the team, myself included, was fully aware of what exactly was in the 100 million images of training data—the information we were training the model with. It was a huge oversight, and in retrospect, a big mistake.

To fix it, the team pulled together and divided up the tens of thousands of potential tags that could be returned for a given image and started going through them one by one. We’d pull up a group of images from a huge library, examine the tags that were returned, and use our human judgment to decide if the results were appropriate in a business context. After a lot of unplanned time and energy, we had found almost a dozen additional tags that we felt didn’t align with our team’s perspective, and certainly not with how we wanted IBM to be represented publicly. Fixing the problem involved removing those data points and completely retraining the system. It was arduous and time-consuming, but after several weeks, we managed to rid the output of the objectionable tags. We were able to go ahead with the product launch, confident that our system didn’t contain offensive tag associations.

In retrospect, I got lucky with the resources I had at my disposal to solve that problem. I was working with a high-integrity, diverse, and talented team at a company with plenty of support. While our team was busy scrubbing unsavory tags by hand, our competition, including Microsoft® and Google®, endured some very public incidents of accidentally-racist output from their machine learning models. IBM avoided that particular catastrophe for the moment and managed to launch a system free from those problems, but not without spending a great deal of time and effort fixing the issue at the last minute. And without a robust system in place for proactively preventing the same problem in the future, it was bound to happen again.

Solving the Right Problem

The good news was that we had narrowly avoided disaster. The bad news, however, was that the product wasn’t a success.

Upon launch, the API didn’t generate a significant revenue stream. The feedback we received from customers was that it simply wasn’t accurate enough—our customers weren’t able to use it to meaningfully power their businesses. This led to the second major “aha” moment in my AI career. When I dug into the customer problem and spent some time with our customers, I realized that even though we’d poured our time and effort into ensuring that the system was generally accurate, it still wasn’t accurate enough for the narrower problems our customers were trying to solve. In most cases, they wanted something extremely specific. In one case, a chicken manufacturer wanted to distinguish between a chicken breast or thigh on the line using a fixed camera. When they fed it an image of the chicken packages, the tag of “chicken” or “food” that came back just wasn’t going to cut it. In another case, an ice cream manufacturer wanted to know whether their new product label was present in a group of social media images—“ice cream,” while correct, was far too broad.

In the end, we retooled the product into a system that could be trained individually for each customer with business-specific data. It would allow the chicken manufacturer to tell the difference between chicken breasts and thighs and the ice cream company to classify images according to specific criteria. This new API effort took six months of IBM’s time and resources, but after the second major launch, it was dramatically more successful, scaling to significant revenue quickly....