What Color Is This?

In order to transform the way people find what they love, we need to understand many attributes about our merch. Should we buy more of it and who should we send it to? We know from our past work that color is an important factor in all of these questions: it affects who will buy the item, what it will go with, and what season it’s popular in. And we all know that it’s easy to understand what color something is. For example, this top: we asked a few people what color it was.

• “Blue”
• “Teal”
• “Turquoise”
• “Dark Teal”
• “An understated teal, medium saturation and brightness”

… Hmm. That should have been easy. Well, we’re data scientists; we can write code to solve this, right? Our algorithm found that the color is: RGB(63, 105, 125). That doesn’t really answer our questions either. Isn’t there just a simple answer to the simple question: “What color is this?”

Not really. When we say “What color is this?” we’re asking many different questions. For example, we have buyers who decide what clothes we bring into our warehouses. A buyer who’s doing a deep dive into shades of blue would like to know some details about the color, like, “understated teal, with medium saturation and brightness.” Another buyer who’s just looking at macro color trends might only care that it falls in the “blue” category. On the other hand, we have stylists who are making the final decisions on which items each individual client would love; a stylist might want something in between, like “teal.” And an algorithm that’s trying to predict the effects of different color components would get the best results with the numeric value.

Throughout Stitch Fix, we believe that humans and computers both help solve problems, and that we work better together: determining colors is no exception. Computers can identify exact points and automatically pick out colors from images, but humans have to provide the images and translate the results into an interpretable form. In this post, we’ll cover the human sides of this process, and our next post will get into the details of the computer vision algorithms that make up the middle of the pipeline.

Human expertise: taking photos

At Stitch Fix, our photographers in our photo studio create images to be used throughout styling, editorial content, and internal tools. Of course, color depends on so many variables, like camera settings and lighting; luckily, our photo studio controls many of these variables. They maintain regularly calibrated cameras and sets with consistent ISOs, apertures, flashes, and camera locations. They then calibrate the images coming from each one in order to maintain consistent exposure, color temperature, tint, contrast, and other settings. An expert retoucher compares every image against the item under a daylight lamp to ensure that the image is as true to life as possible. After that, the image is uploaded to our internal tools and available throughout the company.

Computer expertise: extracting color clusters

We extract colors from images in two ways: a “fast” way and a “slow, but more accurate” way, depending on the use case. The fast way uses Colorgram.py. The slow way, which we built ourselves, uses a couple of clustering steps: Quickshift superpixel segmentation, taking into account each pixel’s color and position, then K-means clustering ignoring the position. We will describe this pipeline in more detail in a future post: in short, we start with an image and end up with a list of numeric L*a*b* values for each color.

Human expertise: naming and categorizing these colors

Our computer vision pipeline takes in a merch image and returns a set of points representing its colors, but people can’t intuitively understand labels like RGB(63, 105, 125) or LAB(42, -8, -15). To make these useful throughout our organization, we have to give them names. Initially, like others before us, we turned to the XKCD color survey list of 949 most common colors; for each of our clusters, we used the nearest neighbor in this space.

However, this list has a few problems. First, it’s too fine grained; buyers and stylists won’t always need to choose the exact right value from almost 1,000. Second, the colors are named by anonymous people on the internet, which leads to some *ahem* interesting names. They’re very descriptive, but arguably knowing that this top is “puke brown” would unfairly influence people’s decision to buy it:

Third, modeling color as a single categorical variable would throw out all the knowledge that’s built into those numeric values; it would remove our models’ ability to learn that “periwinkle” and “light blue” are similar.

How should we classify these colors? Is turquoise a greenish blue or a bluish green? Is “varsity red” the same as “brick,” or is it different? And we specifically care about these as they relate to clothes, not paint or pixels. We need some human judgment.

So when we talked with Ryen Anderson, director of Men’s merch design and multi-decade veteran of the fashion industry, he had an idea: he could use his color expertise to build a taxonomy himself. His goal was to make something that’s usable for stylists, merchandise buyers, and algorithms: something that not only describes what color something is, but also a little bit about where that color is positioned relative to other colors. After hours of categorizing and sorting colors, he built the following color hierarchy, from broadest to narrowest:

Color Family

This top-level space, with 26 colors, is based on the color wheel: primary colors red, yellow, blue; secondary colors orange, green, and purple; and two tertiary colors between each pair of primary and secondary. We also add some neutral values to this wheel (black, brown, grey, khaki, olive, sienna, white, and yellow ochre) because these values are stylistically important but not included in the standard color wheel. (An attribute called “Color Or Neutral” encodes whether each color is a Color or a Neutral value.)

Common Color

Each Color Family is divided into multiple Common Colors. This “Common Color” space, meant for general use throughout the company—including by our stylists—has 121 colors.

Design Color

Each Common Color contains one or more Design Colors. This space, mostly used by our designers and buyers, is a slightly finer-grained space of 240 colors.

XKCD Color

At the finest level, we have XKCD colors; each Design Color contains one or more XKCD colors. These come from the 949 color list from XKCD[1]. For example, XKCD colors “azure”, “cerulean”, and “water blue” are all subsets of Design Color “azure blue.” Every color in our clothes is matched to an XKCD color, but because there are so many, XKCD colors are hard to use effectively.

Color Type

There is also one cross-cutting attribute called “Color Type.” This space represents, roughly, how lightened or darkened a color or neutral is. It has 11 values such as “tinted color”, “saturated color”, or “medium neutral.” Every common color is mapped to exactly one color type.

Putting it all together

We’ve created a hierarchy of colors, from color family to common color to design color to XKCD color. We’ve also determined each color’s type, which can be helpful to understand cross-family issues like saturation and brightness. Explore the entire hierarchy here:

Color names for everyone

We’re continuing to use our color extraction pipeline and hierarchy throughout the company. For example, our buyers are able to see exactly when dark neutrals start downtrending every year or how fast “Gen Z Yellow” is selling. Some of these features will help stylists decide what to send to each client; maybe instead of knowing that a client likes “Green” we can learn that they like Aquatic Green or Primary Green but not Celadon. We’re also using these color features in downstream machine learning models that help us decide what to buy. Finally, our color hierarchy has other fun uses as well, such as when Ryen was asked to predict the Pantone 2020 color of the year; because he could look at our merchandise at the right level of color granularity, he was able to predict it remarkably closely!

Because our color hierarchy has been so valuable to us, and we think that others might have a use for it too, we’re releasing it as an open source package, Colornamer. It includes the data in CSV and JSON formats and a simple Python package that will return all levels of the hierarchy for any RGB point passed in. Hopefully this will make it easier for anyone to understand colors at many different hierarchy levels; we welcome your questions, comments, and pull requests.

And stay tuned for more technical details about that middle part where we process all those pixels!

References

[1]↩ On this site, Randall Munroe (who ran the XKCD survey) mentions 954 colors, but at the end of the page notes: “There are a couple of ‘again’s and ‘darker’s which are survey artifacts (e.g. ‘green again’ or ‘darker blue’) that I missed while cleaning up the table. I can’t regen it now, but I’ve deleted them from rgb.txt.” We used rgb.txt, which only has 949 colors.