You Have Not Yet Heard Your Favourite Song (eBook)

6

THE ROBOTS HAVE NO PLAN

What Algorithms Do and Don’t Do

It’s reasonable enough to fear what your services know about you. Most of what makes this a fear, though, is the implicit anthropomorphism in ‘know’. We knowthings about each other, our weaknesses and embarrassments and vulnerabilities as much as our powers and accomplishments and concealed contours. Facts accumulate into insights, and past observations lead to future intentions. At least, that’s what happens in our brains.

That’s not what happens inside of a computer. The computer may have a lot of your data, but it doesn’t know about you, any more than a refrigerator knows about eggs. Fill a refrigerator entirelyfull of eggs, and it still doesn’t know any more about eggs than when it was empty.

One notable quasi-cognitive difference between a refrigerator and a music-streaming service is that the streaming service has algorithms. Technically speaking, the refrigerator has algorithms, too. Even the simplest refrigerator has twoalgorithms. One is called a thermostat, and says that if the temperature in the fridge goes over temperature Y, run the cooling circuits until the temperature gets back down to X. The other one is called a lightswitch, and says that if something is pushing in the door sensor, turn off the light, and if nothing is pushing it, turn onthe light. The two combine to produce the compelling product-illusion of the refrigerator, which
is that it’s an always-cold and always-brightly-lit box.

Algorithms can get a lot more complicated than that, but they never get any smarter. They do not seek out challenges, they do not think flexibly, they don’t persevere to understand, and they don’t construct or critique arguments. They basically permanently fail at all the foundational Habits of Mind from my kid’s elementary school. They are not minds. They are, at most, symbolic machines. And more often than not, these ‘machines’ are just math. ‘Just’ is a mean-spirited word to use for this, though. Math is very useful. Symbolic machines can do extremely helpful symbolic work, and songs and fascinations are made of symbols.

Streaming music services are full of algorithms. If the most
important thing to know about algorithms is that most of them are just math, the second-most important thing to realize about algorithms is that there is never just one. There is no ‘Spotify Algorithm’, there are different algorithms for almost every feature. Sometimes a single feature involves multiplealgorithms. Almost everything in any online experience involves some kind of algorithm.

This is perhaps easiest to understand in Search. It sounds initially simple to imagine how it works when you type in the name of a song you want to hear. I would like to hear ‘Amaranthe’, by Nightwish. I hit Search in the Spotify desktop app and I type ‘Amaranthe’. Except I don’t really capitalize when nobody is watching me, so what I actually type is ‘amaranthe’.

There turn out to be three songs called ‘Amaranthe’ on Spotify, and 32 more that have ‘amaranthe’ as partof their titles. There are also 2 albumscalled Amarantheand three more with ‘amaranthe’ in their album titles, and there’s a band called Amaranthe, and others called Amaranthe Love and Amarantheum. Even if we assume I haven’t misspelled the part I typed, that’s already 43 potential matches. Arranging 43 things into a sensible order requires an algorithm. We could group them by item-type (song/album/artist), and within each group put the exact matches first. That’s an algorithm. We could even order the groupsby how many matches they have, so in this case the group of 35 songs would come before the group of five albums, with the group of three artists last.

Except I left out playlists. There are 20 playlists on Spotify, as I write this, that are named exactly ‘amaranthe’, in lowercase. There are 577 more called ‘Amaranthe’, uppercase, and several thousand that have ‘amaranthe’ as part of their titles. So by the algorithm we just defined, the giant playlist group should come first. But this is probably a bad outcome, because it seems unlikely that I am looking for one of these hundreds of playlists with identical names, mostly made by other Spotify listeners for their own personal use. So maybe our algorithm should be the other way around: smallest groups before larger ones. That would put artists first, then albums, then songs, then playlists. Which is unfortunate for the specific case we started with, because we are actually looking for a song. And now, instead of me easily finding a fairly famous song that I have played many times, my top result is some totally different band that just happens to coincidentally share the name. Bad Search.

The most obvious, and probably the easiest, way to try to improve on this kind of bad algorithmic result, since Spotify has years of both listening data and search data at its disposal, is to take advantage of historical patterns to try to guess which results are more likely to be what people generally want. In this case it turns out that Amaranthe, the band, have much more listening than any of these other bands or songs or albums or playlists, and are also the most-often picked result by previous people who have typed ‘amaranthe’ into the Search box.

To be fair, most of those people weren’t trying to make a sneaky point about problem complexity, which is what I’m doing. The Swedish gothic symphonic metal band Amaranthe was originally called Avalanche, but had to change their name for legal reasons in 2009. I can’t find any quotes of them saying they were inspired by the genre-defining Nightwish song, released next door in Finland in 2007, but they can hardly have been unaware of it. Which is probably why they were careful to make their name Amaranthe, with an ‘e’ on the end, instead of ‘Amaranth’, without the ‘e’, which is the name of the plant in the Nightwish song, and thus of course the proper spelling of that song’s title. I misspelled it intentionally, to make my explanation more interesting.

If our searching algorithms weren’t already interesting before, they get more complicated pretty quickly once we try to account for misspellings. People often type ‘Drale’ instead of ‘Drake’ into the Spotify search-box. There isan artist called Drale, but they have 11 listeners from their lone song on a 2019 Croatian compilation. That’s not who most of the people typing ‘Drale’ are looking for. But thankfully programmers have been dealing with misspellings in search for a few decades, now, so there are some well-established techniques, many of them involving edit distance, which is the number of changes you would have to do to make one word into another, or if you think about it the other way around, the number of errors that would get you from the word you maybe meant to the one you actually typed. Thus ‘Amaranth’ and ‘Amaranthe’ have an edit distance of one, because it takes only a single error to type one when you meant the other, but ‘Amaranthe’ and ‘Coelacanth’ have an edit distance of six (you would have to drop one letter, change three and add two to ‘accidentally’ confuse those), so the Nightwish song ‘Amaranth’ is a pretty plausible result for the search ‘amaranthe’, but the deadmau5 song ‘coelacanth’ is probably not. If we tweak our search algorithm to include things within an edit distance of two or three, the song I was actually looking for becomes one of the candidates, so that’s an improvement. But the ordering problem gets harder at the same time, because now we have even more results.

Probably our best bet in this case, and usually the second most-common way to try to improve algorithmic search-results, is to take advantage of data about meto try to give me the result that Iam most likely to be looking for, even if that’s different from what most other people are usually looking for when they type this. I’ve played the song ‘Amaranth’ a lot. As it happens, I’ve also played the band Amaranthe a lot. But I’ve never played Amaranthe Love or Amarantheum, and never played most of the other songs whose names match, nor any of the playlists, so personalizing these results makes them quite good: Amaranthe and ‘Amaranth’ are the first two results, and both are great guesses for what I probably meant.

Pretty much everything else you experience online, including in Spotify, is the result of years of iteration like this on algorithms that are trying to manage latent complexity on your behalf, or second-guess your own literal requests instead of just pedantically allowing you to fail. The Spotify Home page has algorithms that try to remind you of the things you probably want to hear now. Spotify’s personalized Discover Weekly playlist has algorithms to try to find you just-barely-unknown songs like the songs you already like. Algorithmic radio stations try to balance the familiar songs you probably expect to hear with maybe-unfamiliar ones you might be pleased to maybe-discover. Charts and seeminglysimple counting still involve algorithms about eligibility and track-equivalency and time-zones and potential fraud.

And although lots of these algorithms are simple, some of them are not only...