One of the most important applications of artificial intelligence in the music industry is music recommendation in music streaming services such as Spotify. The history of music recommendation goes back to the early 2000s, when LastFM’s Audioscobbler and Pandora’s Music Genome Project transformed internet radio into music streaming.
AI in the Music Industry – Part 5: Music Recommendation in Music Streaming
In 1999, when the P2P file-sharing network Napster was hitting the headlines, Austrian Martin Stiksel and German Felix Miller met at a concert. The two computer nerds agreed that Napster was not a good way to discover new music and decided to set up the “Insine” netlabel in London to make new music visible.[1] At a party, the two label owners met Thomas Willomitzer from Vienna, who worked as a programmer for a software company and taught at Ravensburn College in London. The job at the college had been arranged for him by Michael Breidenbrücker from Vorarlberg, who now joined “Insine” as the fourth member of the group. Sometime in 2001, the idea was born to create an Internet radio station that would synchronise users’ listening habits and use this information to create dynamic playlists and recommend new music.[2] The basic concept of LastFM was ready. The beta version of LastFM went online at the end of March 2002. Users could use ratings to create a music profile, which was then matched with other users’ profiles using a collaborative filtering algorithm. This is a nearest-neighbour algorithm, which is also used by Amazon.[3]
LastFM was registered as a limited company in London in November 2002. However, the new company lacked a business model and the music database, which could not be set up legally due to prohibitive licence costs. This is where Richard Jones came into the picture. Jones was in his final year of a computer science degree at the University of Southampton in 2002. He had read about collaborative filtering and was thinking about how this method could be used to discover new music. The basic principle was simple: if you like bands A, B, C and D and someone else likes bands B, C, D and E, then you should also listen to band E, you might like it. To realise this idea, Jones programmed a plug-in for the Windows Winamp player, which made it possible to track in real time which music was being played. This was the invention of Audioscrobbler, which was also his final project for his degree. Anyone who installed the software on their computer could create a music consumption profile, which could then be compared with other profiles via the Audioscrobbler server to identify similarities in taste and recommend new music.[4]
Audioscrobbler’s functionality was the key addition to LastFM’s dynamic playlist. The two systems supported each other perfectly, and in 2003 the LastFM founders and Jones agreed to integrate Audioscrobbler into the internet radio station. Jones was also given a 15% stake in the company, as he told evolver.fm in an interview on Audioscrobbler’s tenth birthday.[5] In 2003, LastFM attracted the attention of the media and a sudden increase in users, which even led to a server crash that took the online portal offline for two months.[6]
In 2004, the company was still struggling financially, but in 2005, thanks to media coverage, investor Stefan Glänzer, who had previously built a successful internet auction platform, joined the company. A year later, with the help of venture capital firm Index Ventures, LastFM was able to secure long-term funding. The company, which was originally founded as the “last” internet radio station, mutated into a music streaming service with one of the first AI-based music recommendation systems, Audioscrobbler.[7]
With its rapidly growing user base, LastFM became a forerunner for Spotify & Co. and attracted the attention of US media giant CBS, who announced the purchase of LastFM for US $280 million on 30 May 2007. By this time, two of LastFM’s founders had left the company, leaving only Miller, Stiksel and Jones to benefit from the windfall, before finally leaving LastFM in 2009.[8] Despite growing user and sales figures, CBS was unable to turn LastFM into a profitable business. Year after year it made losses[9] and in 2011 it was converted to a pure payment service.[10] At the end of April 2014, LastFM’s music streaming service was shut down and the company continued to operate as a social media platform and music recommendation system.[11]
Silicon Valley entrepreneur Will Glaser had a different concept to the founders of LastFM when, in November 1999, he came up with the idea of identifying the DNA of songs based on a number of attributes, to identify similarities between songs. He put this idea into practice when he met music producer and film composer Tim Westergren, who was working on similar issues of music recognition. Together they launched the Music Genome project with Jon Kraft and Jeffrey Stearns and filed a patent describing the technical process as follows: “Each song is also represented by an n-dimensional database vector in which each element corresponding to one of n musical characteristics of the song. An n-dimensional source song vector that corresponds to the musical characteristics of a source song is determined. A Distance between the source song vector and each of database song vector is calculated, each distance being a function of the differences between the n musical characteristics of the source song vector and one of source database song vector. The calculation of the distances may include the application of a weighted factor to the musical characteristics of resulting vector. A match song is selected based on the magnitude of the distance between the source song and each database songs after applying any weighted factors.”[12] Put simply, a song has between 150 and 450 characteristics, known as ‘music genes’, that make it unique. Using an algorithm, songs with similar genes can now be identified and suggested to the listener.[13]
To monetise the Music Genome Project, Glaser, Westergren and Kraft founded Savage Beast Technologies in January 2000. At the time, the goal was not to create a music streaming service based on the Music Genome Project, but to provide a music recommendation system for music retailers and online music distributors such as CD-Now and eMusic.[14] In March 2000, an undisclosed investor provided US $1.5 million in seed funding. This money allowed Savage Beast Industries to become operational and start acquiring customers. The music recommendation system was originally designed as a B2B service, and the first customer to use it was music retail chain Tower Records, which installed kiosks in its stores in New York City, Los Angeles and San Francisco, allowing customers to browse the Tower Records catalogue and discover new songs using Music DNA.[15]
In February 2003, the company struck a deal with AOL, the largest US Internet provider at the time. AOL customers could use Savage Beast to browse music catalogues to discover similar tracks and create personalised playlists.[16] Despite these collaborations, Savage Beast was on the verge of financial collapse and was only rescued in November 2004 by a US $7.8 million investment from a consortium led by Walden Venture Capital US. Walden’s founder, Larry Marcus, had also the idea of transforming Savage Beast into an internet radio station with music recommendations, initially launched as a subscription-only model, but later supplemented by an ad-supported service. As part of the repositioning, the company was renamed Pandora in July 2005 and launched in September that year.[17]
Unlike traditional music streaming services such as Spotify or Apple Music, Pandora is a non-interactive service that does not allow users to search for tracks directly. Instead, you type in a song to start with and the music recommendation system suggests similar songs. It is not possible to rewind or repeat a song. You can only skip songs you don’t like.[18] For this reason, Pandora is also categorised as an Internet and satellite radio station and licences its music through SoundExchange, a music licensing company set up in 2003 and whose tariff structure is set by the US Copyright Royalty Board.[19] This is also the reason why Pandora is only available in the USA and Canada.
The following years were characterised by solid growth. From 7 million active users at the end of 2009, the number more than doubled to 16 million a year later. Revenue also increased from US $19 million to US $55 million during this period.[20] This laid the foundations for an IPO, which took place in June 2011 and provided Pandora with much needed capital.[21] Despite growing user numbers in the following years, Pandora was unable to turn a profit. A net loss of around US $170 million was reported in 2015,[22] which increased to US $518 million by 2017.[23]
In May 2017, Pandora’s financial difficulties led to the sale of online ticketing portal Ticketfly, which it had acquired two years earlier, to Eventbrite for US $200 million.[24] Around the same time, it was announced that Sirius XM, one of the largest operators of satellite radio stations in North America, would buy Pandora. The US $3.5 billion deal was completed in September 2018, saving Pandora from bankruptcy.[25] Despite the new partner and fresh financial backing, the user decline of recent years continued. Pandora, which still had more active users than Spotify in the US in the years before the SirusXM acquisition, fell further behind its main competitor, and by the end of 2022 only had around 50 million active users per month in the US, a drop of 20 million users compared to 2017.[26] It is doubtful that the loss of users can be stopped by the use of new AI-based models. It seems that one of the first AI-based music recommendation systems has reached the end of its lifecycle and must now make way for other AI solutions.
Spotify also has to deal with this problem. The Swedish music streaming service was an early adopter of artificial intelligence in playlist creation, buying AI startup Echo Nest for $50 million in March 2014.[27] Spotify was already a customer of Echo Nest, whose technology powers Spotify’s playlists such as ‘Discover Weekly’.
Echo Nest was founded in 2005 by two MIT graduates, Tristan Jehan and Brian Whitman. Jehan’s doctoral thesis, “Creating Music by Listening” (2005), explored the possibility of using machine learning to recommend music, not only to identify similar pieces of music, but also to create new music. It is no coincidence that his thesis committee included none other than François Pachet, who produced the first AI generated pop song.[28] In particular, in chapters 4 (“Musical Structure”) and 5 (“Learning Music Signals”), Jehan explains how to make good predictions for music recommendations using machine learning based on musical structure analysis.[29]
The theoretical considerations eventually became the starting point for the project “The Music Brain”. The idea was to search the web for music information in blogs, playlists, discussion forums, etc., using an algorithm to detect trends and online music behaviour. “The Music Brain became the core of Echo Nest.[30] The second pillar was the creation of a huge database of music tracks, using only 30-second samples of each song, rather than entire recordings, to avoid copyright issues. In March 2011, Echo Nest, together with Columbia University’s LabROSA, released a music database of more than 1 million tracks that is freely available to software developers.[31] A few months earlier, Echo Nest had raised US $7 million in funding from Matrix Partners and Commonwealth Capital Ventures to stabilise the company’s financial position.[32] In July 2012, a consortium led by Norwest Venture Partners provided Echo Nest with a further US $17.3 million[33] before the company was acquired by Spotify.
As a subsidiary of Spotify, Echo Nest now had access to a huge database of music and users to train its music recommendation algorithm. In an article for TechCrunch in October 2014, Echo Nest provided a rare insight into how the algorithm works. The goal of the data analysis is to create individual taste profiles for Spotify users based on the metadata of the music they stream. To do this, it collects a wide range of attributes from users, such as mainstream orientation, topicality of music listened to, diversity of music listened to, and discovery of music that will be popular in the future. The profiles created in this way are then compared statistically and similarities in usage are revealed. This is the methodology of collaborative filtering, where the vast amount of data available allows much more accurate predictions to be made about the musical tastes of individual users. Spotify also identifies listeners’ favourite genres and assigns artists and bands to these statistical clusters, which are also made available as playlists. Human curation still plays a role in keeping the playlists lively and varied.[34] Aber das ist immer noch nicht alles. Echo Nest has also created an advanced search engine, originally named Truffle Pig. In the same way that the animal is able to sniff out the location of precious mushrooms for truffle lovers, the search engine is able to find the right music for every taste and compile it into playlists. Truffel Pig uses song characteristics and feelings associated with a piece of music.[35]
With Echo Nest, Spotify has integrated a leading developer of AI-based music identification and recommendation into the company, laying an important foundation for its dominance in the streaming market. Spotify currently (as of February 2024) tracks more than 1,500 music genres, which are used to recommend music to its users and can be listened to on the ‘Every Noise at Once’ website.[36]
However, Spotify has also used its AI expertise manipulatively to secure a slice of the streaming pie with fake artists and fake songs. At least that’s the accusation in the media.
Endnotes
[1] Tonspion, “Das letzte Radio. Last.FM-Mitbegründer Martin Stiksel im Gespräch mit Walter Gröbchen”, December 3, 2007, accessed: 2024-02-01.
[2] The company’s eventful history is detailed in an article on Futurezone: “Zwist und Intrigen bei Last.fm”, August 31, 2009, accessed: 2024-02-01.
[3] Perera et al., 2020, p 83.
[4] The Guardian, “The essential selector”, March 6, 2003, accessed: 2024-02-01.
[5] Wired, “The Man Who Invented Scrobbling and Changed the World”, November 28, 2012, accessed: 2024-02-01.
[6] Futurezone, “Last.fm: Absturz und Neubeginn”, September 1, 2009, accessed: 2024-02-01.
[7] Futurezone, “Last.fm: Turbulenzen und Verkauf”, September 2, 2009, accessed: 2024-02-01.
[8] Ibid.
[9] TechCrunch, “Last.fm reveals £2.84m loss – bulk of users still in UK, despite CBS ownership”, December 3, 2010, accessed: 2024-02-01.
[10] Heise.de, “Last.fm: Mobiles Radio nur noch für zahlende Kunden”, February 7, 2011, accessed: 2024-02-01.
[11] Billboard, “Last.fm Pulls Out of Radio Streaming, Plugs in YouTube”, March 26, 2014, accessed: 2024-02-01.
[12] United States Patent and Trademark Office, “Consumer item matching method and system”, U.S. Patent 7,003,515.
[13] Pandora, “About the Music Genome Project”, n.d., accessed: 2024-02-01.
[14] Vator News, “When Pandora was young: the early years”, April 4, 2017, accessed: 2024-02-01.
[15] Ibid.
[16] Ibid.
[17] Ibid.
[18] Pandora, “How Pandora works”, n.d., accessed: 2024-02-01.
[19] Geoffry P. Hull et al., 2011, The Music Business and Recording Industry: Delivering Music in the 21st Century, 3. Auflage, New York and London: Routledge, pp 101-103.
[20] Pandora Internet Radio, Annual Report 2012, pp 40.
[21] Billboard, “Pandora IPO Prices Above Expectations at $16 per Share”, June 14, 2011, accessed: 2024-02-01.
[22] Pandora Media Inc., Annual Report 2015, p 68.
[23] Pandora Media Inc., Annual Report 2017, p 65.
[24] Music Business Worldwide, “Eventbrite to acquire Ticketfly from Pandora for $200m”, June 12, 2017, accessed: 2024-02-01.
[25] Music Business Worldwide, “Pandora acquired by SiriusXM in $3.5bn deal”, September 24, 2018, accessed: 2024-02-01.
[26] Music Business Worldwide, “Pandora now has fewer than 50m monthly active users”, November 2, 2022, accessed: 2024-02-01.
[27] Billboard, “Spotify Acquires the Echo Nest”, March 6, 2014, accessed: 2024-02-01.
[28] Tristan Jehan, 2005, Creating Music by Listening, PhD thesis, Massachusetts Institute of Technology (MIT), September 2005, MIT: Cambridge/Mass.
[29] Ibid.
[30] Vator News, “The Echo Nest raises $7M led by Matrix”, October 5, 2010, accessed: 2024-02-01.
[31] Echo Nest Pressemitteilung, “Echo Nest, Columbia University Launch Million Song Dataset”, March 4, 2011, accessed: 2024-02-01.
[32] TechCrunch, “The Echo Nest Raises $7 Million For Music Personalization Platform“, October 5, 2010, accessed: 2024-02-01.
[33] Billboard, “The Echo Nest Raises $17 Million, Plans to Branch Out From Music Discovery”, July 12, 2012, accessed: 2024-02-01.
[34] TechCrunch, “Inside The Spotify – Echo Nest Skunkworks”, October 19, 2014, accessed: 2024-02-02.
[35] Wired, “Inside Spotify’s Hunt for the Perfect Playlist”, July 28, 2015, accessed: 2024-02-02.
[36] Spotify, “Every Noise at Once”, December 13, 2023, accessed: 2024-02-02.
Comments