An early application of artificial intelligence is in production music. This involves the creation of music databases that can be used by producers of TV and feature films, commercials and games to acquire music at low cost. Another area of music production where AI has become indispensable is the mastering of music recordings, which used to require a good ear and a lot of experience but is now increasingly being taken over by AI applications. In this part of series “AI in the Music Industry”, we, therefore, highlight the role of AI in music production.
AI in the Music Industry – Part 7: AI in Music Production
The starting point for the application of AI is MIDI technology, which found its way into music production in the early 1980s. MIDI stands for Musical Instrument Digital Interface and is a programme for exchanging musical control information that connects digital and analogue musical instruments. Dave Smith came up with the idea after graduating from the University of California at Berkeley with a degree in computer science and electrical engineering. In 1974, he founded a company called Sequential Circuits, which produced a variety of sound-generating machines such as sequencers and synthesizers. What bothered him, however, was that the synthesizers, which were able to play only a few notes, could not be linked to other synthesizers via a keyboard to expand the range of sounds. The solution was to program an interface that would allow audio signals to be exchanged via a cable connection.[1] Smith first presented the MIDI 1.0 protocol[2] to the Audio Engineering Society in 1981, and two years later, at the National Association of Music Merchants (NAMM) show, he demonstrated its technical implementation by interfacing a device of his own manufacture with a competing Roland product.[3] The success was overwhelming and MIDI quickly became the industry standard for connecting musical instruments.
The MIDI interface quickly found its way into music production. Recording studios used it not only to control a wide range of synthesizers and other digital sound generators, but also to connect the instruments to the computer so that music recordings could be edited easily and cheaply. Audio converters were soon used to extract MIDI files from analogue instruments such as acoustic guitars or saxophones. Over time, this led to the creation of huge MIDI databases that music producers could access in their work.[4]
However, the sound quality of these music databases was poor. This bothered film composer Herbert Tucmandl, who had studied cello and cinematography in Vienna and later produced films by himself. He composed his own film music on the computer, using the orchestral music databases available at the time. However, the databases were limited to 6,000 orchestral samples, most of which were of limited sound quality. Tucmandl came up with the idea of recording entire phrases rather than just individual notes, which led to the creation of the Vienna Symphonic Library (VSL) in 2000[5], which grew to over one million samples within three years. The project was financed with venture capital from Swiss investor Markus Kopf.[6] His money was well invested as VSL quickly became the gold standard for music databases and was used by music producers worldwide to produce music for commercials, feature films, TV films and games at low cost.
This type of music is also known as production music. The business model can be traced back to De Wolfe Music, founded in London in 1909.[7] The Dutch-born musician and conductor Meyer De Wolfe had come to London to work as a conductor of cinema orchestras that provided live accompaniment to silent films. He found that it was very time-consuming to obtain inexpensive sheet music for cinema performances, some of which he had to write himself. He then set up a music publishing company which built up an archive of music specifically for silent film accompaniment. When talkies took the world by storm in 1927, De Wolfe Music’s business model was severely threatened. Meyer De Wolfe’s innovative response was to use the optical sound process, in which film music was recorded on a 35mm nitrate film that could be synchronised with the film reel. De Wolfe Music’s music archive was supplemented by an archive of music recordings. This was the beginning of production music. De Wolfe was able to produce inexpensive commercial music for newsreels that were shown before a movie. Customers soon included not only British studios, but also film producers throughout Europe. In the 1940s, De Wolf Music expanded into the US, where television was in its infancy and the company produced music for the first commercials. In 1955, the first advertisement in British television history, for Gibbs toothpaste, was broadcast in the UK with De Wolf music.[8]
Following De Wolf Music’s example, other production music companies have been set up to build their own music databases. To this day, the revenue model is based on the collection of licensing and synchronisation fees for access to the database on the one hand, and royalties from collecting societies for the use of the music on the other. Production music companies have the advantage of controlling not only the copyrights to the musical works, but also the neighbouring rights to the recordings, and can therefore offer licensing packages at a more favourable price.[9]
Not surprisingly, production music, which is stored in large databases, became an early application of AI in the music industry. The sound samples originally created by composers can be produced much more cheaply with AI, and the licensing fees or royalties do not have to be shared with the authors. MIDI databases are particularly suitable for AI-generated production music because the samples are available in a technically simple form. MIDI files are also ideal for composing and recording music. There are now numerous AI music creation tools that work with MIDI files, such as Google’s Magenta Studio or Open AI’s MuseNet, which will be discussed in detail in the next chapter.
Mastering has become another important application for AI in music production. When a music recording is available in a stereo mix version, mastering consists of balancing the individual sound elements so that the final song sounds perfect on all media formats, from mobile phones to high-end sound systems.[10] Until now, this has required a very good ear and expensive equipment to achieve a satisfactory end result. With AI, the mastering process can be largely automated. The mastering company LANDR, for example, offers an online AI mastering tool that is extremely easy to use. A high-resolution audio file in wav or aiff format of the already mixed song can be uploaded online to the platform. Three different mastering styles and volume levels can then be selected. After a few clicks, the mastering process can be started. At the end the song can be output as a wav, mp3 or HD wav file. The mastered recording can then be uploaded directly to streaming and social media platforms or shared with others. And all for just under EUR 20 a month (as of February 2024).
According to LANDR, Warner Music Group labels are among its clients. This is no coincidence, as in 2015 Warner invested US $6.2 million in the company founded by Pascal Pilon in Montréal, Canada.[11] LANDR, the company name is an acronym for Left AND Right, emerged from the AI start-up MixGenius, in which Pilon invested venture capital through its investment company YUL Ventures in 2014. From the outset, Pilon positioned LANDR as an online mastering platform for DIY artists who cannot afford expensive mastering studios.[12] A Billboard article estimates that there are between 500 and 800 mastering studios worldwide, charging several hundred US dollars per mastered recording, which can add up to several thousand dollars for an album.[13] Observing that only one per cent of music videos uploaded to YouTube were mastered, Pilon saw a gap in the market for LANDR by offering the service at a low threshold via the Internet. In addition to online mastering, since 2018 LANDR has also offered digital music distribution, which allows the finished mastered recordings to be uploaded directly to the main music streaming platforms. In the same year, LANDR also made its music sample database available to subscribers, allowing the company to enter the production music market.
In the meantime, a number of other AI mastering studios have sprung up on the Internet, such as multi-platinum music producer and Grammy winner Smith Carlson’s eMastered studio[14] and the open-source Songmastr project.[15] The business models are similar. Ready-mixed sound files can be uploaded to the platform online and an AI algorithm then takes care of the mastering, with monthly subscription models offered alongside free entry-level versions.
Mastering and production music were early areas of AI applications in the music industry, and it was only a matter of time before the core area of musical creativity, music creation, was also conquered by AI.
Endnotes
[1] A full tribute to Dave Smith’s achievements can be read in an obituary by Heise.de on his death on 31 May 2022: “MIDI-Entwickler und Sound-Pionier: Dave Smith ist tot”, June 3, 2022, accessed: 2024-02-02.
[2] The guardian of MIDI technology is the MIDI Association, on whose website you can also read about the MIDI 1.0 protocol.: MIDI Association, “MID 1.0”, n.d., accessed: 2024-02-02.
[3] The founder and then CEO of Roland Corporation, Ikutaro Kakehashi, received the 2013 Technology Grammy for this achievement, along with Dave Smith, who was honoured by Grammy Award winner Dave Stewart: Dave Stewart, “Technical GRAMMY Award: Ikutaro Kakehashi And Dave Smith”, www.grammy.com, December 3, 2014, accessed: 2024-02-02.
[4] Der Standard, “Wie künstliche Intelligenz heute schon bei der Musikproduktion hilft” by Stefan Mey, February 12, 2023, accessed: 2024-02-02.
[5] Vienna Symphonic Library, https://www.vsl.co.at/de, n.d., accessed: 2024-02-02.
[6] Der Spiegel, “Die Wiener Schnipsel-Musikanten”, December 28, 2003, accessed: 2024-02-02.
[7] The full history of the company can be found on the De Wolfe Music website.: “110 Years of De Wolfe Music”, n.d., accessed: 2024-02-02.
[8] YouTube, “UK’s First Television Advert – Gibbs SR Toothpaste”, September 29, 2011, accessed: 2024-02-02.
[9] See the section “Produktionsmusikverlage/Archivmusikverlage”, pp 154-158 in Christian Baierle, 2009, Der Musikverlag, Munich: Musikmarkt Verlag.
[10] The mastering company LANDR, which works for the Warner Music Group labels, offers a good introduction, including a video explaining how mastering works: “What is Mastering? Why Master Your Tracks Before Release?”, n.d., accessed: 2024-02-02.
[11] 2019 konnte LANDR in einer zweiten Investmentrunde weitere US $26 Millionen einsammeln. Siehe: Music Business Worldwide, “LANDR closes $26m Series B funding round”, 16. Juli 2019, Zugriff am 20.12.2023.
[12] Devenir entrepreneur, “How LANDR is revolutionizing postproduction in music. Pascal Pilon’s Story”, November 4, 2016, accessed: 2024-02-02.
[13] Billboard, “LANDR Brings Music Mastering to the Cloud, Helping Major Labels & Bedroom Acts Alike”, February 19, 2019, accessed: 2024-02-02.
[14] eMastered, “Master Your Track, Instantly”, n.d., accessed: 2024-02-02.
[15] Songmastr, ” https://www.songmastr.com/”, n.d., accessed: 2024-02-02.
Comments