Name: Brainstorm AI London 2024: How Deutsche Telekom Used AI to Finish Beethoven’s Unfinished Symphony
Uploaded: 2024-05-31
Duration: 10 min 50 s
Description: Dr. Matthias Röder, Chief Executive Officer, Karajan Institute; Co-founder and Managing Partner, The Mindshift; Director, Beethoven X – The AI Project

Dr. Matthias Röder, Chief Executive Officer, Karajan Institute; Co-founder and Managing Partner, The Mindshift; Director, Beethoven X – The AI Project

Brainstorm AI London 2024: How Deutsche Telekom Used AI to Finish Beethoven’s Unfinished Symphony

Photography by Joe Maher/Fortune Brainstorm AI

Conferences

Thank you, Jeremy. Hey, everyone. 200 years ago, a mildly grumpy, middle aged man with wild hair, stepped out of his apartment in Vienna briskly walked down the street, mumbling, humming and singing from time to time standing writing something in his notebook and continuing on his way. Yes, you guessed right. I'm talking about Ludwig Van Beethoven at the time, the most well known composer of music in Europe and almost completely deaf. At this point. Later at home, he would take out the melodies he found during his walk and copy them into a big sketchbook that was lying on his desk. This is where the real work started, where he started to work on the musical problems that the material posed to him. How should I develop this melody? How should I combine it? Which key should it be modulating to? What is the setting and the structure that is good for this type of material. This was the actual work and Beethoven had amassed many of such books and he would go back and forth. This was his creative Bible, the 10th symphony that he was working on at the time that didn't materialize Beethoven died two years later, having left the work unfinished. 195 years later, I got a phone call not by Beethoven, but Mi Michael Schulz from Deutsche Telekom asked me, Matthias, can we finish this 10th symphony of Beethoven with a I, I don't know, Michael. Uh we got to put a team together, we got to start working on it and we got to see how far we can get well do it. Two months later, I was pretty certain that it was possible to do something whether it was the big thing, I don't know. So Beethoven left 45 sketches for 1/10 symphony. Some of which were a little longer, others were shorter. A musicologist named Barry Cooper a couple of years ago used the longer ones to finish one potential version of the 10th symphony and that worked pretty well. But with the short material, you know, no human could ever possibly take that and turn it into something bigger. Why? Because you would naturally put your own ideas, your own style, your own way of composing into the material. I called Ahmed El Gamal from Rutgers University who is a computer scientist specialized on creating artworks with the help of A I and asked him if he wanted to be part of the project. He said yes, but I've never done anything in music. So I sat down with him, taught him everything I knew about the creative process of Beethoven and how composition works. And he said, let me think about it, called me back next week and said, Matthias, I thought about it. It's quite straightforward. Composing music is just like email, autocomplete. Who I, you can imagine that moment. I thought like, ok, well, maybe the wrong person and this is never going to work. But then he told me no, but look, every note is a function of all of the notes that came before. And it, it makes sense when I have a piece of music, the next note is related to what came before. We can actually try it out. Now together, I'll sing you a melody. I'll stop at some point and you the note that I left out. OK. Are you ready? OK. Ta ta, ta, ta, ta, ta ta. Yeah. You see every node is a function of all of the notes that came before. So we set out, we said, well, let's use this technology, the transformer architecture that Google invented in 2017 and uh create a G BT for music that's trained on late 18th century, early 19th century music and then fine tuned with the works of Beethoven. And then we use the sketches that Beethoven wrote down as prompts. And we let the machine continue these ideas. And then a group of musicologists combine the material, go through it, improve on it, give it back to the machine and this way iteratively create one possible version of 1/10 symphony at the time. Most people thought, oh, they are crazy. This was never going to work. Even some Google folks said that to the CEO of Deutsche Telekom. But we were pretty certain that at least, you know, we would create something that would pass as a potential musical work of that time. So I want to play you an example of that tense symphony which starts with one of these sketches that Beethoven had written down. It's basically 1 to 1 what he wrote down in his book. And at some point, the A I will take over and I invite you to play this game for with me where you raise your hand when you think this is the moment where the A I comes in, of course, I'm going to give you um you know, the solution afterwards. But this is one minute of music, close your eyes, don't get distracted by the people around you and just be bold, raise your hand whenever you think this is where Beethoven stops and A I begins was generated by this algorithm. Uh Granted it was then, you know, going through other algorithms as well. We not only produced an A I that could continue the sketches, we produced one that would find the suitable accompaniment, one that would stitch together pieces of A I generated music and then one final A I that would take the output of all of the other AIS and say this note will be played by that instrument. This note by this instrument, an orchestration engine. This is where we really reached the end of the line because that A I wasn't that successful. So we actually hired a human in our team to orchestrate the final version that you just heard being performed by a real orchestra with real humans. This collaboration started in 2019. As I mentioned in 2020 we were supposed to premiere it. Uh but COVID came along and so we had to postpone, we had one additional year to work on it. And in that time frame, I got a second phone call from Deutsche Telekom. Hey Matthias. Uh Robbie Williams has heard about your Beethoven A I and he thinks he would like to compose with Beethoven. Is that possible? I said, well, you know, what can I say? I don't say no if Robbie Williams wants to use an A I that we produce, so we'll find a way to make it happen. Of course, the question was, how could we do that? I flew to London. I explained it to Guy Chambers, the composer that worked very closely with him and Robbie's manager exactly what we were doing. And they said, well, we'll think about it and they called me back the next day. We want to do angels. Can you recompose angels in the style of Beethoven? And I said, I don't know, I really don't know if that's possible. Um The, the problem was, you see, we had this A I that was trained on 18th and 19th century music and that could very well deal with Beethoven's style. But how do we get Robbie Williams in there? This is when the composer on our team WV, I had this amazingly genius idea. He said, well, we'll take a famous melody of Beethoven and we combine it with bits and pieces from Robbie's Angels. And this combination is what we then use as a prompt. And so by prompting the A I with something that had already the main motific uh content of angels in it. It was capable of producing music that combined Robbie williams' Angels with the style of composing of Beethoven. There was one problem we thought, OK, well, our A I is pretty good at creating long stretches of music, but Guy Chambers told us no. Um you have uh four seconds for the intro and you have another 30 seconds for where the solo is and everything has to happen in there. So we would try out different types of uh combinations of music. And finally, we found one that worked very well. We combined uh the Moonlight Sonata with uh with some people are laughing. Yeah, uh with, with angels and I have another excerpt for you that uh is, is a little mash up of what we did with Robbie. You can listen to the whole thing on his newest album. Um And please let's listen to Angels with Beethoven A I. The point of this was really to collaborate and we started out to create this one A I that could, you know, do something very, very specific. And we ended up providing a tool set to one of the great artist of our time to collaborate with. And I think this is the, the move that we see in a lot of what's happening in generative A I and music right now, we see tools that become collaboration partners and that are really um instances of inspiration of transformation of working bodies for composers. And if you ask me what the future of all of this is, then I'd say all of the tools that right now are sort of text to finished song where you basically say, I want to have a track which is 15th century style with green sleeves and heavy metal combines and then you get finished track. This is, this is all nice and very good, but it will be superseded by tools that will be directly integrated into the software where we compose music where we have our inspirational A I tools. Now, I want to leave you with another excerpt this time from the Beethoven 10th symphony. It's actually a work of 25 minutes and I condensed it for you here to, to a couple of minutes for you to get some sense of how the piece sounds when you hear it performed by an orchestra, we can roll the tape. Um This was done a couple of years ago in Hamburg and the Philammon, this is the Beethoven orchestra conducted by Dirk Kaftan. And there's a little surprise there, an instrument that you wouldn't expect in a Beethoven symphony. So enjoy.