Name: Making Stuff Up: Improvising LLMs and Spontaneous Robots
Uploaded: 2025-05-06
Duration: 14 min 49 s

Improvisation is a skill needed by creatives, and improv can delight theater audiences. Is improvisation the ultimate test for artificial intelligence? Since 2016, Piotr Mirowski has deployed AI as a creativity support tool for live improvised theater performances with a cast of humans and robots, engaging audiences at Edinburgh Fringe about generative AI, storytelling, and ethical co-creation with machines.

Making Stuff Up: Improvising LLMs and Spontaneous Robots

Conferences

Had some great conversations during the roundtables. How many people went? Raise your hand. Yeah, all right. They were both really good. I popped up my head into both of them. Now we're switching gears and getting a little bit more spontaneous. Our next session explores a fascinating idea. Is improvisation the ability to be funny and respond creativity cre creatively in the moment? Something that's hard for humans to get right to, in addition to speaking apparently, the ultimate test for AI to show us what that looks like in action, please join me in welcoming research scientist at Google DeepMind and director of the Improbotics Theater, Pyt Mirovski. What you gonna do? Hello, London. So, hello. Let me leave that thing and unplug the little robot and maybe say a few words about myself. I do have a day job, um, and as a matter of fact, I was studying computer science and improv for a very long time, almost like 20 years, each of those. And I did my PhD at NYU with a professor called Jan Lukan back in the time when he still had time for students and back in the time where AI was a dirty word, we had to avoid it putting in publications in order to get accepted. And in my day job I've been working on various AI research, for instance, learning to navigate or learning to predict the weather, essentially trying to make up things that happen in the future using machine learning. So that's essentially my day job, but in the evening I've been going to theater school, to drama school in London, and also doing improv comedy as well. I really love theater and I love improvisation because improvisation is one way to really engage and connect with the audience. Um, some researchers, some academics have defined improvisation as real time dynamic problem solving. Other academics like Gla Spolin and Keith Johnstone have talked about spontaneity, being a 6 year old again and doing the most obvious thing, which is surprisingly difficult once you become an adult. Improvisation is all about reacting in split seconds to whatever is thrown at you and saying the most obvious thing that comes to your mind. And if you want to see some improv in London, you can go to see our friends from showstoppers who improvise a two-hour musical, uh, every month, fully from scratch. So, this is improv. Uh, one of the reasons why improv is so valued in business environments is that it's all about collaboration and it's about learning to listen to your stage partner. So something happened in my mind about 9 years ago. I was working um at Bing. Um, at Microsoft on the 2nd most popular search engine Bing, and I was working on query autocompletion trying to basically predict what people are going to tell based on the context, and I was doing improv and then I realized that there was a strange connection being delivered two things machine learning, maximizing the likelihood, and being obvious, saying the most obvious thing on your mind, so that. How the idea of improvising with a robot came to be. So I built an AI. I trained a language model on the dialogue from 100,000 films from open subtitles. That was 8 years ago, and I brought this on stage and I called the AI and the robot Alex Artificial Language experiment. And the way this worked, I had a microphone. I would speak to the robot using speech recognition, and it would use a language motor to reply. Nothing surprising for anyone who has used an AI system in the last few years. And of course the AI would take typically, I don't know, 5 seconds to come up with an answer, which seemed an eternity on stage. So I had to actually make up for that. I was improvising alone, but soon after I met someone fantastic, Corey Matheson, a PhD student in robotics in Alberta in Edmonton, Alberta, an improviser at the Rapid Fire Theater who had the same idea. So I wrote to him a 3000 word email. He flew to London the day after. We met. We became friends. It was the beginning of a beautiful friendship, and we started working together. From Corey's perspective, improvisation. is the art of making your stage partner look good on stage. It's not about you, it's about them. He would have this shtick where he would invite someone from the audience who never did improv before, and he would make him look good, make her look good for 15 minutes on stage. So we created this company Inrobotics human Machine, and we started doing improv shows remotely via Zoom or Skype, and we had this show in London and Portland, Oregon, 8 time zones apart of that's what we thought until we realized the day before that actually, you know, daylight setting time is not always coordinated. So that created a bit of a logistical problem. And also, um, the previous show of a run, we didn't have time to check the microphone levels. The robot literally crashed and burned on stage. Everything was going wrong. And the reviewer came. And they wrote 2 stars chaotic, never worked with robots to a PhD student in robotics and to someone who was working on robots in his daytime, daytime. Did we listen? Absolutely not. So we created this theater company where we actually replaced the robot by humans, getting lines the earpiece from the AI and essentially becoming the embodiment, the cyborg, improvising with all the humans on stage, and that's how it looks in practice. We have an AI system that generates lots of candidates, lots of suggestions. There is a curator, a human in the loop who is choosing the best lines, the least offensive, or maybe the ones that make most sense, and we want to show that to the audience because we want to make sure that the audience understands that AI is not about an oracle giving an answer, but it's about a stochastic parrot generating answers all the time, and it's up to you to choose the best one. And one of the improvisers wear a headset, wears a headset, and they deliver the lines with physicality, emotional expression, etc. We have been evolving the formats. We have created 4 different companies in 4 different countries. We have been publishing academic research on the subject in human computer interaction, and we have replaced the earpieces by augmented variety glasses, which we might know later. And we have also been working on telepresence. How do we improvise during COVID where everyone is miserable at home. Everyone gets a green screen, and one of us is live video DJing us to place two improvisers on a horse or in a car or in a bathtub and see what happens next. Uh, so this is a demo of the system during COVID. We have also worked on live translation because that's the origin of large language models, translation, and we have done multilingual shows where one person may speak in Polish, another one in French, and both get understood because they have live translation. So does the audience. We have incorporated image generation, exploring PowerPoint generation, uh, you know, improvised TED Talks, PowerPoint karaoke made by AI. Or even incorporated deep fakes in a very consensual way to basically inform the audience about the potential to actually get a live deep fake in real time and to explore maybe a more fun aspect of it while informing about the potential dangers of all the technologies. This is something I've been doing on my side, but I've also incorporated it in my day job working on social technical aspects of AI. For instance, 3 years ago we worked on scriptwriting and playwriting using large language models before Chad GPT was released, and we partnered with playwrights and screenwriters to explore the potentials of the tools and its limitations. or we have also been talking to comedians and to artists about all the possible uses of AI that empower the artists instead of replacing them, and that's all that I have to say, but I believe that Andrew has a few more questions. So welcome, welcome back, Andrew. OK. All right, so all of this is fantastic. I just want to say, uh, I'm, I'm a drummer, OK, so, uh, you know, we, we start with the fundamentals of jazz when you, when you learn how to play drums and so improvisation is a core component of that, and I love that. Um, I wanna ask you, so how much of AI is art versus science? And I'm thinking more about your day job than I am about this. How much of AI is art instead of science? I think there's a lot of An artistic approach to AI that is very useful in understanding the um affordances. Um, I've been very lucky to, to meet many artists working with AI in 2016. who have been exploring the glitch aesthetic and everything that doesn't work with AI to actually make work of art out of that and and they literally build upon the work on explaining AI, explaining convolutional networks. So there's a very interesting partnership that happens that kind of replicates what poets were doing with drugs in the 19th century. Am I making a comparison? Sort of, yes. Um, you have done a lot of different work in, in building your career, your day job, um, in, in different areas, navigation you've worked on, you've worked on, um, climate modeling. So what have you learned from that that you're putting in this work? Yes, um, directly climate models, there is the stochastic aspect which is vital to climate models. That's essentially how we get forecasts of the potential outcomes. You change initial conditions and you're keeping the same dynamical model, you get radically different outcomes. And so when we look at both extremely scary curves of the rise of temperature over the last 8 years, it might rise by only 1.5 degrees or by as much as 80 degrees. And this, uh, this is an illustration of a stochastic aspect of AI model. You my. With some crackling from microphone, um, that, um, I am trying to convey when coming on stage and in improv this is the same principle that we listen to minute details in somebody's conversation in. Happens on stage. What happens in the audience, and we get influenced by that and we use it in our creative practice to get inspired and to radically change the outcome of a scene. So huge parallels here. Fantastic. Well, we've only got 3 minutes left, so why don't we actually try this thing? What do you think, folks? Yes, let's improv. All right, so what I have here is a little bit of a system that will help Andrew improvise. So Andrew, have you done theater or improv? What's that? Have you done improv and theater? I have done theater in high school, but it's been a long time. Yes, have you been terrified by having to find to remember your lines? Oh, who, who isn't? Who isn't? Whose line is it anyway? Yes, yes. Well, what if I tell them what I'm seeing? Yes, what you're seeing right now on this, on these augmented related glasses is light generated by AI. And from now on, Andrew, if you can hold the phone, of course, you can only, and you can only say AI generated lines. So let's see how this is going and if. All right, so I am typing a prompt, activating the system. Don't look fashionable. All right, rebooting the system. This is a live demo. Yeah. It's coming up, um. Yes, it is coming up in those very long moments of silence on stage. So what I see are a bunch of different options in my uh in my glasses. Um, it's, it had generated um different responses and I can choose which which to deliver to in our conversation. All right. So if I say something, it's saying hello, yes. Hi, AI. Nice to be here. Well, Hello, hello, Andrew. From now on you are only controlled by AI. You can only say lines that you see in front of you. We are here at the convention center and uh we, let's imagine that we are in, um, at a bar. Do you want some drink? Can I help you? Sounds ambitious. All right. I knew you would think that, Andrew. Well, here's your Negroni. Thank you. Right, Andrew, we've been good friends. We've been living in New York, um, 15 years ago and enjoying the city. Is there something that you really remember and that inspired you when you were in New York? Oh, the rats, the rats, of course. Well, and that is the end of the show. Thank you very much. Thank you, Andrew, of course. All right. Let me pull this down and get out of the stage. You've been fantastic. Thank you. Thank you, everyone.