Name: Brainstorm AI 2024: A New Tool to Master The Mix
Uploaded: 2024-12-10
Duration: 9 min 14 s
Description: Jessica Powell, Co-founder and CEO, AudioShake

Jessica Powell, Co-founder and CEO, AudioShake 

Brainstorm AI 2024: A New Tool to Master The Mix

Conferences

Hi, thanks all for coming. Uh I'm Jessica Powell, the CEO and co-founder of Audios Shake and I am waiting for the slides to start. Oh, there we go. All right. Um So today I'm gonna talk to you about sound, but before we do that, we're gonna do a sound experiment. So, in just a moment, I'm gonna ask you all to clap and to cheer as if you were at a sports stadium or a concert and I'm gonna talk and can we get some music? Great. Ok. So louder. Can you hear me? Can you? Great. I really just wanted to applause. Um Great. So now let's see if I had asked you what the person 2 ft from, you had said during that time, you would have no idea or if it had been a concert and you would actually wanted to hear what was happening. Like maybe more you wanted to hear the music and the crowd, you'd be stuck and that's generally how we experience sound in the real world, right? Think about being in a noisy bar or being uh you're on the street and you're recording a video and all of a sudden a, an ambulance comes by or a police siren or since we're in Silicon Valley, let's say you're developing an application that requires voice input. Well, if your user is in a call center on a noisy street corner or simply as toddlers running around in the background, good luck getting the input that you need. But what if that didn't have to be the case? What if we could actually extract what we needed to hear? So let's listen to something that's very, very noisy like this. And now let's isolate the voice and lift off of the space of returning to the playstation. So this is what we do at audios shake. We make sound work better for everyone. We split sound into its different components in order to make audio editable accessible and useful. We're a core audio infrastructure company a bit like a Dolby. Now, this is actually pretty hard to do and let me show you why this is an image of an audio recording and you probably don't know. Is this a whole bunch of people speaking in a crowded room? Is this a music recording? Is it a bunch of sound effects in a movie? And all I'm gonna tell you is that the Y axis is the frequency and the X axis is time and that's about all you're gonna get. Now, what I'm gonna do is I'm actually gonna color this in for you. So you can see what's actually going on. So this is a song recording. Um And you can see we've got some vocals, some drums, some bass collection of other instruments. And again, this is not actually how sound looks. You have no way of actually knowing what's in these different parts. Now, recording engineers have a hack for this developed back in the sixties around the time of the Beatles, which is multi track recording. You send the, uh you know, the vocalist into the studio to lay down their track, then the drummer, then the bass player and you now have all these granular elements that you can do granular audio editing with. You can remix the track, you could put those different sound objects in different perceptual fields. The bass here, the drum here to make a surround sound mix, you could correct imbalances in the audio all because you have those parts. But real World sound doesn't come to us like this, right? It's, it's more like what we just experienced in our little experiment. And so we have to turn to A I and deep learning to help us fix this problem and be able to get at these individual parts. So let me show you how we do that. And in fact, before I move slides, if you look here, uh if you look at the yellow and the orange, the drums and the bass, this is a really great example. So you can see that these are not in one tidy band, they're all over the place, right? They're overlapping with all the other frequencies as well. And remember, we don't see any color in this image. So all we can see we have is essentially sound patterns. So what we do is we train on hundreds of thousands of these parts which are called stems and let's say drums in order to learn what these sound patterns are. So let's say we can, then we can then take an audio recording that we've never seen before and recognize that there are drums in there. This, this matches to the sound pattern of a drum. So we figured out what something is, but now we need to disentangle it from everything else. So let's say we want to do that with the base. We want to isolate the base from the recording, think of an image editor. So what we're gonna do is we're gonna leave everything that we've all the pixels that we've identified as corresponding to the base. We're gonna leave those there and everything that's not the base. We're gonna black out and essentially, we're gonna silence that audio and that's gonna leave us with the base. Now, this might sound, as my parents said to me like a very weird hobby. Um And, and, but actually, it's very, very practical and audio tech works across a ton of entertainment and speech work flows today. So in film and TV, we can isolate dialogue, music and effects to take old film and open it up to localization and other languages or remove on set uh wind and noise in music. We can separate the different instrument stems for remixing or immersive sound in sports. We can boost what a player is saying on the field or remove music from the environment so that teams and leagues can avoid millions of dollars in copyright signs in transcription and captioning. Our tech is used to isolate the voice before it goes through transcription so that you get much higher accuracy. And in generative A I were used by some of the world's largest foundational models to learn human conversation. So for example, disentangling overlapping speakers or extracting the uh huhs and ya that make human conversation human. So let me show you a quick demo of what this all looks like in practice and the s what there a lot of dream come true. Innit. It really did. Did you uh did you, we talked about love and coming into a game like this? It's the magnitude of this game. Massive game. Did, was there a a hangover at all? The result? And the that coming to our lane came to earth, you couldn't be more mistaken but here to help you by deactivating them. The Lord. Yeah, don't have to trust an goodness said Noah, I need at an airport. So I think I came to the airport like 30 minutes before boarding maximum. I wanna walk right back here. Enjoy the lounge. That's why I call the lounge. In fact, there's going to be so much more that you're going to be able to power with sound separation in the future and in the near future. So we've already heard a little bit today about music in music. You're going to be able to remix and reimagine everything and all content including including c is going to be able to be made immersive both through sound separation and things that are happening on the, on the vision side as well. Uh We're also going to be able to do all kinds of measurement and analysis thanks to sound separation. So imagine being able to measure the degradation or the health of an ecological environment around the presence or the absence and the change over time in different kinds of animal sounds and in generative A I, we're going to be able to the foundation models in fact, are going to go deeper and deeper into audio and to the point where they're going to be able to understand audio, the way we do. So all the complexity, the way any human here could have said that was crowd noise and there was a dog barking over there. Computers will be able to do that too and they're going to be able to learn to talk like us as well, which is going to change so many workflows from the factory floor through to kiosks in the amusement park. Finally, because of advances, not just in A I but also chips and hardware, we're going to be able to solve the noisy bar problem. Now, one big technical challenge in all this is speed. So in sound separation, we are taking large audio files, running them through large deep learning models and then output large high resolution audio files. And that's why for the past year, audio shake has been working on building faster and faster models. And today we're launching here at Fortune uh the audio shake voice SDK, which will isolate voice in real time for noisy backgrounds, uh streaming capable on edge devices. And I think while we were all talking and doing our experiment in the back, they were running exactly that or we're about to find out if they did. Uh Can you play the full mix of what that room sounded like when everyone was cheering? All right. And what did you isolate louder? Can you hear me? Can you hear me? Right? I should have done something like you all want a free car or something like that. Um So if you go out today into the noisy hall or in a street corner later or at a bar tonight, watching a DJ remix of track, know that all these experiences are going to be enhanced by sound separation and audio shake is gonna help make it possible. Thank you.