Name: Brainstorm AI 2024: The Everything Agent
Uploaded: 2024-12-10
Duration: 14 min 3 s
Description: Rohit Prasad, Senior Vice President and Head Scientist, Artificial General Intelligence, Amazon Interviewer: Jeremy Kahn, Fortune

Rohit Prasad, Senior Vice President and Head Scientist, Artificial General Intelligence, Amazon Interviewer: Jeremy Kahn, Fortune

Brainstorm AI 2024: The Everything Agent

Conferences

Ravi. Thanks for being here with us. It's great to be here. I'm missing all the bad weather on the East coast. So I'm excited to be here. Fantastic. So big news out of out of re invent last week was Nova, as Andrew said, um you know, you finally have a family of large language models out there. I guess a lot of people are wondering though what took you so long? I thought it was very fast. You know, uh I think uh first, very excited for what we did with Amazon Nova. It's our next generation of foundation models that we introduced to the world last week. Uh Let me take you a bit behind the scenes of uh uh where I sit, I have the words best to thousands of generative A I applications being built by our internal teams. And we are noticing three very important things. One was choice and importance of selection. So if you look at Amazon Nova Foundation Model family, it's not just one model, we actually introduced six models. Five of them are generally available as of last week and they start with one of the models which is a text only model, which is called micro, which by which you mean that you can take any text, input and output text. And then there are three models of varying increases in size and intelligence that are taking multimodal input as an image, text and video and generating text. And then you have two additional models that are using taking text as input or images and generating images and videos. Those are called Nova canvas and novare. And the reason we did that because we found internally, people are using very different models and they had very different needs for their applications. So that goes down to the selection of what you need to do to build. And then secondly, as people move from experimentation to large scale loads, we were finding that cost is becoming increasingly important. So these models are 75% less expensive than their counterpart best performing models on bedrock, which is our service where you have many different model providers available with their foundation models. So clearly cost was big deal and so good to see that the team did massive invention to bring costs down. And lastly, as you know, with these foundation models, it's not just about the model's capability or cost, but if delivered a very good user experience and after time for the user experience factors like latency matters. So all these models on the ones which take uh image video and text as input and output text. They are lightning fast. Uh So those were the things that we worked on and I'm very excited about the future and we are looking to the next generation of Nova as well now. Great. I think with all these models, one of the big questions people have is is data, you know, where did you, where did the data come from? That that was used to train Nova? Yeah, we don't disclose the specifics of the data, but it's a factor of many factors. One is sources, of course the publicly available data. And second, we have a lot of proprietary data and we also license data from many providers. Yeah, that's interesting licensing seems like it's becoming more of a norm in the building of these models now whereas it may not have been in the past. Yeah, I think you want to, to trust with the publishers and everything is super important and we want to make sure that you are using the data the right way and all that is super critical. And I think we, we are very excited about some of the partnerships we have made on the data site as well. So now Nova is out, but a lot of people are still waiting for an update to what is maybe Amazon's sort of best known product when it comes to A I, which is the Alexa Digital Assistant. I know you guys said there would be a new Alexa coming. But you know, when is it actually going to arrive? Yeah. As you know, Jeremy Alexa is near and dear to me, I joined Amazon more than a decade back to build Alexa and it's been a very fulfilling experience. Uh We are right now in the midst of reengineering Alexa with the latest of state of the art foundation models so that it can be the world's best personal assistant. We are going to be true to that. No T A that we set ahead and a lot of people coughed at us when we were looking at that, but that's the reality. You can have your world's best personal assistant available to everyone. And what I see in our labs is super exciting. So all I can say today is stay tuned. Uh Jimmy there, no, no timeline, not, no timeline right now. Just stay tuned. OK. Well, we'll stay tuned. Um And I want to come to the audience for questions in a minute. So please think of your, your questions. Um One of the things with Nova, I mean, you built that those, that family of models in house, but you also have this close partnership with anthropic. You made investments in them, you have a strategic relationship with them and I know they've been helping you on, on some things. Um You know, why in some ways, why build your own own family of models if you have that relationship, philanthropic and they have very capable models of their own. Yeah. First, we love Anthropic. The partnership has been great and I'll go back and reinforce what, where I started, what, what we saw was happening internally. Customers want choice. We have seen that many times in our internal Amazon stores. If you look at how many different selection we have there and same in a, there's not just one solution, there are multiple ones, same. It's turning out to be true in the foundation models. Our customers do want choice and the partnership with Anthropic and what we did with Amazon Nova is just accentuating the the choices available for our customers. Um So the the relationship Amazon has with Anthropic is somewhat similar to a relationship. You know, Microsoft has with open A I, there's been some reports in that relationship between Microsoft and Open eye that there's some tension. What's the relationship like with Anthropic? And is there any, are there tensions in that relationship where they want more compute out of Amazon than you guys can give them or you're not getting what you feel like you need from them in terms of innovation. On the model side, I think as I said, we love the partnership. I think nothing more to state here that the partnerships are going great. And you saw some of that at reinvented last week, right? Um There's a lot of discussion recently about progress in large language models hitting a wall. What's your view on that? I think that's a very debatable question in terms of what wall is coming. I think we have to be very careful about what we have seen in A I, every time, every time we come close to a wall, there's a new invention, I think we'll keep finding that if you see what is happening with pre training, which is where you build a token predictor and post training, where you're giving the model a lot of different tasks. Now it's shifting to inference. We are spending a lot more time as an inference. And at the same time as I mentioned with Noah, we are making inference cheaper, which means I think the scaling is curve is changing from different stages of training to inference. And we'll continue to see that I'm very optimistic that in A I every sometimes every decade, sometimes every five years, every three years, you continue to ship the curve up on performance. And I think that's going to be true again in this coming years. Are you working on a reasoning model that would use your test time inference the same way as the one model from, from open A I does? Again, I won't go to the specifics of it like in terms of the many different ways, what we are really building is for A I to be useful in real world, practical applications. And a lot of that requires reasoning of many different kinds, not just mathematical reasoning, but more utilitarian reasoning as well where you're trying to do tasks in the real world. And that's where if you look at A I agents, you have to get to the point where they're not just solving a coding problem or math problem, but they are calling the Uber for you or making a reservation for you, which are not really high on intellectual proficiency but require a lot of world knowledge to do it. Right? So that's where the feel is evolving. And is that where Amazon is heading towards agents and making agents available? Yeah, I think if you saw like what we talked about with our Rufus, which is a shopping assistant with Alexa, which I think of as a super agent agent because it does many different things. That's where the word is going to go, right? I want to get a question from the audience. Please raise your hand if you have a question, if not, I've got lots of questions for. So keep going. There's a woman down here, wait till we can get a, a mic to you. If we can get a mic handler, just a woman in red, just down here. Bye bye. And if you could please state your name when you stand up and where you're from, that would be great. Thank you. Hi, how are you? My name is Lorenza and um I work currently for the Press uh at the Economist and uh yeah, I would love to. Well, we know that Jeff Bezos is back working uh at uh Amazon full time. And uh yeah, I would love to, to know whether you've had uh you know, work with him in, in certain projects. And what is his vision? Has he mentioned? Any specific vision uh that you can share with us on his mind? I would say you should ask that to Jeff. But yes, he's very involved, as he said, uh I love the partnership with Jeff. I've worked with him early days of Alexa too and he's still very much connected with A I elements of uh where I spend a lot of time and I get to see him. So very fortunate with that partnership and mentorship. OK. Um I want to ask a question about A I's energy demands. We had an interesting session this morning at a breakfast on this topic. How is Amazon looking at, at the huge energy demands that particularly these foundation models seem to require? Yeah, I personally on my front as well. I want a I to be sustainable and I love that at Amazon, especially with Aws. What we are doing is we are looking at the end to end life cycle of sustainable A I. And it starts with, I think our chips. One of the main things where we have our mlx and chips like trainum is essentially to drive better price performance so that you are able to build these foundation models in a much, much better and sustainable fashion. And specifically, if you look at our announcement last week on Trainum three, which is a third generation of trainum that is 40% more price performance than Trainum two. So that kind of uh uh uh step function improvement in price performance is where I think sustainability will be key at the same time. On in parallel in our data centers, we are trying to use as much renewable energy as possible. We have partnership with a company where we are absorbing the carbon that's getting emitted. All of that is key for making computing more sustainable. So and I think as a whole, the industry needs to look at it even more carefully and its model optimization because some people worry that OK, you make the models more efficient, then people just use more of it. It's called the GS paradox, you know, and actually the overall consumption is higher. Is that a concern? Are you seeing that also as you drive down model price, are you seeing usage actually increase over this actually validates what we did with our own models? If you look at the model family, we launched with Amazon Nova, we are taking a lot doing a lot of work to the capabilities of the large model into the smaller font factors or smaller sizes to a suite of techniques. And this is where if you look at where you're going Jeremy with it, like inference is becoming a building block that every application in the world will use. This is why if you thought about it as a per request inference or a per token request, each of that has to go down by a factor of factors. Right. I mean, and of late, we have all we managed to get an order of magnitude faster with our influence and more efficient, but we need to continue doing that work and I'm very optimistic that we'll do so because our customers want it. Ok. Right. This is where instance as a building block is such a key element for us to think about as you're building applications. Thanks. There's a sort of war for talent going on out there. How is Amazon sort of competing in that war for talent? How are you able to kind of retain the engineering talent you need? I think one thing I cherish about Amazon that it has uh world's most customer obsessed scientist and ML talent in the world. Uh because here we are looking at what's the best way to solve and build real practical A I solutions. So I'm very proud of the team that we already have. Uh We just today announced our uh San Francisco A I hub, which is very crucial for this. Like we have had a present in Sunnyvale for a long time, but we said we need to be in the city. So this is why we are in the city now. That's why I was also here today. And so that I think it's great times for A I scientists and A I practitioner. But the good news for all of you in this room is that the bar to build with A I has suddenly reduced. You don't need a phd in machine learning or mathematics to build with A I. This is the way where you now can just prompt these systems or run out a lot of experiments where you don't need to be the A I expert. And frankly, I believe that more and more work will be now at the application layer as well where you will see that again. A phd in machine learning is not required if you could tell the audience a lot of whom are trying to figure out how to get from A I in their own businesses. You know, if there is just one piece of advice, you could give them on how to do that, what would it be? Well, first two things, one, we're gonna run out of time. So first, I think we are seeing a phase where generative A I is moving from experimentation to real production use cases, which is very positive. And what I'm seeing is that if you look at our development environment, we now can save 4500 hours of SD software development engineer time because you don't need to do a Java migration with a lot of people, you can simply do it with the five people in a room. That's incredible. So my last minute advice for you on that one would be focus on getting your data right, your decisions and your daily workflow captured. Because there's the magic in A I is that once you have the data and the decisions, it can learn if you don't have the data and the decisions it's not gonna work. So focus on getting to get your data right. I'm afraid we're out of time, but Rohit, thank you so much for being with us.