Name: Brainstorm Tech 2024: A Roadmap For Converting Technology To Value
Uploaded: 2024-07-15
Duration: 14 min 40 s
Description: Presenter: Erik Brynjolfsson, Jerry Yang and Akiko Yamazaki Professor; Senior Fellow and Director, Digital Economy Lab, Stanford Institute for Human-Centered AI

Presenter: Erik Brynjolfsson, Jerry Yang and Akiko Yamazaki Professor; Senior Fellow and Director, Digital Economy Lab, Stanford Institute for Human-Centered AI

Brainstorm Tech 2024: A Roadmap For Converting Technology To Value

Conferences

Good afternoon. I'm delighted to be here to share some of the research we've been doing at the Stanford Digital Economy Lab and also at my company work Helix, which is trying to take some of that research and put it into practice. There is a lot of excitement about A I you mentioned maybe as big as the industrial revolution. I think that's true. But there's also a lot of hype out there. There's a lot of misdirection. There are so many examples of A I being applied to situations that just aren't going to work and as a result value is not being created. So one of the things I want to do in the next 15 minutes or so is share with you some of the research we've been doing to provide a more data driven approach to understanding where the real opportunities are. And I think they're enormous and also what sorts of things to avoid. So let me just start with some technological benchmarks every year. My team and I put out a report called the A I Index. It's about a 520 page report filled with lots of facts and charts like this one. And as you can see, a lot of technical benchmarks are getting much, much better over time, few things in the world improve as rapidly as A I and the pace of change is improving. And I put there a horizontal line for what's happening human level performance on these different benchmarks. And as you can see in many cases, the A I systems are matching or exceeding human level performance as an economist, that's a very important benchmark. Because when you cross that threshold, like when water crashes, the boiling temperature, it's a phase change and the economy fundamentally changes when you have a new way of doing something that does it better as well or better than humans do, but it's not enough to have amazing technology. We're not translating into business value. Mary Daly talked about some of the productivity gains we hope to see, but that's not going to happen unless we transform work, simply buying technology almost never turns it into business value. And one of the most common questions I get is are we just going to see the end of work, end of jobs as this technology replaces people? That's also not what's going to happen. At least not anytime soon. Jeff Hinton, one of the greatest minds in A I pioneered deep learning techniques and one of the reasons all those lines are growing so rapidly. A couple of slides earlier is because of Jeff's work. He's also has a side gig as an economist sometimes and predicts that we won't need radiologists anymore. Actually, he predicted that back in 2016 team. Well, actually we have more radiologists now, almost twice as many job postings. Now, why is that? There's two main reasons for that one is that paradoxically sometimes when you get better at something, when it becomes more productive and more efficient, instead of having fewer workers, you have more. Now, why would that be the case? Well, think about jet engines, they made airline pilots vastly more productive, more passenger miles, lots of metrics went up. Do we have fewer pilots or more pilots than before jet engines were invented? Obviously, most of you flew in here and jets are a big part of the reason of that. And with many other allergies, you have what's called Jon's paradox that as the technology gets better, it actually increases the demand even faster, not everywhere. But in some places, you have to understand where the second reason is that while A I can do many tasks, it can't do everything you still need humans in the loop for almost every category of work. In fact, the right way to think about it is to break it down to what we call the task based analysis where the task is the fundamental unit of an organization. Think of it as like the DNA based pairs for organisms tasks are that fundamental unit of analysis for organizations and for over a decade, I've been analyzing companies using the task based analysis where you take a job, which is a bundle of many little tasks. And then you analyze each of those tasks individually. You see whether or not A I gen A I robotics or some other tool can help them. So lifting a box, gen A I is not so helpful, but as we'll see later, robotics can help with that. Writing a memo, actually, that's one where gen A I can help. And then there are other tasks where no technology currently can help a whole lot. Once you've done that analysis of the task based analysis, task based level, you can aggregate them back up to the job weighted by wages or even to the entire organization. Now you're getting somewhere. Now you have a roadmap to understand where is this technology going to help? What specific ways and instead of having anecdotes and stories from vendors and things that you read in places, you're going to have a data driven approach to understanding where the opportunities lie. So we did this in a series of academic articles. I wrote one in science. About eight years ago, my colleague Daniel Rock had one come out in science two weeks ago where we used this approach to understand how the economy is going to change. Now A I is not A G which means it can't do everything but it can do certain things incredibly well. And let's take the case of radiology that Jeff Hinton brought up in our initial analysis, we broke the radiologists into 27 distinct tasks. And one of them, machine learning could do extremely well interpreting images, reading the medical images better than humans in many cases. And there's academic papers showing this. So that's great. But there are many other tasks that humans. You want to keep humans in the loop. You don't really turn over the keys to an A I system to start sedating people or do some of the other things that are on this list. And this is exactly what we found in every occupation we looked at, we found some tasks where A I could help other tasks where you really want to keep humans in the loop. Not once did we say A I run the whole table and just replace an entire occupation. So that is what you're going to be seeing in your organizations. You're going to see opportunities to use A I to help, but mostly as a compliment as a partner for humans to do it better. When, when Daniel Rock used to be a student of mine. Now he's a professor at Wharton did this for gen A I. He and his team found on the horizontal axis here is wages, vertical is the implication you can see a sort of upward sloping. What that means is that actually unlike some of the earlier technologies, gen A I dispose affect some of the higher wage occupations. So a lot of managers, salespeople, doctors, lawyers for the first time, they're going to really have their jobs affected, changed, transformed. And that actually is good news. It's where a lot of the productivity gains need to happen. And it also will help with income inequality. Let me give you a concrete example. So you understand a little bit about how this plays out. We did a bunch of case studies of A I being rolled out in different situations. One of them was in a call center customer service and this company Kresta did not try to replace call center operators. It would be most of it get kind of annoyed when you talk to a robot. Instead they had humans talking to the customers. But the A I helped them by suggesting possible answers and the human could use that answer or not use that answer. We had almost the perfect natural experiment to do causal estimates. Half the people got access to the technology. Others did not. We looked at over 5 million conversations, 5000 center agents and we were able to very cleanly and quickly see what a difference it made. The red line. There is improvements in performance of the people who had access to the technology. And you can see within just four or five months, they went from about two resolutions per hour to over three per hour. The folks who did not get access to the technology didn't improve nearly as fast and never got to the same level. So this is a very clean causal estimate of the effects of large language models in a particular application. What's more? We tracked a dozen other Kpis, we looked at customer satisfaction, we looked at customer sentiment, basically the ratio of happy words to angry words in those transcripts, we looked at employee turnover. All these metrics also went up by comparable amounts. So stockholders were happy year, the productivity was up. Customers were happier, more customer satisfaction and higher sentiment and even employees, this was not an electronic sweatshop. Employee turnover went down. The employees seemed to be happier working with the system as well. So it was a situation where you could actually benefit all the groups. It was not a zero sum implication of this. And interestingly as I hinted earlier, the less skilled workers actually were the ones who benefited the most. And so this closed income inequality a little bit and it was a big opportunity to do that better. Furthermore, we found that it did not deskill the workers the way we were worried about. There were at all systems. Sometimes the system went down and that's not great use for the customer. But it turned out to be a great another great natural experiment for us researchers. When the system went down, we continued to monitor the performance of those workers. And to our surprise, the workers who had been working with the system continued to outperform the ones who hadn't, they had learned, they had internalized some of those answers. They didn't do it as well as when they had access to it, but they had captured some of that knowledge. So it turns out to be a very good way to capture tacit knowledge and translate it over into people who hadn't otherwise gotten that. And this is the first time we really have a technology where you don't have to write down every single rule for what you want your workers to do. Instead, the LLM or the gen A solution will capture some of that knowledge and transfer it to other workers. This opens up a whole new set of opportunities that previously didn't exist. Now, I mentioned that you want to keep humans in the loop and this graph shows one of the reasons why you want to do that machine learning. Gen A I is great when you have lots of data, but you don't always have lots of data. There are some tasks that come up over and over those to the left of the chart there. Like how do I change my password? How do I log on? You get those questions over and over and you get to learn what's the best way to explain that answer? But there are other solutions or other questions at the far right of the tail that we only see one or two times or a small number of times in the data set. Well, ja I doesn't have a chance to learn how to solve those, but we humans actually are pretty good at extemporizing at dealing with exceptions. And so there's a natural division of labor. You have the machines do the types of projects that and the types of questions where you have lots of data, they can learn better and better solutions, but exceptions are better done by humans. We're seeing this play out right now with self driving cars where they work for 90 95 90 9.9% of the task. But there's these exceptions that humans still need to intervene in and eventually they'll get there. But in most of most of the projects we looked at, there's a natural division between humans doing the exceptions and machines doing the more frequently done things. Now the example I gave was in call centers, but it was striking to us how that same pattern showed up in all these other applications in coding, writing management diagnosis. In each case, we saw often double digit gains in productivity and coding is actually triple digit gains. In some cases, 100% or more productivity, we saw less skilled workers benefiting more and we saw the performance happening relatively quickly. So while there's a lot of hype out there, there are definitely some situations where gen A I in particular can very quickly lead to productivity gains. And eventually, as Mary was hinting that will aggregate up to the whole economy and we'll see better outcomes in the economy more broadly. Now, I'd like to give you guys a little bit of a road map so you can get to work right away and put some of these in place. Part of my mission is to shorten that time between technology and productivity between A I and business value and to do that, you have to change the way work is being done. And so work he likes the company I started is very much geared towards shortening that time. And I should mention, I think over here, our CEO James Milan was nice enough to come join us here. So feel free to talk to me or him later. But basically what we do is we do this task based analysis we put into software and this is something you can do, you can do it by hand, but it's a lot easier to do it with organization, with, with organizational software because we look at about 200,000 individual tasks and then we roll them up. Every company has a different fingerprint. And so this particular company, you can see software engineers are the biggest opportunity that's actually pretty common but not always others who may be call center management sales. And you see all five of those green dots, there are places that are pretty ripe for applying the solutions. You can double click and get in deeper about what are the tasks specifically that are involved in that like monitoring system or providing advice on project costs? And look at exactly how much of acceleration might be possible. How many hours per week people are doing those, what the dollars spent on it is and where the biggest gains are likely to be in this way. Instead of having just anecdotes to try to decide where to prioritize you get a road map of what's more important to focus on. And this was all done. I should say these initial analyses we do with external data. So things like linkedin burning glass, other data sources that we license, we already kind of know roughly how many software engineers there are in your company. As you can imagine that stuff is posted on the open web. And so if we license it, we can, we can see that, but we can also do a deep dive, which is an internal scan which hooks up to the HR is systems and that allows you to get a more specific understanding of what's going on. And I encourage you to do these kinds of analyses on your own to understand what are the specific tasks that are being done in your organization? And which of those are you likely to be able to have some sort of a, a gen a solution or of A I or robotics solution for? So what should you do tomorrow morning. Well, the first thing you want to develop a plan instead of just picking and choosing a few cases based on the things that your, your fellow executives are pitching at you or vendors are pitching at you do it in a systematic way, use the task based analysis to understand where the opportunities are. And then secondly, you want to start tracking progress. You want to keep track of those different KPIS, like I did customer satisfaction, customer sentiment time handled employee turnover. The set of KPIS is going to be different for every project. But if you do that, you're going to be able to turn these amazing technologies into productivity and avoid some of the pitfalls and some of the hype that you'll also you otherwise might fall into. So if you want to learn more about this, a lot of it's on my website at the Stanford Digital Economy Lab. We have all those research papers that I mentioned there that you can read and get an understanding of the task based approach and more generally how A I is transforming work. I also have some mind my personal website and also the work hes website goes into more depth. So, thanks a lot. I'm looking forward to talking to all of you later this afternoon. I'll be around until dinner time and be happy to chat with you about how you're doing, how you're doing in your gen a transformation and your journey. Thanks very much.