Name: CEO Initiative 2023: The AI-Powered Productivity Boom
Uploaded: 2023-10-04
Duration: 15 min 10 s
Description: Erik Brynjolfsson, Jerry Yang and Akiko Yamazaki Professor and Senior Fellow, Stanford Institute for Human-Centered AI (HAI)

Erik Brynjolfsson, Jerry Yang and Akiko Yamazaki Professor and Senior Fellow, Stanford Institute for Human-Centered AI (HAI)

CEO Initiative 2023: The AI-Powered Productivity Boom

Conferences

Hello, hello. Hello, welcome back. So we want to continue talking about A I because what else have we been talking about all day? So right now, uh let's dive further into the impact of A I and LL MS or large language models on productivity and what it means for workers. Eric Brisson. Great Icelandic name is a professor and senior fellow at the Stanford Institute for Human Centered A I where he directs the digital economy lab, Eric led a study on A I and worker productivity and the results show that A I tools like chatbots helped boost productivity by 14% among those organizations queried. And he recently said something pretty profound. Actually, he said it this morning regarding the implementation of A I and how it relates to humanity. If we play our cards, right, the next decade could be some of the best 10 years ever in human history. And I think after the last few years, we could use a few really good years. So let's hope he's right. Um We were fortunate to have Eric participate as one of the discussion leaders at the town hall this afternoon and he joins us again to share his research on bridging the gap between human capabilities and this emerging technology. And after Eric's presentation, I'll be back to join him for a couple of minutes to take your questions. So get them ready. Now, let's please welcome Eric to the stage. Good afternoon. It's great to be here. This is an amazing time in history and this is an amazing group to be interacting with the full quote from Phil. By the way, was if we play our cards, right, the next 10 years could be the best ones ever in history. But if we mess up, it could be the worst. So I just want to get both sides of that down there. I've seen some people quote one half or the other half. I'm actually mostly on the optimistic side. I think it will turn out well because we can, we can, we can do this and I'm putting my money where my mouth is. I am actually, let's see if I can click there. I've actually made a bet with Bob Gordon who's a famous economist, written some great books, but a little bit of a pessimist when it comes to the future of productivity. So we, he's a friend of mine, we have dinner from time to time, but we disagree about the future and we decided to make a long bet so you can go there and weigh in. This was actually made a couple of years ago and at the time, the bureau, the Congressional Budget Office was predicting 1.4% productivity growth. I think it's so I was willing to say it's going to be more than that 1.8%. He said it was going to be less than that. I've updated since then because of generative ai I think actually I was lowballing it back then. I'm up to like 3% productivity growth now more than doubling what the congressional budget office was. And it's because of the power of these technologies. And you heard a little bit about the studies we've been doing. This is not just hypothetical, this is us going in and looking at particular companies that are implementing the technology and already seeing some big productivity gains comes from some core technological improvements that go back about a decade when Jeff Hinton and other colleagues introduced deep learning techniques. And so you can see here this blue line of the rapid improvement in machine vision, recognizing images on image net. But there are dozens of other things that it's just if anything, the pace is accelerating where they are matching or surpassing human level performance in activity, active activity, active activity. And if you're a CEO or entrepreneur or manager and you have two ways of solving a job, one of which is now better and cheaper than the other, you're going to start changing around or you should be changing around the way you do your work. We're seeing companies do that one that I studied in quite some depth that Phil briefly mentioned was a study of call centers where a company called Cresta introduced a tool, a large language based tool to help the operators. It didn't try to replace them. I think we're all, we know it's not quite there yet where it just replaces them. It gets, can be kind of annoying talking to a bot but it, it, it they did an A B test, there was a Stanford team that started the company, so they're into these A B tests. And so you could see that the people who got access to the technology were about 14% more productive within about two or three months. It happened very quickly. But actually, it was more than that the least experienced workers, the newest workers and the least skilled workers had about a 35% productivity improvement, the most experienced workers, it was almost zero. So it's quite a gradient there. In terms of who benefited the most. You can see it, I can show you lots of fancy regression tables and statistics, but you can just see it in the raw data. It's always good when your results are clear enough, you can just see them without having to do any, any econometrics or anything. This is an example of a set of technologies that are affecting lots of different applications, any place where you've got a bunch of digital data for inputs and digital data for outputs, you can do a mapping now from inputs to outputs. It used to be that we humans had to figure out the relationships and those of you who have coded, you know, that you have to be very precise when you do coding and know exactly what you're doing. There's a new kind of way of doing software. Some people call it software 2.0 where we show the machines, the inputs and outputs and the machines figure out the relationships, machine learning. This is so much more powerful and it opens up a lot of capabilities in areas like, you know, the tacit knowledge really that we all have about whether it's how to ride a bike or tie your shoe or recognize your mother, that it would be very hard to codify in a set of, you know, Python rules. But you can have examples and the machine will figure that out. And this is opening up a whole set of things in companies that previously could not be done a lot of organizational knowledge. It's one of the reasons why that call center works so well. The machine looked at literally millions of transcripts and you could see some of them led to happy customer outcomes and some of them not so happy customer outcomes. And it identified what are the common patterns? How do you solve? How do you answer this particular question? That question? What kind of tone of voice should you be using? What words work better and not. And by looking at these millions of data points, it captured the tacit knowledge of the best operators and transmitted it to the less skilled operators. And very quickly got that productivity improvement. By the way, it wasn't just a productivity improvement. We also saw an improvement in customer satisfaction and quality. If you looked at the sentiment analysis, the number of happy words versus angry words, the customers were using a lot more happy words after they were working with the system. So they enjoyed it better and even the workers seem to be better off. There was significantly less turnover, they stayed in the job a lot longer. So it was a win across all those dimensions. There are a lot of other studies that find comparable results now in software coding in email and in customer service applications. So we're beginning to see a lot of evidence piling up that this is a very powerful tool. As I said at lunch, the most general of all general purpose technologies, especially these foundation models or generative AI. And here's some data showing how they can perform on lots of different tests. For instance, over here, you could look at say the uniform bar exam, GPT 3.5 scored about a 10% GPT four scored about a 90% better than most humans on that. Now, that's not the same as actually doing legal work, but it's a good indicator of some of the rapid improvements. And as I'm sure, you know, GP T five is being trained and would be coming out. And not that long after that, probably GP T six, not to mention Gemini and uh Anthropic Clad and all the other tools that are coming out. So which applications are most suitable for machine learning? Well, this is a question I've been working on for quite a while. I wrote a paper in the journal Science a little while back and we looked at which kinds of tasks we'd be most effective. What we did was we took every occupation in the economy. About 950 of them, like ceo professor, radiologist, primary school teacher, truck driver, and each of them, you can break down into a set of tasks, say 20 to 3 tasks for most jobs. And then we scored every one of those tasks, 18,000 of them as to whether or not machine learning could help you with that task. Once we did that, we rolled it back up to the company level. So you can develop an actionable road map for generative ai earlier at lunch, I asked how many of you had a road map and most of you did. Not hardly any of you did, but since I moved from MIT to Stanford, I decided to start a company. Apparently that's the norm when you come to the bay area. And so we do have a company called Wikileaks that takes this research from science and nature and makes it accessible. And I'll just show you a couple of screenshots of how it works. Basically, here's an analysis of one relatively small company that has about a $35 million opportunity. Almost more than half the workforce has some activities that could be at least 100% improvement in productivity by using generative AI. Here are the top five marketing manager, software developer, pr specialist of this company. If you double click on marketing manager, you can see some of the tasks that Gen Ai could help with them, initiate market reach research studies. You could click on that and you an indication that you probably can't read it too well, but it gives pointers to specific tools, particular activities. So this is a first start. I wouldn't just use it without any human intervention. But if you're a CEO and your board is asking you, you know, what is our game plan for generative AI? This is a tool that helps with that, that first draft of that and identifies what the low hanging fruit is. What the biggest opportunities are that you may want to go after as you do that. So even your boards themselves can get generative ai help. It's every task that I decided to ask GPT before I came up here, whether or not what, what questions boards should be asking their ceos. So it came up with a list of questions for all of them as well. So you can share that with your board if you'd like to learn more about any of this. It's on my website at Stanford, you can download all the papers that I made reference to. Um You can also go to the company work Helix for examples of the software or my own personal website. Has a lot of my own research as well. But I'm hoping that you all are leaning in and embracing this opportunity with generative ai I think it's the real deal. I think it's probably the most important technology of my lifetime. And sometimes when I'm feeling grandiose, maybe the most important technology of all of history. Thanks very much. So I tried to leave about a few minutes for questions or Phil. I don't know if you have questions. I do. I do. Thank you, Eric. Um So let me ask you a self-interest question. Um Say you have somebody who is a little bit older um still about 25 years away from retirement is starting to get into that phase of life where new things are a hassle. This is all in theory, of course, does such a person benefit uh more or less from A I and all its promise than a young, a newbie coming up in the professional world. Well, what we're seeing sadly is that for some people, I guess is that that young people are benefiting disproportionately in the studies we did when we did those A P tests, we looked at the demographics and there does seem to be a noticeable measurable gradient where younger people are getting bigger benefits. I mentioned briefly the less experienced workers at the call center people found similar results for coders and for other kinds of writers that said, you know, hope is not lost. I think there's this is a technology that doesn't require a lot of like new training. You don't have to learn a new programming language, you communicate it with it. In English also, as the technology helps with one set of tasks, there's a whole lot of other tasks that now are not being helped by the technology that may be more important. And Julie Sweet mentioned those earlier things like planning, quantitative analysis, a lot of interpersonal skills that other people might have a disproportionate advantage with as well. Are there any questions from the? OK, let's go over here. Oh, hi, Stefan Ch with a light. Um I love the scorecard and the obvious ones are software engineers and so on. But you mentioned the word CEO and scorecard. So any lessons learned for the role, this is one of my biggest surprises because you know, I've been working with the technology for a long time and I used to have this very set under standing. If you look at my book, Second Machine Age, I kind of had a hierarchy of tasks and you know, less skilled, more routine work tended to be the first to be automated. I was surprised with gender I that a lot of very professional workers and even ceos creative workers, you know, about the writers' strike, these tools can be incredibly helpful for them. A friend of mine is a CEO and he was talking about a board meeting he had to prepare for and he was just like having trouble coming up with what do you call them KPIS, you know, for, for, for the next quarter of what he wanted to, you know, to do. And he had his team help him a little bit. And then he turned to chat GPT and he described his company, you can put in some context and it came up with his great list and he said like eight of the 10 that it came up with were like spot on. And I think it's a little spooky actually that, that the kinds of jobs are being affected. I want to stress though that in none of these examples, would I just turn the keys over to one of these machines? It's always a matter of keeping the human in the loop and knowing which of those eight of the 10-K PS are the ones that you probably want to keep and the others that may be a little screwy. I think everyone's heard about hallucination and some of the other problems that these tools can create uh in our uh last minute. Unless there's a question, the audience, um you know, not everybody is all that excited about tech and, and white collar workers. So let's say because there's a couple of retailers in the room, you're somebody who works in a store and really likes that and doesn't want to be bogged down by technology. How does A I help somebody who's working on a sales floor or who's stocking shelves? Yeah. Well, so one of the things that I'm excited about this technology is that as I mentioned, it doesn't require seem to require the same level of training and specialized skills to some of the earlier ways. So I think we heard that chat GPT went from 0 to 100 million users in a couple of months and that's because most of us can just type in English and, and, you know, pretty much everybody has some of those basic skills. Now you can get better at it and better at prompt engineering. Um The thing that's lagging a little bit is on the physical work side. So like robotics is not nearly as advancing as rapidly as gene and L so it's very uneven uh movements here. Um But I do think that it's something that, you know, according to our analysis, about 80% of jobs, 80% of occupations will have some significant effect. And retail employees actually are, are do are in that 80%. Not so much on the stocking the shelves side, although people are working that, but on the other parts of the thing. Uh Yeah, exactly. Um Well, thank you for joining us and absolutely my pleasure. Thank you.