Bakul Patel, Senior Director, Global Digital Health Strategy and Regulatory, Google
Dr. Monique Smith, Director, Design and Innovation, RockHealth.org
Moderator: Chrissy Farr, Principal Investor, Health Tech Lead, OMERS Ventures; Co-chair, Fortune Brainstorm Health

Brainstorm Health 2023 - The Role of Regulators On Responsible Health A.I. 

Conferences

Bakul Patel, Google
Dr. Monique Smith, RockHealth.org
Chrissy Farr, Fortune Brainstorm Health

The Role of Regulators On Responsible Health A.I. 

Bakul Patel, Senior Director, Global Digital Health Strategy and Regulatory, Google
Dr. Monique Smith, Director, Design and Innovation, RockHealth.org
Chrissy Farr, Principal Investor, Health Tech Lead, OMERS Ventures; Co-chair, Fortune Brainstorm Health

I am so excited to get to moderate this particular panel because I have been on maternity leave the past few months and reading nonstop about generative A I um because I am a total weirdo. Um But now I get to talk about it with these two um who know, honestly, everything there is to know about this, this space um even as it rapidly evolves. So I'm gonna start by picking on you. Um And I would love for you just to give us like a quick 32nd intro to where we are at right now with uh regulating A I, what is FDA doing? What's AJ S doing just like a snapshot? Yeah. So before I left FDA, which is about a year ago, um we had put out an AI regulatory framework that sort of proposed a couple, couple of unique ideas about how do you allow algorithms to evolve and change over time without actually going back or back to FDA for approval. So those, those things are now, well, one of those things which is called a predetermined change control plan is now in law that FDA has in the toolkit now. So that's a good advancement that allows people to sort of iterate their models and, and, and not go to FDA like more frequently. So that's one thing, Nis put out a framework and then the White House put out a principle to document, I think when you start looking at where things are and I believe there's an R out from NTI A to ask like, how should we think about regulating? So those are, those are sort of where things are in the government today. HHS is obviously thinking about health and what should be done. And I think one of the key principles in, in health, like people are like gravitating towards is transparency, like explaining in layman's term to the users, what the algorithm does, how it got to what it got to and what its recommendation really means. I think that's one of the fundamental principals how to get there, what to do. I think still work in progress. Um Yeah, that's definitely helpful and a lot of confusion I think still around which agencies specifically should even be regulating this space that um we will definitely be be watching because it's, it's unfolding as we speak. OK. So um I recently put out a poll on my account which I, I do sometimes um question was in the US, we will regulate A I not nearly enough thoughtfully or with over caution. Can anybody here guess what the answer was? 40 to nice people. I'm gonna go with a caution. You know what, when I see patients in the emergency department, there's this tendency to think that anything that we're doing and decisions that we make don't have enough, um, rigor around them. But I think actually our government tries to do well with that and it's actually gonna go over the bar that as providers, we probably will have less latitude, um, to kind of play with clinical decision, support tools and everything else that we do. We do right now. Um Yeah, I would, I would agree, but I would also hope there's with enough caution, people regulate because there is the, the thinking around how do you sort of push technology to be useful to the benefits part and then the risk part. But if you can, if you were to like, transpose exactly what we do today, it's a lot of the risk sort of angle, right. So I think so, I think there's going to be a balance and it's gonna be a struggle for regulators to think about that. Um So most people disagreed with the both of you. Um, 78% of people thought that the exact opposite actually, of what you thought, um, not nearly enough was the, was the response of touch. So, um, any thoughts on, on why, um, the public or at least, um, however many, uh, folks answered my poll, uh, think this way, I, I think, um, I started to enter I guess, I think if I talk to my friends and family, I think people are really, really cautious about, like, using technology for the sake of using technology. And also you're reading news articles that sort of talks about how, you know, we can ask a question to general and gives you something that you should not be getting or you would be afraid that somebody else, like your, your child should not be getting. So I think that I think there's a lot of that fear built into it. Um As, as, as I imagine it is an emergency doctor, I always think first do no harm, right? And I think a lot of folks are worried about harm and that's the idea of under regulation. The elephant in the room is that there's this idea that what we're doing in medicine might be codified to the extent that there's less choice, there's less autonomy. You know, a lot of our health care system and the beauty of it is built on the fact that we can do what's best for the patient in front of us. And when you layer algorithms on top of that, that might not be on the best data. Um You certainly have challenges. Um Quick reminder, everyone that if you have a question, um look for one of the, the lovely ladies with a mic and ask at any time. Um So, you know, along these lines, you're both doctors, um how do you? Well, are you? You're not, I'm not a doctor. Oh my God. OK. So you are a doctor. Um So I'll direct this question to you, Monique. But um there is definitely some evidence now that these A I algorithms can sometimes outperform physicians, especially, you know, not, not all doctors are the top 1% of the profession. So how do you think the physician community copes with that knowledge? Especially as these algorithms get more and more small over time. So the question is outperform for whom? Right. So each of these algorithms are built on a specific training data set and that data set is a set of people like people in this room and the people who came here today in this room are different than the people who came to another location today in this room. And so the ability to outperform, you have to ask that again on an individual basis, you also have to ask that over time, right? So as our world changes, algorithms don't mix in, that's why as we mentioned, the FDA is thinking about how do we actually allow algorithms to evolve. So I don't worry about my job stability one because I resuscitate patients and no algorithm is going to do chest compressions for you. But also because there's this element of human interaction that matters, right? I don't think anyone's talking about replacing physicians and the role of that knowledge, they're actually just talking about how can we have let you work top of license. How do we make this more efficient, less, painful, less burnt out? And there's a lot to be figured out in that mix, particularly because medicine in and of itself is already a little bit flawed. The data that we have to work on on the baseline already has biases that are embedded within it. You mentioned bias, which I think is, is a key point here. Um Humans are obviously biased. So are the algorithms that we create and we're seeing more and more evidence of this now. Um So as we kind of move these technologies into health care and it may be slower than other sectors, quite frankly, but it will happen. How do we ensure that we don't have all of that kind of bias baked in? So I'll give you an example during early pandemic, our emergency department partnered with our Bioinformatics Corp and we were really trying to solve the problem of, we don't have enough personal protective equipment, we don't have enough doctors, but we need to screen patients. We need to know who's sick and who's not sick. We want to use very well known facial recognition technology to do some very simple vital signs. Do you have a fever? Are you turning blue because you can't breathe and you're short of breath and use that to differentiate where people should go and that's, you know, something that looks at your face and decides, I mean, we had a great group that really put together technology and they said, hey, we're an academic center. This is as far as we can take it based on our training data set. And this is a training data set based on the population in Atlanta, which is reasonably diverse but still didn't have a wide variety of melanin in the facial recognition software which causes problems. When you're thinking about thinking about blueness around the mouth, even for the element of sensing temperatures for facial recognition, you know, much to my chagrin about, you know, four months later, I had a well known and established technology vendor come with me to a company that they had acquired and this company, they were doing facial recognition for vital signs. And I was like, this is great, you know, we're doing really great things in telemedicine. Maybe this can be incorporated. I'm so glad you've solved this problem. Tell me about your training data set. Absolute crickets. I never heard back from them. I had to ask a few months later when they were telling me about another technology and said, you know what that training data set has zero diversity in it. That's the problem, right? So it's a level of accountability that spends all the way from local decision makers, all the way to the FDA. It's an ecosystem, right? I as a purchaser have to make the decision that is this appropriate for my population and not just appropriate at the point of purchase, but also it, six weeks from now, from year, six years from now it's six decades from now. We can't just have a one and done policy and then it stands up to the way you think about, you know, how you actually build for it and whether or not the FDA said it's ok. I think it's a very broad problem that has many, many actors that have to take part to make sure they're all safe. Yeah, I absolutely agree. So, bias, we know huge risk. Another one which we already mentioned is that we might actually not do enough to regulate A I, how do you think all of this plays out as we see other countries globally? Like China and Russia move ahead and start to do quite amazing things. But, you know, potentially things that require more caution, more thoughtfulness, more regulation even. Um Do you think that the States will look at examples like that and think we need to keep up um without kind of doing the work to keep people safe? I, I just have some, some positive hopes that, you know, countries like ours would probably be more responsible and not just followed suit with all the other countries that do. I think there's a lot of, a lot of harm that can be done and for long periods of time. And I think that's the, that's the key risk here. And if you're not responsible in terms of how do you develop technology, how do you sort of make sure that you have the bias thought process in the, in the, in by design and, and not, and not only that, but also like thinking about how do you make sure there are the right filters in place as you can see, like Geneva, like the chap and bars of the world are, are now not giving you answers that they used to give before. Right? And, and that's sort of the evolution that needs to happen. And I I know there is going to be this is kind of in other scenarios that we have seen in the past, in the future that people will race down this path. But I think people who do it properly will probably be succeed in succeeding in the long term. So I feel like the industry, the technologies, the users and states would have to be responsible in terms of how do you sort of get to the next level of using and realizing this potential for A I to be actually be useful for us in health, especially when it's more important. Um You know, along these lines, I was reading about kind of all these technologies like I mentioned on my maternity leave and then my mom actually fell and broke her wrist and we ended up in several different urgent cares that same evening. And of course, given imaging on a disk drive that we then had to take between the two urgent cares and then get a PC P referral and then take the disc drive imaging over to the surgeon. And the whole thing just reminded me of what it is actually like to be a patient in health care today. And that we still live in this era of fax machines, which is a very long winded way of asking, you know, we've got A I, on the one hand, on the other hand, we still use fax machines. Um How do we square both of those things? And um can, can I believe sincerely that um in the next kind of, you know, 1 to 3 years, we true health care will truly be disrupted by this technology when there is clearly so much invested in the status quo. So there's a company that I'm super interested in um that we've, you know, deeply partnered with at Emory in this space to enable us to have, enable your mom to have a much better experience, right? There's this element of meeting patients exactly where they need to be met, that there's a I that can help with that triage. There's an element of being able to use the technology that allows us to communicate in a way that makes sense based on where you are to be able to have a result on the other side that you don't have to carry a CD 1000 places that information travels where it needs to be. And maybe your mom has someone come, if it's her foot that's broken and she can't walk, she have to do that X ray as opposed to having to get up and go to an urgent care. Right. You've got to think about all those different things. There's an emergency doctor. You know, we often are like catching the rest of the health care system. Something we firmly believe in is how do we work upstream of that? And technology enables that right? The ability to catch people before they have like last result, I'm going to wait in the emergency department forever. And I think there's a beauty of A I in terms of the way we triage folks, there's the beauty of like the way technology platforms now engage and appropriately diagnose and care for patients that are appropriate for the platform. And I think there's a lot that's going on in that space and I'm excited to see more happen there. I could you guys remember and probably in the audience when he used to fax things you would just call and and said, did you get my fax? And this is exactly the scenario, right. I'm sure that the image went over the electronic system. But in order to make sure that the patient does carry it with them, the CD was given. So I'm hopeful there's a positive lining into all this that down the road. We'll probably not think about this. Like carrying a CD is a burden, but carrying a CD is like more of that phone call. Like, did you get my facts kind of thing? Although I did, I did call Urgent Kara and he said, well, you just bring back the disk drive and then you take a photo on your phone of us use it like it was just like insane. Like what or you could just use Box or Dropbox. Like haven't those existed for like a decade? The best part is when you fill up in the emergency department by ambulance with the disc and you've got the entire care team saying where's the disc for like 20 minutes and like, oh forget it. Let's just send her an X ray. Yeah, I mean, yeah, anyway, um but I think you're hitting up on something which is near and dear to my heart is in, right? I think a I can only help when the data is there in front of you. It can operate, it can sort of operate on that data on the disk, right? But it does not do the transfer of the disk to point A to point B I think when we, when we start talking about interoperable solutions, I think they're getting there. I think this journey has been there for a very long time. You and I chatted what at least 10 years ago about this? And we are we are still working on this, right? I think c is actually pushing this agenda on inter ability further and we need to continue to push that and it needs, it will probably take all of us the health care systems, the technology vendors, etcetera, who's in, who's in for the challenge? And are these people in the room that might be? So A I, we talk often about the clinical applications, we've already talked about that on stage. Um What about all the back office on sexy stuff? Um It seems like it's probably gonna revolutionize claims in some fashion. Look at all the waste that we see in the health care system. Could A I do anything about that? Um How exciting, how near term do you think these applications are? I I, one of the key, one of the most exciting things that I'm really thrilled about is education. You know, if you think about generative A I and you know, Google's Met product that is, that is basically A I literally like scored 80% in their A S M L E exam sort of test. And that to me is phenomenal. I mean, can you get a machine to like pass the exam that's hard and, and to me that just opens up doors for many other things down the road, right? I mean, you can, can you just imagine a place where you're not supplementing medical training with generative A I I think to me, those are even fundamental changes that you can, you can see and I'm excited about this. I think there's a really interesting space here of what the A I does, right? When you talk about adjudicating administrative items, whether it's a pre authorization or figuring out, you know, should this person be able to, you know, get physical therapy, you have to have this element again of the interaction, right? Because there are personalized pieces might not fit across the algorithm and you're only going to discover that if you're looking for it. So when I think about a I think of it as the assist, right? As like the assistant that no one in medicine has, like you pull together the pieces of information really quickly for you to be able to make a game time decision as opposed to having the backlog, it reduces that down so that people are still making decisions that are relevant to other people and that technology is the assist. Um So I love to kind of leave it on the note of giving people ideas and something optimistic to think through. So if there are any potential founders in this room, what would be an idea that you'd love to have them work on in say 20 seconds or less in this space, to further kind of health care in a in a more positive direction, hopefully get rid of the fax machine too. Uh You know, one of the things that I find most painful. Um, is that when I'm seeing a patient real time really locked into them, there's something that I need for their plan. Right. Like I need them to be able to get this medicine and also have this diet plan and for that to be neatly packaged and appropriate for where they live and their zip code and all those things to be figured out and the ability to offload that thought to make sure that my well thought out plan that gets lost in execution is actually delivered. Yeah, that sounds great. I'd buy that. I would actually go back to your original question. I was like, what all the back end applications that create inefficiencies in health care? Like getting to a doctor getting that care. I think once the, once the patient is in front of the doctor, you get phenomenal care, but getting there and sort of following through, I think that's where all the work is necessary and needed. And I think if we can use technology to sort of remove some of that friction will be great. Amazing. Well, we, we managed to end on time um without the A I so really appreciate you both being here and sharing your thoughts, your perspectives. Um Thank you. Thank you.