Will AI Replace Statisticians? The Future of Statistics in Pharma

Chris Harbron

PoshStats Statistical Consultancy

Chris is a now consultant after having a industrial career of over 30 years as a statistician working throughout the drug discovery and development pipeline for multiple different companies. Chris has published widely and is a regular speaker at international conferences. As well as a long background in Clinical Statistics, over recent years Chris has spent time considering how more modern Data Science, Machine Learning and Artificial Intelligence techniques and technology can work in partnership with Clinical Statistics to derive the greatest benefit.

---

Transcript

00:00
You are listening to the Effective Statistician Podcast. The weekly podcast with Alexander Schacht and Benjamin Piske designed to help you reach your potential, great science and serve patients while having a great work-life balance.

00:21
In addition to our premium courses on the Effective Statistician Academy, we also have lots of free resources for you across all kind of different topics within that Academy. Head over to theeffectivestatistician.com and find the Academy and much more for you to become an effective statistician.

00:49
I’m producing this podcast in association with PSI, a community dedicated to leading and promoting the use of statistics within the healthcare industry for the benefit of patients. Join PSI today to further develop your statistical capabilities with access to the ever-growing video-on-demand content library, free registration to all PSI webinars, and much, more. Head over to the PSI website at PSIweb.org.

01:18
to learn more about PSI activities and become a PSI member today.

01:30
Welcome to another episode of the Effective Statistician. Today I’m super happy because of two reasons. First, I talked to a very well-known statistician, Chris, and second, we talked about a very, very interesting topic, AI. And the topic of today is around a paper Chris wrote, what the future role of statisticians will be in an AI world.

02:00
And do we actually need statisticians in the future? Will we all become redundant by solutions? So before we dive into the discussion, maybe you can introduce yourself a little bit. Yes, certainly. Thanks for the invitation to be here, Alexander. Yeah. So I’m Chris Harbron. So nowadays I’m an independent statistical consultant. Before that, I’ve been in the pharmaceutical industry for many years, working for a variety of roles and a variety of companies.

02:30
Most recently within the methodology group at Roche. I guess within the AI world, my backstory of this probably around about 20 years ago was one of the cycles around the omics technologies, which people started thinking about big data and how that could be used and techniques like machine learning was starting to be thought about in that situation. And I got involved with that and as that developed into thinking around ideas around personalized healthcare.

02:57
Yeah, got involved with those types of things as well as biomarkers more generally. And then of course, over the past few years, there’s been this explosion of interest in AI. So again, it’s natural extension that to get involved and be involved in that type of activity. So maybe I could start with some kind of provocative thing, a protocol and statistical analysis plan, the programs, the report, maybe parts of the manuscripts.

03:27
and abstracts. I can’t all do that with AI. If I would be kind of a little bit worse physician, yeah, couldn’t, or maybe just a medical writer or an assistant or maybe even some kind of AI agent, yeah, could set the AI solution to everything around a clinical study. Well, potentially could in some senses.

03:55
So the that the technologies today, particularly when we look at agentic AI, which is the most recent development, which allows for specialization. So effectively we can think of an agent within an AI spec as being a large language model, that’s actually GPT, which has been specifically trained on one area of expertise, whether it be remote-contacting modeling or biomarkers or whatever. And if we were to do that, we could properly generate all of the documents that you’ve mentioned there. And…

04:23
Initially, they may look quite plausible. Yeah, they may look quite good. But then if we started to read them, I think we’d start to get a little bit worried about things, in all my case. I think we’ve all had the experience when we’re using the large language model where suddenly it traps out something with huge confidence, which is clearly not true, what we call a hallucination. And so we cannot be sure of that confidence of that. I think there’s also another subtle layer below that.

04:51
is that actually, if we think back in the development of all of these kinds of documents, in any team I’ve ever been in, it’s never been a straightforward linear process. There’s always a huge amount of circulation goes around. And I might write the statistics section based on what I understand the protocol to be about. It goes to their physician and they’ve got a completely different interpretation of what the study was around.

05:16
And it’s only by going to that level of detail that we’re actually able to come to this resolution. What do we actually want the study to be doing? What do we want to be achieving in this study? If we are just outsourced that core stats or a large language model or an AI system, then we’re going to lose that kind of discussion. That understanding is just going to be wherever the AI assumes we want it to do, which is probably not going to be correct and certainly won’t have agreement for everybody. So in a sense, these documents, starting with a protocol,

05:45
is not just a document, it is a tool to help facilitate discussions, so to say. Yes, yeah. And I think that the whole process of that, yeah, and especially the way it forces a lot of rigor and real precision about what’s going to be called to that is really critical in designing a good study.

06:09
And I think the adoption of the S demands framework is a great example of that. It doesn’t offer any wiggle room or misunderstandings because we’re being really precise what we’re going to do within that particular study. think another point is also any innovation will be very, very difficult to assume from an AI partner, isn’t it? Yes, I think it would be. Yes. Yeah. Naturally what an AI model does is it just reflects the data it’s been trained on. That’s what they’re very good at doing.

06:38
So yeah, their starting position for any protocol will be all the protocols would be written previously. Now, in some ways that’s not different. Maybe we would write it. If you asked me to write the protocol in the new area, the first thing I’d go and do was dig out the previous one and take parts of it. But yeah, you’re absolutely right. Without a lot of additional input or work, which probably has to come from the human, you’re not going to get an innovation or some different type of design to what’s gone beforehand because they’re just trying to replicate their training data.

07:08
Another point that I have seen in designing clinical trials, you always need to make compromises. You can’t have everything. You don’t have all the time in the world. You don’t have all the money in the world. You don’t have all the patients in the world. there’s always real life conditions that you need to have. And under these conditions, of course, you need to find some suitable balances.

07:38
AI work with that. if I, for example, say to AI, I want to have a study that is not bigger than so many patients and doesn’t run longer than three years, let’s say, and doesn’t cost more than 10 million whatsoever. Yeah. I mean, so again, potentially you could do that if you could specify your limitations about things. I’ve got a budget of 500 million. I’ve got this amount of time.

08:06
I want to maximize the, yeah, but the objective of this study is about minimizing the time to market. Yeah, this is what I could need for a PTS value and so on and so on. It could probably come up with some compromise, which would meet those criteria. Now there’s two challenges there. One is really being explicitly about to state those criteria, which again, is typically something you do lots of iterative discussions about things normally. And the other is.

08:36
Will it come to the trade off the balance that we would want to have? It may be that it’s used that things, but it actually completely cuts out all of the biomarker assessments because they cost lots of money and they’re not relevant. Yeah, I did see them as being relevant. The biomarker scientists on the other hand is going to get very annoyed about that. And yeah, there’s a lot of value which can be achieved from that. So the balance, which it will achieve, it will try and do something sensible because yeah, as everything else.

09:04
It doesn’t necessarily mean it’s what we would want to be achieving as a optimal compromise. I think that is also coming back to the discussion. In team meeting, you can then come up with all kinds of different solutions. Especially through these discussions, you can better clarify what actually the goal is that we want to reach here. Because very often that is not super clear. Another problem that AI can’t

09:33
can’t solve. And also, I think the other point is looking into the bigger picture. For example, we might think about, oh, we’ve just focused on the phase one study and all the discussions about the phase one study. But could we actually optimize things if we also include aspects of the phase one B study into it? Or what kind of implications would that have for the phase two study?

10:01
Can we optimize the whole thing a little bit differently so that we get better through the phase two study and then maybe safe in the phase two study? There’s so many different aspects that I think it’s really, really difficult to just get out of AI.

10:18
Yeah, and I think there’s also something about just the experience which anybody working in drug development has accumulated over time from seeing things, experiencing things, maybe not going quite as they wanted and so on, which is again very hard to capture in any way and put into things. But if you’re in a meeting and you’re discussing things and you’ve got someone who’s got that experience, it really shines through and they can quickly identify where things could be going on in the future and other things. And I think this all speaks to something which…

10:46
I think you always found really interesting that I don’t see the future as being completely AI world, but AI will be part of it. And the real interesting challenge is going to be how we get AI in humans to work better. I’ve always thought it was a bit unfortunate actually, AI’s got the name AI with intelligence in it. yeah, intelligence I think is something humans have, but it’s a different type of intelligence wanting a better word to what we get from an AI.

11:13
And in some ways it’d be better if it different word because people automatically try and think of it as being the same type of things. But I think if we then thought of AI as being something which is a complementary capability and how we work together at complementary capability instead of just being competition, which is the most intelligent, that may give us other ways of the best way of being working together. I kind of think of AI very often as a kind of librarian that has access to a huge

11:42
kind of library of all kinds of different things and can quickly pull out all kinds of different answers from this library. The problem however is your answers will very much depend on the quality of your questions. I think that’s another kind of area where kind of you need to ask the right questions in order to do this. And then of course understand whether what you get back is sensible or some complete

12:11
BS. And we know that there’s a lot of BS actually out there. I just recently stepped over a post that talked about kissing skills and how that relates to the months of birth of you and that there were six specific months that you’re a better kisser and the other six you’re not as good in terms of kissing. I was thinking like, gut feeling tells me that this is complete rubbish.

12:41
No, absolutely. And I think one of the clear takeaways at the moment is that if we are using AI to be generating anything of any importance, then having a really thorough review of it is absolutely critical. Keeping that human in the loop as part of that review process and checking what’s provided is absolutely important. This is something, as we think more broadly about say the medical applications, we can use them in the hospitals and so on.

13:08
Again, fantastic useful, but they realize they have to include those very rigorous review processes. And that itself creates other problems because one of the features about, I think, humans, which may be less important than AI, is that humans are fundamentally lazy. And actually, if we’ve been using AI and it’s generated three outputs and we’re happy with those, often when we look at the fourth, fifth and sixth, we’re not going look at it so rigorously. And how do we keep up the quality of that review in the future?

13:37
That’s one of the big concerns I have moving forwards. I think that speaks to the trust that we have into the AI output. Why should we trust a statistician more than what we get from an AI output? That’s a really good question. I think that we know and understand this is on the AI. There are a whole range of which are involved with things.

14:06
and the outputs. We hope from the statistician, and I think that sort of is a role which all of us have got to keep on developing pushing in there, that there is an added layer of understanding within that, which you don’t get from the AI. We’ve got that additional confidence, which we can provide to people. The understanding why the reasons behind things.

14:27
be able to interact with them and understand their understandings better. And that can be a comfortable time. Sometimes it may be actually a case that there’s a challenging, which is the one of the roles statisticians can provide, which AI, we know can generally often be actually doesn’t like doing, can be very agreeable at times. I think it’s a really good underlying question. There’s always this challenge, even in the pre-AI world, about how we produce trust and confidence in people and how we continue to build that.

14:55
I think for me, if I think about trust, I usually think about three different things. The first is competence. The second is care and the third is character. So competence, well, maybe AI can become pretty good at some point, but AI doesn’t really care. So it doesn’t care whether you succeed on it or not succeed. Yeah. It’s a machine. Yes. Yep.

15:24
And where it will not, for example, just generally tell you something that you don’t want to hear. Yeah. Like this is not feasible. is nobody wants to hear that. But of course it’s something that people need to hear so that you can come up with something that is feasible and meets the demands. When you also.

15:53
something fails, who’s accountable? Yeah, that is absolutely one of the unanswered questions at the moment. And this is potentially some very big in the wider world, some very big questions because it becomes litigation legal standings. And at the moment, it’s probably one of the big blockers in the adoption of AI in a lot of critical systems, because until you’ve established case law to understand who will have to be responsible, nobody wants to take that risk.

16:22
Now, certainly in an internal case, think in to beside a company, companies are still very much structured on human reporting lines. And so I think there’s a very clear accountability lies still within that human reporting line. Yeah, I think it’s fair to say probably in every company that the person who is accountable is a person and it’d be the person who was accountable before. yeah, as in a way we’ve always been accountable for the work, the outputs that we produce. Yeah, I have to be constantly right.

16:50
If they’re produced by AI, then I think it’s still very much going through that same line, the person who is the user of the AI has to ensure that they’re comfortable with that. Yeah. tell you what, yeah, take accountability, take responsibility for it. So if you get rid of your whole statistics department, can’t blame the statisticians anymore. let’s talk a little bit for recommendations for the future. Yeah.

17:17
So for, let’s say, the typical statisticians that working on clinical trials, real world evidence, any kind of data projects, pretty much hands on. What would you recommend for them to learn on to change and maybe to think differently about? I think underlying this would be this idea of curiosity, which I think is something we’ve always been encouraging in any sense. Yeah, there’s this world out there, which is AI.

17:45
Just for people to explore it and read about it, listen to information about it. There’s huge wealth of information and stories about AI. there’s huge opportunities to be learning about that. And to experiment and play with it in any setting like that. So I think that’s sort of the basic bottom line within that. think it’s certainly going to be a case that AI in one form or another is going to be with us moving forward. So I don’t think it’s something we’ve got the opportunity to stick our heads in the sand and pretend it’s not happening.

18:15
And over time, I’m sure in the same way as a Google search was quite revolutionary when they first came out four years ago, was. email, think of the change that’s made to our life. There will be parts of AI which get just adopted into our everyday lifestyle, our everyday way of being as humans, which we ought to even think about in the future. And we want to make sure you’re part of that and able to do that. Otherwise you’ll be operating at a complete disadvantage.

18:44
Completely agree. I tried to test AI for all kinds of different tasks that I’m doing. especially tasks where I need to generate text or documents or things like this. I really like doing it. Of course, I read things and what I also like is to basically ask the AI what additional information they need. I think that was one of the best hacks whatsoever.

19:13
Yeah. So that you can provide better prompts and then refine it and refine it and refine it. That’s also how I’ve written my book. But if you then also have test things out, you can become more confident over time in what to do, how to use it. And that can really accelerate things. Yeah. And I think that’s a great thing of experimenting. And if there’s an activity which you’re doing, which is basically a bit dull, feels a bit repetitive.

19:43
there is probably a way I can help you within that. But again, the experience of doing things and I’m not sure of your experience in writing, but certainly when I’ve used it, I’ve always felt that probably half of the comments it makes, I think, are really valuable. They’re things where it structures the sentence better than I had and you’d say that is good feedback in the future. But probably about half the cases, actually, it makes a suggestion, but it’s not what I wanted. I’ve written in that way because I had a reason for writing it that way.

20:13
And think that understanding is going to be really important for getting there. also use it for complimenting me. Yeah. So for example, you get an email that makes you very angry. Yeah. And we all get that sooner or later in our work. In the past, I would have kind of drafted something, slept over it and then refined it. Thought about, do I need to send it at all? All these kinds of different things.

20:43
But now I can also use AI to write an answer and tell AI to write it with a very different mood than I have. Yeah. So I might be really frustrated, angry, upset, but I can tell AI to write it in a very, very understanding, calm voice, very solution oriented. And which I can’t at that moment.

21:12
But I know I want it, but I can’t. And that’s another kind of very, good thing. Yeah. And no, I think that’s a great example, but yeah, I do think within that, we’ve got to be careful about the authenticity of that. Occasionally you get something which is clearly written by AI and you’ve lost that personal touch. It’s doing something like that, but just not taking verbatim what’s written by the AIs would be a very helpful approach there.

21:38
I agree. I don’t use AI for my social media actually anymore because social media degrades AI written posts, which is in a way kind of weird because all these different social media have AI built into it and suggest you to use it. it doesn’t display it. Anything else that you would like the listener to take away from this podcast episode?

22:07
I think the overwhelming position which I had tried to communicate in my paper was actually, is obviously things will change with AI, but things have always been changing over time in any career. I think statistics is no exception within that. But actually sort of, tried to be positive about this. Why we’re totally going forward is that maybe some of the less interesting, the less rewarding parts of our job, we can actually reduce. We’re going to completely eliminate them. AI will provide a really good tool for doing that.

22:35
But actually at the same time opens up a range of opportunities for other things we’ve been doing with that time. I don’t think it’s the case that there’s fewer, need for statisticians. In fact, I think there’s going to be more need. I think that we’re in a world that is going to be more data, new endpoints which come from AI technologies like in imaging, wearable devices, other analysis, which may become feasible because we’ve got a lot of data we can now use with AI to sort of get additional insights too.

23:03
So I actually see a very promising future, enlightening future then. And probably many careers, think actually statistics is probably one of the best places to able to really ride and thrive within the AI world. And honestly, we work in a very innovative area. So there will be always new things that we are working on that AI hasn’t worked on before and where we need to think differently about. And lastly, we talked a lot about the human aspect.

23:33
that will never go away. No, absolutely. Yeah. And I think those are the skills which become more important and more useful than ever. So yeah, absolutely. Thanks so much, Chris, for this interesting discussion about AI. As we are recording this, here’s the PSI conference. And there’s also, of course, a lot about AI. When that gets published, this conference is already going, but this topic will definitely not go away. That, absolutely. Yeah. Thanks very much for the discussion, Icon.

24:07
This show was created in association with PSI. Thanks to Rain and her team at VVS, who helped with the show in the background and thank you for listening. Reach your potential, leak great science and serve patients. Just be an effective statistician.