A Webinar Recording: Redefining the Role of Statisticians in a World of Real-World Data and AI

Flaminia Chiesa

Transcript

A Webinar Recording: Redefining the Role of Statisticians in a World of Real-World Data and AI

[00:00:00] Alexander: Welcome everybody to this Cytel webinar. I am super excited, said Benjamin and myself as host of the Effective Statistician. Do this [00:00:15] together with Cytel today. Let’s first start with an introduction. I’m Alexander, just I’m host of the Effective Statistician Podcast. Have been 20 years with the pharmaceutical industry, worked in academia as well as.[00:00:30] 

[00:00:30] Alexander: CRO and I love everything around leadership and with that, I will hand over to Flaming. 

[00:00:40] Flaminia: Yes. Thank you, Alexander. My name is Flaming, I’m a [00:00:45] statistician. I work at title since approximately six years. I have a similar background to Alexander. I’ve worked in pharma, academia roles in different countries in the last 25 years.[00:01:00] 

[00:01:00] Flaminia: I’m very happy to be here because I’m also very interested in leadership, management and innovation. So let’s see what we can get out of this chat. 

[00:01:10] Alexander: Let’s turn over to Jannis. 

[00:01:13] Yannis: Hey everyone. I’m really [00:01:15] excited to be here. I’m Jannis Miah, chief Scientific Officer at sel, and a statistician by training similar to Alexander and Nia.

[00:01:22] Yannis: I’ve been in the industry for 20 years about, or, or more. And very excited to see it [00:01:30] evolving, to see how our profession is developing and gaining prominence. We have so much to bring to how clinical trials are evolving and where we’re headed, and so I think this conversation’s gonna be really exciting.[00:01:45] 

[00:01:46] Alexander: And finally, not. Lastly, Benjamin, great to have you on the call as well. 

[00:01:51] Gary: Thanks Alexander. It’s been a while co-hosting this, so it’s a pleasure to have a chance now in a different setting as a webinar and podcast together. [00:02:00] That’s very exciting. And yeah, my background is, you know, statistician by training more than more than 20 years now in the industry.

[00:02:07] Gary: But I skipped all the academics, pharma, I’m right to CRO and stayed there since the last 20 years. So it, that [00:02:15] is my experience and I’m, you know, heading the Our Value Statistics Department. Tel in the project based outs, project based services, PPS, which is the execution of the tribe. So it’s a very exciting occasion to be here and to have this chat [00:02:30] and looking out not only to what has changed, but really what’s going to change.

[00:02:34] Gary: And that is an exciting today. 

[00:02:39] Alexander: Let’s get started. The topic of today is the future of decisions in the [00:02:45] evolving pharmaceutical and healthcare landscape. What kind of changes have we seen in the recent years? I think these evolving trends will have an even bigger impact in the coming years. Yeah. [00:03:00] Do you have seen, in terms of cities changing in the last five years, what are bigger trends that you have seen?

[00:03:08] Flaminia: Different ones, many. First of all, I would say change of the target indications. [00:03:15] So we are. Looking more and more towards rare or rare diseases and more genetically identified diseases, that’s one. So the indications, definitely 

[00:03:29] Alexander: [00:03:30] lots of companies, rare diseases as one of the key target indications. As we know, rare diseases are actually not that rare.

[00:03:39] Alexander: If you can identify them, especially with gene differentiation, that’s [00:03:45] definitely a bigger point. 

[00:03:46] Flaminia: Well, you mentioned the therapies that go out with these rare diseases. The gene therapies, immunotherapies, cellular therapies for rare diseases. The focus shifted from the [00:04:00] clinician point of view to the patient.

[00:04:02] Flaminia: Course by involving more patient advocacy groups and define spec special endpoints that are maybe more targeted to the [00:04:15] heterogeneity of the patients. 

[00:04:19] Alexander: That’s a very good point. So lots of case in terms of guidelines regarding patient reported outcomes. Which are now, the abbreviation has actually [00:04:30] changed in the last years as well.

[00:04:32] Alexander: So before that, we have talked about pros. Now we talk about COAs. That’s yet another change. Jannis, what kind of others changes have you seen? 

[00:04:44] Yannis: Thanks. [00:04:45] I think we’re, we’re talking about RCTs, but really the broader picture is clinical trials in general, because we see a lot of trials that are single arm studies that are used.

[00:04:56] Yannis: External control arms, for example, trying to [00:05:00] leverage all the data that exists in the healthcare system, but is not necessarily channeled through randomized clinical trials or even clinical trials altogether, right? So there’s a increasing need to [00:05:15] use all the data, real world data that is available.

[00:05:18] Yannis: Obviously trying to do that in clinical development, but there’s also an increased. Concern with reimbursement and how to make sure that you get market access. It’s not [00:05:30] sufficient to get the drugs approved. You also need to make sure that the healthcare systems and payers will be able to support that.

[00:05:38] Yannis: We’re seeing a lot more of benefit cost as well as benefit to risk. All this feels like it’s converging. There’s a lot of [00:05:45] innovation in statistics on methodology. We have, of course. The whole field of AI that’s blowing up and people are wondering, you know, how do we leverage this? How do we make the most of it, make our lives [00:06:00] better, simpler, or derive better insights from all this data?

[00:06:04] Yannis: It’s quite incredible the insights that can come from a machine looking at large quantities of data. 

[00:06:14] Alexander: [00:06:15] Completely agree. Benjamin, what would you extend on these kind of points? 

[00:06:20] Gary: Yeah, I think we touched quite a number of things already in terms of what we observed and where we currently are in the process.

[00:06:27] Gary: But what we didn’t really touch on is the [00:06:30] people, right? The statisticians. So what did we expect, or what did you, and now I’m asking Fami again on the, it’s more on the operation side, is how did you see that the, the, the statisticians, the requirements on the statisticians change [00:06:45] over time?

[00:06:48] Flaminia: Well, 20 years ago, the statistician was working in silos in his own small team, uh, and doing his own [00:07:00] work for specific focused objectives. Nowadays, I see this changing greatly. There’s much more need of collab for collaboration. For [00:07:15] flexibility, adaptability, really in terms of mindset. The statistician needs to be open to innovation, like Janice mentioned, so should be up to date with digital [00:07:30] fluency, you know, being digital fluent.

[00:07:33] Flaminia: Know, at least know some of the options that are out there on the market in terms of ai, neural networks, machine learning techniques, not necessarily being able [00:07:45] to use them to do them, to create them, but being able to know when to use them and where collaboration, again, I come back to the first point.

[00:07:55] Flaminia: There’s the need of collaborating with data scientists and developers [00:08:00] of innovative techniques. In terms of personality, a statistician needs to be proactive, needs to be optimistic, and needs to be curious. Curiosity is one of the main [00:08:15] points that drives innovative implementation, I would say as well be open and not resistant to change.

[00:08:25] Flaminia: The state of the art is always scary for everyone, and we should [00:08:30] never underestimate that. As managers, we should try to encourage change and really help sustain the people with their innovative ideas. 

[00:08:42] Alexander: Completely agree. [00:08:45] See from your face that you want to add to that. 

[00:08:48] Yannis: One thing I want to give as background is as tell we.

[00:08:52] Yannis: I did some research last year where we spoke to 30 or 40 different stakeholders in a variety of [00:09:00] companies in North America and Europe, large, small, some regulators, some payers, and asked them, you know, where did they think that we would be five years, 10 years from now in terms of how we run clinical trials.

[00:09:14] Yannis: I spoke about this [00:09:15] idea of augmented. RCTs or reimagined RCTs, where we are bringing in not just the usual clinical trial methods, but trying to augment that with real world data, new technology, new [00:09:30] methodologies, trying to push more innovation. In many cases, a lot of these innovations and ideas have been around for a couple of decades, at least the time I’ve been in the industry.

[00:09:40] Yannis: But somehow they feel like they’re really coming together and accelerating. And I think in the last [00:09:45] couple years especially so. The natural question after that is, of course, who are the people who are going to work in this new environment? If AI is gonna replace certain aspects of programming, then what are we gonna be doing instead of programming?

[00:09:58] Yannis: For example, [00:10:00] you would imagine the idea is becoming more strategic, becoming more multidisciplinary, being able to provide insights, interpret, communicate, convince. Other [00:10:15] stakeholders about what needs to be done and where we’re headed and trying to bring all those pieces together. So I see what we’ve seen in a lot of discussion.

[00:10:25] Yannis: A lot of people already moving in this direction of what we might call a [00:10:30] clinical data scientist who is not only a statistician, but knows some epidemiology like Sal in implants, for example, knows how to bring in. Real world data together with experimental clinical trial data. [00:10:45] We need, as Mia mentioned, people who can feel comfortable with data science, with machine learning, with the idea of synthetic data and digital twins, and of course, programming has been exploding.

[00:10:58] Yannis: The open source community [00:11:00] and things like copilots and AI assistance, it’s a very different ball game from when many of us were trained. In programming some years ago, and I think all the other stakeholders in pharma and [00:11:15] regulators look at statisticians to help navigate all this. We’re the people who understand quantitative methods along with some of our colleagues in epi and health economics and so on.

[00:11:26] Yannis: But this class of quantitative data scientists, if you like, [00:11:30] are really the guide to this new future. And can help to set the boundaries of what’s acceptable, what’s not acceptable, what’s right and wrong. Biased. We’ve been doing that for many years, but I think our time has [00:11:45] come to shine really. 

[00:11:48] Alexander: I think as you mentioned there all these different data rules, roles.

[00:11:52] Alexander: Yeah, like epidemiology, pharmacologists, real evidence, clinical trial [00:12:00] statistics, people that. No. Inside out machine learning, my perception is there’s the convergence here. Yes. The, the, these kind of silos are not that strict anymore. [00:12:15] I also see that these departments don’t sit completely separated from each other in different functions, but I see a couple of different companies who have built an umbrella organization, put all these [00:12:30] different quantitative scientists.

[00:12:32] Alexander: I know a couple of different companies that have went that way. And so it’s much easier to align from an organizational point of view and to collaborate because [00:12:45] it said have been used for example, primarily in real evidence and in epidemiology, like causal inference. And now we use more and more in clinical trials with Theon [00:13:00] framework.

[00:13:01] Alexander: Yeah, so programming, machine learning used in real world evidence, parts or epidemiology is now also used in statistics. Some of those, they will actually [00:13:15] comes from statistically originally. So it’s, I think this kind of this to say, I’m only in working on this, I’m only working on that, and I’m only working on yet another thing.

[00:13:27] Alexander: You can’t say that anymore. 

[00:13:29] Yannis: You [00:13:30] can’t. But I do think that, you know, there are different personality types. There are people who are more detail oriented and more abstract thinkers, and I think you need to be able to touch and understand a lot of [00:13:45] these points and understand how other function, how, how they’re needed.

[00:13:49] Yannis: Yet they’ll also always, or at least for foreseeable future, be some level of specialization where some people are better at. Getting phase three trial passed a [00:14:00] regulatory hurdle versus being creative in exploratory settings where you’re analyzing data and trying to hype generate hypotheses, which I think is tricky, right?

[00:14:11] Yannis: Because you need to be able to juggle a lot of [00:14:15] these things. Yet people are not always comfortable in all these areas because just personality types, right? So connects with this whole idea of leadership because leadership is an area where some people [00:14:30] uncomfortable, right? 

[00:14:31] Gary: Yeah. When I hear you talking about the, you know, the, the mindset and the, the different aspects that a statistician on like quantitative scientists is, is needed to fulfill.

[00:14:41] Gary: I’m kind of worried that we are running out of them. If you filter for the [00:14:45] different attributes, let’s say, that you need to bring along as the future statistician. It will be rare. Right. So that’s why I think we, we have a big need in getting statisticians trained, open up to other [00:15:00] aspects, to other ideas.

[00:15:02] Gary: You know, as Mami said before about the, you know, you know, the thinking and curiosity things where we, where we really have to help as managers, as leaders in the industry to, to support this, [00:15:15] to develop further. ’cause otherwise, looking back 20 years. We have, have the introverted statistician on a computer.

[00:15:22] Gary: That’s not the future role. So we have a lot more to consider and it’s much bigger thinking behind, um, you know, aspect of [00:15:30] in, in, in looking into, into our future as, as data scientists or statisticians. 

[00:15:36] Flaminia: But a statistician cannot be an expert in everything. And because we are statisticians, we are very oriented towards the [00:15:45] details.

[00:15:45] Flaminia: So we really want to know. And understand a model or a test or a method. Uh, so it gives us uncomfort not to know and staying on the surface of things. [00:16:00] Nevertheless, these days, the methodologies, the designs of trials, the mass of data, ask for. This holistic view of real world evidence together with [00:16:15] RCTs, uh, with innovative techniques like adaptive patient methods, which we don’t necessarily need to know by heart, I think.

[00:16:25] Flaminia: Mm-hmm. We collaborate with people that know how to use them, and we have to [00:16:30] know how to sit at a strategic table with the biotech and. All stakeholders to actually find the right solution for that specific. 

[00:16:42] Gary: Yeah, I was just going to say that is probably where, [00:16:45] where the collaboration comes in, right?

[00:16:46] Gary: Breaking down the silos and really have the, you know, the cooperation between, you know, the designers, you know, the statisticians who are more focused on the methodologies, the executors than you know, outside, like [00:17:00] later on. Market access, RWE, anything that comes along across in all directions. Also regarding when we use innovative technologies, it’s not, not necessarily that we need to be expert in it, but it’s really about knowing who and [00:17:15] collaborating with, 

[00:17:17] Alexander: I think I basically see two different trends.

[00:17:21] Alexander: On one hand, we have all these fancy statistical methods like Asian borrowing. We have these digital twin. [00:17:30] Things. We have digital health applications, many more patients reported outcomes, these kind of things. My perception is we have, and I see that also in pharmaceutical [00:17:45] companies as well as some CROs of experts that have a very, very deep knowledge in a narrow space because there’s so much change going on.

[00:17:56] Alexander: To keep up with Cesar from things, you need to have [00:18:00] some kind of focus. Same if you look into Beijing designs and phase one, phase two, yeah. There’s a new public coming out. It fits like daily on a new kind of trial. Design [00:18:15] specialists need to work together with, I think more generalist type data, clinical data scientists that have an overview of what is possible.

[00:18:26] Alexander: Work closely with non-quantitative [00:18:30] functions like physicians, market access people, commercial people, or regulatory people. And those need to have a good way of communicating with these jobs. But I think especially in this letter type, [00:18:45] these kind of more quantitative generalist type says, need to be strong leaders with very good communication skills.

[00:18:56] Yannis: I think we’re sorry. [00:19:00] Would you agree with that? I think what I’m thinking is, you know, we live in really interesting times. If you look at like big picture of what’s going on in society and some people have claimed a comeback of the [00:19:15] Renaissance men, like Leonardo da Vinci that does a little bit of everything.

[00:19:20] Yannis: Ever since the Renaissance has been a lot of specialization and. Not exactly today, but if you look out 5, 10, 15 [00:19:30] years, you’re gonna see a lot. I think of the special, there’s a lot of specialized tasks that you’re gonna get automated where AI agents are going to replace many things that we do where we may not need to know all the details [00:19:45] over time.

[00:19:46] Yannis: In many cases, people are a bit skeptical and don’t trust new technologies and don’t trust black boxes. Over time, their attention fades and people start accepting. There’s so many [00:20:00] technologies that were difficult to adopt and now we don’t even think about because we just live with them. We live with our phones, our computers, our cars, and we don’t give it a second thought.

[00:20:11] Yannis: The first people who were exposed to the car didn’t necessarily [00:20:15] look like the same way that we do. So I think it’s difficult to imagine how these things are going to become. Common and uh, accepted, but over time a lot of these specialized [00:20:30] functions will be available to many people. There’s gonna be more of a focus on this sort of generalized aspect of, you know, do you know, do you understand the bigger picture?

[00:20:41] Yannis: How can you connect the dots between the various [00:20:45] functions or the various quantitative approaches and make them work together? How do you. Communicate and convince people that these are the right approaches, that this is the right interpretation, analyze the pros and cons. There’s a number of things that the [00:21:00] machines are not gonna be able to do.

[00:21:01] Yannis: Right? That we will essentially start toing ourselves. 

[00:21:05] Gary: I, and maybe, and I said that is a good part, to dig a little bit deeper in the goals of ai, right? It’s a password for a number of years now. Right? So when I’m saying [00:21:15] password kind of provocate, because this is. At the beginning, everyone was talking about it without knowing what it really means and the impact that has.

[00:21:22] Gary: We still don’t know what it has. At least we get a sense of where it could be potentially used and maybe it’s already starting. And in some [00:21:30] regards, do you have any more, especially when he’s talking about the research that, that, you know, kind of looking in the future. So where, where do you see AI in general or in specific In our role being the new car that we are driving at some point.[00:21:45] 

[00:21:46] Yannis: It’s challenging to think about that, right? Because I think use cases are coming up with use cases on a regular basis. The technology’s all evolving really fast, and although it has limitations, [00:22:00] you know, like the iPhone 10 versus the iPhone one, a lot of problems got solved, right? So it’s hard to know. I think intuitively as statisticians and so on, we see limitations and problems to the technology and wonder how will they ever get over human [00:22:15] biases.

[00:22:15] Yannis: But people are working on that and thinking about it. And future may look very different. Primary use case in ai, ML is just efficiency. You know, all these sort of processes that can be automated and maybe in the past have been [00:22:30] automated using robotic process automation. People are looking at ai. To do that LLMs or, or machine learning in general.

[00:22:39] Yannis: So synthesizing literature, generating [00:22:45] documents, programming code, even being able to converse with an AI and say, Hey, I need a BA hierarchical model, or a negative binomial regression model. Run my data. Can you do this for me? And have it sort of create the [00:23:00] code and then run with it. There’ll obviously be.

[00:23:03] Yannis: A human in the loop, sort of a QA step. Because of the industry that we work in, we may just start trusting these things more and more as they become more and more reliable. So there’s [00:23:15] all the areas that are maybe a little bit more scientific, which is how can you derive insights from data? How can you use this since diagnostics, which is already, I think ahead of the curve.

[00:23:26] Yannis: Do you other, I’m curious to hear what’s if. 

[00:23:29] Flaminia: Yeah. I don’t [00:23:30] think we should be scared. Ai. 25 years ago I did a master thesis on image recognition for diagnostics on neural networks. The implementation of that is probably used nowadays in [00:23:45] clinics, but in far and biotechs will be probably used as a second eye.

[00:23:51] Flaminia: But let’s remember, we are a highly regulated environment, so any change cannot happen from one day to the other. [00:24:00] Although the technology changes rapidly, the changes in our environment would be slower and we will have time to adapt and to learn natural language modeling. I did that when I was at IQVA [00:24:15] some years ago to read the electronic medical records.

[00:24:19] Flaminia: It’s helpful. But it’s still real world evidence is still not a standardized environment. Uh, the data for, for it, we can use them, but [00:24:30] with care, ideally in the future, a standardized real world data set so that we can easily merge with one arm trials. We are still not there though. 

[00:24:42] Alexander: It’s a couple of different tasks [00:24:45] and you just say kind of on a probability from zero to 100.

[00:24:50] Alexander: How likely that will be done through AI protocol. If you have a protocol, outline, outline of a protocol, will that be done through ai?[00:25:00] 

[00:25:02] Flaminia: 60% of it will be done through ai 

[00:25:06] Yannis: Giannis, I think higher, I think 80% Maybe. It’s not a probability the whole thing will be written. It’s like how much of it will [00:25:15] be AI driven versus expertise, uh, coming into it. Of course, all this depends on the timeframe you’re talking about 

[00:25:24] Flaminia: and depends on asking too many question.

[00:25:27] Flaminia: I I think it depends also the on the [00:25:30] input that you give to AI and input is set up by humans, not by AI themselves, right? So what is the objective? 

[00:25:39] Alexander: But if you have a good kind of document that outlines a protocol, I think 80 [00:25:45] 90% of it can be by ai. Pretty soon, especially thinking about lots of standard platforms that we have.

[00:25:54] Alexander: What’s your take 

[00:25:55] Gary: on that? Well, I agree it’s at, at some point it’s, it’s a [00:26:00] quite high number, so there’s some adjustments needed. But my, my question or my, my question back to you is more on, you know, against the nature of the statistician to just trust ai. Right. As said before, you need to. Get the [00:26:15] data, get on the ground of it.

[00:26:15] Gary: How do we build this up? Protocol is not the, you know, sta document per se, but it’s more, next discussions is probably about the SAP Alexander.

[00:26:26] Gary: How do we make sure that the, the statistician or that this [00:26:30] is a trust worth document, right? So, so that’s why I’m, I’m a little bit, you know, like 80, 90%. I would, I would agree. In principle, however, how much time do you as a statistician or a writer. Spend on [00:26:45] confirming that this is correct or in line with what you expected or experienced.

[00:26:50] Alexander: Yeah, I think that’d be interesting because you can create feedback loops, you can correct things and train large language model to become better. And I agree, the [00:27:00] protocol is just a statistics document, but in terms of the statistical analyst plan, once you have a complete written protocol, yeah. It will be to create ASAP or actually, do we still need send an SAP [00:27:15] do, do we directly kind of, here’s protocol, create all the tables and things from it?

[00:27:24] Yannis: Yeah, I think there’s a question of like, there’s a journey to getting somewhere and then once you’re there, [00:27:30] fluctuation will be different. In order to train these models, refine them, the feedback loop, et cetera, you need an expert who’s gonna say, yes, we got the right answer. No. And that has to be the statistician.

[00:27:41] Yannis: I think when you start thinking about bias, the [00:27:45] various types of biases, being able to check whether the model are doing the right things or not, I think we’re in a great place to be helping that and making sure that we get through that journey. And I think we see that in the industry when you. [00:28:00] You just hear about various companies are working on AI ML initiatives and the statisticians are generally involved because they’re the ones that most of the other participants feel are best PO to deal with.

[00:28:14] Yannis: With, [00:28:15] 

[00:28:16] Flaminia: I would add Jannis AI is statistics, right? That is a probability model. That’s also why we are the best placed to learn about ai, to use it best. But I [00:28:30] doubt that it, we will disappear as statisticians. As you say, there’s still the human eye and the human brain that is needed to really discern, uh, what is correct and what is not.[00:28:45] 

[00:28:45] Yannis: Alexander, I have a question for you. Yeah. I know you give leadership trainings to many organizations. How, how have you seen changes in what’s being requested of leadership given this changing environment and new [00:29:00] tools that are coming up?

[00:29:05] Alexander: Companies including Tel as May, and in the beginning we focused on very much on [00:29:15] communication. Presentation skills, negotiation skills, building teams, building trust, and also confi resolution. In the last years, we were asked to also provide more training in terms of [00:29:30] strategy. What is good strategy, what is bad strategy?

[00:29:34] Alexander: What are elements of a good strategy? How can you look into sources of power within a strategy? So things like focus or leverage or [00:29:45] design, or other aspects that make strategies actually work, and how can you differentiate from a good strategy, from a so-called bad strategy? Strategy [00:30:00] is nowadays used pretty much like a password, and everybody sits in a.

[00:30:05] Alexander: Strategic role and works on the strategic document with strategic discussions in strategic governance meetings and so on, [00:30:15] and not everything said, it just has this adjective. Strategic is actually strategic. And so helping people understand what is actually tactic and what is the strategy. So it’s [00:30:30] definitely very, very important.

[00:30:31] Alexander: Important point. Negotiation becomes much more important as well, specifically because the pressure on statisticians and data scientists has increased [00:30:45] quite a lot and much more exposed to doing lots of different tasks and see the, the companies wanna do more with less. So they need to negotiate about timelines, about [00:31:00] workload, about resources all the time.

[00:31:04] Alexander: This is one of the main pain points actually when I talk to statisticians, is a workload. It’s ever increasing it and demands are really big. [00:31:15] And of course people know that there’s lots at stake. So fund negotiation skills are also key aspects. I. Delegation is worth right. Negotiation.[00:31:30] 

[00:31:32] Yannis: It’s an interesting that we’re increasing efficiency with things like yet at the same time people have increased workflows. 

[00:31:43] Alexander: Al one, one of the key things is, and [00:31:45] that is not going away by the ai, is other discussions, you know. Recently was at a, at an workshop recently, September last year, and someone said, yeah, with AI [00:32:00] we will from tables and figures to clinical trial report or to a PowerPoint presentation much faster.

[00:32:11] Alexander: But the weeks of discussions of [00:32:15] the study team. About these results, what’s the mean and the next steps? I don’t think that you can expedite that a lot with machine learning or generative ai, 

[00:32:28] Flaminia: and that would’ve happened in [00:32:30] the writing 

[00:32:30] Alexander: of the report. 

[00:32:31] Flaminia: Mm-hmm. Right. 

[00:32:33] Alexander: The small part is really affected by generative ai, the kind of the writing part and the summarizing part, and kind of converting documents into different formats [00:32:45] like.

[00:32:46] Alexander: Creating an abstract out of a clinical trial report or creating a PowerPoint presentation out of a clinical trial report or having an abstract and creating a poster for it. I think these tasks that will be solutions that [00:33:00] help us with that, doing these kind of 90, 95% ready, creating a discussion out of all of that and determining whether that is a benefit risk ratio.

[00:33:14] Alexander: [00:33:15] Sets yet a completely different topic. Also, lots of companies are competing against each other on many of these different indications that are kind of lead indications, even if they are rare indications [00:33:30] and they all come work with different designs, and so statisticians will be needed to explain. Why did Company A go with that design?

[00:33:43] Alexander: At your [00:33:45] company with a different design. The company C came yet with another different design because you know, of course you always want to design something perfect for your compound. As compounds are different, [00:34:00] of course have different designs, and that’s also probably automate using language models.

[00:34:09] Yannis: Well also, I think a lot of AI relies on. Interpolation of existing [00:34:15] patterns and data. So when it comes to innovation and new methods and new ideas and how to solve new problems that come along, I think that’s gonna be challenging, at least in the foreseeable future till we get to the next level of [00:34:30] artificial intelligence.

[00:34:30] Yannis: But just with what exists today, it’s hard to see how when you’re confronted with a new problem, a new endpoint, a new statistical challenge. You’re gonna have to rely on people who’ve been trained in statistics and know [00:34:45] how to work these problems out. Okay. I would 

[00:34:49] Alexander: say we stop here and move over to a couple of questions from the audience.

[00:34:56] Alexander: Yeah. We are now 45 minutes, about [00:35:00] 45 minutes into seminar, and there are lots of questions also online. Pierre, for example, asks, what is your web recipe? Enough confidence so that we are able to share evidence generated by the front [00:35:15] experts, like data, find evidence, ology, all of that. How do we get to that confidence level?

[00:35:23] Alexander: What do you think I,

[00:35:29] Alexander: how [00:35:30] do we build confidence that area? I would say we just try and do it. I think you can only build confidence working into it, leaning into it, trying at the out, [00:35:45] and of course, have good discussions with these experts so that you at least understand the big ideas. You don’t need to understand all the details of how to implement the answers, but you need to understand some big ideas.

[00:35:58] Gary: I think it’s the comfort [00:36:00] zone that you need to leave once in a while. Really out of this. That is also where time management comes in, where we give guidance to, to our staff. Maybe open up doors say that is, that is something you need to do, but actually confidence comes with training. With the training of doing it and [00:36:15] not, you’re not born with.

[00:36:16] Gary: So it’s really go out and try, do something new or be afraid of failures, something that everyone needs to come through at some point. Ai. 

[00:36:26] Flaminia: AI works like this. AI does fail and then [00:36:30] starts again from the failures. It learns. So that pouch, 

[00:36:35] Gary: yeah, the 

[00:36:35] Yannis: ai imperfect. I think a certain amount of during, like you said, that people need to have, the other aspect is sort of the comfort [00:36:45] zone is is very interesting piece there that you said, Benjamin, because I think as statisticians we always hedge our bets.

[00:36:53] Yannis: It’s like, oh, well there’s a 5% chance, like I’m wrong. You’re never a hundred percent sure of [00:37:00] your answers really, but you need to act that way when communicating with other people, because others are not that comfortable with uncertainty when they need to make decisions. Just need to go ahead with that confidence, I think has to come from within the same way that we have [00:37:15] confidence.

[00:37:15] Yannis: When we stop a trial of 95% predictive probability of success, we should be able to say, okay, well. I’m 90% sure this is happening. Let’s do it. Right. 

[00:37:25] Alexander: I have another question from Nancy. That’s a very controversial [00:37:30] question. How much should a statistician compromise on basic principles, for example, replacing an RCT, including synthetic data in the analysis plan?

[00:37:42] Alexander: Do you foresee regulators [00:37:45] being with these kind of things? 

[00:37:48] Yannis: Yeah, I, I don’t think in the. Short, medium, future, we’re gonna stray very far from where we are today. You do need to stick to basic statistic and clinical trial [00:38:00] design principles. We’ve seen adaptive designs early on. Question is, when can you use adaptive designs?

[00:38:06] Yannis: And at at the beginning, it was really well, you have to be able to justify that there’s no other way of doing this, or this is the best way. [00:38:15] And, and I think there’s a little bit of that where people are experimenting with synthetic data or external control, arms, uncommon lines. There’s usually a big justification that needs to be done to the regulators on why this is a special case, an unmet need.[00:38:30] 

[00:38:30] Yannis: There’s no way of recruiting the patients. You, you have to have all those arguments aligned. I think as people get comfortable with the methods and the. Reliability of the results that may start to loosen up because we see that with adaptive designs, there’s not as much [00:38:45] justification of why you need to have an interim analysis.

[00:38:48] Yannis: People are starting to accept that as like, okay, we will just do that. So maybe over time that’ll change. But in the short, medium term, I think we’re, and that’s why I think we talk a [00:39:00] little bit about a, a reimagined rct and not like doing away with RCTs. I don’t think the rc t is going away anytime soon.

[00:39:07] Yannis: I. If ever 

[00:39:11] Alexander: I completely agree, the RCT will not go away just [00:39:15] because it is such a powerful tool and not get completely of randomization. When I think about regulators that have been talking about, you know, single alarm trials and [00:39:30] still very, very skeptical about single arm trials, such as then kind of combined with external data.

[00:39:37] Alexander: It’s not two extremes. Having a balanced randomized trial, [00:39:45] the thing alarm for also lot of things in between you can do. Being open for this in between will gradually probably change. I have another question here. Eric mentions that debugging [00:40:00] codes written by ai. Can be quite difficult. What your experience in that, is it, does it actually make more time to debug code this by ai?

[00:40:13] Alexander: Or how do you [00:40:15] foresee the future there?

[00:40:19] Flaminia: I’ve never used AI for a generating code, so maybe, 

[00:40:25] Yannis: yeah. And um. I, I think that’s [00:40:30] true, especially at the beginning of a technology adoption curve that you might spend more time. Yeah. You won’t get those efficiency gains right away. It can be quite difficult, especially if the code is long. I’ve heard some people suggest having [00:40:45] multiple AI models.

[00:40:48] Yannis: I wonder if you could generate code using, I dunno, CHE GPT, but then, you know, have cloud AI. Check for bugs and errors or, [00:41:00] or people will come up with creative ways of doing things like that, or just even interrogating the AI to figure out where the bugs might be, what went wrong. But I think coming back to what I said earlier as well, this is a part where the [00:41:15] statistician is needed because the number comes out or an answer comes out and you need to be able to recognize that’s wrong or that’s right.

[00:41:24] Yannis: Let’s go find out 

[00:41:25] Gary: what went wrong. Yeah. But it’s a good example of why the skillset of the statistic still [00:41:30] needs to be quite high, right? Debunking code without any knowledge, we need to have basic or advanced experience and expertise in the field. Even if two AI comparing, you still not gonna debug if you’re not experienced in that field.

[00:41:44] Gary: [00:41:45] That’s a classical example of how to use it and, and. You know and still why the statistician will be need program and maybe it’s more programming will actually, another question. There are people that think that that AI will [00:42:00] replace statistician in the future while others think AI can have part of the statistician opinion and you know, to justify your, I think we touched part of it, but maybe just to summarize, you know, for FIA and Giannis on that point.[00:42:15] 

[00:42:15] Flaminia: I think AI will give us the freedom to have more time to think about the contents rather than the actual format. So just to emphasize what Janni has said, we will be able to really [00:42:30] evaluate different strategies, methods, and. The interpretation of the results and what Alex said, if a treatment was approached with five different trial designs, we will have more time to [00:42:45] evaluate why these different methods were targeted and used for that specific problem, which we don’t have so much now.

[00:42:55] Flaminia: We are more executors of standard methods, right? And we [00:43:00] spend hours doing actual programming, the actual writing of a sub, researching the endpoints derivations. So we will have more time real statisticians 

[00:43:14] Yannis: [00:43:15] when you go to governance committee to get your trial greenlighted and explain the different trial design options.

[00:43:20] Yannis: You’re not gonna be able to say, oh, the AI came up with this option. You’re gonna have to explain why that’s a good option. Even if the AI helps you get there. People are gonna want to [00:43:30] understand the context. I don’t think that’s gonna go away and might get more intense to try to justify what good decisions are.

[00:43:38] Yannis: A lot of these tools and technologies, AI, about the designs and synthetic data, they’re all decision support [00:43:45] tools. At the end of the day, as human beings we’re making decisions. And we need as much information to make good decisions. 

[00:43:54] Flaminia: So we need critical thinking. That’s our one of the, 

[00:43:58] Alexander: we need critical thinking and [00:44:00] communication tools.

[00:44:01] Alexander: The emphasis on the communication part is increasing. In terms of generative ai, it also depends on how you prompt them and kind of what is the extent to which you kind of let it do something. [00:44:15] I used generative AI quite a lot to write my book. And that worked very well. If you want to write short snippets of code into it, time step and step [00:44:30] after step, you can increase your speed quite a lot by not typing it all.

[00:44:36] Alexander: You need to be very clear on your prompts, and then you can refine it from there. You still of course need to understand what code does. [00:44:45] For me, it was important to read what I had written and spot any bugs, but this is kind of the key part, so how you recognize 

[00:44:54] Flaminia: yourself about your art, 

[00:44:58] Alexander: definitely was [00:45:00] very good with the kind of storytelling part.

[00:45:03] Alexander: Wanted to write a book that’s really written in a story and not so much a fiction book or nonfiction. Okay. As we are approaching the end, let’s give you [00:45:15] some summary statements. Mia, what’s your key takeaway for the audience? From our discussions today 

[00:45:24] Flaminia: as statistician, I suggest everyone to [00:45:30] open to innovation.

[00:45:32] Flaminia: Not scared. Ask for support whenever needed. Uh, we will interact more and more between departments and be over [00:45:45] different specialties, and that’s the beauty of brainstorming, of being a statistician. We can only grow through this collaboration. 

[00:45:57] Alexander: Thanks so much. What about you? [00:46:00] 

[00:46:00] Yannis: I think we’re living through exciting and challenging times, critical thinking like we outlined and decision making are critical and things that we should focus on as well as communication.

[00:46:12] Yannis: Those things go together because [00:46:15] obviously you make decision, you don’t communicate it. Things go wrong very quickly or convincing people to make certain decisions. We’re living in a world where it’s cheap to generate data in some way. We have to remember there are people involved and it’s a burden. You [00:46:30] can’t just collect data all the time if you have no purpose for the data.

[00:46:35] Yannis: I think we’re living in that sort of tension right now because these tools are saying, if you’ve got a bunch of data, you’re probably gonna derive some insights. But at the same time, we really should be thinking [00:46:45] carefully about what data we want to collect, how we want to collect it, what we’re gonna do with it, and there are regulations and rules in place for that.

[00:46:53] Yannis: So you can’t just. Collect stuff. Just to collect stuff. So I think that’s where [00:47:00] statisticians also have a role to play in figuring out what are essentially the experiments you wanna run. Randomized or controlled experiments that may be just collecting data from real world data, but how do you design [00:47:15] the experiment, the study to answer the questions that you’re interested in answering and making the decisions that you need to make.

[00:47:23] Yannis: So I think we have to keep our eye on the ball, even though there’s all these things happening and a lot of buzz and advancing [00:47:30] technology very quickly.

[00:47:36] Gary: That was really good speaking with you and getting your insights. It’s exciting, right? It’s really exciting and what’s coming up, and we didn’t have the chance today [00:47:45] to touch a lot of topics that are impacted this. Started to think about how RCTs are changing. We observe the changes over time. There’s a lot to come.

[00:47:54] Gary: And that is not only ai, even though we talk about this quite a bit, but it’s also about [00:48:00] the, the mindset. It’s about the extension of, of topics that are, that will be on the, you know, at the desk of the statisticians and the responsibility of the statistician. Or at least in the, you know, in the responsibility of interaction then.

[00:48:13] Gary: So that is exciting. When people ask [00:48:15] you why I’ve become, became a statistician, I’ll continue to give statistician, and it is about the, the different types of requests coming in every day. It’s about different studies, different type of studies, different methods, responsibilities, et cetera. And that [00:48:30] is, that is just not ending, right?

[00:48:31] Gary: We see that this is becoming more and more. Statistician is becoming more and more important in the in, in the current standard world. Excited bury.

[00:48:44] Alexander: [00:48:45] I’m also super excited about these these things because mundane things will be easier, and I think there’s a lot of tools that can do things so much easier. I’m just thinking about AI and how it helps me [00:49:00] to generate the slides that I want to have. Condense it from hours into minutes. So these things will go away, mundane tasks and free up time or do new.

[00:49:13] Alexander: And the innovative things we’ll [00:49:15] definitely need to keep learning the technical elements, human skills, the leadership skill, skills. All Z will play an absolutely vital role irrespective of whether you wanna work [00:49:30] more on the generalist types, the Leonardo da Vinci kind of renaissance, you know, everything a little bit type, or whether you are person focused on one specific area where you know [00:49:45] a lot of what’s going on.

[00:49:47] Alexander: Thanks so much for this awesome discussions that we had today, and thanks so much for s for organizing this. 

[00:49:55] Flaminia: Thank you, Alex. Thank [00:50:00] [00:50:15] [00:50:30] you.