Simulations – your most powerful study design tool

Transcript

Simulations – Your Most Powerful Study Design Tool

[00:00:00] Alexander: Welcome to another episode of The Effective Station. I am super happy to have two great people with me here. Two great statiscians that have a lot of experience in simulations. I actually stepped into simulations very early in my career even doing studying because when you do anything on nonparametrics you always need to simulate things because nothing works out really like it should at the beginning and you need to test lots of different things, but of course there’s a lot of other things you can use simulations for.

But before we dive into this, I want to shortly introduce you to our two guests today. So first we have Kim Hacquiol who has been on the show before, maybe for those who missed the episode with you, you can explain a little bit what you do.

[00:00:59] Kimberly: Hi, Alexander. Yeah, it’s lovely to be here. Yep. It’s Kim Hacquoil. I’m the Chief Data Scientific Officer at Exploristics. I’ve got lots of years experience working as a statistician in the industry. And I have done a lot of clinical trial design and more often than not, most of the time I will be using simulation to do that. I’ve worked in pharma companies and I’ve worked here at Exploristics and yeah, love designing studies and learning about different ways to use simulation to do that effectively and efficiently.

[00:01:35] Alexander: Yeah. Awesome. And then we have Jamie Inshaw, who I first shortly met when whole big group was in Exploristics went through the effective statistician leadership program, which I found really cool because Aiden and others, Aiden who is the CEO O of Exploristics, by the way, he has been on this show before. Did I say he’s the CEO O of the show? No, he’s the CEO of the company. Exploristics. He has been on the show before talking about some cool things at Exploristics doing. They all felt that Investing in these skills will help a lot within the company, but also help with communicating with clients.

So Jamie, great to have you on the show as well.

[00:02:25] Jamie: Great to be here.

[00:02:27] Alexander: Tell us a little bit about yourself.

[00:02:30] Jamie: Yeah so my career started in academia. I started as a clinical trial statistician. I decided to take a break from clinical trials for a few years and do a PhD in statistical genetics, which was a real kind of change of pace. And then my love affair with simulation really began about three years ago when I joined Exploristics and since then my kind of, my primary focus really has been on using simulation to inform optimal clinical trial design.

[00:02:59] Alexander: Awesome. Very good. Let’s start with a very simple question. What is a simulation actually, and why and when should you use it?

[00:03:11] Kimberly: So I can start with that one. Simulation is essentially. Oh, I making it up as it were. Simulation is trying to create a virtual environment that you can then test, you can then think about what if I do this differently? Or what if I do that? And if you think about you know what a simulation is when you’re doing a fire drill, you are practicing what might you do in a real kind of situation. You’re simulating a pretend fire drill and running through what you would do and then you can adapt and you can Make things better for next time if you’re maybe not quick enough getting out of the building.

In clinical trials, simulation is used at many different stages, but in specifically in clinical trial design, it can allow you to essentially run clinical trials in a virtual environment before you actually run it for real. And the real benefit of that is you save potential patients because you can test lots of what if scenarios really test your design factors that you can that you can choose whether to go down one route or the other. And it allows you to really optimize before you actually run your real clinical trial.

[00:04:25] Alexander: What has typical clinical trials where you would simulate things upfront. Is there a specific areas that you would most focus on?

[00:04:35] Jamie: Do you want me to take this one? So just I guess before going on to that maybe another angle on why simulation is useful and why I think simulation is so useful is that when you are simulating a clinical trial, you are in complete control of the truth. So you can generate a virtual population. Of which half of the population are on the active treatment, the other half of the population are on a placebo. You can generate a treatment effect that you know exists because you’ve generated the data yourself and then the BC is, it really highlights to you. In reality, you’re gonna have that clinical trial playing out once. But what if you were to play out that trial over and over again? Actually you won’t have success every single time because just the nature of statistical probability distributions. And what simulation enables you to do is just understand in advance of a trial starting your probability of being in that section of trials that are successful.

Yeah. And so you are in complete control of the truth, and then you let the statistics do the rest of the work. So I think that’s why simulation is really powerful and so answer your question about when it’s useful. So often if you have a traditional sample size calculation and you’re expecting to see an endpoint that you expect to follow a fairly standard distribution, then that’s, that may suffice. But the reality is that real world data are messy, real world data are and when I say real world data, I don’t mean. We would rather just mean data collective. Yes. Yeah. So you’re subject to all kinds of different things that could happen over the course of a clinical trial. And simulation enables you to factor these things in absolutely as you expect to observe them. So a really simple example would be missing data. Now, typically people would handle, people would think about missing data, but they would just inflate their sample size calculation by a particular factor. But that, makes the assumption whether people know it or otherwise, that the missing data are equally distributed between your treatment arms.

And so with simulation, you can very easily just increase or decrease the proportion of business in each of the different arms. And then you suddenly have to, you have a little bit of control over what you are going to see, and then you can actually see what your probability of success might be of your trial in advance of starting. So whenever there is some other factor that you consider might be important for your probability of success. Simulations a really effective and actually very easy way of testing those critical or potentially critical assumptions.

[00:07:08] Alexander: So things like missing data weird distributions of C data. What are other factors that you can look into in terms of trial simulation?

[00:07:19] Kimberly: So actually I’ve just got outta a client meeting where we were discussing some of these elements and the one that they really honed in on actually was a kind of repeated measures piece where, they’re in a setting where they could take multiple measurements on patients Or just one measurement at the end, or one at the beginning and the end. And they were asking how do you build that into your design of your study? And simulation makes it really easy to answer that question where you say, is it better to have more observations? On fewer patients or actually just, is it better to have more patients and only take one observation?

And so using simulation, you can very easily build that in and actually build in more importantly, the correlation structure that you inherently have when you are taking repeated measures on an individual. And so that’s one very relevant one, which I have literally just come out of a meeting discussing.

[00:08:20] Alexander: That’s interesting. Never thought about that once, because of course, that’s a driver in terms of how much burden you put on physicians, on patients within the study. And if, yeah, any additional visit is always costly. Always, puts more or more data into the system which, yeah, statisticians generally love, but of course that all comes with the cost.

[00:08:44] Kimberly: And it’s very relevant in areas where like rare diseases where you might not have the luxury of a massive sample size, for example. You might be in a kind of constrained setting where it’s only feasible to recruit, 50 patients or something like that. So you really do need To make the most out of those patients that you recruit and really set the kind of study up for success in the measurements and the observations that you’re doing on the patients.

[00:09:11] Jamie: Other examples. So if you’ve got a primary outcome interest in a clinical trial, that might be some questionnaire that’s bounded between zero and 10. For example. And you’re expecting to see a three point difference between your treatment and your placebo group. If you were to just run a power calculation, it almost certainly wouldn’t take into consideration the fact that you can’t go above 10 or below zero in there, in that distribution. And simulation is just enabling you to understand if that traditional or simplification matters, so you can run the simulation as if you would’ve collected the data. So with all between 0 and 10. And then you can see whether or not that traditional power calculation is valid for your expected set of data. So that we could probably talk for quite a while about examples of where simulation could work or could be useful above traditional approaches.

[00:10:04] Alexander: Yeah. So probably pretty much everything that goes into your analysis piece. Yeah. Like populations visits, dosing, stopping rules, all these kind of different things all go in there. So that’s pretty cool. I very much also like that you can simulate everything about missing data, so also can much better explain what’s the different estimands will be. And that’s another really nice approach so you can actually show people, okay, this is the truth. Yeah. And if we assume there’s no other, things happening, any post baseline events, everything is fine, then you know, this is what we’ll see. And now we’ll imagine we’ll have patients that die or that, stopped you to adverse events or whatsoever. Yeah. Any other, any intercurrent events and we can put into these distributions of the intercurrent events that we also know and then can show, Okay. What will happen. And you, I think this will is a really nice way to also show what estimates are and how to work with them.

[00:11:19] Kimberly: It very much does Alexander. And I think obviously estimands is it’s, it was a hot topic a few years ago. It’s still a hot topic really, isn’t it? And I think we’re continuing to learn as we go along here, but simulation really is the only way to think about those intercurrent events, which are not missing at random. That’s really the only way you can build it into your study design. And you can like Jamie said about the missing data that might be unbalanced between different treatment groups, then you can build in exactly the same thinking With regards to intercurrent events not being balanced across different treatment groups or even the relationship between different intercurrent events and how that factors in, into the estimands framework.

[00:12:04] Alexander: Yeah.

[00:12:04] Jamie: And it’s really if you are exploring different estimands strategies in advance of the trial starting, you want to be a hundred percent convinced that your estimands strategy that you are choosing for a particular intercar event is not inflating your type one error. Yeah. And so regulators are gonna be very interested to see that in your simulation, where the truth that you have simulated is that your treatment is not working. You are only gonna get a positive result a controlled proportion of times. So I think that, yeah, it’s really It’s a really nice use of simulation to really think about these estimates in advance and what you are proposing in your protocol. I, is that a fair thing to propose? And the regulators may have a thing or two to say if that type one error is inflated under the norm.

[00:12:49] Alexander: That’s really nice aspect as well, so that you can use it for discussions with regulators. Now, if I hear all these kind of different things, yeah, you can do this and this and this, and you can, play with all these kind of different things. For me, that looks like a huge number of simulations. I need to look into kind of, if I’m just thinking about all these different factors and say, have all kind of various levels and how many years do I need to simulate to get through all of that? And I don’t know so probably will burn my placebo before it gets all the simulations back. Is that as complicated as it sounds? Jamie?

[00:13:34] Jamie: I was gonna let Kim start and I’ll add on.

[00:13:36] Kimberly: Okay, that’s fine. Obviously the more factors that you look at in study design, the more complicated things you get, which then can add time to that planning or the design stage, but I would very much argue that is time worth well spent. In the upfront planning of a trial, if you think how much trials cost, how long they run for. You think of the failure rate generally of clinical trials a little bit more upfront planning in that area. Could save you, a lot in time, money and resource.

So I think it is worth spending more time looking at all those different design options and, building out that virtual population, understanding all of the characteristics, operating characteristics of the designs. There are tools out there, there are ways that you can efficiently simulate the patient level data and the clinical trials to do that in a much more streamlined approach. And I would say as well, planning is key. But planning and being adaptable because inadvertently we will be working with clients and they think they want something, you go away and do it and then you obviously come back and present the results. And that process ignites another thought or another avenue that you want to go down. So I think it’s really important as you are building out simulation plans and when you are actually doing simulation work that you build in a certain level of flexibility in what you do. But that doesn’t mean. You still need to plan and try and discuss as much as you can upfront so that you can understand the potential avenues a piece of work might take so that you don’t spend ages building something as it were that will ultimately not get used or only used a small proportion of it.

 If you’re not a skilled coder, stimulations can take a lot of time. And you run the risk of doing it quite inefficiently. Even not necessarily coding it in a way that uses paralyzation, for disenfect that could mean the difference between, a couple of minutes and a few hours, or even up to a day of actually running the simulations, let alone the time spent coding it.

So I think there are tools out there that can really optimize that process of trying to speed things up and at the end of the day, allow statisticians to spend time on the interactions with the project teams, which is really where the value comes of interactions with them on the design piece.

[00:16:09] Alexander: Yeah. I completely know what you’re talking about. If I remember back my days at university I was, would block all PCs. That we had in the department over the weekend and then run simulations on them sometimes just to find out on Sunday evenings that something was wrong. It’s really important to have something that is sufficient and fast and they can get results much more quickly because yes, it’s innovatively an iterative process. And you always need to check with your clinicians, with your regulatory people, with your operational people, because you yeah. Optimizing step by step. Jamie, what’s your experience?

[00:16:55] Jamie: I was just gonna say I completely agree with, Kim so even if the complexities of running all these simulations seems off-putting we’ve had examples in the past where we’ve run simulations for people that thought they had 80% probability of success for their trial. The simulations were run and the most likely scenario in those simulations, but they would’ve had about 50% probability of success, which is a coin flip. And so if you don’t, if you’re not armed with that knowledge, You think you’ve got a particular probability assess and you could be wildly off. And so I think you, it’s really money well spent in my opinion.

And then the other thing I was gonna say, which is probably less relevant but will maybe resonate with some people is it was happened today. I was trying to do a power calculation, so something very simple, something not complicated at all, and I was trying to do it in a statistical package SAS and myself and my colleague just couldn’t quite figure out the right syntax to put in to get the power calculation that we wanted. And one of us thought, It was a particular syntax and the power from SAS came out as 7%, and the other one thought it was another syntax, and the power from SAS came out at 90%. And so of course that’s a real range. And so we said, let’s just for confirmation, simulate a very simple set of, Simulations that just have a a binary outcome. I think it was a crossover design. And when in about 20 minutes of generating that data, we found out that the probability of success was posted at the 7%.

So it doesn’t have to be extremely complicated. It’s really useful for doing little sense checks like that. But yeah, give you confidence that if you send something off into what could seem like a black box you, it makes sense.

[00:18:39] Alexander: Yeah. So one question is, of course when you put your assumptions into your simulations where do you get these assumptions from? Yeah.

[00:18:50] Kimberly: We get asked this a lot. So when we’re talking to clients that is, cuz obviously when you are generating subject level or patient level data you need to get data to inform that. And quite often, they’ll ask that question, where do you get your data from? And Depending on the conversation, sometimes it goes down the route of if you were doing a power calculation, where would you get your assumptions from for that? Yeah. Yeah. And I think people don’t always, Recognize that actually what you are doing in a power calculation is making assumptions. And they think of it very differently as if they were trying to generate patient level data. And actually it’s not a million miles away. Obviously if you’re generating patient level data, it would be great if you had some subject level data that you could use, some historical data that you could then pull out.

Correlations between different risk factors or correlations between different end points, that would be the kind of optimal thing. We’re not obviously always in that space where you have the luxury of having historical clinical data that is really relevant. But there are other ways that you can look through the literature and pull out summary measures and still utilize that and bring that together as part of a meta-analysis and or augment it with any patient level data that you might have.

There’s obviously ways of more structured ways like prior solicitation where you can elicit expert belief around what you believe is the true treatment effect. And you can use that in then generating subject level data sets. So there are lots of different ways, and it depends on the problem. Or what you’re trying to solve as to which route you might go down, as well as the relevant data that you might be able to get your hands on. And, there’s an explosion at the moment of utilizing real world data and real world evidence. In the design of clinical trials, external control arms as well as historical control arms are things that people are actively looking at and trying to utilize that information. But with that real world data comes other challenges with the messiness of the data, and it’s making sure that you can pull out the insights and pull out the bits that you need to then help generate that simulation data.

[00:21:13] Alexander: That’s interesting. But as you talk about real world evidence if I have historical data, yeah. So not real world evidence that I collect alongside the clinical trial, am I actually allowed to use that in the simulations for the study itself?

[00:21:29] Kimberly: But the analysis you mean? Or for the design?

[00:21:31] Alexander: No, for the simulation. for the design..

[00:21:35] Kimberly: So you, I don’t see why you wouldn’t be able to use it for the design, because the design is to inform how what you think the study is going to be like. There’s obviously different techniques if you’re then going to use that in the analysis or using it instead of a kind of internal control arm. Yeah. Or augmenting it with an internal control arm.

[00:21:56] Jamie: The only restriction on that might be if you are looking at the outcome data of your external control arm in advance of running a externally controlled clinical trial, or you you run the risk of biasing yourself or yeah, knowing too much about the data prior to, you lose your ability to pre-specify before seeing all the data. So you have to be careful about what you look at before doing a clinical trial that combines data with an external control arm. It’s, there’s a load of regulatory guidance that’s come out semi-recently on that. Yeah.

[00:22:31] Alexander: Okay. Yeah, that is I think their pre specification is maybe not the optimal way to adjust for bias, but I think sometimes people. I think pre specification is the holy grail. There’s lots of other ways you can control bias in a way. You can choose that your results were robust. Pre specification is just one. And in the area of purely world evidence analytics pre specification usually doesn’t work because you don’t know what you’ll really get.

That’s the other topic, but that is a little bit of a sidetrack. Now I’ve got all these different things. I’ve worked with Kara Clark, who I got some very easy and fast, and now I have lots of different dimensions that I looked into. How do you actually help your clients, your non-clinical, your non statistical partners to dive into these data and use these results?

[00:23:29] Jamie: We were talking about correlations earlier, a scenario where you’ve got no correlation that exists between two data points in a given individual, and a scenario where you’ve got a high correlation that exists between two data points in an individual where you can compare the two scenarios and see directly the impacts on your study probability success of factoring in that correlation into your design and how in reality, your probability of success would go well up or down, depending on the direction of the correlation should that exist in reality. And so you can answer lots of different questions, pair wise. And then ultimately drill down into the trial that gives you the highest probability of success or the design that gives you the highest probability of success under a given set of assumptions, under a given truth.

[00:24:13] Alexander: That is good. Yeah. Interactive data visualization is definitely, is a way to go. Yeah. Instead of having lots of different tables you can very easily see any patterns there, you can directly see, okay, these are all the design spaces we can ignore and we need to focus here on this red green, whatsoever kind of part in your data visualization. By the way, not use red green, because that’s bad for colorblind people. But Kim, what’s your experience?

[00:24:42] Kimberly: I think. I think gone are the days where as a statistician we are asked to go away and provide a sample size and come back and just, there you go. That’s your sample size. I think, the designing of studies is now, and quite rightly, is now much more of a whole team exercise and and obviously that’s a great thing and it’s the way it should be, but what it does mean that is that there’s a lot more pressure on a statistician as it were to be able to communicate those results. To go away and look at loads of different options and be able to then drill down, as Jamie was saying, onto particular bits and be very flexible within a meeting to do that. It does put you quite a lot in the hot seat as it were when clinicians will say, oh, what about this? Or Have you thought about that? Or, what about if we did this instead? And sometimes you’ll need to turn around and say I haven’t actually. Been able to look at that, I’d need to go away and come back to you on that or other time says, yep, if we look at this and we compare it to that.

So I think there’s a lot more free styling probably that a statistician needs to do nowadays than, 10, 20 years ago where it was people weren’t interested in that sample size section of the protocol. Now it’s so much more pivotal. And, it’s a lot more of a collaborative activity.

[00:26:05] Alexander: I love it. If you have these interactive data visualizations and you have a discussion with the team, sometimes it’s good to have to come with two statisticians is my experience. One that is working on the interactive data visualization and the other one is moderating and leading through the discussion because doing it both at the same time can be quite demanding. And I’ve worked with a similar setup in also with two opinion leaders and things like this and that can be quite nice. It actually can also be quite impressive to people when they see wow, you did this and you can answer this question and so on.

So that’s pretty cool. So, How many iterations do you usually go through until you be into the end of your design space?

[00:26:55] Jamie: Do you know how long a piece of string is as the answer? They, if it’s a simple simulation, it can be a one or two iterations, but when you start to get complexity within your simulation. And, each different assumption is scrutinized and the impact of each assumption is really realized. These things can run and run, but we tend to say two or three iterations should be enough to give people a good idea of where their affordability of success would be. But it really depends on, so I’ve recently done a set of simulations and this is one area that we didn’t mention earlier actually, that is a big advantage of simulating this is in adaptive designs. So the client was trying to understand what the optimal futility rule should be. An interim analysis and an optimal futility rule is one that stops the style the trial as many times as possible if the treatment is not working, but doesn’t stop the trial as many times as possible if the treatment is working.

I’m looking like it’s in line with your assumptions that you set out at the start of the study. And of course you can come up with infinite number of different futility rules, and each of them will have slightly different operating characteristics. And I feel like two or three iterations in those conversations may go on for some iterations more. But you have other examples where yeah, one set of simulations answers the question broadly and they don’t need much more.

[00:28:16] Alexander: Awesome. Yeah. Now, Kim?

[00:28:19] Kimberly: And yeah. And to add to that, I suppose it, it does depend how many times the goalposts are moved. We’ve all been there where you have a certain discussion with your project team. You think you’ve got to the design that you’re gonna use, and then there’s either some external competitor information or maybe some internal resource or other constraints or other factors that come into play. And then the project team have to rethink things. It how long does a piece of string and then also how many times the goalpost move, is we need something to think about there as well.

[00:28:57] Alexander: No need to do it. And half the time we now have 10 millions less in budget we now have whatsoever. Yeah,

[00:29:04] Kimberly: I think we’ve all been there. Yeah.

[00:29:07] Alexander: But sure, still I wanna turn around to some things that is quite helpful for the listeners. And that is a course that Jamie, Kim and I are now rolling out it’s called Mastering Study design and Strategy with Simulations. And the goal for that goal, of course, is to get you really up to speed with simulations and make sure that you feel confident about it, that you have, all the different aspects of it covered. Kim, Jamie, can you tell a little bit more about this course?

[00:29:47] Kimberly: Yep. So the course is gonna cover the standard kind of bits around what is simulation, when should I use it and how, in the sense of, those steps of a simulation plan. But it’s also gonna be fueled with lots of examples cuz I think examples are the easiest way to learn new topics or also really understand things that you might have a kind of high level knowledge around.

So there’ll be lots of examples going into kind of details of where simulation has really made an impact to help you think. A little bit more about when you are in certain situations, how you could use simulation in clinical trial design. It’s also gonna cover a little bit about regulators, cuz ultimately they are one of our key stakeholders when it comes to designing clinical trials, especially obviously later phase ones.

So it’s really important to understand what they’re thinking and where they see opportunities and what they’re willing to accept in that space, as it were. So they’re the kind of main topics. And then we’ll also cover a little bit what we’re talking about there, around how do you actually communicate the results and how do you interact with your team to make sure that you are really being effective in getting across what is, a lot of work and sometimes some quite technical, statistical aspects to non statisticians.

[00:31:13] Alexander: Yeah. And what I really love about this course it is from people that actually do this day in, day out. Yeah. You just heard from Jamie and Kim. Lots of different examples, lots of use case stories. Lots of real world experience. Yeah. And I think that makes it so valuable. It’s not, some kind of ivory avatar research on simulations is practical things that you will need for the design of your studies.

And as you have heard, it’s if you’re working in the clinical trial space, it is one of the aspects that you absolutely must know about. Yeah. I think there’s no idea, there’s no way around it to be able to use simulations for trial design. And you know what? When we need to focus on something, What are the three main, most important parts within clinical studies?

Design, design, and Check out this program. This this course you will find the link into show notes will actually run this as a life course for the first time. We will have two recordings and two kind of webinars during July of 2023. If you are listening to this before July of 2023, you can still enroll in the course and get it live delivered.

Of course, that comes with the benefit of you. We will be asking questions and I think it’s also always a little bit more engaging. If you listen to it only afterwards, then check out the Effective Statistician Academy, whether you can still find it there. I’m still finding it there because, we have I know this kind of podcast episodes get listened to years and years in the future. Maybe we have something better by the time you listened to this, maybe in 2027 or whatsoever. Any final words, Jamie, Kim, from your end?

[00:33:22] Jamie: Not from me. Looking forward to seeing whoever wants to come to the course. See you there.

[00:33:28] Kimberly: No, just actually there was one thing when I was asking people who I know worked in simulations around when would you simulate, and I still love this one the best. It was like always I simulate all the time. And it comes back to your point, Alexander, the it really is a, an important skill for statisticians to have. So yeah, people generally should be always simulating. It’s great.

[00:33:52] Alexander: Yep. Always keep simulating. Love it. So, see you on the program in the course, and Jamie, Kim, great to have you on the show.

[00:34:03] Jamie: Thank you very much.

[00:34:04] Kimberly: Thank you.