The past, present and future of estimands!

The past, present, and future of estimands! Interview with Chrissie Fletcher and Mouna Akacha

00:00
You are listening to the Effective Statistician, episode number 38, the past, present and future of estimates, an interview with Chrissie Fletcher and Mona Akatscha.

00:19
Welcome to the Effective Statistician with Alexander Schacht and Benjamin Piske, the weekly podcast for statisticians in the health sector designed to improve your leadership skills, widen your business acumen and enhance your efficiency.

00:34
We are creating an online course for improving your leadership skills as a statistician even if you have no direct reports. So register now your interest on the page see effe slash course. In today’s episode we’ll talk about estimates.

00:55
why it’s important for us, why the concept itself is important, and what are the implications. We’ll talk about the terminology of the estimate framework and also explore how it relates to missing data terminology.

01:11
The guests are absolutely fantastic. They are key players in this field. And if you go to conferences, you will see Chrissie’s and Mona’s name very, very often presenting on this topic. Um, very, very nice interview guests. So stay tuned for this amazing episode. This podcast is created in association with PSI, a global member organization dedicated to leading and promoting best practice and industry initiatives.

01:40
Join PSI today to further develop your statistical capabilities with access to special interest groups, the video on demand content library, free registration to upcoming PSI events and much much more. Visit the PSI website at psieweb.org to learn more about upcoming PSI activities and become a PSI member today.

02:13
Welcome to another episode of the Effective Statistician. This is Alexander Schacht, and today I’m, without my co-host, Benjamin, because his child is sick. So I’m alone here from my side, but I have two very, very nice guests. Muna from Novartis and Chrissy from Amgen. Hello together. Hi there. Hello. Okay, very good.

02:43
Maybe we start with a little bit of an introduction first of what have brought you to this topic, what you’re currently working on and what’s your kind of feeling about estimates. So, Munda, maybe we can start with you first. Okay. So, thank you first of all for having me here today. So, my name is Munda, as you’ve already said. I’m…

03:11
I’m working in the statistical methodology group at Novartis for around eight years now. And regarding your question, what brought me to STMAT, I think this is already going back like four to five years by now. So when the concept paper was published 2014, I remember that my manager came to me and said, you are our expert on missing data. So my PhD was in missing data and somewhat of an expertise in that area.

03:39
And he essentially said, so you’re an expert in missing data, so how about you try to collect the comments on this concept paper and wrap your head around it. And well, admittedly, that was the very first time I heard about the word estimate per se. But ever since, I really enjoyed the topic. I think very quickly we figured out that it’s much broader than missing data. So it was brought to me as such, but I think we all appreciate by now that it’s

04:08
broader than that. And yeah, I’ve been working on some aspects around it ever since, currently mostly around recurrent events and time to event estimates for some consideration around that type of endpoint. Okay. So maybe I stop here. Okay, very good. So Chrissie, what’s your involvement with that?

04:31
Hi, so I’m Chrissie Fletcher and I work at Amgen. I’m a TA Head in Biostatistics for non-oncology disease areas. I got involved in estimates when we had the draft concept and a few of us reached out and discussed with Rob Hemmings as the lead for the draft concept, what were some of the challenges he was introducing or thinking about before he was submitting it to ICH. So that was through the FSBI PSI Regulatory Committee.

05:00
So we got involved quite early into some of the thinking processes behind the concept paper. And as I’m a member of the FPEER Clinical Development Expert Group, I was able to get involved and become one of the FPEER representatives on the ICH E9 Working Group. And it’s certainly been a journey that has been very interesting and challenging and obviously still continues to this day.

05:29
Okay, very good. Mona, you mentioned already said, you were put on this problem because of your experience with messing data, but you mentioned that it’s much broader than that. So, I know that this is the origin, but why has it developed into something that’s as much broader than handling missing data?

05:56
That’s a good question actually. I think from the background that I’ve heard from some of our regulatory colleagues, but also internally of people who have been in industry for a bit longer, I think a lot of the challenges that motivated the concept paper really stemmed from missing data problems. So sometimes you would have, let’s say, submissions where we would choose a certain way of handling missing data. And then sometimes…

06:25
the regulatory agencies would maybe disagree on that approach or would prefer an alternative approach. And very often these discussions would happen among statisticians, let’s say. And in all that discussion, sometimes I believe it was maybe forgotten that we sometimes don’t really ask ourselves very carefully why are those data missing.

06:51
is the fact that they are missing themselves already telling me something about either efficacy or safety of the drug. So just as an example, so when I joined from an sort of academic head, let’s say, when I joined the industry and I consulted on a couple of projects, it wouldn’t even cross my mind to ask, are these data really missing or are we just considering them not useful for this analysis? So take a diabetes drug, for example.

07:20
where patients are observed over several weeks. And some of them then start taking rescue medication when the HbA1c values are not well controlled. And I think the practice in the past was to disregard the data after intake of rescue, although we would sometimes collect that, and then do something like less observation period forward. So I would only get into the problem once.

07:48
people talk about missing data and so on. But I think with this whole estimate discussion is becoming more clear that missing data is just one facet of that. But actually all types of data that are, let’s say, painted by either other medication or that where the patients are not fully following the protocol while they are dying, let’s say, so the data doesn’t exist, it’s not missing, it simply doesn’t exist. So all these things were sometimes

08:18
put into one bucket, the missing data bucket, and we statisticians deal with it. And I think this whole estimate discussion has helped us now to take it back out of the bucket, let’s say, to talk about it a bit more clearly. And sometimes the mere fact that a patient is not willing to take a drug for 12 weeks or 16 weeks is the key outcome itself. So the information is not missing, although the symptom score itself may be missing. Does that help clarifying?

08:48
Chrissie, do you want to add to that? Yeah, I mean, I think if you think of ICHE9, the key principle in ICHE9 is intention to treat. And those of us who understand E9 will say that intention to treat is all patients randomized and followed through to the outcome. And I think we seem to have maybe lost a little bit of that over many years in that, again, the missing data report from the NRC back in 2010

09:16
really emphasised the need to follow through on patients whether they were adhering or not to the treatment. And as Munna says, we sort of handled issues with the data as though it was a statistical issue only. So estimates is really taking us all back to square one, is what is it we’re trying to estimate? And that’s the core in the estimate. And I think it’s making sure that if you do want to take a treatment policy in perspective and follow patients after they’ve been randomised,

09:46
and we need to make sure we try to minimise truly missing data. But perhaps there are other clinical questions where we do actually want to know what happens to the treatment effect for patients who can tolerate the treatment or do adhere to treatment. That’s a different estimate of our treatment effect. So I think this new framework is giving us more clarity about what it is we’re trying to address, what’s the key question in a study. And this isn’t just a statistical issue, and we do need to think of it as a…

10:15
a study objective, what is it we’re trying to estimate? So if I understand it correctly, on the surface, it was some missing data problem. But actually, by digging deeper into the actual problem, we identified that we don’t have a missing data problem. We also have a missing data problem. But actually, our problem is that we weren’t clear on what we actually wanted to measure and what we actually, which

10:44
objectives we actually wanted to answer. Because I think commonly we wrote just objectives like we want to understand the treatment effect of drug A versus drug B in this patient population by that measure or something like this. So we weren’t really kind of clear on what treatment effect actually means. Yeah. So if I can start, I think it’s even got a bit worse than that. I

11:14
Intention to treat in the protocol, but we actually are not taking an attention to treat approach So we have wanted to say we’re looking at the effect of a treatment We’re going to be looking at the full analysis set but in our analysis methods We actually haven’t been applying the intention to treat principle So what we have been calling an intention to treat estimate is actually not intention to treat estimate because of the way we’ve handled data Not just missing data, but how we’ve handled patients who for example took rescue

11:44
or what happens to patients who discontinue. And all these different events matter in terms of what we’re estimating. Okay. You mentioned already ICHE9 and that actually currently needs an update and this update is currently working on and hopefully becoming live next year. There’s currently the…

12:14
the time where the responses and comments to this draft guideline has closed and currently people are working on these comments. So Chrissie, can you explain shortly what ICHE 9 is about and why it needs an update? So ICHE 9 is introducing a new framework for how we design, conduct and analyse clinical trials.

12:41
And it’s a logical framework that hopes to really bring together the study objectives to define what it is we’re estimating, to make sure that the planned analysis methods are aligned to that what it is we’re estimating. And we also have aligned sensitivity analyses. So the E9 addendum, which is an update to E9, is going to sit as a separate document to E9.

13:08
is really trying to realign ourselves to make sure that if we are, we are going to be clear in terms of what we’re estimating. But also a big part of the end is redefine what we mean by sensitivity analysis. So ICHE9 currently says that we might use different analysis sets to test the robustness of our trial conclusions. But if we take a different, if we, if we

13:37
that in the new framework is going to be a different treatment effect of interest, a different estimate. So we need to make sure that whatever we’re doing for sensitivity analyses are now going to be aligned to our estimates of interest. Yeah, and just for the listeners who are not used to what ICH actually means, that is a regulatory consortium across the world that seeks to harmonize how

14:06
Regulatory submissions are done across the world, so Japan, US, Europe, but also lots of other regulators involved in that. And E9 specifically speaks about lots of these statistical problems that we have.

14:27
So we talked already a lot about estimates. And I think the origin of this, as we talked about, is this missing data. And we already talked about treatment policy and maybe other aspects of that.

14:48
And if you look into the updates, there’s, I think, currently five different approaches mentioned to these intercurrent events, which could be drop-offs or could be rescue medication or these kind of things. But whenever we talk about statisticians as about estimates, it seems to be very focused on this intercurrent event. But that’s not the only key aspect of the estimate, isn’t it?

15:20
So the intercurrent event is one consideration. And we need to understand what intercurrent events are of interest for our study. But yes, that’s one of at least four attributes that are going to be really important for an S demand. The other attributes include the population. The population means the target population that our label is hoping to achieve. We’re also interested in the variable or the endpoint of interest. What is it we’re going to be basing the treatment effect of interest on?

15:50
We’re also concerned also with the, well, how are we going to summarize the treatment effect? And is that going to be some kind, for example, a difference in means between treatment groups or treatment regimens? So all in all, there’s a number of different attributes to an Estimand, including the Intercurrent Events. And together, these will make an Estimand description. But this Estimand description will hopefully enable all stakeholders, including regulators and

16:17
payers and HJ agencies to understand what it is we’re estimating and how we’re handling events that happen in trials in terms of that and how that leads to an estimate of our treatment effect. So let’s go a little bit deeper into that. If we start with population, so you mentioned that the kind of label population, but I guess, you know, within this label population, there could be also

16:43
different subpopulation for which you would have different estimates. So you could have maybe one where you look for patients with a specific comorbidity or patients with a specific kind of severity of a certain aspect of that disease or things like that. So that is always one of the key elements. Yeah. Yeah. Yeah, I think that’s an important one. Yeah.

17:13
And if we think about it, this population, is that, I think we often very much think from the inner exclusion criteria of the study, that is it the study population, or are we more talking about the population in which these patients will be prescribed?

17:39
It is the latter. We’re looking at the target population, what are the patients we hope to treat, and we hope that our study is representative of our target population through the inclusion exclusion criteria. And of course, once we’ve recruited all our patients, we need to demonstrate that that trial population is actually representative of our target population. So if you don’t have that, there will be some questions that you’ll need to address.

18:08
So imagine you have, let’s say, a US population. This is maybe one target population. And then for the next regulator, it’s maybe, let’s say, Korea. And in Korea, the patients are, let’s say, on average, 20 kilos lighter than in the US. And maybe weight actually plays an important role.

18:35
in your treatment effect modifier. So what you need to then kind of have different estimates for these different populations.

18:48
I guess you could, right? I mean, if you were looking at a subgroup analysis, they are focusing more on, let’s say, Asian patients, then that is certainly still in the remit of what we do. And then actually by defining the population accordingly, which as you said, may be implemented in terms of the analysis of ocean statistics through a model that includes rate or another type of surrogate for that.

19:18
Personally think that that’s still certainly possible. I don’t know, Chrissie, would you agree? Yeah, it comes back to the design. I mean, we also now have ICHE 17, which is a multi-regional clinical trial guideline. So again, if your null hypothesis is that you have a consistent treatment effect across all the regions you’re doing the study, then you’re assuming a common treatment effect. But if you don’t have that, and weight is gonna be an issue.

19:44
And if you haven’t maybe stratified for weight, then you may have some additional analyses to understand whether you do have a consistent effect between different regions. So it depends on how you design the study, whether that subpopulation is something that you do plan for. You may have a separate estimate specifically to look at Korea, for example, if that was of a question that you wanted your study to address. But the main thing is that you specify that upfront.

20:12
and you don’t wait for the planned methods of analysis to say that you actually want to look at a subgroup in South Korea, whatever it is, you need to have it upfront and aligned to the study objectives. Okay, okay, very good. Then the second point is we talk about the endpoint. Let’s say you have a diabetes study, as you just mentioned, and we are looking into HbA1c.

20:38
Would then we just say HBA1C is the endpoint, or how would that look like?

20:46
So I guess it would depend on your aim, let’s say. So let’s look at APO-NC at the endpoint, together with the fact that many patients may take rescue medication. So some people may argue the mere fact that you require rescue medication is a sign for lack of efficacy. So it is a failure to the initially randomized treatment. And you could even think about defining the composite endpoint.

21:14
where you require a certain improvement in terms of HbA1c. And in addition, you have to stay on treatment at least for a certain duration. At the end, diabetes is a chronic disease. You want patients to be on treatment for far longer than 12, 24 weeks. And I think this whole estimate discussion, at least in my mind, has also helped in.

21:40
phrasing the endpoint in a more clear and transparent way. So sometimes, for example, you would read our endpoint is HbA1c, and then in the missing data handling section, you read we do non-responder imputation for patients that take rescue. So really the endpoint is a composite endpoint where you require a certain improvement on HbA1c and you have to say on treatment. But in the past, sometimes by saying these are in separate section of the protocol,

22:09
you actually have to look at different places to really know how to interpret the treatment effect estimate that you get at the end. So the endpoint also is affected by intercurrent events in certain settings.

22:25
Okay, so these four different aspects that we are currently talking about, they are somehow related to each other. They can’t kind of, you can, very often if you change one, you also change something else. Yeah. Okay. So coming back maybe briefly to a point that you said earlier, that a lot of statisticians when they talk about estimates, mostly focused on intercurrent events. So I think in a way that’s almost a good thing because really that…

22:53
I think a lot of the discussions around STMEN boil down to how do we account or how do we capture these intercurrent events, which themselves show us a different perspective on the treatment effect. And they can impact the population, they can impact the end point, they can impact the summary measure or the inter… like how to account for intercurrent events themselves. So given that in the past we have already focused quite a bit of…

23:22
that’s normal, or it becomes normal to us to focus on population, endpoint and summary measure, the focus on inter-current events is actually, I guess, the good new thing with this estimate framework.