If you ask someone within the pharma world of statisticians on what is the hottest topic, you most likely will just get: Estimands!

Many scientific events, conference sessions, publications and presentations are organized around this topic. Thus, I’m super happy, that we have Mouna and Chrissie on the podcast as they have presented about this topic at various occasions.

In this episode, you will learn about

  • the importance of the concept itself and it’s implications
  • the terminology of the estimand framework
  • and how it relates to missing data terminology

The episode covers the following questions and gives some examples on the different parts of the framework:

  • What is the history of the estimands and how is it related to missing data? 
  • What is ICH E9 and why does it need an update now?

  • What are the different parts of an estimand and how do they relate to each other?
  • Is the estimand framework only applicable to clinical trials in the regulatory setting or is this something to be considered beyond the regulatory approval of new medications?

We also cover the PSI conference 2019, which surely will include sessions about estimands as well as the upcoming 1-day event organized by PSI: New Emerging Topics around Estimands and ICH Addendum. Click here to find more about this event and the registration page.

The ICH E9 (R1) addendum on estimands and sensitivity 5 analysis in clinical trials to the guideline on statistical 6 principles for clinical trials can be found here.

A nice interview with Mouna around estimands as well as a great list of resources was published by Cytel here.

Mouna Akacha

Mouna Akacha is a consultant in the Statistical Methodology Group of Novartis Pharma AG, based in Basel, Switzerland.

In this role she provides internal advice for clinical projects across all development phases and therapeutic areas. One key aspect of her work is to make complex statistical problems and methods accessible to a wider audience. In addition, she is engaged in developing and implementing innovative statistical methods for clinical projects. Her role also includes training of internal statisticians and collaborations with external statistical centers and researchers.

Mouna has a wide range of research interests including topics on missing data, longitudinal data, recurrent event data and dose-finding studies. Before joining Novartis, Mouna studied mathematics at the University of Oldenburg in Germany and holds a PhD in statistics from the University of Warwick in the UK.

Chrisse Fletcher

Chrissie is a Regional Head in Global Biostatistical Science at Amgen and she leads a Health Technology Assessment (HTA) Biostatistics group. Chrissie is also leading the development of Amgen policies and processes for sharing clinical trial data with external researchers. Chrissie has worked in the Pharmaceutical Industry for over 25 years and has experience of developing and commercialising new medicines from a variety of therapeutic areas across all phases of clinical development.

Chrissie is currently the Vice-President of the European Federation of Statisticians in the Pharmaceutical Industry (EFSPI) and the EFSPI Communications Officer; a member of the Statisticians in the Pharmaceutical Industry (PSI)/EFSPI Regulatory Committee; chair of the PSI/EFSPI HTA Special Interest Group (SIG); member of the Integrated Data Analysis (IDA) SIG and member of the EFSPI data sharing working group. Chrissie is a member of the Clinical Development Expert Group for the European Federation of Pharmaceutical Industries and Associations (EFPIA), and she is one of 2 EFPIA representatives on the ICH E9 Revision 1 working group that is developing an addendum to E9 on estimands and sensitivity analyses. Chrissie is the Industry co-chair of the Innovative Medicines Initiative (IMI) ‘GetReal’ initiative Work Package 4 which is developing mathematical models and analytic tools for synthesising clinical evidence and predicting effectiveness of treatments based on data available from clinical trials and observational research databases/studies.

Chrissie is a Chartered Statistician and Chartered Scientist of the Royal Statistical Society (RSS). Chrissie has an MSc in Applied Statistics and a BSc (Hons) in Statistics with Management Science Techniques.

Transcript

The past, present and future of estimands! Interview with Chrissie Fletcher and Mouna Akacha

00:00
You are listening to the Effective Statistician, episode number 38, the past, present and future of estimates, an interview with Chrissie Fletcher and Mona Akatscha.

00:19
Welcome to the Effective Statistician with Alexander Schacht and Benjamin Piske, the weekly podcast for statisticians in the health sector designed to improve your leadership skills, widen your business acumen and enhance your efficiency.

00:34
We are creating an online course for improving your leadership skills as a statistician even if you have no direct reports. So register now your interest on the page see effe slash course. In today’s episode we’ll talk about estimates.

00:55
why it’s important for us, why the concept itself is important, and what are the implications. We’ll talk about the terminology of the estimate framework and also explore how it relates to missing data terminology.

01:11
The guests are absolutely fantastic. They are key players in this field. And if you go to conferences, you will see Chrissie’s and Mona’s name very, very often presenting on this topic. Um, very, very nice interview guests. So stay tuned for this amazing episode. This podcast is created in association with PSI, a global member organization dedicated to leading and promoting best practice and industry initiatives.

01:40
Join PSI today to further develop your statistical capabilities with access to special interest groups, the video on demand content library, free registration to upcoming PSI events and much much more. Visit the PSI website at psieweb.org to learn more about upcoming PSI activities and become a PSI member today.

02:13
Welcome to another episode of the Effective Statistician. This is Alexander Schacht, and today I’m, without my co-host, Benjamin, because his child is sick. So I’m alone here from my side, but I have two very, very nice guests. Muna from Novartis and Chrissy from Amgen. Hello together. Hi there. Hello. Okay, very good.

02:43
Maybe we start with a little bit of an introduction first of what have brought you to this topic, what you’re currently working on and what’s your kind of feeling about estimates. So, Munda, maybe we can start with you first. Okay. So, thank you first of all for having me here today. So, my name is Munda, as you’ve already said. I’m…

03:11
I’m working in the statistical methodology group at Novartis for around eight years now. And regarding your question, what brought me to STMAT, I think this is already going back like four to five years by now. So when the concept paper was published 2014, I remember that my manager came to me and said, you are our expert on missing data. So my PhD was in missing data and somewhat of an expertise in that area.

03:39
And he essentially said, so you’re an expert in missing data, so how about you try to collect the comments on this concept paper and wrap your head around it. And well, admittedly, that was the very first time I heard about the word estimate per se. But ever since, I really enjoyed the topic. I think very quickly we figured out that it’s much broader than missing data. So it was brought to me as such, but I think we all appreciate by now that it’s

04:08
broader than that. And yeah, I’ve been working on some aspects around it ever since, currently mostly around recurrent events and time to event estimates for some consideration around that type of endpoint. Okay. So maybe I stop here. Okay, very good. So Chrissie, what’s your involvement with that?

04:31
Hi, so I’m Chrissie Fletcher and I work at Amgen. I’m a TA Head in Biostatistics for non-oncology disease areas. I got involved in estimates when we had the draft concept and a few of us reached out and discussed with Rob Hemmings as the lead for the draft concept, what were some of the challenges he was introducing or thinking about before he was submitting it to ICH. So that was through the FSBI PSI Regulatory Committee.

05:00
So we got involved quite early into some of the thinking processes behind the concept paper. And as I’m a member of the FPEER Clinical Development Expert Group, I was able to get involved and become one of the FPEER representatives on the ICH E9 Working Group. And it’s certainly been a journey that has been very interesting and challenging and obviously still continues to this day.

05:29
Okay, very good. Mona, you mentioned already said, you were put on this problem because of your experience with messing data, but you mentioned that it’s much broader than that. So, I know that this is the origin, but why has it developed into something that’s as much broader than handling missing data?

05:56
That’s a good question actually. I think from the background that I’ve heard from some of our regulatory colleagues, but also internally of people who have been in industry for a bit longer, I think a lot of the challenges that motivated the concept paper really stemmed from missing data problems. So sometimes you would have, let’s say, submissions where we would choose a certain way of handling missing data. And then sometimes…

06:25
the regulatory agencies would maybe disagree on that approach or would prefer an alternative approach. And very often these discussions would happen among statisticians, let’s say. And in all that discussion, sometimes I believe it was maybe forgotten that we sometimes don’t really ask ourselves very carefully why are those data missing.

06:51
is the fact that they are missing themselves already telling me something about either efficacy or safety of the drug. So just as an example, so when I joined from an sort of academic head, let’s say, when I joined the industry and I consulted on a couple of projects, it wouldn’t even cross my mind to ask, are these data really missing or are we just considering them not useful for this analysis? So take a diabetes drug, for example.

07:20
where patients are observed over several weeks. And some of them then start taking rescue medication when the HbA1c values are not well controlled. And I think the practice in the past was to disregard the data after intake of rescue, although we would sometimes collect that, and then do something like less observation period forward. So I would only get into the problem once.

07:48
people talk about missing data and so on. But I think with this whole estimate discussion is becoming more clear that missing data is just one facet of that. But actually all types of data that are, let’s say, painted by either other medication or that where the patients are not fully following the protocol while they are dying, let’s say, so the data doesn’t exist, it’s not missing, it simply doesn’t exist. So all these things were sometimes

08:18
put into one bucket, the missing data bucket, and we statisticians deal with it. And I think this whole estimate discussion has helped us now to take it back out of the bucket, let’s say, to talk about it a bit more clearly. And sometimes the mere fact that a patient is not willing to take a drug for 12 weeks or 16 weeks is the key outcome itself. So the information is not missing, although the symptom score itself may be missing. Does that help clarifying?

08:48
Chrissie, do you want to add to that? Yeah, I mean, I think if you think of ICHE9, the key principle in ICHE9 is intention to treat. And those of us who understand E9 will say that intention to treat is all patients randomized and followed through to the outcome. And I think we seem to have maybe lost a little bit of that over many years in that, again, the missing data report from the NRC back in 2010

09:16
really emphasised the need to follow through on patients whether they were adhering or not to the treatment. And as Munna says, we sort of handled issues with the data as though it was a statistical issue only. So estimates is really taking us all back to square one, is what is it we’re trying to estimate? And that’s the core in the estimate. And I think it’s making sure that if you do want to take a treatment policy in perspective and follow patients after they’ve been randomised,

09:46
and we need to make sure we try to minimise truly missing data. But perhaps there are other clinical questions where we do actually want to know what happens to the treatment effect for patients who can tolerate the treatment or do adhere to treatment. That’s a different estimate of our treatment effect. So I think this new framework is giving us more clarity about what it is we’re trying to address, what’s the key question in a study. And this isn’t just a statistical issue, and we do need to think of it as a…

10:15
a study objective, what is it we’re trying to estimate? So if I understand it correctly, on the surface, it was some missing data problem. But actually, by digging deeper into the actual problem, we identified that we don’t have a missing data problem. We also have a missing data problem. But actually, our problem is that we weren’t clear on what we actually wanted to measure and what we actually, which

10:44
objectives we actually wanted to answer. Because I think commonly we wrote just objectives like we want to understand the treatment effect of drug A versus drug B in this patient population by that measure or something like this. So we weren’t really kind of clear on what treatment effect actually means. Yeah. So if I can start, I think it’s even got a bit worse than that. I

11:14
Intention to treat in the protocol, but we actually are not taking an attention to treat approach So we have wanted to say we’re looking at the effect of a treatment We’re going to be looking at the full analysis set but in our analysis methods We actually haven’t been applying the intention to treat principle So what we have been calling an intention to treat estimate is actually not intention to treat estimate because of the way we’ve handled data Not just missing data, but how we’ve handled patients who for example took rescue

11:44
or what happens to patients who discontinue. And all these different events matter in terms of what we’re estimating. Okay. You mentioned already ICHE9 and that actually currently needs an update and this update is currently working on and hopefully becoming live next year. There’s currently the…

12:14
the time where the responses and comments to this draft guideline has closed and currently people are working on these comments. So Chrissie, can you explain shortly what ICHE 9 is about and why it needs an update? So ICHE 9 is introducing a new framework for how we design, conduct and analyse clinical trials.

12:41
And it’s a logical framework that hopes to really bring together the study objectives to define what it is we’re estimating, to make sure that the planned analysis methods are aligned to that what it is we’re estimating. And we also have aligned sensitivity analyses. So the E9 addendum, which is an update to E9, is going to sit as a separate document to E9.

13:08
is really trying to realign ourselves to make sure that if we are, we are going to be clear in terms of what we’re estimating. But also a big part of the end is redefine what we mean by sensitivity analysis. So ICHE9 currently says that we might use different analysis sets to test the robustness of our trial conclusions. But if we take a different, if we, if we

13:37
that in the new framework is going to be a different treatment effect of interest, a different estimate. So we need to make sure that whatever we’re doing for sensitivity analyses are now going to be aligned to our estimates of interest. Yeah, and just for the listeners who are not used to what ICH actually means, that is a regulatory consortium across the world that seeks to harmonize how

14:06
Regulatory submissions are done across the world, so Japan, US, Europe, but also lots of other regulators involved in that. And E9 specifically speaks about lots of these statistical problems that we have.

14:27
So we talked already a lot about estimates. And I think the origin of this, as we talked about, is this missing data. And we already talked about treatment policy and maybe other aspects of that.

14:48
And if you look into the updates, there’s, I think, currently five different approaches mentioned to these intercurrent events, which could be drop-offs or could be rescue medication or these kind of things. But whenever we talk about statisticians as about estimates, it seems to be very focused on this intercurrent event. But that’s not the only key aspect of the estimate, isn’t it?

15:20
So the intercurrent event is one consideration. And we need to understand what intercurrent events are of interest for our study. But yes, that’s one of at least four attributes that are going to be really important for an S demand. The other attributes include the population. The population means the target population that our label is hoping to achieve. We’re also interested in the variable or the endpoint of interest. What is it we’re going to be basing the treatment effect of interest on?

15:50
We’re also concerned also with the, well, how are we going to summarize the treatment effect? And is that going to be some kind, for example, a difference in means between treatment groups or treatment regimens? So all in all, there’s a number of different attributes to an Estimand, including the Intercurrent Events. And together, these will make an Estimand description. But this Estimand description will hopefully enable all stakeholders, including regulators and

16:17
payers and HJ agencies to understand what it is we’re estimating and how we’re handling events that happen in trials in terms of that and how that leads to an estimate of our treatment effect. So let’s go a little bit deeper into that. If we start with population, so you mentioned that the kind of label population, but I guess, you know, within this label population, there could be also

16:43
different subpopulation for which you would have different estimates. So you could have maybe one where you look for patients with a specific comorbidity or patients with a specific kind of severity of a certain aspect of that disease or things like that. So that is always one of the key elements. Yeah. Yeah. Yeah, I think that’s an important one. Yeah.

17:13
And if we think about it, this population, is that, I think we often very much think from the inner exclusion criteria of the study, that is it the study population, or are we more talking about the population in which these patients will be prescribed?

17:39
It is the latter. We’re looking at the target population, what are the patients we hope to treat, and we hope that our study is representative of our target population through the inclusion exclusion criteria. And of course, once we’ve recruited all our patients, we need to demonstrate that that trial population is actually representative of our target population. So if you don’t have that, there will be some questions that you’ll need to address.

18:08
So imagine you have, let’s say, a US population. This is maybe one target population. And then for the next regulator, it’s maybe, let’s say, Korea. And in Korea, the patients are, let’s say, on average, 20 kilos lighter than in the US. And maybe weight actually plays an important role.

18:35
in your treatment effect modifier. So what you need to then kind of have different estimates for these different populations.

18:48
I guess you could, right? I mean, if you were looking at a subgroup analysis, they are focusing more on, let’s say, Asian patients, then that is certainly still in the remit of what we do. And then actually by defining the population accordingly, which as you said, may be implemented in terms of the analysis of ocean statistics through a model that includes rate or another type of surrogate for that.

19:18
Personally think that that’s still certainly possible. I don’t know, Chrissie, would you agree? Yeah, it comes back to the design. I mean, we also now have ICHE 17, which is a multi-regional clinical trial guideline. So again, if your null hypothesis is that you have a consistent treatment effect across all the regions you’re doing the study, then you’re assuming a common treatment effect. But if you don’t have that, and weight is gonna be an issue.

19:44
And if you haven’t maybe stratified for weight, then you may have some additional analyses to understand whether you do have a consistent effect between different regions. So it depends on how you design the study, whether that subpopulation is something that you do plan for. You may have a separate estimate specifically to look at Korea, for example, if that was of a question that you wanted your study to address. But the main thing is that you specify that upfront.

20:12
and you don’t wait for the planned methods of analysis to say that you actually want to look at a subgroup in South Korea, whatever it is, you need to have it upfront and aligned to the study objectives. Okay, okay, very good. Then the second point is we talk about the endpoint. Let’s say you have a diabetes study, as you just mentioned, and we are looking into HbA1c.

20:38
Would then we just say HBA1C is the endpoint, or how would that look like?

20:46
So I guess it would depend on your aim, let’s say. So let’s look at APO-NC at the endpoint, together with the fact that many patients may take rescue medication. So some people may argue the mere fact that you require rescue medication is a sign for lack of efficacy. So it is a failure to the initially randomized treatment. And you could even think about defining the composite endpoint.

21:14
where you require a certain improvement in terms of HbA1c. And in addition, you have to stay on treatment at least for a certain duration. At the end, diabetes is a chronic disease. You want patients to be on treatment for far longer than 12, 24 weeks. And I think this whole estimate discussion, at least in my mind, has also helped in.

21:40
phrasing the endpoint in a more clear and transparent way. So sometimes, for example, you would read our endpoint is HbA1c, and then in the missing data handling section, you read we do non-responder imputation for patients that take rescue. So really the endpoint is a composite endpoint where you require a certain improvement on HbA1c and you have to say on treatment. But in the past, sometimes by saying these are in separate section of the protocol,

22:09
you actually have to look at different places to really know how to interpret the treatment effect estimate that you get at the end. So the endpoint also is affected by intercurrent events in certain settings.

22:25
Okay, so these four different aspects that we are currently talking about, they are somehow related to each other. They can’t kind of, you can, very often if you change one, you also change something else. Yeah. Okay. So coming back maybe briefly to a point that you said earlier, that a lot of statisticians when they talk about estimates, mostly focused on intercurrent events. So I think in a way that’s almost a good thing because really that…

22:53
I think a lot of the discussions around STMEN boil down to how do we account or how do we capture these intercurrent events, which themselves show us a different perspective on the treatment effect. And they can impact the population, they can impact the end point, they can impact the summary measure or the inter… like how to account for intercurrent events themselves. So given that in the past we have already focused quite a bit of…

23:22
that’s normal, or it becomes normal to us to focus on population, endpoint and summary measure, the focus on inter-current events is actually, I guess, the good new thing with this estimate framework.

23:39
Prisci, do you want to add to that? Yeah, and I think that the other thing that the Estimand discussion is now bringing out is are we being clear on what are the treatments that we’re comparing? Some trials, all patients are on a background therapy. It may be truly placebo-controlled in naive patients. You may have a more complex study where patients are randomized in oncology. They have some kind of induction therapy and move on to consolidation therapy.

24:07
Well, in all these different scenarios, I think another highlight to me is making sure that we’re being clear on what it is we’re comparing between. So is it just a placebo versus a new active? Or is it standard of care plus new versus standard of care plus placebo? And I think over time we’ve lost in some way what is it, how we label some of our treatment regimens in our studies. So the estermans will also now…

24:35
make it clearer in terms of what is it we’re comparing against and at what point we’re comparing these treatments. The different strategies to handling intercurrent events are also enabling us to take different approaches, so you’re not just taking the same approach for all intercurrent events. There’s lots more flexibility allowed in terms of how you want to handle each intercurrent event. But again, the challenge will be coming up with a very nice estimate and description.

25:04
but also being able to then use appropriate methods to estimate at Estaband. And as anything we know, statisticians, there’s a number of methods we can use, but what’s the most unbiased method? What’s gonna be the most appropriate method? And what assumptions are being made? And actually, how can I then verify these assumptions by doing different analyses, as far as my sensitivity analyses, to check that I’m still aligned? And I actually do.

25:33
generate a robust estimate at the end of the day. Yeah, I think this term bias, I think, is a really important one because it also helps us to think about…

25:46
Bias means we are systematically estimating something that we actually don’t want to estimate. And there’s a difference between what we want to estimate and the estimator and the expected estimator. But for that, first we need to be clear on what we want to estimate. And I think that was one of the big…

26:11
problems in the past, we weren’t really clear about that. And I think my perception is all this framework helps really with vocabulary to explain what we are really doing. And what I got from your comments, Chrissie, is that maybe another kind of dimension to the estimate framework is actually treatment or what we are actually comparing, which could be kind of

26:39
treatment, but it could also be kind of treatment, a treatment policy or something like that. And that kind of also is associated with population, isn’t it? Because maybe you’re studying in a specific population that has a specific kind of pretreatment or concomitant therapy.

27:09
So that feeds into this treatment as well, isn’t it? Yeah, that’s exactly a good example where, and as Munna says, all these things are interrelated. So patients coming into a trial may have had to fail some prior therapy. They may have had specific previous lines of therapy, and you may or may not need to be very specific in your population, but that is what you’re studying in your trial.

27:37
Sometimes the prior medications is really important to highlight as part of that population statement. Or you may be clarified that it is patients who have a headset in therapies that you want to be in your target population. And of course, sometimes in our clinical trials, we have patients who come into the trials that actually failed an inclusion exclusion criteria. Well, what are we going to do with those patients? How does that impact what we’re trying to estimate?

28:03
And so the population is really trying to make us clear what it is that, what patients are we treating in our trial and how did that relate to our target population? Okay, okay.

28:17
So third aspect in the estimate framework is about how we actually then measure what kind of statistics do we use. So there we need to be very specific in terms of whether we look into response differences or odds ratios or relative risks or these kind of things, isn’t it? That’s correct. So we call it the summary measure.

28:45
It is what it is we’re going to be using to differentiate the treatment effects. And there’s been a lot of discussion, particularly in time-to-vent analyses, where, for example, if we’re looking at a survival endpoint, we often will use the hazard ratio as the summary measure. But of course, we all know that in order to use the hazard ratio, we’re assuming proportional hazards.

29:09
And we don’t often have that in our clinical trials. So there’s lots of discussions around what would be an appropriate summary measure in time to event kind of analyses. I think in our analyses, it’s quite straightforward. But we were looking at change from baseline. We would look at the difference in means, again, assuming normality. And we would like to test that. And if we need to transform the data, we would obviously do so.

29:40
But Meena, would you have any thoughts? No, I think that was a nice summary. I guess, I mean, the summary measure is essentially asking us to specify what is the particular aspect of these curves, let’s say, that you want to focus on. In survival, that could be like the survival difference at a certain time point, it could be the hazard ratio. But as Christy has said, this comes with…

30:05
some interpretation of challenges unless you really have the proportion has assumption holding which of course at the design station phone you won’t I mean you don’t know in general so it’s sort of appreciating that at the end of the day while we have all the data and the totality of that information we want to combine it somehow and condense that and the question is on which aspect.

30:31
Is it relevant and appropriate to condense it so that we have something that’s interpretable, not only to us the decisions, but also to the clinicians? At the end it will be in the label, so to treating physicians and patients. And while it seems almost trivial, so we have to state the summary measure, I think through the discussion around the estimates, the framework, I think some of these things that we have taken for granted throughout the last…

31:01
decade let’s say, so some established practices are in fact being sort of revisited and sometimes even questioned whether sometimes maybe we have to rethink how we can summarize these curves or the totality of the data in an appropriate way. So that’s another plus I see, although it seems very simple to start with. Yeah, it seems really simple but I think it’s

31:28
In this simplicity, it helps us to be also more transparent and more also that it’s very clear in what we want to do. For example, if we want to analyze binary endpoint, we are upfront saying how actually we want to measure it, whether we look into odds ratios or risk differences or whatsoever.

31:55
I think very often that would only appear then maybe in the specifications of the table, what you actually present, rather than somewhere in the protocol or the SAP. You may just say, oh, we’ll do a logistic regression. But what do you present then from the logistic regression? Yeah. And to me, in fact, I think it would be pretty important. For example, the art ratio, right? I mean, that’s a term we use in many clinicians.

32:23
probably also know what it means. But it would almost be nicer if we could sort of put it in layman’s words, what does it mean? So that the estimate ideally, at least in my mind, should be something that we write together with the whole clinical team. And if we then give it to a treating physician or even a patient, ideally they should be able to understand what it means. And art ratio, to be honest, is not something that my mom, I guess.

32:51
would pick up and think, yeah, of course, I understand that. So it also gives us this opportunity to sort of try and phrase things in an array that we can communicate it in a transparent and clear way with patients and so on. And I think that’s going to be a key focus going forward, is that we’ll have a number of different estimates that we will want to get out of our trials. And that’s perfectly fine.

33:19
there are different stakeholders who are interested in different treatment effects. Me as a patient, I could be interested in one esterator treatment effect, which is different to a clinician who’s treating a whole group of patients. And that’s different to a healthcare system. Well, what is, what is the population going to be able to achieve if I prescribe this treatment? But we do need to specify these all upfront and be clear why are they different. And

33:47
by that description will understand why they’re different. But as Munir says, they need to be understood by a lamer and we can’t make it technical. There could be a technical version of an estimate, but we do need to be able to describe estimates very clearly so that all our stakeholders, including patients, can understand what it is we’re estimating. That’s actually a very, very good topic. So it’s not only something for regulatory interactions.

34:15
it’s much broader than that. So it basically forms the basis that we can communicate with many, many stakeholders. You mentioned prescribing physicians, patients, payers, insurance companies or government agencies that inform payers on a national level. So it really affects everything in our industry.

34:44
And I think this is so fundamental to what we are doing, is that every statistician really needs to be well-trained on these kind of things, because it affects setting up studies, it affects reading studies, it affects analyzing studies, it affects communicating about study, publishing about study, interpretation, critiquing other studies, all these kind of different things it has an impact on.

35:14
And even if you work only, let’s say, not only, if you work on integrating studies from different sources into something, you need to be very, very clear on what you’re trying to estimate. And I think there are in these, even in these network meta-analysis kind of areas, something that more works along these lines would help a lot.

35:43
Although they actually have the PICO statement, which more or less goes into the same direction where they say, okay, which studies they want to look into, which interventions they want to look into, and so on. It goes a little bit into this direction, but I think it’s not as elaborate as the estimate framework. And the estimate framework is much more…

36:10
more detailed and clear in terms of what’s being done there. So I think there are links between PCOS and the Estimans framework. There’s not exactly a match per se, but there’s a lot of synergy. I think people should realize now though that the Estimans will impact design of future studies, but also how we do synthesize evidence. But we have that problem now that where people are calling

36:38
an intention to treat estimate and merging it with other intentions to treat estimates across studies, but actually we don’t really know, and unless you go deep into the methods, how was the estimate of treatment effect actually derived. So I think we’ve been using labels a lot, but now we need the statistical community to actually look into the details. What is it that from that particular trial that’s published, how was the treatment effect derived?

37:06
especially if we’re going to be doing meta-analyses or network meta-analyses, making sure that we are comparing apples and apples. Unfortunately though, we may only have the published information, so we may need to understand what has been estimated and appropriately use that in our subsequent meta-analysis. But even in designing these studies going forward, think about non-inferiority trials. We have to look at our non-inferiority margins.

37:33
often we’ll go back to previous studies and look at the placebo control trial for the comparator that we’re using. Well how was that estimate defined and how does that compare to the estimate we’re going to be finding in our new non-inferiority study? So I think again going forward we’re going to have better clarity at what we’re doing and how we’re doing it and understanding maybe some of the nuances that do exist now but we’ll be more transparent about them.

38:01
Yeah, and I think it gives us the opportunity to interact much more meaningful or more meaningful with our non-statistical colleagues, be it on the regulatory side or on the medical side or the medical writing side or whatever, because now we can pull it out of the

38:23
stats own section and really put it where it belongs into the objectives, into the discussion part of papers and speak there about it and not just have it labelled as sets the stats part and the statisticians talk to the other statisticians about it. It’s something where all the different stakeholders need to be involved. Yeah and if this one take-home message today from this podcast is please do not think of estimates as a statistical…

38:52
issue only. It is a cross-functional effort that we need to make sure that the estimate is aligned to our study objectives. The estimate does not belong in the statistical analysis plan only. We need it specified in the protocol aligned to our study objectives. Of course we’ll have much more details about it in the statistical analysis plan but do not let your colleagues say oh this only just sits in the statistical analysis plan.

39:21
We will be providing some further guidance through Transcelerate. So Transcelerate is a consortium of industry partners who have developed a common protocol template. That has been updated recently, and a new version is coming out, which has actually incorporated Estimands into its template. So I’m hoping that will also give more credence to making sure that the Estimands is described alongside the objective.

39:50
and it does not just reside in the statistical analysis plan. Yeah, and that’s actually a very, very nice point. I’m planning to record another episode about TransCeleraRate with one of my colleagues here at Lilly’s that is working on exactly that point. And so stay tuned to this episode in the future. Speaking about events in the future,

40:18
We talked that there’s a huge need for all institutions to be trained on these kind of different topics. What’s going on there? Can you speak a little bit to what PSI is doing in terms of that to help training people and get everybody up to speed regarding this?

40:38
the estimate. So as part of the scientific committee, I can share what the scientific committee is planning for 2019. So following up from 2018, where we had a couple of sessions around estimates at the conference, which I think were very well attended, but also we received good feedback for those. And usually at the end of the conference, we then asked

41:03
for interest in additional topics for next year’s conference and next year one day meetings. And that surprisingly estimate was yet again one of the hot topics that people were requesting and interested in. So we therefore decided to run an additional PSI one day meeting early next year. So the date is fixed for the 29th of January, where we will be focusing on new emerging topics.

41:32
the round estimates and the ICH E9 addendum. So as you have, Alexander, you and Chris, you have briefly mentioned before, so the draft addendum is currently being revised. The 1000 plus comments are being read and I’m sure the working group is having a busy time trying to incorporate these into an updated draft. So one of the aims of the meeting in the beginning of next year is really

42:00
to share the feedback from the public consultations and maybe to discuss some future steps. So what are maybe some of the changes that are to be anticipated.

42:13
Then also related to something we discussed before, so while I believe the ICH9 addendum focuses on confirmatory clinical trials, the topic itself affects a much broader set of studies, but also different stakeholders, and we briefly touched upon HTA. So HTA considerations will also be covered at this one-day meeting, so we will have Anja Schiel from the Norwegian Medicine Agency,

42:42
a link between estimates and HTA considerations. And then I think something that many of us have realized in the past, and to be honest, I have only, after being into the estimate topic for around a year where I thought this was somewhat new, I sort of realized that in the causal inference community, this has been there a long time. So the estimates, the defining estimates, as well as being clear.

43:10
around when can you even estimate certain estimates, what are the assumptions that are needed, what are appropriate sensitivity analysis. So a lot of these considerations have already taken place in the causal inference community. So not surprisingly, there’s a lot to be learned from that community as well. And therefore, an additional aspect of this one-day meeting will be around causal inference and how that fits with estimates and clinical trials.

43:41
We will of course have some case studies being presented that are at that interface. And yeah, I think it promises to be a very interesting day. So that is certainly one activity of the PSI. And then there will be sessions at the PSI conference. So the next year’s conference, as you all know, is the beginning of June, I believe, in London. So there will also be case studies hopefully being presented.

44:08
and also a more dedicated session where the PSI RSS winner on that award that was given for work on estimates will be presenting some of the work as well. So I would just say stay tuned. I’m sure there will be even some additional things coming up. And I’m also sure that some other associations in Europe and US and beyond are actually…

44:37
thinking about some conference sessions and so on on the topic.

44:43
Within FPEA we also have a working group of about 70 people now, both combined with FPEA and SBIE. So that’s enabling us to work directly with clinicians. And the ICH have also released 200 slides of training materials on the draft addendum. So the link went round in the SBIE newsletter. So look at the ICH website under E9 and there’s a series of training slides that are accessible to everybody.

45:13
I would encourage everyone to use those within your companies to train cross-frontally the topic of estimates. Very good. The links to these different things will also be in the show notes. Just go to the effe check out this episode, which will be called the past, present and future of estimates. There you will find the link to the…

45:41
PSI one-day event at IQVIA and UK end of January, as well as other documents that we mentioned today. So, thanks so much Chrissy and Muna for this very, very nice interview on this super relevant and hot topic at the moment. And hope you enjoyed this show. If you have any kind of suggestions.

46:08
for future topics you would like to learn about. If you have suggestions on how to improve this overall podcast, please let me know. Just go to thee and leave a comment there. So with that, thanks so much to Moona and Chrissie. Thank you, have a nice day. Bye bye. This show was created in association with PSI. Thanks for listening.

46:37
Please visit thee to find the show notes and learn more about our podcast to boost your career as a statistician in the health sector. If you enjoyed the show, please tell your colleagues about it.

Join The Effective Statistician LinkedIn group

I want to help the community of statisticians, data scientists, programmers and other quantitative scientists to be more influential, innovative, and effective. I believe that as a community we can help our research, our regulatory and payer systems, and ultimately physicians and patients take better decisions based on better evidence.

I work to achieve a future in which everyone can access the right evidence in the right format at the right time to make sound decisions.

When my kids are sick, I want to have good evidence to discuss with the physician about the different therapy choices.

When my mother is sick, I want her to understand the evidence and being able to understand it.

When I get sick, I want to find evidence that I can trust and that helps me to have meaningful discussions with my healthcare professionals.

I want to live in a world, where the media reports correctly about medical evidence and in which society distinguishes between fake evidence and real evidence.

Let’s work together to achieve this.