If you’re a statistician working in a regulatory setting, you’re probably familiar with estimands as a way to frame the clinical question of interest. But what’s the role of estimands beyond regulatory approval, in a Health Technology Assessment setting? That question is especially important these days where the framework for pan-European HTA is taking shape. In this episode, we’ll touch on some HTA body views around estimands, and reflect on how current draft guidelines for EU HTA do and don’t address estimands.
Anders Gorst-Rasmussen
He received his PhD in statistics from Aalborg University in 2011 and worked with pharmacoepidemiology and RWE in academia prior to joining the Development organization in Novo Nordisk in 2015. At Novo Nordisk, he has held roles as project statistical lead on various late-phase projects. For the last 2 years, has been working in the HTA area as functional lead for a small team of dedicated HTA statisticians. He is an active member of the PSI/EFSPI HTA Special Interest Group.
Arthur Allignol
He is currently HTA and medical affairs statistician at Daiichi Sankyo and is actively involved in the PSI/EFSPI HTA Special Interest Group. Before joining Daiichi Sankyio, he worked as Real-World Data Scientist at Merck Healthcare KGaA and in academia, specializing in observational studies and complex event-history analysis. Arthur Allignol obtained his PhD in biostatistics at the university of Freiburg in 2013.
Transcript
What is the role of estimands in the EU HTA?
[00:01:58] Alexander: Welcome to another episode of The Effective Statistician and this is the second podcast episode in a series of four episodes where I’m talking with members of the HTA Special Interest Group. And today, I have Arthur and Anders here. So maybe you can first introduce yourself. Arthur, give it a start.
[00:02:23] Arthur: Hi, everyone and thank you for the invitation. So I’m actually new. I’m an HTA Medical Affairs Statistician working at Daiichi-Sankyo Europe. And what concerns us today is mostly HTA. I’m dealing with all the supplementary analysis that are required, asked by the different payer bodies, and I’m also helping countries on their dossiers and replying to questions. Before that, I worked at Merck, in the Epidemiology Department as a real world data scientist.
[00:02:58] And before that I was in academia in Ulm for a postdoc and before that, it was in Freiburg for my PhD. It was in Germany.
[00:03:05] Alexander: Very good, awesome. Anders.
[00:03:08] Anders: Yeah, my name is Anders Gorst-Rasmussen. I’m working within HTA Statistics in Novo Nordisk. So we have this small HTA Statistics Group which has now been in existence, I think, for a couple of years, a little less than a couple of years now.
[00:03:22] I think also let’s see, all these increased requirements we also have for a special handling of HTA statistics tasks. And before, I joined this HTA Statistics Group, I was actually working within the regulatory space as a regulatory statistician for a little more than five years actually.
[00:03:39] And before all that, I was working within academia, actually working within real world evidence. So more specifically within pharmacoepidemiology.
[00:03:48] Alexander: Yup, I think both your CVs speak to the point that when you work in this area, it’s pretty important to have a [00:03:58] broad background and to not have only worked on clinical trials, but also epidemiology, real world evidence.
[00:04:07] It’s very often I see people being successful in this area have this much more diverse, let’s say, non-standard CV, which is great. So let’s talk about a combination of two really hot topics. On one hand it’s a health technology assessment and all what’s going on there. And if you don’t know what has been going on there, then scroll back to the first episode where we talked about a lot of the current trends and the hot topics.
[00:04:40] In terms of HTA, EUnetHTA and so on, it doesn’t tell you anything. Have a listen to this one and to on one hand, the HTA and on the other hand, estimands, really hot topics that have been here for quite a while and [00:04:58] has evolved over the last decades, I would say. First, it was really a missing data topic, and now it has evolved into much more than that, into something like really better understanding what we want to research, what we’re really interested in it, and of course, that has
[00:05:17] implications not just on the regulatory side, but also on the HTA side. So with that in mind, Anders, maybe you can start to explain a little bit, what is the current situation with estimands and the HTA bodies?
[00:05:34] And so, thank you, Alexander. And I think first things first, one of the reasons why we thought it could also be interesting to take this estimand and EU HTA discussion also on the podcast here is because I think it provides a really nice intro to some of the things that we are dealing with in the Special Interest Group.
[00:05:50] You’re talking just before about the importance of having the breadth as a statistician also working with this. And I think estimands and how it [00:05:57] fits into the HTA context is really about understanding and navigating, I could say, the different evidence standards of HTA bodies regulatory.
[00:06:06] And then I think there’s, they’re very much also an aspect of the processes that are involved in this from an EU HTA perspective. What do they look like? How do they align with, for example, the thinking around estimands? And then finally, I think, the awareness and the dialogue that we are trying to raise from the Special Interest Group perspective is also very much reflected in this estimand discussion.
[00:06:26] So I think it’s a very nice case of showcasing what this discussion regulatory versus the HTA perspective is all about. So you could say that very briefly and I’m sure most listeners of your podcast will be well aware of this but I guess most pharma statisticians will be quite aware of what estimands really are at the moment.
[00:06:45] So the basic idea of an estimand is that, okay, instead of asking these sort of fairly vague open-ended questions like, okay, what’s the effect of treatment X on mortality? You have to be more specific about what it is that you’re asking. So [00:06:58] you need to formulate your questions in a more precise manner.
[00:07:01] So the estimand idea is basically saying that, okay, you need not just this vague kind of question, you need to be explicit about what kind of population you are looking at? You need to be explicit about the treatment condition that you’re looking at. So what are you comparing? What treatment are you looking at?
[00:07:17] What are you comparing it against? It needs to be explicit about the endpoint. And then two important things, at least for this discussion. You need to be explicit about the summary measure that you’re using to do the comparison. So is it the absolute scale that you’re looking at? Is it a relative scale? And then finally, I guess, the most key thing, or the new thing about estimands, which is this idea of needing to have a strategy for handling your intercurrent events.
[00:07:43] The key thing here is, say that we want to address the effect of some treatment on mortality in a given population. So we would ask the question, okay, in this population, what is the effect of treatment regimen X versus regimen Y on one year [00:07:57] mortality as assessed by the relative risk? And then comes the handling of the intercurrent events.
[00:08:03] So you’d say, okay, you want this effect regardless of treatment discontinuation and intake, or for example, additional medication. So this aspect here, and this is the kind of estimand we refer to as a treatment policy estimand, or a free estimate days. You talk about this as the ITT effect. So usually when you go and talk to HTA bodies around this, I mean there’s, I think it’s fair to say there’s a strong preference for the ITT effects when you’re looking at things from an HTA perspective.
[00:08:34] For example, back when the ICH nine addendum came into review and was commented on by various stakeholders around the world, for example, the German Institute so IQWiG, the Institute for Quality and Efficiency in Healthcare, which is the German Institute that takes care of health technology assessments in Germany commented that
[00:08:55] they thought that only this treatment policy estimand [00:08:58] or a composite estimand would be relevant for doing, you could say, the main analysis for things that would be used for HTA purposes. And so the treatment policy that would be of the greatest interest, I think, it’s important to understand what would be the alternatives, what kind of other estimates might you be interested in?
[00:09:15] So that could be, for example, an hypothetical estimand. So an estimand saying that, okay, had patients not initiated rescue medication, for example, so addressing this hypothetical scenario of a patient doing something that did not happen during the course of the trial and the IQWiG argumentation here is okay.
[00:09:34] Why would you be interested in something that does not act current, real life? Something that is of more interest to us is actually what the patients were actually doing within the trial. And importantly, and I think as a key criticism also of going beyond the treatment policy estimand is if you have to do these assumptions, if you’re looking at hypothetical scenarios, you have to make additional assumptions.
[00:09:57] You have to make modeling assumptions. So there’s a risk of incurring additional bias by going to things like hypothetical estimands. And of course, that’s an issue. You cannot necessarily estimate things without additional modeling assumptions in the setting. And so it is preferable for, from this perspective of making sure that you have a high degree of internal consistency to stick to the treatment policy kind of estimand.
[00:10:22] So I think IQWiG is probably the HTA body where we can most explicitly see their thoughts on what is appropriate in terms of the treatment effects that you’re trying to estimate. But again, back to my initial point, I think it’s fair to say that there is a preference currently also with HTA bodies for having this intention to treat effect corresponding to the treatment policy estimand.
[00:10:45] Basically for the reason that you can estimate, it reflects in principle what’s going on, what you see in the actual trial, and it’s something that we can estimate without risking a lot of bias.
[00:10:56] Alexander: Yup. How is that [00:10:57] different to what the HTA bodies have been coming from? Because when I started in this area, it was very much driven by Cochrane and evidence-based medicine, and there everything was just about the PICO statement, population, intervention, comparator, and outcome.
[00:11:21] How’s that different from the estimand approach?
[00:11:24] Arthur: In a way, they are complementary. So the PICO puts the frame into the research question you want to address in a way in which population you want to work. Comparing which interventions on which outcome. Well, I think both approaches are complementary. It’s because the estimands then add a precise way to formulate these research questions.
[00:11:48] So in particular, what the PICO doesn’t mention are the intercurrent events and the summary measure. Yeah. In this sense I tend to argue that these are very [00:11:58] complementary approaches.
[00:12:00] Alexander: Yeah, I think it speaks to when we do systematic literature reviews and things like this, we don’t need to just look into PICO.
[00:12:08] We need to look into all the different elements of the estimand, because otherwise we’ll not be able to collect all the data that we need for indirect comparisons and things like this. And we might, you know, have an estimand from one study and a different one from the other study, and we shouldn’t just match them together.
[00:12:29] I think that’s a really important insight.
[00:12:32] Anders: Yeah and I think that’s also a reason why that I think historically there’s also been making sure that we have all the data that we need for doing our assessment. That has been one of the reasons why there’s been so much of a focus on ICC basically because if you insist that you as a sponsor need to continue collecting data also after patients have discontinued
[00:12:50] treatment then you have the broadest possible data to make your inferences on. Whereas you can say, if you start opening up for other kinds of estimand, [00:12:58] then that might be seen by some as a loop for collecting less data. And that’s at least a concern that has been raised also by HTA bodies, by going to something else than just a treatment policy set up.
[00:13:10] Alexander: So in the last episode, we talked about EUnetHTA 21. If you don’t know what these are, go back to the other episode. So EUnetHTA 21, what’s their point of view on the estimands now?
[00:13:23] Anders: So last episode, we were also talking about this. You mentioned before, Alexander, the PICOs which are really the preferred way of formulating questions or clinical questions of interest for the purpose of EU HTA.
[00:13:36] So the basic idea being that, okay, once you’re about to start your assessment, a questionnaire will be sent out by the assessors and the co-assessors to the member countries asking them about, okay, what questions are you interested in within a local context about this new treatment? What will be your population, your intervention, your comparator and your outcome?
[00:13:56] So in that sense, there’s not, you [00:13:58] could say any direct mention within just this basic scoping process of the concept of estimands. So there’s also no mention of, you could say, intercurrent events of any kind. That’s not to say that EUnetHTA 21, so the draft guidelines that we’ve seen so far ignore the concept of estimands. It is actually described here and there in the current draft guidelines that we’re seeing.
[00:14:20] But you could say it’s not necessarily very closely linked with the way that these questions are formulated. I think it’s nested somewhere in that guideline under sensitivity analysis that there, you see a description of the concept of estimands taking, I would say, quite a bit also from the ICH E nine addendum and encouraging important things like, okay, you need to be clear about what is your main analysis.
[00:14:44] You need to be clear about what is supplementary analysis, and you need to be clear about what is the sensitivity analysis associated with this. And then it says, importantly that the member countries, of course, need to be aware that it’s important during the [00:14:57] assessment whether or not the
[00:14:59] the estimands that have been included in clinical trial protocol, the extent to which they actually measure up or match up with the PICOs, that are part of the scope. And that sometimes, of course, member countries might be interested in different estimands that are not necessarily part of this. And that can be addressed during scoping and that somehow, that’s buried currently in guidelines.
[00:15:20] So you could say there’s not, if I was a member country, I would probably just formulate the PICO. Whereas you could say, if you suddenly start having a need for different handling of intercurrent events, for example, it’s not exactly obvious how that will go down.
[00:15:35] Alexander: Arthur, anything to add from your perspective?
[00:15:38] Arthur: No, just a code that’s in the traffic guidelines we’ve seen, estimands are mentioned in some sessions, but it’s, I don’t see clear requirements on whether estimands need to be used, mentioned, et cetera.
[00:15:55] Alexander: Yeah, I think it’s, what I hear is probably, [00:15:58] it would be great if they would have a much more prominent role.
[00:16:02] Already in the scoping process, it said there’s about what is really needed. It says that we have a clear debate and clear requirements about this kind of different things because otherwise, as we mentioned before, it made pooling things together that you shouldn’t pool or provide answers to questions that weren’t asked, missing the answers to the questions that were actually asked, but not clearly described because of these missing parts in the PICO statement as compared to the estimand framework.
[00:16:33] So let’s go a little bit in terms of the relevance of this, let’s say, more hypothetical estimand. Is it really completely out of the window here? Just because IQWiG doesn’t like it? I remember once talking about someone at the IQWiG and thinking about efficacy, and he was very adamant on the treatment policy [00:16:57] approach.
[00:16:58] And has since switched the coin and said, “And what about safety?” What you’re saying also signs all safety events here, and what that means is that if there’s a standard of care and then you switch the standard of care to something else, all the new safety events would also be assigned to standard of care. And then there was some kind of, “Hmm, interesting. Maybe it’s not as black as white.”
[00:17:23] “Why should we be maybe a little bit more open and flexible around the estimands and not only rely on the treatment policy or the composite strategy?”
[00:17:33] Arthur: Like already mentioned, there is really a strong emphasis on the treatment policy estimands and also in the draft guidelines. We can think of a couple of use cases where hypothetical estimands would be interesting to look at.
[00:17:50] So for instance, one would be the, when we look at overall survival in oncology and patients may switch over to [00:17:57] the experimental treatment after. That’s a case where you might think that’s when hypothetical estimands might be of relevance because they control patients switching to the experiment. So treatment, if it works, you would expect a lowering of the treatment effect.
[00:18:17] So actually, for instance the NICE is interested in these estimands, especially for cost effectiveness modeling of what happens after treatment switch and is also interested in any case to see this trend of effects, but other payer bodies are a bit less open to these approaches. I think one reason is what Anders mentioned already is that they require more modeling and supplementary assumptions and yeah, that needs to be documented and judged and assessed.
[00:18:49] Alexander: Yeah. I think it’s also relevant to think about exactly that case that you just mentioned, whereas the standard of care is switched to the [00:18:58] experimental treatment, then the question is very much about do you directly start with experimental treatment or do you wait?
[00:19:06] And is that really the comparison that HTA bodies are interested in? Because it basically assumes that you make the experimental treatment available, because otherwise that comparisons are of no help. So if you don’t make it available, the standard of care, it’s really about standard of care and not standard of care and then switching in case it doesn’t work.
[00:19:28] So I think that is, it really needs to be looked into case by case scenario. Also what data is actually available and that makes a lot of things much more complex. It’s as usual, if you look into the details, then it becomes much more kind of harder and not as black and white anymore.
[00:19:50] Now, if we think about this case, then of course it’s not looked into just from an HTA perspective. All the [00:19:57] timelines are not changing. So you said, we need to look at it more from a holistic perspective, regulatory and HTA. And what do you think we can do to make sure that we have everything we need, both from a regulatory and from an HTA perspective when things really happen in parallel, more or less?
[00:20:20] Anders: I think that’s a really difficult one, right. I think we should also acknowledge the fact that, of course, there will be a need for asking different questions from an HTA perspective. So in some sense you can say, I think maybe one of the things that is also a concern is that if there’s too much of an estimand focus within an HTA setting, then maybe you as a, as if you are a member state, you need to ask a question, you would be more inclined to look at the estimands that actually have been defined within the study.
[00:20:44] Instead you could say posting questions that are relevant for the health policy that you are interested in looking at within your specific country, right? But I still think that accepting that there will be different questions, I think what we need to do is of [00:20:58] course, to make sure that we have these discussions also at a much earlier stage.
[00:21:02] So even though there will still be estimands that are specific for regulatory purposes and questions that are specific for shared purposes, what we of course need to do is to make sure that we have the discussions about what the respective needs are at an earlier point, and then that we’re able to cater for all of them.
[00:21:16] Alexander: Yeah, and I think especially in the case that you just brought up with this oncology study and which to experimental treatment or could be also outside of oncology, I guess, that’s the design topic that you need to discuss before you start your phase three study. And so having a discussion with the HTA body at that point already is really important because when you have done the study, a lot of opportunities are lost thereafter.
[00:21:44] And if the, of course, things like operational and ethical considerations also need to be in place just to say, oh, it would be nice to have that. But if you can’t really put patients into such [00:21:58] a study, then that’s nice to have, but it’s not feasible. And having that kind of discussion is also important.
[00:22:07] Awesome.
[00:22:08] Anders: Yup. And I think one other thing that we need to be aware of also when we’re asking questions using PICOs is, of course, that we intrinsically bake a lot of uncertainty into the PICOs that’s not really present with estimands, right? For a given PICO, because you haven’t specified things as crispy as you have with estimands, you could have multiple estimates that are matching up with that.
[00:22:30] It could be multiple estimands as, sorry, multiple endpoints. It could be multiple time points and so forth. There could be multiple ways of dealing with intercurrent events. And in principle, when you are a manufacturer, then you would need to deal with all these different forking paths in dossiers as well.
[00:22:44] So you’re basically taking a lot of uncertainty and baking it into the question that you’re asking. And the question that really what we try to do also with what we’re doing with EU HTA, if there’s uncertainty left on the table that we could address by being a little more [00:22:58] crisp about the questions that we’re asking, then I think it’s definitely a venue that’s worth exploring as well.
[00:23:04] Alexander: Yup, Anders, I think that was a brilliant closing statement. There is still a lot to do for our statisticians to influence these areas. As discussed in the last episode of the series, there’s a lot of opportunity at the moment, in and of the next few years to influence this process and sign up for the HTA SIG. Contact the people there. Go to the seminars.
[00:23:29] Provide input into the draft guidelines, into updates. Get in contact, and also raise awareness within your organizations about this really important topic. We, as statisticians, play a crucial role here, and in order to make sure that we have great medications, not only passing the regulatory hurdle, but also the HTA hurdle, we need to be at the table when these things are discussed.[00:23:58]
[00:23:59] Thanks so much for both of you and all the best with your continued work on the HTA SIG. You’re doing quite a lot of really cool things. I’m very happy that you invited me to help you with this podcast series.
[00:24:14] Arthur: Yeah, thanks a lot for the opportunities.
[00:24:17] Anders: Thank you.
Never miss an episode!
Join thousends of your peers and subscribe to get our latest updates by email!
Get the