The 15% Threshold in Germany and Why You Should Care

Transcript

The 15% Threshold in Germany and Why You Should Care

[00:00:00] Alexander: Welcome to another episode of the effective statistician. And today I’m really excited to speak about a topic that is really important for everybody who wants to bring new medications to the general public beyond regulatory approval. And for that, I have an expert here that has a lot of expertise with this specific topic, but also in general with all the post regulatory approval market access topics around in Europe, Michael Henning. How are you doing Michael?

[00:00:43] Michael: I’m fine. Thanks. Excellent. Thanks for inviting me, Alexander.

[00:00:46] Alexander: Very good. So before we dive into the technical topic, maybe you can give a little bit of a description of yourself and introduction of yourself. What have you been doing so far? And you are actually quite long career overall.

[00:01:03] Michael: Yes. Happy to introduce myself. Actually I’m from education. I studied statistics in the wonderful university of Dortmund and not almost fine. And since then I have been at different locations also within the clinical development process, starting for Pharma company who was mainly caring about metabolic diseases where I learned to plan phase three studies and analyze them, did some programming also and then I was also at university at the Institute of Medical Statistics, where I did a lot of consulting for the doctors at the clinic home who wanted to plan their studies.

We were also responsible as a statistical analysis center for a large phase 4 study at these days. And yes, after a couple of further stations, I finally, yes, arrived in the very special arena of HTA, Health Technology Assessment, where it was all about what happens to a drug after it is approved, after it’s on the market. And I think not only in Germany where my focus was this is topic which is very much discussed and where it is all about showing the edit benefits of a drug. And this is what I did over the last 10 years and which was also, yes, showing me that the perspective you have when you design a phase three study and are successful by launching a new medicine.

This is good, but this is not everything you need. Then for HTA, the story starts again, actually. And this is what made me very exciting, excited about this aspect. Because I believe this is also a very important arena for statisticians. It’s not sufficient to bring only good drugs with good statistical expertise to the market. But it’s also important to care about the time after the drug is on the market. And this is what I did after the during the last 10 to 12 years.

[00:03:16] Alexander: Yeah. It’s not sufficient to have a label. It’s really important to also make sure that the drugs are reimbursed. And in most countries where you have a national healthcare system, that goes through the health technology process. And through which you look into, first, is there some medical benefit over the existing treatments, not so much about placebo, but existing treatments, and is the cost also in line with this added benefit? If you look into the US, you have actually some aspects that, go into the similar direction.

You have the ISA Institutes there, which looks very similar into how much more value you get. for every dollar you pay. And so that is what we are discussing here today as well. We are not going into the dollar so much, but more about the added benefits, the added medical benefit for the moment. So the German system has a very specific topic and Germany is really important because it is by far the biggest market in Europe and there are some further consequences of it in Europe.

First, the prices of many other European countries are connected to the price in Germany through a reference price system that we have here. So the price in Germany does not only affect the people in Germany and your reimbursement status here and your your business opportunity in Germany here as a sponsor, but it affects countries across Europe. And if you see Europe as a whole, it’s of course a huge market with hundreds of millions of patients and therefore very irrelevant.

Now, in Germany, there is another aspect that is really important, and that is with the emerging EU HTA harmonization effort that is going on. Since the UK left, Europe, so at least from a political point of view, not from a geographical point of view, of course. Germany is by far the biggest weight in terms of HTA bodies here in Europe.

And if you look into the participants list of the new HTA regulations and how that is set up, you can directly see that. In Germany, we have two players. The first is the GBA, which is the federal official political decision maker. And then you have also the IGWIG. Which is the most scientifically guided, specifically very statistically oriented consulting body to the GBA. And build have a lot of representatives in this overall process actually more than double the number of the representatives, then the next big country in this overall process. And therefore you can probably assume that everything that’s going on in Germany will have a big impact. Also from a methodological point of view across Europe. And that’s why this topic we are talking about today is really important.

If you work on phase two and phase three studies, because a lot, what we will talk about cannot be saved thereafter. And stay tuned for this really interesting discussion. And now let’s talk about the problem itself. Michael, what problem are we talking about?

[00:07:44] Michael: I think the problem we should talk about today is about how to assess whether there is real and added benefits for the patients. Because I think this essentially is the key question, which is to be addressed in the HTA process. And therefore of course, There are various ways in how to address this question. And one, one way is in to in assessing the additional benefit of a new drug by a so called minimal important difference.

And this minimal important difference, sometimes also called MCID, Minimal Clinical Important Difference, is this difference, which essentially the patient feels and is really caring about. And there is a lot about the MID, also in the statistical literature, because it’s a key question on how to determine this threshold. And therefore this I think is a very important topic, how to come to this MID and how to perform the statistical analysis around this MID because this at the end of the day gives you an impression on how a drug works with regards to the patient relevance which I think everything is all about especially in this process.

So this is a topic I think which is of a major interest and where also we as statisticians play a very big role in how to deal with this construct.

[00:09:29] Alexander: Yeah. And we are talking here about not the group level clinical difference, but what’s happening on the patient level. That’s a little bit complex thing because if you look into Minimally Clinical Important Differences, you can look into many different things. You can look into group differences. You can look into a group differences before and after giving treatment, or you can also look into differences. over time for a single patient. And the last thing is what we actually talk about. So the problem occurs if you have something of a questionnaire or some kind of continuous outcome, where you see an improvement over time for a patient and what happens then?

[00:10:26] Michael: Yes, this is exactly the point to to see how we develop this difference on a patient level, because at the end of the day, it’s about an end point, which is binary. Either you see an a relevant effect within a patient or you do not see a relevant effect. And what threshold to use? This is the key question on a patient level. And typically what is done in elaborating on these threshold is to to ask the question to the patients on how they feel about a change and how they realize a change in a specific question I have been, I think it’s always easier to make this and set a specific example.

I was working very often in the pneumology area where the patients had either asthma or COPD and here there is a very established questionnaire, the so called SGRQ, the St. George respiratory questionnaire, where Patients are asked on their symptoms with regards to their breathing capacity. And this is a questionnaire which essentially gives you back a scale from zero to 100 zero meaning there is totally large difficulties with breathing 100 everything is fine. And here the question is really what is relevant for a patient and typically you do a lot of studies in which you ask the patient.

And what is really felt by them. Is it already one scale which is felt by the patient is it is a two, is it more. And so essentially it all starts with involving the patients on asking them what is felt by them as a relevant difference. And this is a starting point in establishing these so called MCIDs or MIDs.

And this is Yes, a long and important scientific journey you have to take because it’s not only about asking a couple of patients and then setting something. Of course, you have to consider the variation between the patients. You have to establish the psychometric properties of these. At the end of the day after a quite some analysis steps.

You establish an MCID on a patient reported outcome level. So you do this for the SGRQ, as I just mentioned the example. You do this for other questionnaires but the key point is you have to do this for every single patient reported outcome. By asking the patients and by analyzing all these data you get from this adequately. And so there is a rich literature also about these MCIDs for various endpoints, in the various indications. And this is typically the starting point you have to consider. When you address the question what is really relevant for the patient is the change you observe relevant or is it not relevant?

And statistically speaking, once again, you have a dichotomous Yes, decision, so either a change for a patient is relevant or not. And then you compare these relevant changes from the one group with the other group, and then see whether you really have yes, a benefit. Of the drug with regards to this MCID.

So this is in a nutshell how it works. And this is I think good scientific practice to involve the patients to do a lot of statistical framework about, having a certain yes, certainty also with regards to the stability of this MCID. And this is how you typically should do it from a scientific point of view.

[00:14:46] Alexander: Yeah. And if I now listen to this my statistical heart kind of breaks a little bit, because if we take. optimize a continuous endpoint, of course, we throw a lot away information. Yeah. And. That of course leads to less precision, less power, all these kind of different things. However, here as a sponsor or as a C R O as, or as a consultant that you work with, you need to play by the rules of the game and so these kind of things are pretty much set in stone and you can’t really argue about it. Yeah, I think it’s another field to change these rules overall on a more kind of policy discussions.

But if you are in the discussions about getting reimbursement for a specific treatment, you can’t change the rules. So even though you might think, Oh, that’s a really bad idea to, create a binary endpoint here. Welcome to the real world. This is how it’s done. So the problem, of course, is when you create these binary endpoints, if you have everything established.

Then I think that’s a good thing. But if you have not, what happens then? If you come with your not established MCID, or if you just come with your continuous endpoint, what would the Equix say?

[00:16:28] Michael: First of all you already mentioned within equi there are a lot of statisticians working, and I think also for them looking at a binary endpoint which is created based on the continuous endpoint is perhaps not the optimal thing but also looking through their eyes.

I think what I have learned in the, in discussions with them is, of course, it’s statistically not optimal, but on the other hand, you have already an endpoint, which includes this these relevance, because if you would do this over the classical way by considering as a continuous endpoint, then at the end, you also would then need a decision on whether a difference you have observed on the continuous endpoint is of relevance and so they put it the other way around.

I also believe that this is not the very best way, but as you mentioned, this is how the environment works and of course as someone who works in this environment you, at a certain point, you have to accept this. And therefore this is a way we, which is fine, but as you mentioned, if you do not have an established end point already, or if you do not yet have a minimal important difference you always have a problem. You can go back to the continuous endpoint and then compare these groups on a continuous levels. And then it’s always up to the question whether the observed difference even if it’s statistically significant, what we believe now is really of clinical relevance.

But this clinical relevance is then on the group level. It’s no longer on a patient level, which we discussed before it’s on the group level. And therefore you have to yes, include experts in asking them, do you believe whether this observed mean difference of X points is of clinical relevance?

And then you turn the problem to the clinical experts which should provide you an answer on this is key question. And so this is option number one, if you do not have something and typically the Equic says they, they’re going to have a look at this. They’re going to consider it, but very clearly their preference is on this dichotomy, this binary thing we discussed before.

And therefore they very recently, I think now it’s a one and a half year ago they established a threshold on their own, which they think if you do not have something, you can use this threshold. They do not call it any more an MCID they call it a plausible threshold for a relatively small.

But scientifically certain noticeable change. So I had to learn this phrase also. And they said looking at their, looking at the literature on this, they identified a threshold. And this threshold is 15% of the scale of the endpoints. So if you have an endpoint coming back to the example from the SGRQ with a range from 0 to 100, they say you can use a threshold of 15%, 1 5. So whenever there is a patient who develops a change, which is 15 or greater, this patient would have would be considered as a patient with a noticeable change. And a patient with a value below is a patient without a noticeable change. So this is how the Equic recently addressed this the this problem.

When you do not yet have established a threshold, they introduced a new threshold, which is quite high. If you compare to the literature and coming back to the SGRQ, the typically established MCID for the SGRQ is four, four points. And there was a lot of literature around this and showing that this is the order of magnitude, which is felt by the patient.

And this four points, yes, compares to 15 points within the equic threshold. So the factor is about four. So it’s four times higher. If you apply this threshold compared to to the established threshold. And this of course was a matter of large debate with within Germany over the last years as you can imagine, because a lot of statisticians who were starting the discussion with the equic and also with the GBA on this topic and but at the end of the day, this is now a value which has to be considered in the process of Germany.

And of course not everybody is happy about this because it really, in some situations, it really leads to two values, which are very high compared to the traditional values. And therefore yes, within the German statistical community, colleagues of mine and myself try to make a lot of noise about this.

We published quite, quite some papers on this also within the Eastport community to raise the awareness about this, but at the moment, this is still an official. Statement of the EQIC, which is to 100% followed by the GBA who finally makes a decision. And therefore yes of course, this is how it works right now.

But looking at the European HTA assessment, as you already indicated in the introduction, Alexander yes, makes me feel a little bit concerned whether this is also taken on the European level because I believe This is not the best scientific way. Of course, it’s an easy to handle way. Because it’s a very easy to calculate threshold. You can apply this for any end point. So it’s very easy. You don’t have to do many studies about this. You don’t have to ask many patients about this. So it’s easy to apply. But is it really the best way? Let’s put a question mark at this point.

[00:23:04] Alexander: Yeah, there’s a couple of problems, of course, with that. So for example, if you look into the scale that you just discussed with zero being worst and 100 being best health, and 15 point change that is clinically meaningful, then basically everybody beyond 85 points on the scale can never experience it. Yeah. So if you go from 86 to 100, you go from impaired to perfect. That’s not meaningful for them. Yeah. The other problem also happens if you, for example, have very skewed scales, so I’m just thinking about scales that is used in psoriasis, the P A S I or just PASI called it goes from zero to 72, that’s highly skewed. Yeah, the usual cut-off and the mean value is usually something around 20, and higher means worse in this case. But there’s largely nobody beyond, let’s say, 35 or definitely beyond 40. Because 72 would mean your complete body is covered in sick, scaly red plugs. And that’s just not what happens, beyond 50 is nearly you can’t live with that.

Yeah. . There’s only outliers in that area. Yeah. So it’s a real range. Doesn’t go from zero to 72, I don’t know, 99% of the patients are from zero to 40 or something like this. But still, of course you would use the 15 between zero and 72, which is 10.8 points change. If you imagine, yeah, that the, just the entry criteria to go into a study is 10 or 12, Yeah, that is in about that range.

Yeah. Now, the good side is we have some biologics that really help quite dramatically, so lots of patients still get onto this, but if you look into let’s say topical treatments and things like that, yeah that’s a completely different story. Or if you look into more mild patients, so mild patients are those that have less than 10 on that scale. Yeah, they have less than 10. So by definition, they never can, get a clinically meaningful improvement, which is completely weird. Yeah. So you have a disease and you cannot reach it by definition. How weird is that?

[00:25:55] Michael: Yes, exactly. And I totally agree. And this was also a matter of debate we had in the discussion because we put it that way that in the population you investigate, you typically do not have the entire range because the entire range. From zero to 100, you may observe them. Yes, in the whole population, but you are obviously here in a population of patients who suffer from the disease and therefore they do not cover the entire range from zero to 100. They only cover a certain range because if they had zero in this example, there would not be in the study because they would not suffer.

And therefore exactly, this is a point you have a difference between the theoretical scale and the practical scale within the population, patient population of interest. And this is not considered by these 15% because once again, these 15% goes on the entire range refers to the entire range and not of the, in the range of interest. Yes but this is exactly 1 of the few points. We criticized on this procedure.

[00:27:08] Alexander: What are other critique points that you have mentioned?

[00:27:12] Michael: I think one fundamental point is the patient missing patient centricity, because what I understand from the typically process, which I highlighted at the beginning is that you have to ask the patients on what is really meaningful, which is meant by him and with the procedure suggested by the equic. There is no longer any patient voice within there. It’s just a set threshold of 15%. It applies for all diseases. It applies for all patientry patient reported outcomes. So it’s a one size fits all approach, which is not considering the patient aspect. And of course we know that the patient aspects Vary from disease to disease from end point to end point, but of course, also from patient to patient.

We know about this variation. I think as a statisticians, we also know how to handle this variation, but to put a threshold for all patients. In all diseases, in all endpoints, I think this does not make sense because there are variations and therefore one should consider these variations.

There may be difference in, in oncology compared to psoriasis or compared to pneumology. It’s not all the same. It’s not one size fits all. There are differences and there may be even situations where a change of more than 15% is more meaningful, but this is 15%. I think it’s not meaningful, especially from this patient perspective, because at the end what we are doing here is trying to assess the added benefit for a patient. And therefore I think it’s always a good. It’s a must to ask the patient on what is meaningful. And this patient voice is left behind in this 15% approach.

[00:29:18] Alexander: Yeah. However, you can get around it if you do your research well. Yeah. So this only happens if you come to the market, haven’t done exactly this patient centric research before. Yeah. If you have not consulted with statisticians that know about everything about market access. Yeah. If you have, just focused only on the FDA and the EMA, not considered what happens thereafter. So when would actually be a good time to look into these kind of different things so that you avoid these kind of situations. Once you have market authorization?

[00:30:10] Michael: Of course you should start this at the very earliest when designing a study, you should make you, you should definitely involve statistical expertise by investigating these properties of the MCIDs right from the beginning. What we did very often in studies where we investigated a new end point is that we also set up a small pilot study in which we did this additional exercise where we ask the patients and where we already, where we really created solid evidence, which showed us which. Changes are we talking about in these patients, which are of interest to us.

So whenever you have a chance to do this in a pilot study or an extra study, this is definitely a good choice because then you have the evidence, which you can use also for the HTA purposes. And this is actually also an advice I would give that you apart from these must have 15% which the authorities need to look you should create as much as meaningful additional evidence to show also with appropriate sensitivity analysis on the robustness of the results.

And this starts already at the very beginning of a study. And it is not enough to yes, to run the analysis after the study has has finished. Coming back to your point, start as early as possible involve statistical brains as early as possible by yes, doing additional exercise by creating evidence to have a rich body of evidence which you should share with the equic.

[00:32:05] Alexander: Yeah. And so that means basically, if you talk about a study it’s potentially already a phase two study. Yeah. Latest, when you plan your phase three study, you should have these discussions with statisticians that also know about also market access topics.

[00:32:24] Michael: Yes, exactly. And this perhaps is also a good point to, to address also one observation I made in the past in general about when to involve statistics very often statistics, statisticians are involved at the very late stage when you have to rescue your data, let’s put it that way to make an analysis, which which is appropriate. This I think is not the best way to involve statisticians. You should do so from the very beginning to create the evidence which really answer those questions you have.

And the question we have here is what change is relevant for a patient. And therefore I think it is very important to, to do this by involving statisticians right from the beginning. And not only at the end I think for us, Alexander, this is a no brainer, but I wanted to use also this platform to, to address this topic once again, because very often we become into late, unfortunately.

And of course we can rescue some things, but it’s even better if you are the arch architect of the body of evidence, right from the beginning.

[00:33:37] Alexander: Yes, I don’t remember who said it. The most, the three most important points a statistician should focus on are design, and design.

[00:33:49] Michael: Exactly, nothing to add to this.

[00:33:51] Alexander: And yeah, as all the statisticians listening to this probably can completely relate to this topic. And, yeah, one thing that is really important is we have so many different statistical experts nowadays. Yeah it’s, as a statistician, it’s really hard to be an expert in all different areas.

Yeah, maybe you’re an expert in phase two dose finding studies. Maybe you’re an expert in phase three studies. Work with other statisticians that have expertise in the other areas. You don’t need to be someone that knows it all. Pull in experts from other areas from, experts that know about psychometric development of new questionnaires.

Many of the big companies have specific organizations for that. And for me, it was always really important as a statistician to work with these patient reported outcome or nowadays called COA core groups. Because it’s a whole new world looking into this. Thanks so much, Michael. That was awesome to talk about this.

[00:35:04] Michael: Thank you.

[00:35:05] Alexander: What is your last kind of thought that you would like our statisticians listening take away from this episode?

[00:35:17] Michael: Yes, perhaps in addition to what you already mentioned, Alexander, that you have to get statistician, you should network also with other statisticians. If you are in a large company, you should certainly also connect with those people within your company.

But very often people are working in smaller companies where they do not have the statistical expertise. Go out and connect. Also cross industry. There are a large number of organizations which do both on a national international level. I did this research work here with the ISPOR colleagues from other companies where we are the experts.

We were also connecting with Psychometric with people with more psychometric experience than I have. So do connect also with other folks around. There are so many organizations within Germany. We have also the APF, Arbeitskreis Pharmazeutische Forschung. So there is so much statistical brain and knowledge around.

It is really worthwhile to connect with these expert, because as you mentioned, you can’t be expert in every single field, but it’s important to connect and to talk to those people who have this experience, because at the end everybody can get something out of it. It’s definitely always a win-win situation. Use these expertise and go for this experience. I think it’s for the sake of your own development, but also for the sake of the question you want to address. And this would be my very final advice.

[00:36:57] Alexander: Awesome. Yeah. Not only go to the conferences where you talk to the people that work in the same area, also go to these others like ISPOR Cochram Colloquium, these kind of areas where you meet statisticians that have a very complimentary background to your background.

Thanks so much.

[00:37:18] Michael: Thank you, Alexander. Thanks for having me. And it was a pleasure to talk with you.