Top 8: The Single Arm Studies and What are the Alternatives?

Anja Schiel

Senior Advisor Senior Advisor, Methodologist in Regulatory and Pharmacoeconomic Statistics at Norwegian Medical Products Agency (NOMA)

She has studied Biology at the Johannes Gutenberg-University, Mainz, Germany. She received her PhD from the Free University in Amsterdam in 2006 and worked several years as Post-Doc on a range of subjects focusing on oncology, immunology and molecular biology, first at the University of Leiden and later at the University of Oslo, before starting at the Norwegian Medicines Agency (NoMA) in 2012.

At NoMA she is working as Special Adviser/Statistician/Methodologist both on regulatory (EMA) and HTA projects. She has been Chair of EMA’s Biostatistics Working Party 2017 – 2019 and from 2019 – 2022 she was Chair of the Scientific Advice Working Party (SAWP) at EMA. She continues currently as alternate member of the SAWP and is member of the new Methodology Working Party (MWP) recently established at EMA.

Since it was established in 2020 she has furthermore been a member of the Big Data Steering Group, working on utilising the full potential of the vast amount of health care data for regulatory decision making.

In her role as Team-leader international HTA (iHTA) at NoMA, she has been heavily involved in EUnetHTA JA3 and its successor, EUnetHTA 21. Her particular focus is on parallel EMA-HTA scientific advices, now in the role as Vice-Chair of the JSC-CSCQ.

She is furthermore involved in several academic and PPP projects as member of Scientific Advisory boards on subjects such as RWD, Patient reported outcomes, rare diseases, paediatric drug development and decentralized trials and digital tools.

Transcript

Top 8: The Single Arm Studies and What are the Alternatives

[00:00:00] Speaker: If you’re listening to the Effective Statistician Podcast, the weekly podcast with Alexander Schacht and Benjamin Piske get designed to help you reach your potential lead great science and serve patients without becoming overwhelmed by work.[00:00:15] 

[00:00:17] Speaker: Today I have a very special guest, Anja. She’s from the Norwegian regulator and the HTA body. She has a lot of influence across Europe. [00:00:30] She speaks at lots of different conferences, webinars, all kind of different things. She is one of the key drivers of the UNITA initiatives of harmonizing HTA [00:00:45] across. And therefore is really important just generally to our field.

[00:00:50] Speaker: And it’s great to have such a outspoken statistician at one of the regulators. So stay tuned for this great discussion where [00:01:00] we will talk about sing alarm studies. I’m producing this podcast in association with PSIA community dedicated to leading in promoting the use of statistics within the healthcare industry for the benefit of patients.

[00:01:13] Speaker: Join PSI today [00:01:15] to further develop your statistical capabilities with access to the video on demand content library free registration to all PSI webinars and much, much more. Visit the PSI website@psiweb.org to learn more about PS i activities. [00:01:30] Become a PSI member today.

[00:01:36] Speaker: Welcome to Raza. Happy to, and today I’m really excited about a guest said [00:01:45] I was thinking about having for a very long time. How are you doing, Anja? 

[00:01:50] Speaker 2: Fine, thanks. 

[00:01:52] Speaker: If you’re working in the HTA area, Anja probably doesn’t need further introduction. Anja, maybe you [00:02:00] can explain a little bit what you do and how you got there.

[00:02:04] Speaker 2: Yes. I’ve been working at the Norwegian Medicines Agency since 2 20 12. I have a background in biology, actually. Mostly I drifted off into [00:02:15] statistics once the microray came onto the picture. Dataset started looking at more complex than the usual 10 mice in our experiments. And I would say as a woman with an interest in statistics, I was [00:02:30] very strongly pushed into that direction by many people because I found that in strange combination, most people don’t like statistics.

[00:02:38] Speaker 2: The real reason I ended up doing the job I’m doing is because my agency was looking for someone who [00:02:45] could explain complex statistical. Problems to people that don’t speak statistician. 

[00:02:52] Speaker: And 

[00:02:52] Speaker 2: that is where I think lots of my colleagues, we have six point now at the agency. There’s barely [00:03:00] anyone who’s a real statistician.

[00:03:02] Speaker 2: They are all coming from sideline backgrounds. I would say for the very reason that we need to communicate the translation. So it’s not the facts, but rather we explain other people what it really means for their interpretation instead of [00:03:15] packing everything in a language that nobody understands. That’s how I ended up there.

[00:03:19] Speaker: You also have a couple of additional roles now. Tell us a little bit about Unata 21 at your role there. 

[00:03:28] Speaker 2: I’m one of the vice chairs of [00:03:30] Unata 20 one’s Joint Scientific Consult Consultation Committee. The joint scientific consultations are the follow up of what used to be called the parallel advisors.

[00:03:41] Speaker 2: Which is a bit confusing term because there’s also parallel advisory [00:03:45] with the FDAs. Before that, it was called Early Dialogues. The idea is that for many years, people like me that work at the border between regulation and HTA have understood that there are problems. When those two [00:04:00] stakeholders don’t communicate, believe the industry is trying to serve the two kings almost.

[00:04:05] Speaker 2: And that can not really work unless these two kings and their kingdoms are connected and understand each other. And that is the main reason why we have these joint scientific consultations. [00:04:15] They have been one of the two elements of the three joint action programs that Unda went through in the last decade.

[00:04:25] Speaker 2: Two of the products that the industry most strongly felt were needed [00:04:30] and the joint scientific consultations had enormous popularity with the usual problem that popularity kill. The HDS have limited capacity to do these kind of advices. The demand was much higher than what we could offer, but [00:04:45] now with renewal legislation coming in 2025, the joint scientific consultations are one of the cornerstones together with the joint clinical assessments.

[00:04:54] Speaker 2: Um, 

[00:04:56] Speaker: I hope that both on the [00:05:00] regulatory side as well, on the HTA side, there will be a lot of new opportunities for statisticians to work on the non-formal side in the future across Europe, we get a much stronger representations there [00:05:15] at some point. I think that’d be. Pretty cool. We wanted to speak about a very specific point.

[00:05:22] Speaker: We both stumbled over the same LinkedIn post, said it was there following up the ISPO [00:05:30] 2022 in Europe. Someone says, yeah, we need to convince payers of single arm studies. And you said something like, yes, single arm studies should maybe not be the default options for [00:05:45] developing drugs and getting it through HTA, yet on the other side, having a complete head to head, head-to-head study is maybe also not the perfect solution either, but there surely other ways to move forward.

[00:05:59] Speaker: Let’s start [00:06:00] by talking about single arm studies. Where do you see most of these single arm studies coming in? 

[00:06:07] Speaker 2: Unfortunately, they’re coming in where they are not supposed to be. 

[00:06:10] Speaker: Okay. 

[00:06:10] Speaker 2: There’s a lot in the oncology field, the biggest area where we have a [00:06:15] problem with a single arm trial the way it is used 15, 20 years ago, you would see a single arm trial as a phase two hypothesis generating perfectly, okay?

[00:06:25] Speaker 2: If you want to do that as a pharma industry. When it comes to [00:06:30] approval, it starts getting a problem. It should be a problem, but it’s almost a no go for HTA because it’s obviously, if one understands what HTA are supposed to do, which is we first need to assess the internal [00:06:45] validity of a study, then we have to establish relative effectiveness, not just effectiveness, relative effectiveness.

[00:06:52] Speaker 2: And that in itself already tells you there has to be something to compare to. Then in a last step when we have to make this [00:07:00] decision on whether to buy something or not, we have to run through an exercise which kind of establish the external validity, meaning that you have to contextualize your results in the context of a national healthcare system.

[00:07:13] Speaker 2: That explains why 27 different [00:07:15] member states have 27 different. External validity realities they need to compare to with single arm thought trials, it starts simply by the internal validity. If you’re really honest about it, they are nimble usually also because they are [00:07:30] almost always of small sample sizes.

[00:07:31] Speaker 2: If you had enough patients, we all would agree that you shouldn’t do a single arm trial to begin with because there is the risk of selection bias, and that is a very huge problem. Single arm trials rely on a ton of assumptions. [00:07:45] Unfortunately, they often, like any justification, it’s just making a statement.

[00:07:49] Speaker 2: And as a matter of fact, we have seen in recent time that some of these assumptions have simply been wrong. Once you get more data, you realize they were wrong to start. This [00:08:00] 

[00:08:00] Speaker: examples 

[00:08:01] Speaker 2: probably do remember the publication about the accelerated access from the FDA. They said that for some of the accelerated products, either the survival data were never produced, or it was shown that there was no [00:08:15] survival benefit.

[00:08:15] Speaker 2: Mm-hmm. And in worst case, you even have a detriment on the long run when you finally start doing the real analysis in the real data and you have comparisons. And that is because this concept of single arm trial is really one [00:08:30] of a go no go decision making. This is not how either regulators or HTA are working.

[00:08:35] Speaker 2: We are not go, no go. We have a much com more complex problem, and we also have this issue that [00:08:45] the endpoints that you can use in single arm trials, they are not necessarily considered clinically relevant by HTA. Observed response rate is something that is very highly debated, and there are enough clinicians that will tell you it has no [00:09:00] clinical meaning for patients.

[00:09:01] Speaker 2: There are patients that claim it has clinical meaning for them. It depends on who you ask, but for HTAs it is a huge problem. The endpoints that we are usually willing to accept the time to [00:09:15] event endpoints cannot be interpreted in a single arm trial. Simple. 

[00:09:20] Speaker: Let’s say, so there’s a lot of question. So first is with surf rates, things that is, that has this name [00:09:30] observed in it.

[00:09:30] Speaker: So patient’s assessment, physician’s assessments, all these different things are prone to different biases as we know. They are much more susceptible to biases from investigators. You have different [00:09:45] investigators in a clinical trial compared to that. You have in. Real world. This is a typical ones. You have probably different sites, different ways to assess things.

[00:09:57] Speaker: Maybe you look more regularly, you look [00:10:00] more closely, you look more WhatsApp, all these kind of things. Complicated more. Now, if we would have something like survivor rates, how would that help then? Or which kind of problems [00:10:15] would you see remaining? 

[00:10:17] Speaker 2: The endpoint is just one aspect. The real issue for us is the comparison.

[00:10:24] Speaker 2: It’s not even the randomization because the randomization is just an [00:10:30] insurance that we don’t have this selection bias. Yeah, 

[00:10:33] Speaker: yeah. 

[00:10:33] Speaker 2: But once randomized. Everything happens in a trial. You never have full control. We understand that patients have different experiences, they do different things. Maybe they violate the [00:10:45] protocol without telling you.

[00:10:46] Speaker 2: So all these things are quite accepted. What I as a statistician find, the problem is I can live with variability, heterogeneity, protocol violations, but I need to be able [00:11:00] to assess them. I can only assess the degree and the impact if I have a comparison. This is just all assumptions, promises by someone.

[00:11:10] Speaker 2: No, it didn’t make any difference. I can’t assess this, and as a statistician, [00:11:15] that’s the only thing I can’t accept. I can live with all kind of wild assumptions, then it’s my task to prove that they are wrong or that they are overly optimistic or overly pessimistic or whatever. But in a single arm trial.[00:11:30] 

[00:11:30] Speaker 2: I’m left with nothing that would allow me even to prove that it’s wrong or biased, that the problem that I see as a statistician, if I have to go and take my health economist role, [00:11:45] there are so many additional levels that are coming in. But just to start with, as a statistician, I’m against single arm trials in a AL setting because I cannot make an inferential conclusion, and that is what I’m supposed to do.

[00:11:56] Speaker 2: That’s at least how we work normally. 

[00:11:58] Speaker: Okay, so let’s [00:12:00] speak about the comparison. In the literature, we have seen response rates of 2%, and here in the study we have a response rate of 20%. Salesforce is a breakthrough treatment. 

[00:12:12] Speaker 2: Yes, and that’s [00:12:15] definitely a problem. Nothing is more frowned upon If you look into the HTA statistician’s literature than what they call naive comparisons, it’s even the same company.

[00:12:27] Speaker 2: Running the same trial has [00:12:30] difficulties repeating their own trials. How am I supposed to come to rely on something that comes from a literature description? They had some similar inclusion, exclusion criteria, and anyone who has ever done a network meta-analysis [00:12:45] realizes that this is a guarantee for absolutely nothing because they’re going to throw out more studies with the same inclusion, exclusion criteria than you’re going to include.

[00:12:55] Speaker 2: It tells you something about the fact that randomization is at baseline [00:13:00] for the very second. You put this patient into the voice recorder randomization program, and after that it doesn’t hold anymore. It’s only the assignment to the treatment. Then everything has an impact. Your [00:13:15] treatment centers, is it multinational?

[00:13:17] Speaker 2: Is it national? Regional preferences, local preferences. And patients still have rights when they are participating in a trial so they don’t have to behave like they’re little robots. And that’s exactly the point. [00:13:30] So no, you cannot compare trials with each other unless you are really willing to go all the way trying to figure out how good are they matching, do we have additional well that we can use for it?

[00:13:43] Speaker 2: We see that quite often [00:13:45] in HTA admissions. We live with these indirect treatment comparisons and what I use them for is. To analyze how poor the fit actually is, because if you end up with, let’s say 20 patients out of 200 [00:14:00] that you can really match, it tells you something about how uncomparable the data sources actually are.

[00:14:05] Speaker: Yeah, that’s a good way. I’ve just recently seen the A PF meeting in Germany where someone talked about propensity scoring and [00:14:15] need to look into the overlap. A way to look into this is the effective sample size, and if that kind of decreases dramatically, like you just mentioned, by 80, 90%, then the overlap is not that big.[00:14:30] 

[00:14:30] Speaker: And then it becomes a lot of extrapolation and a lot of additional assumptions that it gets really tricky. So there’s a lot of thinking about, okay, let’s start with some kind of target trial. Yeah, the optimal trial. [00:14:45] So let’s say optimally you would have a one-to-one randomized study with a comparator.

[00:14:53] Speaker: Let’s call the comparator just for kind of standard of care at Simon, and you have, and that [00:15:00] would be your ideal role that you would like to run. On the other hand, you have C one on study where you only have your arm. What would be, if you go back. Away from the one-to-one randomized study. [00:15:15] The next version from that, that working towards the is one arm study.

[00:15:21] Speaker 2: I have to start with a statement that as a regulatory statistician, I agree one-to-one as an HTA statistician, I would say nope. [00:15:30] Doesn’t necessarily have to be like that. 

[00:15:32] Speaker: Okay, 

[00:15:32] Speaker 2: so our ideas about what is good evidence. Not the same as the framework regulators have established. So neither do [00:15:45] we actually insist on any P value or alpha level called it’s okay for planning purposes.

[00:15:50] Speaker 2: It’s okay to agree that this is a success criterion, so if you don’t make that, you shouldn’t call your drug a success or your drug development program or [00:16:00] your trial. In the HTA world, that really doesn’t count because if you come with a primary endpoint, that is irrelevant for us. We have to look at secondary endpoints.

[00:16:10] Speaker 2: They have not been controlled for a multiplicity most of the times, so [00:16:15] we actually look at the data and whether there is a difference, we will look at the confidence in the world and we start instantly complaining when they overlap. No matter if you somewhere at some point had a statistical significant [00:16:30] result, that’s not gonna save it for you.

[00:16:32] Speaker 2: And that’s why the RCT is not the optimal tool for HTAs per se, but we have to live with the fact that you have to have some standards. Regulators have made these standards. This is a gold standard, [00:16:45] but the gold standard isn’t the one-to-one or the P value in itself. It’s the comparative aspect.

[00:16:54] Speaker 2: Concurrent control. These two Cs are the core of what we want to [00:17:00] see. 

[00:17:00] Speaker: If you speak about current control, what’s that? What does that mean for you? 

[00:17:05] Speaker 2: It means I don’t accept something that was generated 10 years ago. 

[00:17:09] Speaker: Okay, so you, no matter how I 

[00:17:11] Speaker 2: always say that, every drug we approve is like a mini [00:17:15] atom bomb.

[00:17:16] Speaker 2: It changes the world forever because the same patient population doesn’t exist anymore once this drug comes to the market. It’ll have an impact, and drugs that have been used before are not [00:17:30] having the same efficacy relative of effectiveness in the real world anymore because of this new kid on the block.

[00:17:38] Speaker 2: It changes everything from that time point on. You might have a selection towards patients with a [00:17:45] better prognosis, getting the new drug. Patients with a poorer prognosis might be more conservative. Physicians might be more conservative in some indications. Many physicians have a tendency to say, the new is not always better.

[00:17:57] Speaker 2: Let’s be careful. Uptake it very slow. In [00:18:00] others like oncology, everybody jumps on everything new. This by definition better whether it has proved or not. And that’s exactly where I say you have that these impacts cannot be underestimated in this whole picture. 

[00:18:13] Speaker: Yep. And 

[00:18:14] Speaker 2: that, [00:18:15] and that’s where concurrent is really concurrent means.

[00:18:18] Speaker 2: Me patient in the trial, someone looks like me, has to be my control at the same time point as I am in the trial, not five years before, not even a year [00:18:30] before, because my healthcare system changes and this has to be taken along. That’s what I mean with concurrent. But concurrent doesn’t mean included in a randomized clinical trial per C, there are options that people are not sufficiently exploring.[00:18:45] 

[00:18:45] Speaker: If you enter areas where there’s multiple options in a clinical trial, I rarely see anything that has more than one active comparator. Maybe you have an active and placebo, but that you have two active comparison. I don’t know [00:19:00] that whether I’ve, I’ve seen that in HIV, but that was very specific. If you wanna compare to all the different drug out there, especially if later the, all the health economics and these kind of things come into place.

[00:19:13] Speaker: You can’t do that within [00:19:15] just one trial. Usually you would need to re, you know, go back to indirect comparison, MacBook, metaanalysis, these kind of things, isn’t it? 

[00:19:24] Speaker 2: In a way, yes. But on the other hand, there is no real, it’s a perception that you have to [00:19:30] compare yourself to everything out there. We just had a DIA meeting about the famous Pico, and the industry claims that it’s gonna be like 400 Picos in the European context.

[00:19:42] Speaker 2: That’s really not true. The majority of us have [00:19:45] one PO. That one describes likely our preferred first choice comparator, but for pretty much anybody who does cost utility analysis, the idea is that you need an [00:20:00] anchor. Yeah. From which to build onto the other comparators. That’s what we are looking for.

[00:20:04] Speaker 2: That’s why we want head-to-head comparisons, not. Versus placebo even. We don’t like placebo per se. We’re trying to explain to the industry already for quite some time [00:20:15] now that there is a strategic choice you can make because regulators will accept several comparators in your study. But if you are really smart, you will choose your comparator for your randomized clinical trial [00:20:30] based on an exercise beforehand.

[00:20:33] Speaker 2: Trying to figure out which comparator would give you the strongest network meta analysis or indirect treatment comparisons options for the HTAs. And that might not always be the [00:20:45] last drug, but rather one where there is more evidence available. And if you make that exercise or beforehand doing this network meta-analysis, you can easily figure out if there is a comparator.

[00:20:56] Speaker 2: That has huge advantages because there are many other [00:21:00] studies that would allow you to build a stronger network. That should be your first choice. Yeah. Because in the end it might be one of the picos, and the idea is that we can formulate Picos with different populations [00:21:15] or different comparators we tend to say, or meaning that we have a preferred comparator, but we would also accept another comparator and a third comparator in worst case.

[00:21:28] Speaker 2: And if any of those [00:21:30] are the ones that you can pick. That help you to then make this extrapolation to the other comparator, the contextualization that is required for different countries. Then you can make your job easier by doing your homework beforehand, not [00:21:45] afterwards. You run your trial, discuss it with a regulator.

[00:21:48] Speaker 2: They said, yes, this is okay. They don’t tell you something else will also be possible. They just say, no, this is okay, because that’s a question you’re asking instead of saying to yourself, what’s the strategic [00:22:00] best choice? And would that still be acceptable for the regulators? And probably it would be.

[00:22:05] Speaker 2: They wouldn’t say no to that either. But then if you make your homework, you realize that it does pave the way for the other analysis that are needed [00:22:15] for the 27 plus every other country in the world that does cost utility analysis. It’s not just US 27. There are many other countries that do the same. So.

[00:22:25] Speaker 2: This is strategic thinking and it means that you have to start thinking reimbursement from day [00:22:30] one before you start your drug development. When you’ve decided the goal, this is going to be a drug we want to develop, then you have to start thinking reimbursement, and how do I get it to the patient, not how do I get approval.

[00:22:44] Speaker: Having worked [00:22:45] for about 20 years in the pharma industry, I can tell you that these statisticians. So work on regulatory and the statisticians that work on HTA and work much closer together and have an [00:23:00] advantage here. The stats community within these companies is usually quite small compared to all the other communities, the medics, the the HTA market access people, they little bigger than the H than the [00:23:15] STAs department.

[00:23:16] Speaker: That can be an advantage. You can know each other quite well and help each other. You need to reach out. You need to learn from each other. I’ve yet to come across a statistician that [00:23:30] knows both worlds, the regulatory and the HTA world inside out. I’m just not sure that one career is long enough to become such an expert.

[00:23:41] Speaker: Yes, this is always the need for working together. [00:23:45] So as soon as you start thinking about your phase two, phase three plans, from a regulatory side, it is so important to internally work together and to have someone from the [00:24:00] stat side and they can also think like these kind of strategic things. There’s always usually someone that is new product planning, HTA market access person, at least in the bigger companies.

[00:24:13] Speaker: That have already [00:24:15] one drug on the market and have went through all the pain, have someone within your stats department that works on that. That is so important. So you don’t need to compare to everything within a clinical trial. You can compare to [00:24:30] one that gives you a lot of strategic options. One area is that I’ve worked a lot in is psoriasis, for example.

[00:24:38] Speaker: And there are two drugs. One is etanercept and the other one is Ustekinumab. [00:24:45] Lots of studies have been run against these, most studies have been run against placebo, but if you look into the network, these are the ones that really stand out. Now, etanercept is maybe not so [00:25:00] much the standard of care anymore.

[00:25:02] Speaker: Kinumab is a little bit more kind of recent. It’s probably still the workhorse of many dermatologists out there, so that might be a good choice for a comparator where you [00:25:15] get a lot of bridge comparisons against all the new treatments out there because there’s a lot of studies against Ustekinumab, and that gives you strengths you need.

[00:25:25] Speaker: While also giving something relevant for [00:25:30] day-to-day physicians. Maybe not the avant-garde dermatologists that always jump to the newest ones, but dermatologists are more conservative. They stick with their standard treatment for quite some time. Maybe that’s also another problem, but we have lots of [00:25:45] opportunities.

[00:25:45] Speaker: Okay, very good. So we have the relevant comparator. It’s maybe one to one randomized. What would be the first step towards a one arm study? 

[00:25:56] Speaker 2: I always think in terms of what’s [00:26:00] the question you need to answer? I am really a lot in favor of thinking adaptive design, because that really does allow you to stop your control arm, for example, once you reach the [00:26:15] thresholds crossing, or you can define.

[00:26:18] Speaker 2: Hallmark reach points that you have to reach in some of the rarer diseases. You can probably best define something where you say, okay, now we’ve seen enough and we’ve seen enough [00:26:30] this, where sometimes the statisticians are getting in the way on the regulatory side. They wouldn’t get in the way on the HJA side because enough is not the same as statistically significant.

[00:26:42] Speaker 2: So yes, you cannot [00:26:45] accept a certain risk accepting you’ve seen enough, which might not be the same as the inferential framework. Uh, and you need to be willing to discuss it. What is enough? When do I feel that I have seen the data that make me [00:27:00] certain enough on safety or an efficacy endpoint that I would accept that now the controls can go out.

[00:27:06] Speaker 2: So it depends also on your claim, and I think it’s a bit broken in our system. Patients, payers, HTA, we all want [00:27:15] better drugs, but if we can’t get better doesn’t mean that someone who is as good couldn’t come to the market, preferably with a lower price or preferably with another advantage. So yes, there are reasons why you [00:27:30] want additional drugs on the market, but the idea seems to be that everybody says we have to be better, but that’s so difficult.

[00:27:40] Speaker 2: To avoid having to be better, we do a single arm trial. Yeah, which [00:27:45] is a completely unlogical choice, and that’s where I am most frustrated with the system. It doesn’t have to be better, it can be just as good. The point is that you have to prove it, and the proof is only possible when [00:28:00] it’s comparative.

[00:28:01] Speaker 2: It’s never possible by just claiming that I’m potentially better, theoretically better. No, bring me data. Let’s discuss what this data has to look like. And there are many options to [00:28:15] decide on. Maybe your side effects are different, or the frequency of administration is preferable for some patients, but not for others.

[00:28:23] Speaker 2: The issue is the proof. You have to prove any claim of better, just as well as you have to prove any [00:28:30] claim of not worth then, and that has to come from, and there always has to be some. R CT part in your development program? How big that has to be. That’s a different point. Uh, your trial is going to [00:28:45] be enormously big if you have to include a very heterogeneous patient population.

[00:28:50] Speaker 2: The discussion has to be not around. How can we minimize the costs and the burden to the companies in the development program? We want to spare patients. [00:29:00] The point is, which questions? Can only be answered by comparative data and which questions can be potentially answered by a different approach. For example, for me, and many have published stuff on it, including me and a couple of [00:29:15] colleagues on hybrid designs, where we say, not everything has to happen In the RCT, you can find additional information, for example, on populations that you do not want to include in the trial due to heterogeneity.

[00:29:28] Speaker 2: You can find information [00:29:30] on the natural history. Please not from 20 years ago 

[00:29:34] Speaker: about hybrid designs. It means you have some external control. 

[00:29:39] Speaker 2: Yes. Observational aspect I would call them. And there are many options. Few of them are [00:29:45] discussed because at the moment it is pretty much the thinking that is it go observational.

[00:29:51] Speaker 2: Then we do that in phase four, you know, and everything is already broken. Created a problem by design. We did [00:30:00] not do the right trial for everyone. We have missed collecting information on subpopulations or on other comparators, on different endpoints that are more relevant for others. But now we want to fix this problem after the facts, and that’s [00:30:15] never gonna work.

[00:30:15] Speaker 2: We all know that’s not gonna work because concurrent is the miracle word when it comes to this, and that’s where I think statisticians have a huge potential. To show everything between the RCT and the single ARM [00:30:30] trial, there’s a myriad of possibilities. What you could do, that minimizing number of patients that has to be potentially exposed to some they might find unattractive.

[00:30:42] Speaker 2: But keep in mind, unless you have proven [00:30:45] it black on white, that you are better or as good as any claim with. Drawing the equipoise of the development program are really misplaced. Patients also need to understand that no is [00:31:00] not always better. I always say there’s a reason why we have double blind as an aim, because physicians are equally bad in judging what is really going on.

[00:31:10] Speaker 2: They also mostly see what they want to see. As a statistician, [00:31:15] when I tell people what we are doing, I always say we are the science of. Trying to help people to avoid seeing things that are not there. That’s what we do. 

[00:31:27] Speaker: Seeing the things that we wanna see, 

[00:31:29] Speaker 2: trying [00:31:30] to give them a possibility to understand no matter how much they want something to work, there’s an alternative explanation, and if there’s enough evidence that the alternative explanation that it doesn’t work.

[00:31:42] Speaker 2: Is supported, then they have to simply accept that fact. [00:31:45] That’s what statisticians are supposed to do. We have to explain to them the potential of misinterpretation, the risk they’re taking. If they make a decision based on something that’s not solid enough and not robust enough, understanding the dangers [00:32:00] of not having a control arm.

[00:32:02] Speaker 2: And yes, at the very end of this discussion, when we have offered them many alternatives in between, that would all have. Benefits in terms of their design being more robust, more attractive, [00:32:15] and still attractive enough for patients, then we can in the end come to the conclusion that in some instances, a single arm trial is the only option, and I agree on that.

[00:32:26] Speaker 2: But an important aspect is always you need [00:32:30] to make a difference between generating just evidence for some signal of efficacy. The wish to generate scientific knowledge. I think trials have to be the latter, not the [00:32:45] first. They need to contribute also to the scientific knowledge. Doing a minimalistic design is when the industry refuses to contribute to the building of scientific knowledge.

[00:32:57] Speaker 2: We have to protect participants in trials [00:33:00] as best as possible, but then again, a trial is an experiment. There is no human right to participate in a experiment. It’s your free choice. There’s also no human obligation to participate in an [00:33:15] experiment if you don’t want to. What the experiment should lead to is information for future patients.

[00:33:21] Speaker 2: At the moment, with the way we are developing drugs, we are sacrificing the future patients with the argument that we have to protect the participant in [00:33:30] trials from having to do something unpopular. I keep wondering all the generations before us that had to go through all the randomized clinical trials to give us the drugs we have today, have they all been idiots?

[00:33:44] Speaker 2: No, [00:33:45] they haven’t. And do we have an obligation to try to at least participate in the these experiments for the greater good? Yes, we do. If you don’t want to, you don’t have to. But if you do, then please participate in a good trial. 

[00:33:59] Speaker: Yep. [00:34:00] And actually I think it is. I would put it the other way around. It’s ethically questionable to put someone in a bad study.

[00:34:09] Speaker: You put them at risk without a lot of benefit in the end. I really like the approach [00:34:15] with the adaptive design because that gives you lots of opportunities, especially if you at the same time also run a prospective observations where you get exactly what you talked about, the [00:34:30] concurrent treatment. If you get patients from other sites that don’t participate in the study, you get maybe older patients from the sites that don’t fit the exclusion criteria are certain vulnerable populations whatsoever, [00:34:45] and that helps you to exactly establish that framework that you talk about in terms of external, what did you say, contextualization.

[00:34:55] Speaker: And it also helps you to understand. What’s really going on in the [00:35:00] clinical practice? Usually we have different ways of treatment in different countries, in different regions of the world. And if you’re talking about the rare disease, that also a great opportunity to get in touch [00:35:15] with all the different researchers around the.

[00:35:17] Speaker: I wouldn’t call that my primary objective here, but getting in touch with all these people that care about the same patients that you wanna treat in the future. And so that one that should benefit from ment in the future [00:35:30] has never been a bad idea. I think that’s a very good approach. By the way, these types of studies help you get data on many more.

[00:35:40] Speaker: Things that you will get off throughout the HTA process, [00:35:45] any EPIT data, any kind of burden of disease data, any data about what are the typical treatments, I would say what are the treatment patterns? Where are the problems? What are the patient, what do patients care about? [00:36:00] All kind of different things you can learn from this observational data that you actually cannot learn from the RCT and I completely agree.

[00:36:09] Speaker: Observational studies shouldn’t be only run in phase four. I think they are [00:36:15] very well placed before approval. In an observational study, you can’t have your experimental work, but you can collect lots of data that help you contextualize your experimental work data. 

[00:36:26] Speaker 2: Yes, absolutely. Those areas where everybody agrees that maybe a single [00:36:30] arm trial is the only option in those orphan diseases where you have a hundred patients globally.

[00:36:36] Speaker 2: You want these physicians and these patients to build a network because most likely they will show you that they have poor data on [00:36:45] their natural history, which kind of speaks again against the single alarm trial. They have an extremely heterogeneous treatment option because everybody tries something, but nothing really works.

[00:36:59] Speaker 2: That’s the [00:37:00] disadvantage of being so rare instead of. Trying to identify the evidence gaps at the start, work towards it and say, okay, maybe it isn’t this generation. We can bring a good treatment, but for the next generation we [00:37:15] can build a basis on which we can finally identify a working treatment.

[00:37:19] Speaker 2: That’s in the often area where you really wonder, is the treatment really not working, or is it just because the data are so poor that we can’t see? It might be working. [00:37:30] That’s where you run into this, where you say, okay, that is just pool planning from everybody’s side. It’s not identifying early enough what kind of evidence is needed.

[00:37:39] Speaker 2: And that evidence is not just the trial for my drug. It is [00:37:45] this larger context that you’re describing, what I call the scientific knowledge. And scientific knowledge is actually what is needed for good decision making and decision making falls. Into the category [00:38:00] of HTAs payers, physicians, patients, their families.

[00:38:05] Speaker 2: We all have decisions to make and we all feel like we have to make them on insufficient information [00:38:15] coming from drug developers. And that’s where the line is where we say this has to change. It simply has to change because I feel. Uncomfortable. I’m getting old enough. I’ll be a patient sooner or later for something more serious.

[00:38:28] Speaker 2: If I go to my [00:38:30] doctor and I have the idea, they look at me and say, Hey, yeah, we have this armamentarium. I have this box. 10 drugs in there. Make your pick. Take your favorite color or the package size, because as a matter of fact, I cannot tell you. You [00:38:45] should, you as a patient should take first the green ones, then the blue ones, and then the yellow ones, because I have evidence that supports this.

[00:38:53] Speaker 2: Instead, they’re just offering me all of this and it’s again, back to where we started, trial and error. [00:39:00] That was a reason why we invented statistics in the first place. Why we invented RCTs, why we are hammering on that. We need better evidence because we don’t want to have destroyed an error yet. When you go into the clinic, you start feeling like, [00:39:15] okay, there’s an awful lot of tried and error going on.

[00:39:18] Speaker 2: Awful lot of, in my experience, coming from your doctor and I’m really like, I don’t want experience. I want evidence. 

[00:39:27] Speaker: Yeah, [00:39:30] completely agree. You basically just summed up my personal vision. I want to make sure that payers, physicians, and patients. Especially patients and as their caregivers, their parents, [00:39:45] their kids, if they are elderly, have the right evidence.

[00:39:49] Speaker: It’s the right time and in the right format to make the right decisions. It’s not sufficient that the evidence is hidden behind the paywall of a journal [00:40:00] or. In clinical trials.gov said nobody other than us that decisions. And sometimes even we can’t understand what’s really in there. Even if the evidence is there.

[00:40:12] Speaker: It’s hardly communicated well [00:40:15] throughout the system. Through the, we can make informed decisions. Data literacy is surely one part there, but as a industry we can definitely improve that. 

[00:40:25] Speaker 2: Yeah, absolutely. And I think one of the things that I. And I was really [00:40:30] happy was I know that not everybody’s so fond of the Estiman framework and feels it’s just adding another layer of complexity to everything.

[00:40:38] Speaker 2: But I did discover a couple of studies that didn’t use some kind of sensei acronym as [00:40:45] a title of their study in clinical trials.gov anymore, but they actually use the estimate description. Oh, and that is something that I would really love to see everybody do. Don’t give me the Wonder or [00:41:00] Miracle acronym that you torture out of whatever description you have given to your trial, but tell me really explicitly on the first page in the description, what your trial is actually doing, because that would help me make a better selection.

[00:41:13] Speaker 2: No reports, no data [00:41:15] reported. 99% of everything I look at apparently. I’m really poor at finding studies with results, I would say. So that’s another step forward. Everybody has to learn, and that’s where the Esteban framework really is still not [00:41:30] reaching the right audience. That’s all. But Einstein said, if you can’t explain it simply, then you still haven’t understood it yourself.

[00:41:36] Speaker 2: Yeah, you need to be able to explain what your trial is going to do. Which question are you going to answer? [00:41:45] Then you can go to others and say, is this actually a really relevant question for you? And that includes patients and they will, I think, very often tell you, I don’t really understand why this is a relevant question to begin with.

[00:41:58] Speaker 2: That’s where the dialogue starts. [00:42:00] That’s where you start understanding, okay, I’m doing something. It isn’t helping someone else to make a decision. So why is that the case and is there something we could do to improve that? At least it doesn’t have to be perfect. You cannot ask patients what they find [00:42:15] important when that is something you cannot operat nurse in a trial.

[00:42:20] Speaker 2: I understand it, but you can still at least then start thinking, okay, if that is so important for patients, can we somehow generate evidence around that topic in some other [00:42:30] way? It doesn’t have to be an endpoint in the clinical study per se. Can we find some other way or can we simply at some point start designing studies that are never meant for approval by FDA or EMA?

[00:42:42] Speaker 2: But these are studies that [00:42:45] are meant to provide information, relevant information for others, and they don’t need to follow some kind of statistical rigorous framework. If you’d be honest about it and say what you really want to do, you can still use the statistics. [00:43:00] You don’t have to be a sucker for the alpha 5%.

[00:43:03] Speaker: We touched a lot of stuff in this nearly one hour chat about HTA regulators, one on studies, clinical [00:43:15] trials that are had to head what is actually good comparator. What is this, uh, strategic choice? The overall evidence that you need to have that is not just your clinical trials. There’s a lot of companies that talk about it.

[00:43:29] Speaker: The [00:43:30] integrated evidence plan, you are in that as a statistician or as a STA function. I think we can drive that very nicely and we talked about a couple of different design options as well to get there. Thanks so much, Anja. That [00:43:45] was an outstanding chat. If you would like the statistician listening to this with one key takeaway, what would that be?

[00:43:55] Speaker 2: We have to see ourselves as facilitators [00:44:00] for trial designs that answer scientific questions. We are scientists and we should stand for that, and I know it’s very hard to pick up that fight for anyone. It’s a hard fight [00:44:15] on the regulatory side. It’s a hard fight on the HTA side. It’s definitely a hard fight within pharma companies.

[00:44:22] Speaker 2: I know the argument. Why should we do something more? Because our competitor got away with a single arm trial. [00:44:30] That’s where we have to really push extremely hard and say even if nobody does it wrong is wrong. Even if everybody does it. That should be our motto for 2023, I think. 

[00:44:43] Speaker: Yeah, [00:44:45] upskill your leadership and influence skills and then let’s do that together.

[00:44:51] Speaker: Thanks so much. Have a nice time. And maybe we speak again. 

[00:44:56] Speaker 2: Would love to. Thanks Alexander.[00:45:00] 

[00:45:02] Speaker: I do hope you enjoyed this discussion with the show was created in association with PSI, thanks to Reine and her team at VVS, helping with the show in the background. And thank you [00:45:15] for listening. Reach your potential. Really great science surf Asian. Just be an effective [00:45:30] statistician.