Combining RCT and RWD – Applications in Rare Diseases and Practical Recommendations

Transcript

Combining RCT and RWD – Applications in Rare Diseases and Practical Recommendations

444_Combining RCT and RWD – applications in rare diseases and practical recommendations

[00:00:00] Alexander: You are listening to the Effective Statistician podcast. The weekly podcast with Alexander Schacht and Benjamin Piske designed to help you reach your potential lead great science and serve patients while having a great [00:00:15] work life balance.

[00:00:22] Alexander: In addition to our premium courses on the Effective Statistician Academy, we [00:00:30] also have. Lots of free resources for you across all kind of different topics within that academy. Head over to the effective statistician.com and find the [00:00:45] Academy and much more for you to become an effective statistician. I’m producing this podcast in association with PSIA community dedicated to leading and promoting use of statistics within the healthcare industry.

[00:00:59] Alexander: [00:01:00] For the benefit of patients, join PSI today to further develop your statistical capabilities with access to the ever-growing video on demand content library free registration to all PSI webinars and much, much more. [00:01:15] Head over to the PSI website@psiweb.org to learn more about PSI activities and become a PSI member to pick.

[00:01:29] RIchardus: Ladies and [00:01:30] gentlemen, it’s my pleasure to introduce our next speaker, professor Tim Friede, who I’ve known personally for quite a number of years. I can really attend to his expertise. Professor Friede’s, bio statistician [00:01:45] has made significant contributions to the field of medical statistics like clinical trial design.

[00:01:50] RIchardus: Since 2010, if I’m not mistaken, Tim Salt is a professor of biostatistics at the University Medical Center in Ingham, where he leads the Institute of [00:02:00] Medical Statistics. His academic journey began with a diploma in mathematics from the University of Kazu, followed by the doctorate from University of Heidelberg, and he has been working across Europe in Lancaster University, Novartis Pharma in Basel and [00:02:15] Warwick Medical School before his tenure.

[00:02:20] RIchardus: Needless to say his expertise is widely recognized in 2014, he was awarded biometrics and medicine certificate by the German Society for Medical Informatics Biometrics [00:02:30] and Epidemiology, and the German region of the International Biometric Society. Not only that expertise involvement extends to Thesal German Institute for Social Computing and the Compost Institute [00:02:45] Data Science without for real due.

[00:02:48] RIchardus: Please join. Welcoming Tim Frieder to state is yours. 

[00:02:53] Tim: Tim, thank you very much, Richard for the very kind introduction. [00:03:00] Tim, you can go. Thank you very much for the kind introduction. Let me share the screen. I just saw that we are now over more than 40 attendees in this session. There are still some people coming in.

[00:03:14] Tim: [00:03:15] Let’s make it a slower start so that people can join. I hope you can hear me well and see the video, so then I can share now my screen. Yeah. So the subject for today is [00:03:30] combining randomized control trials and real world data. Applications and rare diseases and some practical recommendations. Thank you very much also for the invitation to present at the Effective [00:03:45] Statistician for conference.

[00:03:49] Tim: Here’s an example that got us involved in all this and made us thinking, of course, this is the publication and personal thinking didn’t start with the publication. It [00:04:00] started earlier at a group of neurologists from our university hospital. That I think that say, approached us at some point and said they had embarked on running a randomized control trial and early [00:04:15] cords for disease CJD to test to clean as a drug in that indication.

[00:04:21] Tim: Actually they didn’t manage to randomize many patients, but at the same time, those that declined [00:04:30] participation in the. Randomized control trial. Several of the patients accepted to be followed as part of a registry. So they had now this data from the RCT and some observational data from the registry.

[00:04:44] Tim: They [00:04:45] approached us for some advice on how to analyze the data. The acro disease is fortunately, very rare with the prevalence of something like one to nine cases in a million people. So [00:05:00] by definition of the EU, it qualifies as a rare or maybe even ultra rare disease. As I said, they conducted a double blind randomized phase two trial, originally plans to randomize [00:05:15] 150 year patients or something like that.

[00:05:17] Tim: They ended up with a rather small number of randomized patients, but at the same time, they recruited patients into an observational study. So you can see here, 88, there were these two [00:05:30] data sources, and I think that was our main contribution to the publication, pooling all the patients across the RCT and the observational data that received doxycyline and compare them against [00:05:45] placebo or no treatment with doxycyline and the registry data as I think the clinic clinical colleagues had in mind.

[00:05:52] Tim: We advise them to keep the stratification by study as you would as a statistician. [00:06:00] And so then also we in particular we help them not only doing the analysis, but also advise them on how to sort go about analyzing some observational data here. We use some propensity score based [00:06:15] methods to account for the non-randomized nature of that observational.

[00:06:20] Tim: So if you do these analysis, of course, both will give you, let’s say, lock hazard ratio with a standard error. You can then combine these two [00:06:30] effect estimates with the standard aerosol confidence intervals in a meta analysis to yield one overall. Result, combined effects with confidence interval. We suggested to them, because these are [00:06:45] different data types, et cetera, we said, okay, let’s use a random effects, a meta-analysis to do this.

[00:06:51] Tim: But as you can see, the estimation of the heterogeneity resulted in zero. So the variance component for the treatment effect [00:07:00] heterogeneity of the two data sources results in zero. So the confidence interval that we got. For the combined effect excluded one that just stopped below at the P value. Here, it’s rounded [00:07:15] to two decimals, but it was actually 4.9%.

[00:07:18] Tim: Those that work with clinical colleagues know they like P value smaller than 5%, so they were easily convinced that this was a really good idea. I thought I’m a [00:07:30] great statistician. And that’s basically what went into the publication. At the same time, we also started, we’re working here on a EU funded project in Inspire, and so it’s a, was a project on [00:07:45] research methodology and rare diseases or small populations, rare diseases and pediatrics.

[00:07:51] Tim: We also had some interest in how to combine like a small number of relative heterogeneous [00:08:00] studies. And so first the point was really to rethink whether this is an appropriate sort of meta-analysis of these two studies. So we looked for some alternatives, maybe one step back to combine [00:08:15] data from different sources for studies of the same type RCTs or registries, but also combining data from different types of studies.

[00:08:26] Tim: We typically use hierarchical models because we have [00:08:30] this structure, we have the data on the patients, and they are clustered in certain studies, registries, RCTs, et cetera, or maybe also electronic health records stemming from one particular clinical [00:08:45] center. Then we want to combine the evidence across these studies.

[00:08:49] Tim: One standard model to do this is the normal hierarchical model, which is the standard model for random effects meta analysis where we [00:09:00] allow some heterogeneity of the study specific treatment effects. These are ct. I may come from a distribution around the grand mean. We can note here by mule with some variance.

[00:09:13] Tim: Tall square. [00:09:15] Then on the study level, we observe a treatment effect, why it comes from a normal distribution given the study specific treatment effect theta and the standard errors [00:09:30] sigma of the treatment effects In study I do to be, no. And so this just a, of course could discuss whether it’s always a reasonable assumption, but it’s a model that’s very commonly used.

[00:09:42] Tim: Because many of the effect measures that we [00:09:45] use follow approximately normal distributions, lock hazard ratios, lock S ratios, et cetera. So this is the standard model. But if we use some standard methods like the Morgan LA standard [00:10:00] methods for random effect, meta analysis, then we know that if we have a small number of studies, basically anything below 10 or particular, if it’s below five studies.

[00:10:10] Tim: The coverage probabilities are well below the [00:10:15] nominal level. In this simulation, the nominal level was 95%, and the action level, depending on how pronounced the heterogeneity was sometimes even well below 90%. In particular, if you look here where you have three or four studies, [00:10:30] or in the case presented only two, we know that in situations where there is some heterogeneity that I combine.

[00:10:38] Tim: Like a randomized control trial and the registry and very different analysis approaches. Yeah, [00:10:45] so I would expect some heterogeneity. I want to account for this in the analysis, but with standard random effects, meta-analysis approaches, this won’t work well. So what else can we do? [00:11:00] And one idea, it’s not new, it’s around, also around for some time is to.

[00:11:06] Tim: Course use patient analysis, and here the idea is really to use, and the way we use it is to use a weekly [00:11:15] informative trial prior on the between trial heterogeneity and use an uninformative prior on the treatment of pax. So we basically say we don’t wanna introduce any information regarding the treatment [00:11:30] effect itself.

[00:11:31] Tim: But we want to acknowledge that it’s very difficult to learn anything about the, between the study heterogeneity from a small number of studies. So we want to reuse, [00:11:45] if you like, some experience from previous meta-analysis. Yeah. So we have some idea on what’s a reasonable range of heterogeneity and use this for the analysis of our next meta analysis.

[00:11:59] Tim: The [00:12:00] advantage here is we avoid zero estimates of the between trial heterogeneity, and more importantly, we account for the uncertainty and estimation of the heterogeneity. There’s the problem with the, so what I call standard [00:12:15] methods, where we use like normal ALS to construct the confidence interval that we basically.

[00:12:22] Tim: Condition on and observed between trial heterogeneity, we estimate the to and then pretend it’s null now. [00:12:30] And in a situation where actually there is a large uncertainty or there’s little information on the size and the patient framework incorporates how we deal with the uncertainty and the estimation automatically.

[00:12:44] Tim: So that’s great. [00:12:45] Yeah. But from practical point of view, you might argue well. If they I need some MCMC sampling, then I need some experience and diagnostics of MCMC sampling, et cetera. And [00:13:00] that may be means that some colleagues with little or no experience shy away from using this type of methodology.

[00:13:07] Tim: My colleague Christian River had a great idea for this specific model, normal hierarchical model [00:13:15] to fit it in Beijing. He derived a different algorithm. So this is very stable and it doesn’t need any inspections on convergence as you would need from MCMC sampling at the same time. It’s also very fast because it’s [00:13:30] just made for very specific situation.

[00:13:33] Tim: And this basically enabled us to run a lot of simulations. I think that was the first thing as a second advantage. I think, as you will see in the following, it means that the [00:13:45] application of a patient, random effects, meta analysis, is really easy and most people are convinced by just showing how few lines these are and when they see I can do it, then most people are convinced.

[00:13:57] Tim: They also will manage. [00:14:00] There is an R package available, developed and maintained by. Ian River Code base matter. You will see some lines of code as we go through the E examples, and you will see how straightforward it is to conduct this [00:14:15] type of random effects, meta analysis. The key point really is how to choose the prior for the heterogeneity parameter.

[00:14:24] Tim: This is a almost a running gag and internal seminar series [00:14:30] where. One of the colleagues always asks, where does the prior come from? And I think that’s appropriate question of course. But in this particular setting, we have not only theoretical arguments, but we also conducted large scale [00:14:45] simulations.

[00:14:45] Tim: We explored certain choices of prior distributions and their characteristics. Not only Beijing characteristics, but also in particular also frequentist properties. Actually, and this is the paper here [00:15:00] below. You can also use empirical data in this case, databases of meta-analysis to then empirically inform the prior distribution.

[00:15:11] Tim: So that’s also something, as you can see, we have done with [00:15:15] some colleagues here, Stewart Vendor, who worked for the German HTA agency called I You and for the I. It’s very important that they can pre-specify their analysis and that [00:15:30] all the analysis have a good foundation and that it’s not perceived as being subjective.

[00:15:35] Tim: Actually, in a follow up paper to this one, they applied the method described in this paper to their own database and [00:15:45] derived the prior distributions for random effects analysis that they’re going to use future assessments of new interventions. So I think this also has some clear, practical implications.

[00:15:59] Tim: Yeah. [00:16:00] Now back to the CJD example. The results that we’ve seen earlier are the randomized control trial observation study here now the effects and represented on the lock scale with their standard [00:16:15] errors. And now we conducted the evasion analysis with a half normal prior for TOR on the scale 0.5. So there’s also something that we, for this type of effect, measure, low hazard ratio would recommend.[00:16:30] 

[00:16:30] Tim: And then you see you get a much wider interval than the one we had seen previously. The previous one stopped just below zero. Now it overlaps the now hypothesis. With this analysis, we are less certain about the effect [00:16:45] size compared to the standards, random effects, meta-analysis. We applied and published in the paper and the previous analysis Tor was estimated to zero and here we used this prior for Tor, and [00:17:00] we introduced basically some uncertainty and the get a wider interval.

[00:17:05] Tim: So at the bottom we also plotted the prediction interval. So what do we mean by prediction interval? So that’s the 95% [00:17:15] range where we would expect theta I star, let’s call it, for a new future study that’s even wider. Of course, always wider than the credible interval for the combined effect. That’s also sometimes of interest in [00:17:30] communication with the clinical colleagues.

[00:17:31] Tim: You need to be careful because it’s not the effect to be observed in the future study, but the true effect in the future study to lie within this range. The effect to be observed in the future [00:17:45] study, the interval would be even wider and would depend also on the size of the future study. These are things that people sometimes look at.

[00:17:53] Tim: Some people recommend presenting prediction intervals to emphasize the heterogeneity [00:18:00] between different studies. For us, it meant we thought once again about, is it really the combined effects that we are interested in? In this particular situation where we embark on an RCT then only conducted the [00:18:15] small scale.

[00:18:15] Tim: RCT used some observational data to support. RCT. That’s maybe not the situation where we are really interested in the combined effect. The diamond at the bottom that we presented. Yeah, as a [00:18:30] confidence interval in the frequentist analysis or as a credible interval invasion analysis, it’s what we look at typically in this kind of standards have wise.

[00:18:41] Tim: Meta-analysis. Sometimes we are interested in other [00:18:45] quantities and we talked already about the effect of a future study using the prediction interval. But maybe in this setting here, we can also make the case that we are actually interested in estimating the effect of an [00:19:00] individual study in the light of the other studies.

[00:19:02] Tim: Yeah, so look at the shrinkage estimator. We say, okay, we want to estimate the effect in the RCT, but we are not doing this now by only considering the data of the RCT, [00:19:15] but we are looking at the RCT and we borrow some information from the registry. So that’s what we are going to look at next. Actually, this technique here, it’s introduced in the setting where we have an [00:19:30] RCT and we combine the RCT with real world data.

[00:19:33] Tim: These techniques could equally be used, for example, to borrow between subgroups in a basket trial or for bridging status. It’s the same idea of [00:19:45] using this meta-analysis framework and then do some boring through this hierarch model. And let’s go back to the example here. We have a plot that’s similar to the one we have seen before.

[00:19:59] Tim: [00:20:00] It’s again on the. Hazard ratio scale. So we see in dark the lines we have seen before. It’s the estimates with a 95% confidence intervals. And we have this here for the RCT, the wide interval, [00:20:15] and we have the shorter interval because we have more information from the observational data. The registry 12 and 88 patients, as we mentioned before.

[00:20:25] Tim: This credible interval here at the bottom is also the same that we have seen [00:20:30] before. What’s new are these lines in gray that we introduced here, and these now represent the effects with 95% credible intervals. From what we argued earlier, we would be [00:20:45] interested in the RCT effect in the light of the registry.

[00:20:49] Tim: So this effect with this interval, no. Why is it called shrinkage interval? Because the effect is shrunk towards the overall mean. The effect [00:21:00] changed a bit now, but the more pronounced the effect nearly is the on the length of the interval. Now, if we only consider the data from the RCT, we have this quite wide interval.

[00:21:13] Tim: If we estimate the effect [00:21:15] of the RCT within the meta analytic model, we get a much shorter. Interval. The RCT shrinkage interval is only 66% of the original length, so we shortened it, [00:21:30] and you can interpret it in terms of sample sizes where we say, okay, we did an RCT here with 12 patients and we have 88, and the observation and the registry.

[00:21:41] Tim: You would also expect from the trial that includes. [00:21:45] 27 are randomized patients, so these 88 patients from the registry basically count like additional 15 randomized patients. So there is quite some substantial down weighting going on here. [00:22:00] This down weighting is also dynamic, so we always down weight by the degree that we are down.

[00:22:06] Tim: Weighting will depend on the heterogeneity between the estimates from the randomized control trial and [00:22:15] the registry, and thereby being robust to some extent. That’s what we want to look at next. Here is first before I go into the robustness properties, [00:22:30] I just show a few lines of code here to demonstrate how these things are applied.

[00:22:35] Tim: Using the base meta package, first of all, here we have a part at the top where we hand over the data. Now we have the two [00:22:45] studies, so we have two lock hazard ratios with their standard errors. So that’s basically all the data here. Then the analysis is done using the base meta function. If you have used other meta-analysis [00:23:00] packages, such as metaphor or meta, then you will find the structure here very similar.

[00:23:05] Tim: We hand over the vector with the treatment effects, the vector with the standard errors. We can also hand over some labels. Yeah, so just [00:23:15] observation, randomized in this case. And because this is now based analysis, we specify the prior here in the example, we use half normal prior with scale 0.5. This will produce some standard [00:23:30] outputs.

[00:23:30] Tim: In the addition, we can also outputs request outputs for the shrinkage estimates and do the calculations that I just presented regarding the interval length. So that’s quite straightforward. [00:23:45] I’m also aware that there are some comments coming in, in the chat and maybe I just also break very briefly here to respond to some of the comments.

[00:23:57] Tim: I see comments by innocence from [00:24:00] South Africa. I know him through my work as chair of the Representative Council of the International Biometric Society, where we collaborate. And so I think one point here is the, if I go from the back and take [00:24:15] the last question he asked, any comment on the effect sizes, minus 0.5 versus minus 0.17.

[00:24:23] Tim: How reliable is the weight that average in this case? So I think what innocent is [00:24:30] pointing out is the fact that there is some difference. Observe the effect in the observation data is more pronounced than the effect in the RCT and course it looks like it’s bias towards a larger [00:24:45] effect size.

[00:24:45] Tim: And I will comment on this in that. Next slide. The other question is regarding the direct algorithm. I don’t comment really in detail on the algorithm, but you can think of it more like a numerical [00:25:00] integration type of technique rather than a random Monte Carlo simulation sampling technique. And I think then there was another question by Tina where she writes with rare diseases and in brackets [00:25:15] and associated low sample size.

[00:25:17] Tim: How do you overcome a potential issue with low power? By default, you are bound to use the different studies available, but heterogeneity must have wide an effect [00:25:30] on the available power of the overall analysis. Yeah, I think there are several good points. One is actually the problem with small studies in particular.

[00:25:41] Tim: If we. Face problems with [00:25:45] randomization. So I think that’s exactly what I’m talking about. I agree. That can be a struggle. So we are using the registry here to help the RCT to gain efficiency here, presented in length of the [00:26:00] interval, but we could also turn it in terms of. Power. But I think there’s another comment here, and that’s very valuable.

[00:26:07] Tim: Of course, in the planning, you might not be able to really characterize this heterogeneity well and all those people that [00:26:15] do random effects, meta analysis, they know adding in additional studies does not always mean the resulting confidence interval or credible interval is getting shorter. More data can [00:26:30] sometimes mean.

[00:26:31] Tim: Even less information if the additional data introduce additional heterogeneity. I think that’s also an important point to note. Thank you very much for that comment. We can come back to this in the [00:26:45] wider discussion later. Let me now continue with the question regarding robustness. This is a graph that we have in the paper that is quite complex, but I think it gives some good intuition into [00:27:00] what’s going on.

[00:27:01] Tim: So first of all, we have the effect estimate of the RCT. We assume it’s zero, doesn’t matter, and we have a confidence interval around it. That’s the effect of the RCT with a confidence interval. [00:27:15] What else do we have? We have the green line. This is our Y two. That’s the effect observed in the registry. Also with an interval that you see.

[00:27:25] Tim: This interval is much narrower in comparison to the wider interval from [00:27:30] the RCT. We have much more data in our registry do have in the RCT and in this point here where the difference between. The effect observed in the registry and the effect [00:27:45] observed in the RCT here, the difference is zero, and what happens now is.

[00:27:50] Tim: This Y two is moving away from Y one. So the effect in the registry is getting larger and more different from the effect observed in [00:28:00] the RCT. We want to study what’s happening in this situation now. As a blue line. We have the shrinkage estimator with an interval. The interval plotted here as dashed lines.

[00:28:11] Tim: If you look at the width of this shrinkage interval, [00:28:15] you see this is shortest. Yeah. At the point where the two treatment effects are the same. So if you get the same estimate from the registry as you got from the RCT. The registry is supporting the [00:28:30] RCT, which makes sense because we think it’s the same thing going on, same effect size as the effect in the registry becomes more and more different from the effect in the RCT, it starts to pull away near the [00:28:45] RCT effect, the shrinkage effect is more being away from the.

[00:28:48] Tim: Effect observed in the RCT, but at the same time, the interval gets wider and wider. But you see also this pulling away, this effect is leveling off now. [00:29:00] So at some point, first the free judge estimator follows the effect of the large registry, but then at some point. It introduces more and more heterogeneity as the effects become more [00:29:15] different, and so that means the weight given to the data and the registry is getting smaller and smaller.

[00:29:21] Tim: So it’s leveling off here the effect that you see, the width of the interval. Is as wide as the one if we only consider [00:29:30] the RCT data by themselves. In some sense, this procedure mimics what you would naturally do if you are giving true information to a piece of information, one from RCT, one, from some other [00:29:45] data source, like a registry, and you try to put these things together mentally.

[00:29:50] Tim: So if they both look the same, you feel very convinced and safe. You’re inclined. To relieve the effect in the RCT, even if it’s a small one, because you see the [00:30:00] effect replicated in the larger registry. If things become and look quite different, then you might have a preference for the effect estimated for the RCT in the RCT because you think it’s unbiased.

[00:30:14] Tim: But [00:30:15] at the same time, you have a lot of uncertainty because it’s a small study and you’re not giving much waste to the registry. So in a way, I think. This procedure mimics what we would naturally do. It’s to some [00:30:30] extent also robust, and the way you can calibrate or attune it is through the choice of the prior distribution.

[00:30:37] Tim: Of course, you can also investigate the behavior of this procedures in simulation studies, which in simple [00:30:45] settings run quite quickly. Yeah. Yeah, there are some comments on the weights because I think some kind ago also had some discussion at the regulatory statistics workshop in Basel. There were [00:31:00] some comments from regulators that had some concerns that in settings like the one that we just looked at, where the registry is much larger than the RCT.

[00:31:10] Tim: The registry might just overwhelm, overrule the [00:31:15] RCT and we thought, okay, is there anything we can say about the weights of these different sources in a particular, the minimum weight that the RCT will have in the final shrinkage? Estimate these weights. You can [00:31:30] also get. From the base matter package, Christian extended the package.

[00:31:34] Tim: With this regard, if you look for a boundary that holds for any choice of prior distribution, you end up with the weights that you know from [00:31:45] common effect or fixed effect matter analysis. Yeah, and they of course, will be not very reassuring because in this example here. We have a standard error for the RCT of 0.8, and then another [00:32:00] standard error for the effect in the registry, which is only 0.2.

[00:32:04] Tim: So this is a factor four. So factor four in the standard errors means factor 16 in variances or maybe also sample sizes. [00:32:15] And then you end up with a minimum here of one over 16 plus one, one over 17, only 6% now, and that is what makes you worry. But why would you look for the minimum weight for [00:32:30] any choice of prior, because that would also include all sorts of very unreasonable choices of prior distributions.

[00:32:37] Tim: We have actually quite clear recommendations on how to specify the prior, and if you do that, [00:32:45] then you see the minimum is reached in situations where the two effects are the same. Yeah. So now you give only fairly little weight to the RCT, but that’s okay because the effect from the registry looks exactly the same.

[00:32:59] Tim: [00:33:00] So that’s not of any concern with the prior, we have used in the previous example, half normal scale of 0.5. It turns out it’s at least 29%. If we use larger scales, then also the weights [00:33:15] will be higher. Our conclusion is with reasonable choices of prior distribution for the torque, these concerns are not warranted.

[00:33:25] Tim: So we have this worked out for the example where you can [00:33:30] see that the actual weight with the analysis I was showing was something like 40%. So this is quite substantial now. So the RCT had 40% weight on the calculation of this shrinkage [00:33:45] estimate that we present. Also a comment that you can get these weights easily out of R using the base matter package.

[00:33:55] Tim: I alluded to simulations earlier. They can be very useful in [00:34:00] studying the properties of your procedure. Often in a regulatory context, you might want to evaluate some frequentist properties of this vision procedure. No. Can do this in terms of coverage of the [00:34:15] resulting so as a frequent dis coverage property of the Beijing credible intervals.

[00:34:20] Tim: But you can also look at it in terms of type one error rate and then you can also choose scaling parameters, for example, for your half normal prior, [00:34:30] so that over range of value scenarios that you think are reasonable. Yeah, you can achieve error rates that would be acceptable. Not only to you, but also to regulators you are discussing the issue [00:34:45] with.

[00:34:47] Tim: Sometimes people ask me, where do we get the observational data from? I’m talking not only about external controls, but having external data that includes the intervention arm, the [00:35:00] experimental arm, and the control arm. And I think in particular, in rare diseases, is I think a good idea to think too.

[00:35:08] Tim: Almost gather this type of data by design. Yeah. And one design that’s actually around for [00:35:15] some time, but actually originally with some other intent or purpose, is the comprehensive cohort design. Where first the patients are checked for eligibility and then asked to consent to [00:35:30] randomization.

[00:35:30] Tim: If they do, they enter the RCT. They’re a randomized between. Their treatments if they do not consent to randomization. Now again, the example that we had from CJD, you might still consider including them in a registry [00:35:45] where they might be offered to the alternative treatments or they get the treatment that’s agreed on between their physicians, healthcare professionals, and patients themselves.

[00:35:55] Tim: But I think I see this because sometimes people argue a lot about whether [00:36:00] an RCT is feasible or not, and you can just lack patients. The investigators basically decide themselves rather than deciding upfront. An RCT is not possible in many cases. People just run some single arm [00:36:15] studies, I think is a good idea.

[00:36:17] Tim: To embark on an RCT even if you run some risk that this will be small scale or might be only large enough to stand by itself if the effects are substantial, but at the same time. [00:36:30] Build up some observational database that can be used to support the RCT in an approach like the one we have just shown.

[00:36:41] Tim: The original intent of this type of design was to [00:36:45] assess external validity. New idea cloud looked into this in the 1990s and even then. They didn’t claim it was new. They got the ideas from somewhere else. And basically all we are saying is now bring in [00:37:00] this data integration idea right from the beginning with the planning, with the design.

[00:37:05] Tim: Now say there will be an RCT part and there’s an observational part with the intent to, in the final analysis, combine the [00:37:15] different data sources. We have applied this sort of thing, not quite the way I described it, but. We have done a study where we did an RCT and at the same time, gathered some observational [00:37:30] data through an open label arm and used some available cohort data from a natural disease cohort, build additional observational data.

[00:37:41] Tim: This is another rare disease from Alpo [00:37:45] Syndrome. It’s an inherited. Disease that leads to kidney failure at a young age, meaning the patients need transplantation or dialysis. So it’s really severe condition. Here we looked into the effects of fear. [00:38:00] That’s an ACE inhibitor, so it lowers the blood pressure.

[00:38:04] Tim: And the idea was to look into the efficacy and safety of this intervention in fairly young children. So it is a pediatric study and it was placebo controlled, so it was a [00:38:15] randomized placebo controlled trial, which is of course, difficult to conduct in this type of population. So you see the sample size and the randomized control trial was only 20.

[00:38:26] Tim: At the same time, we had 42 patients with open [00:38:30] label treatment, 28 from a natural disease cohort. So we have some RCT evidence, we have some real world evidence combined in exactly the way that I described earlier. Here are the results for [00:38:45] the efficacy analysis. If you look at the RCT for the primary endpoint disease progression very wide.

[00:38:52] Tim: Confidence interval. The interval, the confidence interval or credible interval is much shorter. Incorporating [00:39:00] the data from the real world data part also. The weights here were quite similar to what we saw in the CJD example. I think we are coming towards the end of the talk. Maybe I just [00:39:15] want to include one or two warnings you have seen.

[00:39:17] Tim: I use some propensity score based methods and the analysis of fairly small registry. Of course, I’ve done it myself and we learn as we do things here in some work with Zara Friedli [00:39:30] motivated through some. Small scale, non-randomized studies. You remember, for example, the Chloroquine, the example during the pandemic where Chloroquine was recommended in the Oval Office for use in COVID-19.

[00:39:43] Tim: If you looked it up, what [00:39:45] studies were available, and these were non-randomized open label studies. They were naively analyzed and we thought maybe we can improve the analysis by using more sophisticated causal inference. Then learned that many of these [00:40:00] methods don’t really lead to satisfying results in terms of properties that these have, if small sample sizes are small.

[00:40:09] Tim: So we derive some recommendations actually, how to deal with relatively [00:40:15] small example registries in this setting that we considered. Here’s the conclusion slide. I think hierarchical models are flexible statistical framework for evidence synthesis. I hopefully convinced you of some advantages of the [00:40:30] BE framework if you were not be before coming into this talk today.

[00:40:35] Tim: I think cross design synthesis is an important and quite promising, and particular in settings like rare diseases. I think it needs more [00:40:45] practical and regulatory experience, so we need to interact and discuss these different approaches. I talked about the balance for the weights. I think I don’t have really any concerns as long as the procedures set [00:41:00] up properly in terms of choosing the prior, the number of alternative approaches available.

[00:41:06] Tim: Then this setting, there are some equivalences between them. With this I close and leave some time for questions. Thank you [00:41:15] very much for your attention. 

[00:41:16] RIchardus: Thank you very much, Tim, for the thought provoking presentation. I hope we’ll see more of this in regulatory practice as well. You already answered three questions that were in the chat.

[00:41:28] RIchardus: We have about five [00:41:30] minutes left for further questions.

[00:41:37] RIchardus: So no questions yet, Tim. Just one question. From your experience with [00:41:45] military agencies, how is their uptake of these methods? 

[00:41:53] Tim: Yeah. So I think in the discussions with regulators, it’s important to conduct [00:42:00] simulations, to demonstrate frequentist properties such as type one error raids, coverage of intervals, et cetera.

[00:42:08] Tim: I think that’s an important aspect. It probably won’t fly in common disease settings, but I think in [00:42:15] settings where we are talking about pediatrics or rare, ultra rare diseases, there is some acceptability of the methodology as long as it’s well understood. So that needs some discussion, and I think sometimes [00:42:30] all.

[00:42:30] Tim: Parties involved need to take the time to explore and understand. We have done also projects on a European level, inspire, asterisk, ideal. Now, I think I have one of the next slides in terms of [00:42:45] resources of these previous projects that also had some interactions with regulators, like we had a workshop at EMA to discuss these approaches.

[00:42:54] Tim: There’s certainly some interests. We are now starting the [00:43:00] events project, for example, is the chair of the methodology working party of CHMP is involved and also yeah, the Euro Met project is running parallel. And I think all these projects have also involvement of [00:43:15] regular regulators, not only from drug licensing, but also from HTA bodies, which I think is important to get them on board.

[00:43:25] RIchardus: Okay. No, that says perfectly. Okay. There is one more question in the chat. [00:43:30] You mentioned some robustness based on the weight. Could you comment on this in the context of sidley’s robust down weighting methods? In case of discrepancies? Yeah, so I. That’s a 

[00:43:40] Tim: very good one. It’s from Cornelia, I guess the Cornelia from Heim [00:43:45] that we know.

[00:43:45] Tim: Thank you for the question. I presented it here as a shrinkage estimator from a meta-analysis. You could also present it as having the registry, providing Mac prior meta analytic predictive prior [00:44:00] coming from meta-analysis. Random effects metaanalysis of only one study. There is equivalence. Between the two approaches, the meta-analysis approach that I presented here and the meta analytic predictive prior, actually here we [00:44:15] have only two data sources, so we don’t need the extra uninformative component.

[00:44:20] Tim: We have the parameter here in the prior to the scaling. Now that would set the things up, but it’s basically also falls [00:44:30] within the framework of the meta analytic predictive prior, I should have mentioned that. Thank you for that. Question, and then there’s another one by Diane. How difficult is it to access registry data?

[00:44:41] Tim: Do pharma companies pay to access registry data? [00:44:45] That depends on the setup of the registry of some will generally charge for data extraction. Some don’t Access to this type of data is getting much better compared to the past, past 10, 20 years ago. [00:45:00] Actually here at the department, we are running a project where we look into applications of target trial.

[00:45:07] Tim: Ation framework to estimate treatment effects from registries. We are working with a number of registries. Our [00:45:15] experience was that they were very willing to take part in the project and they’re not paid, but of course we are not farmer. Yeah. I think there is some increasing awareness, openness to collaborate in this type of project.

[00:45:27] Tim: In particular in chronic [00:45:30] rare diseases where you’re often very strong. Patient representatives and organizations that also help to facilitate these processes. Thank you, Richard. 

[00:45:41] RIchardus: Okay. Very much, Tim. There are no more further questions. We are [00:45:45] right on.

[00:45:49] RIchardus: Thank you very much. I think you’ll all go to sessions and enjoy the, anyone from the. Effective Decisions. [00:46:00] Team wants to say something, but if not, I’ll close this session. Thank you very much.

[00:46:09] Alexander: This show was created in association with PSI, thanks to Reine and [00:46:15] her team at VVS. Well, with the show in the background, and thank you for listening. Reach your potential lead grade science and serve patients. Just be an effective [00:46:30] statistician.