Understanding and Mitigating Endpoint Bias in External Control Arms

Benjamin Ackerman

RWE methods | Causal Inference | PhD Biostatistician | Director, Real-world Biostatistics at GSK

Ben is passionate about developing and applying quantitative methods for health research and drug development. He strive to answer statistical questions embedded in real-world problems. His goal is to impact how we translate research into practice by improving the tools we use to analyze health data.

His research focuses on methods for combining randomized trials and real-world data (RWD) to improve evidence generation. His applied public health interests also include oncology, mental health, and LGBTQ+ health.

---

Transcript

00:00
You are listening to the Effective Statistician Podcast. The weekly podcast with Alexander Schacht and Benjamin Piske designed to help you reach your potential, great science and serve patients while having a great work-life balance.

00:22
In addition to our premium courses on the Effective Statistician Academy, we also have lots of free resources for you across all kind of different topics within that Academy. Head over to theeffectivestatistician.com and find the Academy and much more for you to become an effective statistician.

00:49
I’m producing this podcast in association with PSI, a community dedicated to leading and promoting the use of statistics within the healthcare industry for the benefit of patients. Join PSI today to further develop your statistical capabilities with access to the ever-growing video on demand content library, free registration to all PSI webinars and much much more. Head over to the PSI website at PSIweb.org.

01:18
to learn more about PSI activities and become a PSI member today.

01:31
Welcome to another episode of the Effective Statistician. Today, I’m super excited to talk with Ben Ackerman about one of my favorite topics, these are external controller arms. But before we dive into this, Ben, maybe you can give a little bit of a background where you’re coming from and how you got interested into this topic. Sure. Happy to, and thanks so much for having me on your podcast to chat today.

02:02
So I am a PhD biostatistician by training. I came from a background and interest in public health. And so I’ve always been motivated by statistical questions and problems that are rooted in real applicable issues. My focus in research has always been on propensity score type methods and ways we can combine.

02:27
trials with observational studies and multiple data sources to learn things from disparate sources together. And so this concept of the external control arm study came about on my mind through that work and as a way to bring in external information to support how we learn from trials. So I work at Johnson and Johnson currently as a statistical methodologist and an internal consultant.

02:54
I guess I worked at Flatiron Health, which is an oncology real world data vendor. I’ve had various exposure to ECAs and different types of designs throughout those experiences. Very good. Awesome. Yeah. It’s definitely very hot topic. So it’s kind of inter interface between real world data and clinical trial data. Also, I don’t really use the word.

03:21
real world because of course clinical trial data is also real. sees actual patients in way. Yeah. But it’s one is more kind of experimental setup and the other one is observational setup. So I like the term observational data actually much more, but real rotators, the buzzwords, this is all around. Yeah, I totally agree with you there.

03:47
try to use the lingo and the terminology that the field is using. yeah, certainly trial data are patients and it is still considered real world. I know there’s a lot of debate on these terms and whether observational or non-experimental because of course, to some extent trials are also observational, even though they’re interventional, we’re also still observing how patients are progressing in their disease and treatment.

04:16
So yeah, I like to use any and all of those terms. I feel I typically describe them as experimental versus non-experimental or when it comes to the electronic health records that I typically use to talk about real world data, whether it’s retrospective or prospectively generated and collected, but for all intents and purposes, I tend to use real world data just to date with the times in the field.

04:41
So you recently presented about quantifying and especially also mitigating different biases between such real-world data, so from, for example, from claims databases or other observational settings and clinical trial endpoints. What are these biases about?

05:00
Yeah, that’s a great question. So when we think about external conform analysis, one of the most important pieces, I think, when doing this type of analysis are making sure we have apples to apples comparisons. So when we identify a comparator arm, whether it’s from real world data or electronic health records, or even historical trial data, I think typically there’s a lot of emphasis on how we make them apples to apples in terms of populations.

05:29
that we want to emulate a study that’s randomized and make the two groups comparable in terms of who are recruited and compared. This research focused on another apples to apples comparison and issue of how endpoints are constructed and measured. So not only do we want to have similarity between who is observed in each group, but also similarity in how and when they’re assessed for disease in the two settings.

05:59
So this issue, think mostly comes up when combining a trial with some say retrospective electronic health records based data set where, know, typically in a trial, we have very rigorous standards and control over how assessments are done and when they’re conducted. So we have a strict schedule of events where patients come in at regular intervals and

06:27
clinicians will use specific instruments or tools, whether it’s imaging and oncology or like solid tumor oncology or validated scales for certain outcomes to be able to assess the patient’s trajectory and their disease. In real world data, we typically don’t have access to that wealth of information. We might just have, for example, a clinical document that is ingested

06:55
in some EHR system that says patient disease progressed or patient is doing better. And how does that compare to that trial standard? That’s really the issue that we’re tackling with this research, trying to understand what are those differences in the ascertainment of the endpoint. And then how can we model and make them more comparable to one another so we can make more of an apples to apples comparison.

07:24
and be more confident that the benchmarking against this comparator is really due to the difference in treatment, not due to the difference in ascertainment of the outcome. So it’s basically two big points. One is when it’s collected. And then the other thing is what is collected and how precise is it collected. I know of course, in a clinical trial, you have these regular schedules, but still there’s some kind of variation around it.

07:53
If you say, well, we want the patients to come in after 12 weeks. Most patients will not come in exactly after 12 weeks, but plus minus something. And so this variability, of course, you may have something similar in the external controller. How have you, how can you know if okay, there is some bias? Yeah. For example, you have in the health records, you can only see improvement or worsening or no change.

08:23
Yeah. And in your clinical trials, you have some things that is real scales that kind of gives you much more kind of information on how much improvement or how much worsening. How can you now mitigate these differences in terms of bias? Yeah, that’s a great question. think the short answer is it depends on what data.

08:43
you have and how well you understand the comparison between what is measured in a trial and what’s available in these, whether it’s clinical notes or what’s available in the comparator arm. So to that point, I think the best way to mitigate those differences is to have some validation study conducted or some data available where you can, for each patient, directly compare these two different

09:13
methods of the outcome where you can say, okay, on this visit there was imaging done and per certain criteria, it was determined that the patient was responding to treatment. And then you also have some version on that same day for that same patient that says what sort of is documented in that EHR system. And you can then compare how often do they draw the same conclusion, how often do they not.

09:42
And then you can use those metrics of sensitivity or specificity in this validation study and use those metrics to adjust your analysis in future studies. So the best way would be to have some data that studies these two phenomena together and use that to correct for it.

10:05
That’s obviously easier said than done. And in a lot of disease settings that might not be easy or even feasible to design. So there are ways in which simulation studies can be helpful, partnered with working with clinicians and experts who understand how these disease endpoints are measured and what might cause differences between the two ways of measuring them. And there are ways you can use simulations to

10:32
say, okay, well, what if this difference is this large or what if it’s maybe negligible? And do some sort of quantitative bias analysis or tipping point analysis to see how off are we if we assume certain differences. But I think even, you mentioned, what if we assume there are these biases? I think that’s an even more critical first step is to stop and think, are these issues going to manifest as bias or not?

11:01
And that’s also where simulations or descriptive analysis can be helpful. And what we’ve seen in our research is in some cases, some of these issues matter. In some cases they don’t. And in some particularly tricky cases, it might matter and you might have strong opposing forces at play. So you might have where there’s strong bias favoring the Alkman one direction due to

11:29
certain false positives of the outcome. And then you might have strong bias in the other direction due to other types of errors or due to the difference in when patients come in, the assessment scheduling. So I think even before we go into the mitigation, there’s a lot of important work that needs to go into understanding in any one case or any given data. When is this actually an issue? Because

11:56
You might find it’s not, and then the types of mitigation strategies might not be necessary. Yeah. So, and you mentioned one important point, the kind of the size is important, but of course the direction is probably more important. Will it shrink the current treatment effect or will it increase the current treatment effect? So that’s a very, very important point. Now, one of the things is that always comes to mind when we talk about

12:23
combining external control, arms with clinical trial data is when do you actually do these things? And speaking about claims data or registry data, these data exist already. You can have a look into these and better understand what is actually in there to also inform what you want to collect in your study. When, for example, there’s certain data available, make sure that you also collect that in the study. And then you have at least one problem less.

12:52
in terms of that. And if you also need other further data from a regulatory point of view, you already have some kind of internal validation study in it because you collect those types of data. So that is super helpful. So that’s actually one of my questions. Would you look into Rebell Data before you run the study to inform the protocol or would you run that kind of independent? How would you work on that? Yeah, that’s a great question. I think you raised one interesting

13:22
an idea of how can the selection of real world data or the use case inform the design of the single arm study. I think one thing that could be advantageous is looking at how endpoints are ascertained in the retrospective databases. If you’re going to use something retrospective that’s already off the shelf, you license from a real world data vendor, getting some information on not necessarily the outcomes themselves and not biasing.

13:51
conclusions you might draw, but understanding the data generation. Are certain vendors using clinical notes and abstracting endpoints based on they see a clinician said patient progressed and that’s what’s represented in the data model. Or are they using algorithms or deriving the outcomes using labs in other ways? Understanding how it’s ascertained can actually be helpful. And if it is.

14:21
say like a clinician documented response, one interesting idea could be to bake that into the trial. So in addition to the rigorous standards of, of using whatever validated scales or imaging to look at the primary endpoint, there could be ways to, as you alluded to, bake in a little validation sample or say, okay, for X percent of patients or for all, we’ll also ask the clinician

14:51
to write this note as if in an EHR document or have some sort of measure based on similar rules of whatever’s collected in a real world data set. And that can help understand the relationship between the two. I think there are logistical considerations with that. Obviously, when planning a trial, you don’t want to place too much burden on collecting certain values or measures. But I do think that’s one way

15:21
to improve the single arm design to anticipate some of these questions of the endpoint misalignments. I think with the timing of assessments, it’s a little maybe tougher because there is strict protocol elements that might be needed, but there is also increasing evidence of introducing pragmatic elements into trials that are beneficial. So it could be feasible to

15:50
bacon more flexibility of assessment timing if you think that’s more aligned with clinical practice and what you might see. But I think this is also coupled with the important role of pre-specification and transparency. You certainly don’t want to look too much at the data before you use the data for the actual analysis. So there might be ways in discussions when designing the study and picking which comparator

16:19
data source you will use to understand the underlying generation mechanisms from the vendors without looking too much and biasing yourself with the data itself, if that makes sense. Yes, that’s an interesting discussion in terms of what does pre-specification actually mean here. I would argue that anything that you do before the first patient visit of the first

16:46
patient and the study is pre-specified, irrespective of what we have done with real-world evidence data. Because we could use these real-world evidence data all the time. And also it’s just from a practical point of view. Nobody could ever control what you have done with this external real-world evidence data. Or one of your European leader that informs the clinical study design has done with that data. Or maybe they publish it or whatsoever.

17:14
Yeah, so there’s this kind of external data I think is not so critical from a pre-specification if you run all the analysis beforehand. Yeah, and you actually include them also into your article and into your SAP and so said, okay, this is how the data will look like. And that is basically now the hurdles that we want to jump over. Yeah, in a sense, it’s like doing one sided statistical test. Yeah.

17:43
where you give yourself that specific hurdle and say, well, that’s what we want to jump over. But then of course, you can’t add uh stuff later on. Yes, that gets a little bit more tricky or you need to pre-specify how exactly you will do this. But that’s definitely a really interesting thing here. I don’t know whether protocols are really up to date in terms of that, in terms of just thinking about templates and things like this.

18:13
how that is all included. Definitely something for a good discussion to have with your regulators. And to discuss what that means, what your kind of your basis is and with your decision rules. Because in the end, that’s what we really want. We have clear, transparent decision rules and we know exactly when we have jumped over what and when that study is clear.

18:43
In terms of that, do you have any experience in terms of documenting these things and pre-specification and achieving transparency? Yeah, that’s a great question. I think when it comes to this issue of endpoints and how endpoints are measured, one thing that we, or at least I have noticed with the sort of issue of measurement error is it’s often recognized as in the room with us and

19:12
possibly an issue, but not often addressed in practice. And so I think one of the goals of the work that we underwent with really laying out these biases due to differences in how and when patients are assessed and building some tools was an effort to help encourage teams to think more rigorously and quantitatively about these issues. So I think doing further R &D on how

19:42
these problems manifest in different disease settings and help bake in some of these concepts when pre-specifying external control arm analysis. So I think at least from what I’ve seen, and this may not be the case completely, but it seems this problem of endpoint misalignment is listed as a limitation or a possible source of an issue that is left unaddressed. our hope is that

20:08
This can be incorporated further when designing a study or writing up an SAP to really make space in the protocol to say, in addition to these primary analyses, we will evaluate how the endpoint definition affects our results. To really proactively address some of these questions. And there has been at least one, if not more regulatory responses I’ve seen where

20:37
These questions of endpoints have come up in feedback to submissions where it’s been documented. FDA has said, oh, but this end point might be measured differently or patients might be assessed at different time intervals than your trial when using this retrospective data. So I do think moving towards prespecifying how those questions will be addressed for the feedback would be really critical. And that’s where at least I hope as a

21:06
a field with ECA as we’re moving towards. So being more proactive rather than reactive. And whether that’s using simulations to put in a protocol, we think the timing of our assessments will have negligible impact on bias and here’s why. Or we think the real world endpoint might be biased towards later times based on how the end point is ascertained and here’s why. Even those

21:33
pieces of information or simulation can be helpful when writing up a protocol. I think it’s really important to understand this. uh In general, we think measurement error, if it’s at random, then we just need to have more patience or it’s in a way conservative. But this kind of term conservative, I’m not super happy with because these can also be safety endpoints. If it’s not clearly documented, we may actually overlook.

22:02
certain safety signals. It’s like always better data. Yeah. And when it comes to this issue with ECA is one, one sort of big assumption in the framing that I make here is that the error is not random by design. If we think of the way things are measured in a trial are not perfect themselves. But if we treat the trial as the gold standard or the way of ascertaining the end point.

22:30
that we want our comparator to align to, then any difference from how the endpoint is ascertained or when patients come in relative to what happens in the trial is error, not random because it’s quite systematic. So there’s an issue because of these differences. And yes, there are certain types of issues that may land towards biasing towards better or worse treatment benefit. It’s like, for example, if you think that

22:59
the comparator way of measuring the endpoint is always going to under count or delay when they detect it, then you might think that your comparator will be biased towards later times and that might make your treatment effect look diluted. And so to that sort of conservative point. So there’s some sort of intuition between what types of errors may lead to what types of biases.

23:28
But again, you never really know to what extent until you dig into the problems. Thanks so much. Now, if the listener wants to learn a little bit more about this and dig deeper, what kind of resources would you recommend or what kind of areas would you guide the listener to? Yeah, I think first and foremost, the regulatory guidance documents from agencies like FDA, EMA are great places to

23:58
get some sort of high level understanding of these issues and how agencies are thinking about them. I know EMA is, I think, working on draft guidance on ECAs. They just had this workshop to solicit input and generate ideas in November of last year, which was where I presented some of this work. So I think looking at regulatory guidance, as well as available

24:26
regulatory response to see how agencies are responding about these problems are good places to learn about the regulatory perspective. I think also there’s a lot of rich literature in epidemiology journals, pharmacopidemiology journals that talk about measurement error or misclassification, maybe not in the context of external control arms, but

24:55
that are highly relevant. And then we have a few papers that have been published through this research. Our third and final paper is actually about to be published in the American Journal of Epidemiology. day now, where we walk through in the context of external control arms, we have one paper that lays out the framing of these issues that describes these types of biases and errors. A second paper.

25:24
on a statistical technique to adjust survival curves when we suspect the outcomes are misaligned. And then this last paper that’s being published in AJE is more focused on quantitative bias analysis techniques and how we can address these problems even if we don’t have the data to do so and want to contextualize our results in the presence of measurement error. So.

25:51
I’m happy to share links to those after we speak to make resources available to listeners. I think we’re really just at the tip of the iceberg with trying to uncover these issues and developing tools in the context of ECA is to address them. I completely agree. This ECA topic is a huge topic. This is not the first episode where I’m covering this. And today we talked really about more the end point.

26:20
problems, not so much the population problems and comparison problems, but really understanding the biases around the endpoints, as well as touching on pre-specification and a couple of other more practical topics. We’ll definitely put all the links into the show notes here, so you can just head over to theeffectivestatistician.com, search for Ben Ackerman, and then you’ll find this episode. Thanks so much, Ben. Of course.

26:48
I get one, one just last quick point. Yes, today we focused on these end point alignment issues. think it’s also important to see that all these biases that come up in external control arms, whether it’s the population or the end point, they’re all happening simultaneously. So these analyses have these constellation of biases and it’s even

27:16
more important to understand how these pieces work in isolation, but also how they interplay together. So bias due to measurement difference might also counteract bias due to population differences. And so we might imagine a case where we do an ECA, everything looks balanced and unbiased, but under the hood, there are all of these strong opposing forces. So

27:46
Contextualizing and understanding how these pieces function on their own helps us put the puzzle pieces together and do more rigorous research. Thanks so much. That was a very, very good final sentence. And I have the feeling we’ll talk again. Yes, definitely. And thanks again for having me and looking forward to seeing where the field continues to move with these types of studies and designs.

28:15
This show was created in association with PSI. Thanks to Rain and her team at VVS for help with the show in the background and thank you for listening. Reach your potential, leak great science and serve patients. Just be an effective statistician.