RWE and the FDA (Part 2)

Transcript

Alexander: You’re listening to The Effective Statistician podcast, a weekly podcast with Alexander Schacht, myself, and Benjamin Piske, designed to help you reach your potential, lead great sciences and serve patients without becoming overwhelmed by work. Today, we are talking again about real-world evidence and the viewpoint of the FDA on it. So if you haven’t listened to part one of this two-part series then just scroll back in your player a little bit and look for the other episode. I’m speaking with Josephine Wolfram  today again so stay tuned for a really really nice episode. 

If you are interested in real-world evidence, there is a lot of stuff going on. There are a lot of webinars on the PSI homepage that you can have a look into and there’s also the special interest group. The special interest group on real-world data that is led by Anny Stari is really, really new but it’s also really active already and so there’s a lot of things going on there. If you’re interested in this special interest group or any other special interest group, just head over to the PSI homepage. PSI is a community dedicated to leading and promoting the use of statistics within the healthcare industry for the benefit of patients, and you can join this community too. It is only 20 pounds for non-high-income countries and only 95 for high-income countries per year of course, so head over to psiweb.org to learn about this and become a PSI member today. 

Welcome to another episode of The Effective Statistician and today, it’s again with Josephine. How are you doing? 

Josephine: I’m doing well, thank you. How are you? 

Alexander: Very good. It’s awesome to talk again. This is the second part of the interview with Josephine about real-world data and real-world evidence if you haven’t checked the first one and scroll back in your podcast player and look for the upper other episode and then maybe listen to this one first. In that episode, we talked about the FDA’s perspective on real data and real world evidence and also how it might be a bit different from other aspects. And we talked about reliability and relevance. And we also talked quite a lot about exposure because that is so important and how you can ascertain it, how we can validate it, and from there, I want to go into the same part of this podcast episode and talk a little bit about the outcome ascertainment and validation. So what are the critical pieces we need to think about its ascertainment of outcomes there?

Josephine: Before I jump into answering, I probably should share my disclaimer a second time, right? 

Alexander: Okay. Yeah, just be on the legal safe side. 

Josephine: The opinions expressed in this podcast are solely those of myself and not necessarily those of my employer Astellas and Astellas does not guarantee the accuracy or reliability of the information provided herein. So coming back to outcome ascertainment and validation, what FDA describes when they talk about outcome ascertainment and the first thing you need to do is to generate what they term, a conceptual definition for your outcome. And this needs to be a definition that can be operationalized in real-world data sources. So to take a sort of very typical example in oncology, the clinical endpoint in trials would tend to be tumor progression based on resistance. That’s not something that’s typically captured. So you need to develop an operational and firstly a conceptual definition of something that you will be able to operationalize in your data source. For example, time to the next treatment, might be an alternative way to tumor progression. And then you need an operational definition using diagnosis and procedure codes such as ICD-10 codes or SNO-med codes or laboratory test codes that you can use to define your outcome. And this can be using coding systems that are part of the structured data. You may also try to extract from the unstructured data. So, from physicians’ encounter notes, or radiology and pathology report to really curate the endpoint you are looking for.

Alexander:  Let’s double-click on this a little bit. So in terms of concept, is this concept always kind of coming from the clinical trial thinking, or could it also come off from other aspects, kind of, you know, a physician or patient perspective on things?

Josephine: It needs to be an outcome that is being captured as part of routine care. So, in that sense, you will have that physician’s or patient’s perspective on what is being monitored in routine care. And that’s not the case with many of the clinical trial endpoints that we might use. So normally when you go to the doctor, you’re not issued a 25-page patient-reported outcome to fill in. 

Alexander:  Yeah. And also the physician doesn’t fill in such a questionnaire himself. So I think that’s an important point to have a look into. So even though you may not have the same kind of things as in clinical trials, that’s not necessarily kind of a limitation. So the nice thing is you have something that has actually been done, actually measured. And you don’t necessarily always need to come from a clinical trial perspective and then search for something that you can do on the real-world data. You can also come from the real-world data perspective and see what is possible there from a content perspective and what is meaningful for physicians and patients and of course, it changed therapy. Most likely means something has been working well. Unless it’s kind of, ‘oh you’re changing from oncology therapy. Now, treat the donor’s blood pressure or something like this because your tumor is gone. But if you kind of add another kind of oncology treatment or kind of increase the dose whatsoever, it’s usually a bad thing. And so that may be something that is much more kind of drastically something that has an impact then maybe there’s a couple of the other things that you would measure a kind of lab changes or whatsoever that doesn’t directly have a heavy clinical relevance that way. 

Josephine: Right. And real-world data will perhaps lend itself better to the sort of harder types of outcomes and then perhaps these little softer types of outcomes. 

Alexander: Yeah. And very often you have larger sample sizes that make it possible and also to still see something. Very often we have these more specific outcomes in clinical trials because there are small samples. And we need to see something much more fine-tuned than in this real-world data where we maybe have tens of thousands of patients. 

Josephine: If you think that cardiovascular outcomes may send points like this that can be very challenging to study in a clinical trial, that kind of an endpoint can be operationalized pretty well in the real-world data sets. 

Alexander: Yeah, very good. That is about ascertainment. What about validation? 

Josephine: Yes, the validation is the exercise of confirming that the operational definition that you use to extract the outcome from your data source has correctly captured the concept that you define for your outcome. FDA expects validation of the outcome variable to minimize outcome misclassification. They state here and I quote that ‘ideally through complete verification of the outcome variable, each subject is assigned an accurate value of the outcome variable to minimize outcome misclassification and improve study internal validity.’ In practice, a more commonly used approach is to assess the performance of an operational definition in validation studies. So that means using a sample of patients and comparing your classification of the outcome to the original records. 

Alexander: Yeah, I think what is there probably is also important to understand other factors that would drive misclassification. It’s there, for example, when you want to compare treatments. Hopefully, treatment is not a factor that drives misclassification. Do you want to understand what is a bias you see? And then which direction does it go? Generally, misclassification, when it’s not driven by any factors, heights things, it’s probably something that has an impact on shrinking treatment effects and things like this. Only when there are variables that are so to say influencing and confounds with treatment then can you have the opposite effect. So these sub-studies are really, really interesting things. What would such a sub-study look like? 

Josephine: Well, you may go through an exercise of, if the data source supports it actually going back to the original patient medical record and then comparing what you find there versus what you’ve derived. So perhaps if your study is conducted with claims data, you’re able to link that for a certain number of patients with the EMA and compare what you find in the two sources. 

Alexander: Yeah. That’s a good way. It’s like any other diagnostic tool, you need to find some kind of gold standard that you can compare against. And that gold standard needs to be good enough so you can know that you finally find misdiagnosis, miscategorization that’s really important. 

Josephine: And then your next step would be to look at things like the sensitivity and the specificity, the positive predictive value or the negative predictive value. And they have a sort of a discussion about which of those statistics you would be sort of aiming to assess in your validation study according to, whether perhaps you’re looking at a rare outcome or more frequent outcome, etc. 

Alexander: So refresh your knowledge about diagnostic statistics quite a lot.

Josephine: With covid, we’ve all been refreshing our minds. 

Alexander: Yeah. For sure. Probably everybody has a discussion about the kind of sensitivity and specificity and then positive negative predictive values of covid tests here for sure. Awesome, very good. One other really important thing that I wanted to touch base on in this second part of the podcast episode is the specification. What’s the FDA’s approach to pre-specification with real-world data? 

Josephine: This specification is a strong theme that’s come through in their draft guidance. Pre-specification in the clinical trial setting I think we all understand what that means because before you actually collect your data, you say what you’re going to do with your data at the end, or at a minimum you have a very specific plan before you run blind. But how can we really pre-specify a study using data? That’s actually already been collected. It’s the same principle, what they want to avoid is that they research the data and actually know the answer to those research questions before declaring what their objectives and methods are. So here, pre-specification means you know in advance of conducting the analyses and this is important to avoid cherry-picking results. I think a statistical audience is very familiar with it, but how are we going to make sure this happens? So it’s a little bit harder to really demonstrate pre-specification in the situation where that data is sort of already existing but the FDA do sort of have a couple of key recommendations around that to help I guess, reassure them that what you’re doing really is pre-specified and so that you have a kind of trial to demonstrate that. 

Alexander: Yeah. I think this is really, really hard. It’s really hard because imagine you work on your treatment. And there are multiple treatments and the same class that is all developed. So probably you’re not the only one looking into this data because all the other companies also have the same thing. You’re going to CRO that has the access to this data and uses this data, they might have just done more or less the exact same stuff for another sponsor. So having this pre-specified is really, really tricky because you can never really 100% be sure that people haven’t seen the data before. The other thing is, I think you need to have a certain level of expertise on the data to actually conduct the analysis and pre-specify the analysis. So you need to have some kind of knowledge about the data, about the database, the structure of the database, and how many patients are in there. You usually do some kind of feasibility study before you actually do all this analysis. So through that, you’ll always dip your toe into the water already. And I think it’s really hard to come up with pre-specification. However, I think pre-specification is for me, one way to address this problem of cherry-picking. And I think it’s just one way. The other way of assessing cherry-picking is looking more into how robust your analysis is. If you change a little bit here and there, your assumptions. Does it make a big impact? If you think about it, let’s say your outcome concept here, if you change things here and there, does it dramatically change the analysis? If you think about your exposure if you change something in there, does it dramatically impact the analysis? Are the effects that you see really big? Or are they kind of just kind of detectable? If it’s really big then probably no matter what you change there’s still another big. But if you need to do a lot of fine-tuning here and there on your parameters, I’m not sure a pre-specification will save you. And say, ‘well we have pre-specified all these kinds of 25 different definitions, and only this combination of all the definitions we get to a p-value of 0.45. If we treat anything we don’t see anything anymore. What’s your point on that? 

Josephine: Yeah, you raise lots of great and really interesting points there. So picking up on the, you know, you can’t really not look at all that your data source before you commit to using it for your study. Yes, you’re right. You need to do some kind of study of feasibility. How are you going to know that you have enough patients to meet your study objectives if you haven’t at least queried the data sufficiently to know how many patients have your disease under study? The guidelines do distinguish between data access for purposes of checking study the feasibility versus purposes of the actual analysis to support study objectives. I think the golden rule is don’t look at outcomes by exposure group during the feasibility. And they actually suggest that in your study protocol, you should describe all the data sources that you accessed when designing the study and the results of the feasibility of valuations on these sources, and then describe your justification for which sources you use for the study. So that’s, you know, quite a piece of work actually to have that nicely laid out, I guess. 

Alexander: You need to be really transparent with what you do, I think that is really important.

Josephine: Yeah. Transparency, that’s a keyword. And then you talked about sensitivity analyses basically and I think that’s also another key way of making your results reliable should we say. So it’s pre-specified, it’s more convincing if your results are robust to a series of ideally also pre-specified sensitivity analyses that I think is also going to be strongly supportive of the conclusions. And as you said, the size of the effect is important in terms of feeling of a study, being able to support decision-making. And there are methods where you can sort of look at how much buyers would then need to be to take away my effect. So like a sort of e- value, it’s a bit like a tipping point type analysis. 

Alexander: Yeah. I really like this kind of tipping point analysis. How much would we kind of need to torture the data so to say to skip, you know, get to a completely different outcome and then says, ‘is it realistic that the data is that wrong and if that is kind of pre-unrealistic. So I think you have to make a really really good claim for your data. Awesome, very good. So these were two outstanding episodes about real-world data, real-world evidence, what the FDA’s thinks about it and what you can do to run a really really good real-world evidence study. Josephine any final thoughts from you that helps us as readers and terms of working with this guideline. And by the way, we’ll put a lot of references on all of these into the show notes. So any final points in terms of FDA and real-world evidence, your thoughts on that? 

Josephine: Actually, we were just talking about pre-specification and transparency, perhaps one piece I didn’t mention there is they are asking for sponsors to publicly post their protocols in advance. So I think that’s also the best practice that may become more of a standard, shall we say, so clinical trials, etc. Any last thoughts? Well, I guess maybe just to mention that we talked mostly about using electronic health records and medical claims data to support regulatory decision-making. And there’s also a guidance document about using registries which has very similar themes and they talk both about designing a registry and reusing data from existing registry. So these can be a really great and rich source of real world data studies. I think we can expect more guidance coming in the future around the statistical design aspect so I’m quite excited to see what comes through on that. 

Alexander: Yeah, it’s definitely an area where there’s a lot of things going on. And if you are still kind of thinking about what kind of opportunities for your career, moving into real-world data is actually quite a fascinating area. The amount of data is only going to increase, the real world data use is only going to increase and also from a statistical point of view as you just saw,  there are lots and lots of challenges that need to be addressed. And so working with data in this field for me is actually quite a lot of fun. So if you like that, dip into it and if you want to engage with real world data, then also connect with Josephine on any PSI homepage at PSI web.org, where you can find the special interest groups and then just go to the real world data sig. Thanks much Josephine for these two outstanding podcast episodes. All the best for you. 

Josephine: Thank you very much. 

Alexander: This show was created in association with PSI. Thanks to Reine and Kasey who help with the show in the background and thank you for listening. Don’t forget to head over to psiweb.org to learn more about these kinds of things, and reach your potential, lead great science and serve patients. Just be an effective statistician.