The FAIRification Of Research In Real-World Evidence: A Practical Introduction To Reproducible Analytic Workflows Using Git And R

The FAIRification Of Research In Real-World Evidence: A Practical Introduction To Reproducible Analytic Workflows Using Git And R

[00:00:00] Alexander: Welcome to another episode of the Effective Statistician. Today I’m super happy to actually have a listener [00:00:10] on the show. It’s always great to have people that benefit from the podcast to also provide things back to the [00:00:20] community. And yeah, with that, I would like you to introduce you to Janick Weberpals. So Janick, tell us a little bit about your career and [00:00:30] where you’re up to now.

[00:00:32] Janick: Hi, Alex. It’s great to be here. So my name is Janick. I’m an instructor in medicine at Brigham and Women’s Hospital and [00:00:40] Harvard Medical School. But training, I’m a pharmacist and epidemiologist with particular interest in real world data analytics and also methodology. So it’s probably unusual for you to [00:00:50] have someone on the podcast who’s a pharmacist.

[00:00:54] Janick: My interest actually in my interest in statistics and data science and statistical programming came [00:01:00] from really working in, in practice. Because at some point when, when I was at the end of my pharmacy training, I was really wondering like, how can I quantify treatment effects? [00:01:10] Besides like the pharmacology and biological basics I learned and how can I determine if like drug A or drug B is the better and safer choice for the patient that is in front of me.[00:01:20] 

[00:01:20] Janick: And so I, at some point I started teaching myself, like our programming and also did research stay in the U S and this is how I. Got into touch with [00:01:30] the field of pharmacoepidemiology, which is basically the academic discipline behind real world evidence and real world data. And so I decided to do a PhD at the [00:01:40] German Cancer Research Center and then also did a postdoctoral fellowship in data science and early drug development at Roche and then also worked for some time in industry.

[00:01:49] Janick: [00:01:50] for having me. And then in 2022 decided to come back to academia and here I’m now focused on working on real world evidence questions and methodological [00:02:00] questions in the area of oncology. 

[00:02:03] Alexander: Yeah. And we already had a colleague from you on the show a couple of months [00:02:10] earlier, no, maybe already a year earlier.

[00:02:12] Alexander: Shirley Shirley Wong, who talked about RCT Duplicate, and that project is about how you can [00:02:20] potentially recreate data from clinical trials and replicate them in real world evidence data. And you’re now doing something, also something similar in [00:02:30] the oncology space, which is very, very interesting.

[00:02:34] Alexander: All these different therapy. or therapeutic indications definitely [00:02:40] have big differences in terms of how they replicate in campus databases. So I’ve seen that already from [00:02:50] the other RCT duplicate papers. So that is really, really interesting work to help us better understand What is possible and which indications [00:03:00] and where are the strengths and limitations.

[00:03:02] Alexander: So please continue to do these kind of things. It’s absolutely fascinating. 

[00:03:07] Janick: Yeah, yeah, absolutely. I, I, I fully agree. [00:03:10] And as I said, like in these different therapeutic indications and areas, really different problems come or it’s like you have all, all sorts of different problems [00:03:20] because in the, in the first, like RCT duplicate, it was more about like cardiovascular and diabetes which you can study pretty well in, in claims data.

[00:03:29] Janick: [00:03:30] But then when you move to oncology, you need really like access to special information like biomarkers and electronic health records. And then you have all the problems like missing data. And so it’s another another type, another [00:03:40] type of, of problem set that you have here. 

[00:03:43] Alexander: Yep, definitely. So I’ll look forward to maybe that we speak about that at a [00:03:50] Different in a future episode, because today we want to talk about much more kind of fundamental problems that is all around [00:04:00] programming and transparency, reproducibility I had already a couple of different Guests on the show, and today I want to [00:04:10] talk with Janick because he actually provides some really kind of hands on experience.

[00:04:17] Alexander: So when it comes to [00:04:20] reproducibility, why is that actually important? 

[00:04:28] Janick: Yeah, I mean, that’s a very big [00:04:30] question that you’re asking, but like so Shirley Wang and I, we published together a paper that introduces a specific [00:04:40] software that’s called Git that is like really like the Swiss knife tool to use for reproducibility when you do like hands on coding and implementation [00:04:50] of statistical programming.

[00:04:52] Janick: And we had different, like a different, or several motivations for that paper. So first of all, there’s guidelines for study pre [00:05:00] specification like templates of protocols to write which is very important. And then there are also reporting guidelines like strobe and record and so on. But there’s not [00:05:10] really something about the implementation back best practices when you implement a study, because there are so many things that can go wrong and the [00:05:20] That can go wrong.

[00:05:21] Janick: And there’s really a need to have a better transparency and also reproducibility for code. So in the most ideal setting, you either [00:05:30] like provide for example, like a software package, like an R package that comes as packaged. Tool to reproduce certain analyses all over again. Or if [00:05:40] you have, for example, a if you did analytics on a specific study the most ideal way is that you can, you could basically just use the same code and [00:05:50] redo the analysis.

[00:05:50] Janick: The analysis all over again without changing anything. And that’s really the, the, the problem set. And I think we’re still, especially in, in, in pharmacopoeia and verbal [00:06:00] evidence, we’re far away from that. So I was at a conference last year. And in terms of like, for example, code sharing a group did a systematic [00:06:10] review.

[00:06:10] Janick: On code sharing and it was, they found that just 5 percent of all real world evidence studies actually share code. So we really don’t know what’s going on behind the [00:06:20] scenes because you can write a lot of things in the, in the protocol and the protocol is very important, but what it really comes down to is how you program it because you can write it one way in the [00:06:30] protocol. But If you implement it really the same way, that’s the big question that we’re all after. 

[00:06:36] Alexander: Yeah, and especially with real world evidence, yeah, real [00:06:40] world data is so messy. There’s so many kind of different aspects you can do it. And if you then want to replicate your analysis, [00:06:50] yeah, for example, especially with real world evidence, you have New data, you have more data, you have updated data.

[00:06:58] Alexander: Yeah. Then you [00:07:00] want to do it again. And yeah, maybe, you know, you have left the institution. Yeah. [00:07:10] And now someone else needs to pick it that up. Yeah. Or there is another institution that works on the same. Data set, you [00:07:20] know, and wants to kind of do maybe some modifications to it. And in order to modify something, you need to start with your baseline.

[00:07:28] Alexander: Yeah. So you first start to [00:07:30] replicate the same things. And if you have ever tried to do that on a clinical trial, such as already hard, On real world evidence, it’s [00:07:40] way more complicated. Yeah, because very often you have much more messy data and not only more messy, [00:07:50] but just more data. So that also makes it much more complex.

[00:07:55] Alexander: So in your paper that you wrote with Shirley, you [00:08:00] call it a verification of research in real world evidence. So, and there is the capital FAIR [00:08:10] in it. So can you quickly speak about what FAIR stands for? 

[00:08:16] Janick: Yeah, definitely. So FAIR is actually comes from data [00:08:20] so that data is findable accessible interoperable and reproducible. And we thought that we extend that not [00:08:30] only to data, but also to the code that you use to produce a certain statistical output. Because all the code should be findable, accessible, interoperable, and [00:08:40] like reproducible. So that the, the intention of the study that you wrote in the protocol is like really reproducible.

[00:08:46] Janick: And That, that does not like just that [00:08:50] actually starts with also how you, for example, label and name the, the programs where you coded up your analysis, because for example, as a, as a PhD student, [00:09:00] I often found that things were labels like main analysis, one main analysis, two main analysis final. And then the question is like, really, which is the [00:09:10] actual script that produced the output?

[00:09:11] Janick: I don’t know. And this is just where it starts. So this is not even going into, into the topic of. Git and, and like having, having a track record [00:09:20] of, or an audit trail of, of changes that you did, but just like, which is the actual code program that, that produces the output that I want. 

[00:09:29] Alexander: Yeah, [00:09:30] so findable, this kind of first thing that on, you know, most of the statisticians and pharma will say, [00:09:40] well, that’s easy that, you know, you have you have the tables and the footnotes and the footnotes point to the source, yeah, the [00:09:50] path and so on.

[00:09:52] Alexander: The problem, of course, is this path is not. readily [00:10:00] available to lots of people. And of course, these tables will then be used further, [00:10:10] for example, in publications and slides and all kinds of different things. And then people will use these slides [00:10:20] to create other slides. And from these other slides, they will create a Further stuff like promotional material, and from the [00:10:30] promotional material, another publication is done, or whatsoever, yeah?

[00:10:34] Alexander: So, very often, kind of, these things travel through multiple [00:10:40] layers. And, later on, even if you don’t have a footnote, yeah, that tells you, well, that [00:10:50] is in folder XYZ, Yeah. You still want to find this stuff. Yeah. So just [00:11:00] relying on, Oh yeah, we have these tables with the footnotes. is not enough. Also if you work with CROs [00:11:10] and you know, these kind of areas, it becomes even harder. Yeah. So making sure that your code is really [00:11:20] findable. Makes a huge difference. Then also, your results are findable. Yeah, because, well, with the code [00:11:30] comes usually also the results. And that’s, that’s yet another thing.

[00:11:35] Alexander: So that is the F. Can you repeat what the [00:11:40] A stands for? 

[00:11:42] Janick: Yeah. So that’s accessible. So that means that the code is not like siloed maybe on like [00:11:50] personal drive or, or somewhere where no one else has access to it. But that you, for example, have all of the things that are important to reproduce your [00:12:00] study are compiled into one, what we often call like repository.

[00:12:04] Janick: And that repository is then can be, for example, found in a [00:12:10] remote for example, in the cloud. So there are often cloud remote repositories where you can connect your code to and synchronize it with often this is like [00:12:20] GitHub, GitLab, Bitbucket. So these are like the three big players that most of the people use.

[00:12:25] Janick: And that’s The, the, the cloud instances where you can synchronize your code [00:12:30] with, so that everyone really has access to this one central repository where all of the important things are stored. 

[00:12:37] Alexander: So that means, for example, in the case of R, [00:12:40] also any macros, packages, and please forgive me if I’m not using the right names because I’m not an R programmer all of these kind [00:12:50] of Other things that help you to run the code are also all in there.

[00:12:56] Janick: Yeah, exactly. So so, so when, for example, I [00:13:00] implement the study, something I also have there is, for example, the study protocol. I have all of my scripts that I use to clean the data, that I use to do the descriptives, the [00:13:10] main analyses, the sensitivity analyses I often then have all of the table outputs.

[00:13:15] Janick: And for example, the figures stored there I have general [00:13:20] information about the project, like what’s the overall scope, what’s the IRB information the instructions on how to install the software, for example, our packages and the [00:13:30] corresponding versions that I use to do that. To, to run the analyses and like everything that is required so that someone else could just very [00:13:40] easily grab that code and reproduce my analyses.