One-armed observational studies: fake science?

Transcript

One armed observational studies

[00:00:00] Alexander: Welcome to another episode of The Effective Statistician Today, it’s Benjamin and myself again. Hi, how are you doing?

[00:00:10] Benjamin: Thanks, Alex. How are you?

[00:00:11] Alexander: Very good. As we are recording this, it’s just four days away from Christmas. It’s pretty cold here in Germany and we are in the last treks of the year.

[00:00:25] Benjamin: Yeah, it used to be colder last week. So it’s now even above zero and it’s raining instead of snowing.

[00:00:32] Alexander: Yeah. Weather outside. It is pretty 30, I would say. And the story we talk about today is actually something that fits the weather conditions quite nicely. Because it’s a little bit, no, not just a little bit, it’s the ugly side of science that we will talk about today, or some people actually call it fake science, so science in happens, sad to say.

We will talk about one armed observational studies today and why they’re, in most cases, not a good idea. And going back the very early to my career as a statistician when I was still pretty new in the pharma world and maybe a little bit naive. I actually was working on one of these.

[00:01:27] Benjamin: Me too. It was actually, at that time, I didn’t even bother too much about why or why not. So all the considerations about using these open label single arm trials wasn’t, it was just, yeah, they were there. . We did the best we could do. And maybe it was a bit naive.

[00:01:45] Alexander: Yeah. So let’s first talk about what it actually is. Yeah. One armed observational studies are studies where only one treatment, one therapy is observed. And you can also have, of course, one armed clinical trials. But here’s the one armed observational studies, this is usually studies that are run, these are always studies that are run after the launch of the therapy, and very often they’re even run very close to the time of launch. Yeah. So directly when the treatment comes onto this market, or very shortly thereafter, these studies come.

And so the observational nature of that allows companies to include many different patient. And of course also physicians. Yeah. Treating physicians into these study speakers with observational studies. The treatment is not paid by the sponsor, but through the usual system. Yeah. Being it through, paid insurance or whatsoever. Yeah. So that makes it cheap. The other thing is also because it’s an observational study, you don’t have as many regulations as for clinical trials. That makes it also cheap. And then on top of that, very often the amount of data that is captured is not as much as within clinical trials. Yeah, you surely don’t have as many visits. You don’t have as many. Once you collect what, which is actually sensible for observational studies in general. Yeah. But because you just wanna observe what’s happening. But all these kind of transference things. And of course, there’s no, not so much of security regarding data quality and source data verification and all these kind of things. And it’s not so much kind. Work around clinical operations. All these make observational studies pretty cheap compared to clinical.

[00:03:50] Benjamin: Yeah, no remember that we had, especially when you talk about data quality, we had a lot of interesting considerations regarding safe evident changes. So basically this, the statistician put into the SAP a section about how to handle data. So there was, which was in, which is something we often see our days now in whatever you call it, like centralized statistical monitoring or any support where consider any pattern and or outliers, et cetera. So that’s a different topic. But, so that is where I saw the, my personal beginning in the, in handling statistics like data or seeing the data quality from the statistical side of things.

However, I think one thing you didn’t mention, what I see, what I experience quite often is that in these settings and the settings of single arm trials, we had a usually very high demand for publications. So we get to this in a little bit later. We usually had very strong involvement of one specific investigator, which the key, some key opinion leaders, some key opinion leaders on that one. So that is a characteristic, what I remember and. But on the other hand, I think the objectives and the data collection I’ve experienced differently so that there was still a lot of considerations put into what, how can we make the best use of expanding the population? Or having maybe different, a little bit off-label sometimes. It’s going into off-label treatment for some of the, for some patients or some combinations of, let’s say if it’s oncology, it could, maybe it was first and second line now, maybe third line, whatsoever? So differentiation from the original label. And there were also experiences where given the results from a new label extension studies, so which were usually observational studies, but then in a randomized setting where we’re initiated after the seeing the results to it. So that is on. Since we talk about the bad weather outside, there’s a little bit of sunshine there as well.

[00:05:54] Alexander: Then we will, at the end of the episode, we can also talk about some cases where actually it might be relevant to have these studies. This is always the exception to the rule, of course, but very often the key driver is basically to purchase. Yeah. To, basically, the system is very easy. Yeah. The prescribing physician gets money for documenting what he sees, and if that money is an incentive for him and he can get it, only if he perhaps a certain treatment, then it starts to become bribing. Yeah. And I know about cases where. Tens or more than 100,000 patients were included in such studies. And where even companies didn’t invest some money to put these at the time. Paper case reports forms into electronic forms. So they says, oh, we should just take every force from 25. And there you can see the real rational behind it is not scientific. Yeah. And these are the things that really hurt the overall reputation about pharma. Just decades ago, Ben Gold Acre was publishing a lot about bad pharma and selective publications and all these kind of different things. Yeah. And how pharma companies bribed all kind of different people and that is one part of it. And sometimes you still see these kind of things happening.

Yeah. So if you work within a company and you get this kind of request, oh, we want to start an one armed ob study, be careful about it. Yeah. Understand where it’s really coming from. These are the things that can much more hurt the reputation of your company and the overall industry than being helpful.

[00:08:07] Benjamin: And also from the statistical side of things, it is questionable in terms of how to use and how to interpret the data.

[00:08:14] Alexander: Yep.

[00:08:15] Benjamin: So it’s, I might say it’s not working, for example. Yeah. Also seen that there are examples of studies where we had a comparator outside of study. Basically it’s that we’ve had external information that we. To compare against the results from the study. At the end, it was a little bit difficult to meet this because of the variety within the population, but at least it was one of the objectives for the study to select the correct, like a subset of patients as population to really compare again. So there, there are also like some interesting parts of how to use the results of the data statistically, but usually it’s a lot of fighting back against the publication team to say, no, no P value. There’s no P value, and so that was.

[00:09:04] Alexander: Yeah, and people come up with all kind of crazy ideas. Let’s look into whether the changes from baseline are significant. And then say the changes from baseline are significant and point in the right direction. Soon the treatment is working. Nope. , you can’t say that. Yeah. And you get into all kind of different problems, like you said, if different populations further say external controls. You might have different background effects. You might have the, your external data might be from different time periods. There might be different healthcare implications. Maybe you don’t have the same endpoints that are you collect in your observational study compared to your external data. You surely have different data quality. Than in clinical trials, you maybe have different time points where you have collected data, which leads to lots of problems. And also when you have these observational studies at the start of the launch period, you have very different patients included. Yeah, different patients and different physicians.

First is if you launch a new medication, there are usually a certain demographic of physicians that try these out. These are the early adopters the people that really wanna be on, on top of everything. Maybe the people that have participated in previous studies. Yeah. So phase three of phase three B investigators, they are already used. To these kind of medications and therefore will more likely prescribe it. By the way, another reason why some study, some studies are implemented in clinical trials just to get physicians, some experience on the drugs is that they are, let’s say, more likely to prescribe it in the future so that you have different patients, different physicians at the start of a new therapy in the market than a couple of years later.

[00:11:16] Benjamin: Yeah. And also to talk about, again, for on the data is kind, you said that we don’t really have a lot of data available simply because it’s new on the market. So that means that also the responsibility for us to work on this, and especially on the safety side of things, to have a quite, have a close look of what’s going there is immense. So it’s, and on the other hand, the risk that we are seeing something which we, which isn’t planned because I, we don’t collect the data, we don’t collect background data as much as we do for clinical trials. So you don’t really know what the patient is or who the patient is entering this study. So there’s a lot of risk and responsibility associated to, especially for a statistician or not especially, but also for a statistician for you and working on such things.

[00:12:08] Alexander: Yeah. And talking about safety. Yeah. The problem is you can’t really win. Yeah. As I pointed out, if you see some kind of efficacy, yeah. Oh, you don’t really see effective efficacy. You see maybe positive changes in certain endpoints you can’t really interpreted, and if you then see something on the safety side. Yeah. Then it’s even worse because you don’t know whether that is for real or whether that is something just because you. You have bad luck because there’s different communication, or these are different patients or whatsoever. Yeah. And then you create a safety problem or safety signal that maybe isn’t real. Therefore, I always favor. Comparative observational studies. Yeah. Where you at least have some proportion of the patients that get on other treatments. Yeah. On that, on standard of care that are on alternative medications whatsoever.

Yeah. And you observe these over time as well to understand what’s really happening. Yeah. And then you have an internal control. You still have, of course, all the problems with usual problems with observational studies. Yeah. So you need propensity scoring and all kind of other things here, but at least you have collect the data through the same channels, with the same endpoints, with the same time points over the same time period with the same investigators and all these other things. Yeah. So lots of problems about biases you can always get rid of. And you also have patient level data. Yeah. So that is immensely helpful.

Yeah. And then you can actually see, ah, yeah. The patients on the new therapy look different to those that are on older therapies. Yeah. They are more severe, they are more pretreated. They are don’t know younger whatsoever. Yeah. And that helps you to get a much better sense of what’s going.

[00:14:21] Benjamin: Yeah, I think you are right. It’s, there are limitations to phrase this nicely, to the whole setup. And I think there that is a differentiator between a seriously, like a scientifically valid observational single arm trial. So where you really put restrictions on the population on inclusion, exclusion criteria on background of the patients to make it comparable and to really have a valid scientific approach to it versus a marketing driven study.

[00:14:54] Alexander: Yeah.

[00:14:54] Benjamin: And, but yeah, but still, it’s still, we have to remember that if you have a good treat, It’s worth the marketing.

[00:15:02] Alexander: It’s, yeah.

[00:15:04] Benjamin: That’s don’t you as a statistician put your signature under the scientific part of it. And so while I appreciate that the successful molecule compound treatment is worth to be present and first in the market.

And first, that is what we all work for, right? To get the right medication out there. But still at the end, it’s your signature on the sap, on the statistics, on the results as a statistician. And that is where you also have a voice to consider your legal knowledge and background and make some improvement to what we observe.

[00:15:38] Alexander: Yeah. And also it’s your name on these publications. Yeah. That is really important. And I talk a lot about anything leadership. Yeah. This is a space where you as a statistician can influence others. Yeah. Where you can understand where is really the value come from and where you can speak with relevant peoples and help ’em understand, hey, this is usually it’s, it doesn’t helpful to say you are trying to bribe people here. Yeah, that’s probably not helpful. But if you explain to them, this can actually hurt much more than it. And here are alternative ways. Yeah, let’s spend also some money on collecting other patients on other treatments. That will help a lot than later on with also use of these data. I can tell you publishing these one armed observational studies in anything that is peer-reviewed is really difficult unless you have these low quality paid publications, and of course you usually can, you can get it to, as opposed to that, some conferences, but getting it, really with some impact is a big difference. And a couple of my highest impact factor publications are actually from comparable observational studies. Yeah. Long-term observational studies that are years longer than usual. Observational study observation time for clinical trials, much bigger, looking into more vulnerable subpopulations. All these kind of different things you can do with observational studies. Let’s at the end, talk shortly about a couple of few exceptions that where I would see observational studies might be helpful. One is, for example, if you want to look into some kind of specific behavior that is only relevant for the new therapy. For example, you wanna check with a certain com medication that must be used is really used. Yeah. Or whether you are interested in whether only the correct population is looked into. Yeah. And whether the dose that you recommend is really used, or is it up ration or whatsoever? Yeah. These kind of questions that are specific to your therapy. . It makes sense there to have a one armed study, but then probably you don’t need to collect a lot of other stuff then just this information.

[00:18:24] Benjamin: And also, if the focus is on safety, I think if the safety, if their questions answered from safety, I don’t know, rational for forum.

[00:18:34] Alexander: I disagree. As I just said, if you then see something in safety, you want to explain it.

[00:18:38] Benjamin: No, you have to put the right frame, right? So you have to collect the right data for this as well. It’s not if you do a marketing study and then just get the safety data out of it and try to use it, that’s the tricky part. But if you design the population and the inclusion criteria and compare this comparable with others, with previous studies and for long term follow up, et cetera,

[00:18:59] Alexander: but even there, I really would like to have comparative data. Yeah. So just to put things into perspective. Yeah. If I have, just thinking about, let’s say antidepressants. Yeah. And you wanna understand what is the safety longer term? Yeah. You wanna understand how is it different to safety of other medications, of the same class of other classes, so that you can put things into perspective. Imagine you have more suicide. What do you do with it? Yeah. You don’t know. Is that really lack of a efficacy or is there something other things going on with your antidepressants? And if you have comparator arms, then you, and at least see ah! Is a suicide consistent across all the different arms, or do I have a detection bias or whatsoever? Yeah. And all these problems you can at least mitigate if you have comparator. .

[00:19:52] Benjamin: Yeah, no, I agree. But usually if you are new in a market, you already have the comparative data.

[00:19:56] Alexander: Yeah.

[00:19:57] Benjamin: It depends. It’s always, it depends, but it’s still, there’s, there are rationals outside. What would the, just making this always like a double, like a two arm trial or compared to arm compared to trial? Well, whatever. Overall it is a little bit sunshine in a, on a rainy day, so it’s,

[00:20:12] Alexander: yeah, . It would definitely sleep better if I know that I have some kind of internal control. Even if it would be just, let’s say a third of the patients. Yeah. So that would help me sleep better and also surely helps with the publication. Okay. Very good. Any final point about this?

[00:20:35] Benjamin: No, actually I think we, we talked this through, so it’s interesting to have to understand that we both had a similar experience in the beginning being naive and new and being thrown into this and realizing it late, that there is a bit of a scientific gap in executing such studies. But actually that’s why we are here and talk about it.

[00:20:53] Alexander: Yeah. And so if you are younger in the pharmaceutical industry have a look into what happened 20, 30, 40 years ago. Pharma did a lot of really bad things. That’s why lots of pharma companies now have a code of conduct and all these other things. Yeah. Where, and you need to do these trainings of not bribing people and stuff like, It’s because people did it, not because pharma is so great that it fights against, yeah, we have these codes of conduct, not because we are great, but because pharma did a lot of bad things in the past. Think about where we are coming from. That helps you to understand a lot about what’s going on and gives you perspective, especially when you see something like this coming your way on your desk. Thanks so much. Have a great time.

[00:21:50] Benjamin: Thank you.

[00:21:51] Alexander: And talk to you soon again.

[00:21:52] Benjamin: Bye bye.