Estimands for Repeated Continuous Outcomes

Estimands for Repeated Continuous Outcomes

[00:00:00] Alexander: Welcome to another episode of the effective statistician and today I’m super happy to have Oliver Keene on this show, and I can’t believe. He hasn’t been on it before. A couple of years ago we had a really, really nice discussion about estimands as part of a PSI conference where we did some kind of Role play and he was one of the key person involved in preparing all of these and we had these different viewpoints from the sponsors, the regulators, the HTA guy and, and all these kinds of different viewpoints.

[00:00:44] And I remember at the time I was kind of defending the. HDA position on S demands at that time and was strongly favoring pull treatment policy. And Steve Ruberg was the patient voice and he [00:01:00] was favoring kind of this on treatment approach because all the patients that he talked to said, well, if I take this medication to the end, how will it work?

[00:01:11] And so it was Really, really insightful discussion. And I think if you scroll back quite a lot of episodes, there should be some discussion with Steve Ruberg about this. And of course, you can check out the PSI video on demand part. I’m pretty sure it’s on there as well as part of a conference. Who was that?

[00:01:36] Hi, Oliver, how are you doing?

[00:01:39] Oliver: Hi, Alexander. Well, first of all, thank you very much for inviting me on your podcast. And congratulations on having 300 episodes, I think it is.

[00:01:49] Alexander: Yeah, over 300 episodes. Yeah, yeah.

[00:01:53] Oliver: I think it’s a fantastic way of getting statistics discussed more widely among [00:02:00] people. And you’ve covered such a, such a lot of ground, had some very eminent speakers on, so I feel privileged to be part of it.

[00:02:06] Alexander: Yeah, I had, honestly, I had speakers from all kinds of different areas and also experience levels, you know, from people that are New grads up to people that, you know, had decades of careers, professors’ senior leaders from companies, but also kind of, you know, projects that stations, you know, that work in the trenches.

[00:02:33] So, and that, that I think makes me really, really happy. You have, have a long career already. Tell, tell us a little bit about that. Yes. Yeah.

[00:02:45] Oliver: Well, I, well, I worked for over. 30 years in in the palm seed industry. Working for GSK as a, as a statistician. I’m now an independent consultant. I mean, it was, it was, it was a long, [00:03:00] a long, a very varied career.

[00:03:02] But I always, I always enjoyed it. It’s always been challenging. The industry has never, never stayed the same over that period. It was, it was always evolving. I always felt I was learning new things. So yeah, and I continue to enjoy working, working in the industry.

[00:03:17] Alexander: Yeah, so if you want to work with a really experienced consultant, Oliver is definitely a person to go to and of course I’ll connect his LinkedIn profile into the podcast show notes. , Today, we’ll talk about estimates. So, and for sure, this hasn’t been, you know, it’s the first time that we have talked about estimates. And we’ll specifically talk about estimates in the situation where you have continuous endpoints collected over time. Yeah, like and then psychiatry, for example, you have a lot of these different [00:04:00] questionnaires, but could be any kind of continuous endpoint.

[00:04:04] And I remember quite some time ago Craig Mallincrotch did a lot of research about different ways to. Estimate things like MMRM, LOCF that was, you know, kind of about 20 years ago, a big, big topic. And at the time he’s, he was strongly favoring MMRM over LOCF and made this. Estimation technique, a very, very prevalent thing, and it’s nowadays also very easy to implement both in R and SAS, of course, and so that has helped a lot to make it widely used.

[00:04:50] Now, recently actually this year in 2023 Ting published an article and advocated [00:05:00] Yeah, for our discussed what could be kind of different estimates around this. And as a response to this Oliver together with a couple of other authors David Wright, Daniel Breton, Thomas Drury, Sunita Rihal, and Ian White, so a couple of really, really famous names, responded to this, and this is what we want to talk about today.

[00:05:25] So Oliver, from your point of view, what are, what’s the first most important point you have about the paper from Ting? Okay,

[00:05:38] Oliver: well, let me first of all, you know, start, you said this was a collaborative work, and I just wanted to say how much I enjoyed working with my co-authors Sunita, Dan, Tom, David, and Ian, and it was really great to be part of a project with, you know, people who are such experts on the subject of S Demands.

[00:05:57] I mean, estimates have always [00:06:00] been something that I’ve, I’ve been really interested in. Right back from when the, the US NIH published, published the theirs, their tracks on. Missing data. I mean, it’s always interested me. You know what treatment effect you’re actually estimating it in an in a clinical trial.

[00:06:18] And this comes back to, you know, even before estimands, the idea of intent to treat analysis and what exactly did that mean? So I really welcome the, you know, the introduction of the estimates framework. I think that’s, that’s been really important. And obviously we saw, the group saw this article by NaiteeTing and we came together, you know, this is Ian White works in academia and all of us work for different, well, we work for different companies.

[00:06:46] We felt Quite strongly, the article had some misconceptions about the ICHE 9 estimates framework and we felt uncomfortable with the arguments that were, that were being put forward for a, for a broad [00:07:00] adoption of last observation

[00:07:01] Alexander: analysis. Yeah, that is the first point. What actually is last observation analysis?

[00:07:08] I know last observation carried forward analysis, but what is the difference to last observation analysis?

[00:07:15] Oliver: Well, in practice, I think the points Natty the article is that essentially they’re the same thing. I mean, conventionally, traditionally we’ve always thought of last observation carried forwards.

[00:07:25] You know, that’s the idea that if somebody discontinues the treatment. You take the last observation on the treatment and use that as, you know, for future time points right through to the, to the final, final time point of the study. I think the distinction he was making was he didn’t see any kind of carrying forward that he sees the last observation that is the.

[00:07:46] the kind of the outcome for the patient is the last observation. So he’s trying to make that that kind of kind of distinction. But that really then brings into question is that is that really a treatment, an appropriate estimate for a treatment [00:08:00] policy approach.

[00:08:02] Alexander: Okay, so let’s untangle this a little bit.

[00:08:06] So the first point I think is that estimation is one thing and defining the estimate is a completely different thing. What should drive what?

[00:08:22] Oliver: Yes, I mean I think current views is you define, you define your estimate first and then consider what’s an appropriate estimation strategy for your, for your estimate.

[00:08:34] Within the paper, some of this gets a little confused, I think, between, you know, starting with an estimation and then kind of justifying it with a, with the estimand and the strategies. I think, as you say, I think it’s important to define your estimate first and then consider, you know, what are the appropriate estimation strategies.

[00:08:56] Alexander: Yep. Yeah. I completely agree [00:09:00] with that. You always first need to start with your intent mind. Yeah. And then it’s about kind of, okay. And how do I know? Approach that. Of course, if you then come up with something that is not estimable, then well, probably you need to go back. But,

[00:09:20] Oliver: yeah, yeah, that’s, that’s always the, you know, there’s always this kind of iteration between, you know, what you would ideally like and what you can, what you can, what you can estimate.

[00:09:30] But I don’t think, I mean, people seem to think that, for example, that treatment policy is easy to estimate. Well, actually it’s not. When you’ve got missing data, it’s actually one of the more, it is quite difficult because you have to make assumptions about the missing data. So. Okay. Yeah. Sometimes there’s misconceptions about what’s easy to estimate as well.

[00:09:50] Alexander: So let’s dive a little bit into Treatment policy here first. Yeah, sure. Treatment policy and correct me if [00:10:00] I’m wrong, basically speaks about you decide at the start of your treatment, how exactly you will treat patients. Yeah. And then you analyze accordingly. So for me, is that what basically mean? Okay, let’s estimate.

[00:10:21] Yeah what is the outcome of a decision we make at baseline? and this decision is, okay, we start with treatment A versus we start with treatment B. Or we start with treatment A and in case something happens, we’ll do this. Or versus B and in case something happens, we will do this. Yeah, so for example, it could also mean we start with treatment A, and if it doesn’t work, we increase the dose, yeah, and then we observe the outcome, or we have treatment B, and if it doesn’t work, we [00:11:00] increase the dose, yeah.

[00:11:01] But the, the, the policy needs to be clear, isn’t it?

[00:11:07] Oliver: Well, I mean, I think it’s, it’s interesting the way your kind of described it but I think, I worry about the way, the way you’re thinking treating policy. Because that assumes that that would be true if everybody followed the protocol, if everybody did what was said in it.

[00:11:25] And the whole point of the esterhans strategy is you have people who discontinue medication, who take medications that in theory are not allowed by the protocol, who once they discontinue the trial can do what, discontinue medication, they can go on to whatever medications they want. So I think, I think that’s one of the difficulties, you know, with I’m thinking of treating policy in that sense in that way, in the way that you’ve described.

[00:11:50] I mean, I would, I would worry about that. I think treating policy more, it’s better to think of it as an effective assignment. So all you’re thinking of is, you know, if we originally assign [00:12:00] a patient to these different medications. What is their ultimate outcome? You can try and control as much as you as you can in the protocol in terms of, well, if this doesn’t happen, but in real life, as we know, trials, people discontinue their medication.

[00:12:16] They don’t follow what’s in the protocol. They don’t you know, if a fails, they don’t necessarily go on to, you know, That’s true. So I think that’s one of the difficulties, the misunderstandings about treatment policy. It’s you’re going to end up, you know, once people have discontinued the treatment, they can do, you know, it’s, it’s the discretion of the investigator what, what they get.

[00:12:36] Alexander: Yeah, of course it is, but of course you could have treatment defined as something that is based on some kind of patient post baseline variables. You have said all the time, yeah, so that you, trig to target or things like this. Yeah. Or you have certain kind of ways to manage [00:13:00] communication or things like this.

[00:13:02] Yeah. So you could include that in your treatment decision and your treatment. Well, not in your treatment decision and how you define treatment. Yeah. So I think that is the. And when you think about the estimate framework, one of the things you need to be really clear about is about what is treatment a.

[00:13:23] Yeah, what is that really? And it could be, well, starting with this molecule, but it could also be a certain strategy, yeah, that you, of course, if then patients or physicians Deviate from the strategy, you could get missing data. Yeah, that is completely, completely see that. Yeah, but I think that and then managing the missing data is yet another point.

[00:13:54] First, you need to define what is really a treatment strategy.

[00:13:58] Oliver: Yeah, but you also, I mean, it just. [00:14:00] So you will also have data. You may not have missing data. You’ll have data, but you’ll have data from people who are following different treatment policies. You know, they’ve, they’ve taken it. And, and, you know, treatment policy, if you, if, if, if your strategy is treatment policy, you have to, that’s, that is the data.

[00:14:17] That’s the data you’ve got to use regardless of what, what, you know, what’s led them to those particular, what policy, what set of treatments they’ve used.

[00:14:25] Alexander: Well, I’m not sure. I’m not sure. Because, so, so, if, that, if you say your treatment policy is treatment A, you start with that, and then whatever the, the physician describes or does is your treatment policy, yeah, then you have, yeah, then you never have any missing data, of course.

[00:14:50] Yeah, because you will always have complete data, and then you can analyze as you randomize. Very, very easy. Yeah. The drawback, of [00:15:00] course, is that it’s very, very hard then to later on describe what was, what is the treatment. Yeah, especially if things change in terms of, you know, background and all these kind of other things.

[00:15:15] Yeah. and you don’t know at the start, of course, how many patients will deviate. Yeah. Oh, well, they don’t really deviate because you, you include any deviation in your treatment definition. Yeah. So then you basically move the, the problem from the intercurrent events into the, into the treatment definition.

[00:15:41] Like what you sometimes do with the confidence at the endpoint, you move it into the endpoint definition. Yeah. So these are of course, both, both possible scenarios.

[00:15:53] Oliver: Yeah, I mean, I think it’d be interesting, you know, perhaps we can get on to talk about composite and, you know, other strategies I think obviously, you know, going [00:16:00] back to the Ting paper is, is very dismissive of any, any strategy other than treatment policy or, or interestingly, while on treatment, um, and particularly the paper is very dismissive of the hypothetical strategy.

[00:16:14] And I think sometimes that word, Hypothetical is kind of problematic to people because it sorts of sounds like something that didn’t happen count. You know, well, I’m only interested in the facts. But I think, yeah,

[00:16:26] Alexander: the word is not really helpful.

[00:16:29] Oliver: No. You know, people, people react against it. You know, I’m a scientist, I’m interested in facts.

[00:16:33] I’m not interested in what might have happened. But I think what I think a better way to think of a hypothetical strategy, you know how I’ve been trying to think more, more recently. I mean, the important thing is it can address causal questions of what, what is the treatment actually doing? You know, these, I think these are important scientific questions.

[00:16:52] They’re important clinical questions. I think, you know, because we think so much in regulatory terms and HTA terms and, you [00:17:00] know, what is the policy, you know, what was the effect of this policy? And you know, but to an individual patient, I think, and for science as well, I think it’s important to try and address questions of what does it, what does that treatment itself, what is the cause and effect of that treatment?