Patient-reported outcomes and the FDA Patient-Focused Drug Development Guidance

Interview with Rachael Lawrance

The FDA is developing a series of four methodological patient-focused drug development (PFDD) guidance documents to address, in a stepwise manner, how stakeholders can collect and submit patient experience data and other relevant information from patients and caregivers for medical product development and regulatory decision making.

In today’s episode, we discuss the process with the FDA publishing a timeline of planned public workshops, draft guidance documents and final guidance documents for 4 documents, originally planned to finish in end 2021 and its objective to explore the factors that need to be considered when developing COA-based endpoints.

Stay tuned as we also talk about the following points:

1. What’s in the 4 guidelines
2. What factors should be considered when constructing COA-based endpoints?
   • Effect of disease type (e.g., acute, chronic) on endpoint selection – acute symptoms resolve within weeks
   • Treatment objective (e.g., cure, symptom management)
3. Population
4. Endpoint
5. Intercurrent Events – What can affect your measurements interpretation
6. Population level summary – What Is the Final Way All Data Are Summarized and  Analyzed?

References:

• FDA Patient focused drug development guidance
• PFDD Guidance 4 workshop discussion document
• Public Workshop (Dec 2019)
• Meeting slides

Listen to this episode and share this with your friends and colleagues!

Rachael Lawrance

Director and Functional Lead – Statistics | Adelphi Values, Patient-Centered Outcomes UK

Rachael leads a team of statisticians at Adelphi Values Patient Centered Outcomes group, involved in working with clients in all aspects of strategy, analysis and interpretation of PRO data in clinical programmes. Prior to Adelphi, Rachael worked at AstraZeneca for 16 years, including 6 years as a late phase clinical trial statistician on oncology projects. Rachael has particular interest in the topic of estimands in oncology, leading the PRO task-force within the Estimands on Oncology PSI/EFSPI Working Group and co-chairs the ISOQoL Statistical SIG workstream on estimands. Rachael is also a Director on the Board for PSI, leading the scientific programme of events for PSI.

Transcript

Alexander: You’re listening to The Effective Statistician Podcast, a weekly Podcast with Alexander Schacht and Benjamin Piske, designed to help you reach your potential, lead great science, and serve patients without becoming overwhelmed by work. 

Today, we are talking about Patient-Reported Outcomes or PROs and the FDA Patient-Focused Drug Development Guidance. A really really nice interview with Rachel Lawrence, so stay tuned for that. 

Patient-Reported Outcome is really an important thing. Everything is moving more toward patient-centric in terms of medicine. And I think that makes a lot of sense because patients are just much more experienced and educated and want to have a say in treatment and to Patient-Reported Outcome is a really important topic. And of course, the FDA is a really important player. So listen to this awesome interview with Rachael. 

I’m producing this podcast in association with PSI, a community dedicated to leading and promoting the use of Statistics within the healthcare industry for the benefit of patients. Join PSI today to further develop your statistical capabilities, with access to the video on demand Content Library, free registration to all PSI webinars, and much much more. And of course, there is the PSI conference coming up in Gothenburg. 

So if you have not registered yet, this is a great opportunity to do so. Rachael and I will be there and lots of lots of other people. So, head over to psiweb.org and learn more about this and see you in Gothenburg. And now let’s go to the episode.

Welcome to another episode of The Effective Statistician. And today, I’m talking with a good friend, Rachael, how are you doing? 

Rachael: Thank you, Alexander, very well. Thank you. 

Alexander: Yeah, it’s really great that we’re talking together on the podcast. We have been talking to each other through PSI for a very very long time and have met quite often on all kinds of different PSI-related occasions, though it’s great to have you on the show today. 

Maybe you can talk a little bit about your career up to now, what brought you here? And what brought you to this topic of Patient-Reported Outcomes?

Rachael: Yeah. Thanks very much for the intro and for the invitation to talk to you on the podcast. And yeah, I actually started my career, not as a statistician at all, I did Biochemistry and Genetics degree and I started working at AstraZeneca as a Molecular Geneticist in their genetics lab, but actually, I found rather than doing ptr’s which everyone now knows exactly what they are.

I like analyzing the data and understanding that. So I should have moved into Statistical Genetics. And that’s sort of what kick-started more, my statistics, focus in the career, and then the diploma in Statistics while I was at work. And while working in the research area of genetics. I started working on Polymorphisms of variations and drugs that might lead to different responses to treatment and we collaborated with, the clinical teams on some of the projects, with a clinical trial, to understand If that was indeed reasons for some of the differences we were seeing in treatment effect. 

So that was kind of how I got introduced into the clinical trial side of the statistics Department at AstraZeneca, and I moved from the statistical genetics research area into the Clinical trials section, and in terms of working as a more standard clinical trial Statistician, where I met many colleagues and really grew and developed, in terms of statistical, the application working clinical trials. So, I worked in a spirit tree disease area on a few projects, and then I mainly worked on oncology projects after that. 

So, yeah, but that was a really good basis in terms of all Oncology and points. And what we did at AstraZeneca.

Alexander: Cool. Where and when did the PRO interests kick in? 

Rachael: It was kinda mixed. So, actually, in my last projects at AstraZeneca, I was the lead Statistician, So I’d be really sure about endpoints. And you know, I worked together with colleagues in the PRO department on some of those endpoints realizing that they weren’t very, you know, we weren’t very good at that time. Really defining what we were going to do with the PRO endpoints and you know that they just produced lots and lots of tables of descriptives and handed them to a different department and nobody really worried about it.

But I did work really closely to integrate back and these projects into the CSR. So I had that background in AstraZeneca and then. Well, what really happened was AstraZeneca said they were going to move the location down from where I live. At the edge of the peak, to Cambridge and I decided that wasn’t for me, so I left AstraZeneca.

And then for a while, I worked a little bit freelance with my own consultancy work for a little company while my children still small and that was the time I got more active with PSI because I wanted to really keep connected to the community of Statisticians and bummer even though then I wasn’t at AstraZeneca and then through after a while of working on my own as a freelance. I really miss working in a team environment and I supposed it was coincidental that I discovered that there was our company near my house. That was looking for a Statistician who works from Patient-Reported Outcomes. 

So it was kind of synergistic. They were looking for somebody to come in and work with a clinical trial, and clients in terms of that focus. So, that’s when I joined Adelphi Values which was about four years ago now. I can’t believe that time flies since joining them and they became my background in terms of clinical trials, you know, really useful to apply that and learn the specialism in terms of patient-reported outcomes. And I think being able to learn from experts at Adelphi in terms of the nuances and the things that go behind patient-reported outcomes makes you appreciate the richness of that data and how it can be used. 

Also, you can see it from the other side, you know, coming from the Pharma side of what priorities can be in terms or perhaps regulatory or others like HTA, you know, or the audience. 

Alexander: Awesome.

Rachael: That’s probably how I got where I am today. 

Alexander: Awesome. Yeah, and PROs is a hot topic for already quite some time. And there are also some episodes that I already recorded about it. And there is, of course, one Stakeholder that is always important in the field and that’s the FDA. 

And so it’s really great to talk today about the FDA and their kind of current view on PROs as they have released draft guidance on it. But before we go into this draft guidance, maybe you can talk a little bit about the history of how that evolved with PROs and Regulatory. 

Rachael: Yeah, I think as part of the FDA’s efforts in terms of relationship with what they published in their 21st-century cures act, the FDA sort of set out in terms of wanting to really develop some guidance. In detail on Patient-Focused Drug Development, all the way from all aspects. So not just how we use it in regulatory decision-making but really what does that even mean in terms of the patient’s input into the whole process as well as development validation or fit to use questionnaires. 

So they put together a statement saying they were going to work on methodological guidance documents to sort of address in a stepwise manner, how-to, you know, collect and submit that relevant data so that it could be used in the FDA decision making. 

And they want to touch on a few recommendations for methods. And as well as analysis interpretation aspects. And I think that they started a while ago, so, it was in 2017, that the FDA published their intent to do these guidance documents and there is an older FDA guidance document in 2009 in terms of PROs in clinical trials. The idea is that the series of documents will in the end surface or supersede that and move things forward from the FDA’s point of view. 

Alexander: Okay, very good, that’s a nice overview. And by the way, if you listen to this and you’re just thinking, oh, that’s a lot of references. You will actually find all of these references on our homepage. So just head over to theeffectstatistician.com where you can learn more about the references and also see what Rachael is up to, otherwise and kind of read her bio and so on. 

Okay, so in terms of these different guidelines, where are we at the moment? 

Rachael: Yeah. So as I mentioned that the FDA started the process in 2017 and they did put us on this timeline this guidance should have been finished by the end of last year, and I imagine that the global pandemic has changed timelines. But as of today, Guidance 1 has already been finalized, and actually, when we’re recording this podcast Guidance 2 has just been released last week at the end of February so they’re kind of finalized there. Guidance 3 and 4 are still available on their website and the FDA has a draft format. Although public discussion meetings for these happened back in 2019. So I don’t know exactly when we’re going to get these final, you know, final guidance documents, but I imagine that they will be coming soon. But, you know, there’s a lot of information even in the draft materials, in the discussions, and a little bit with what you can glean from the direction FDA of travel and the sorts of topics that the FDA is going to have in these guidance documents. 

Alexander: Cool. So let’s go into these guidance documents a little bit closer, one by one. And when we think about the first one, what are the key topics for the first one? 

Rachael: Yeah, so this is very much early on in terms of collecting patient input. So this is very much about thinking, even at all about concepts, things that could be of interest to the patient. So it’s very early to give up in terms of thinking about possible research questions. And methodology about how you could, you know, who you are going to get input from that’s relevant to that disease area. So whether it’s patients themselves, caregivers, whether it’s parents or carers in certain situations, it’s very early on in terms of thinking in terms of the questions that are included in that and the guidelines in terms of thinking about that patient input. 

Alexander: Okay. So this sounds like a really kind of qualitative research that you would do at the beginning to understand, okay, what are actually Concepts that are important for patients? Is it something around sleep? Or is it something about itching or pain? What are all the different things that the patient might be concerned about for specific indication? 

Rachael: Yeah, the thing is the guidance goes into more and more of those methods, for how you can get that state information from a patient. Now, how is that qualitative research? How do you do interviews and develop an interview guide? How do you even think about selecting questions for surveys? And you know, how they considered being applicable across different countries and demographics and those sort of qualitative research topics. So yeah, Guidance 2 is a lot more information on these methods and is appropriate for that sort of qualitative research. 

Alexander: Okay, very good. So that is Guidance 2, interesting. What do you think are the main problems there that need to be answered? 

Rachael: Yeah. I think that something I really learned a lot more when joining Adelphi was having no sort of concept ahead of time of all the work that actually needs to go into a sort of developing, those questionnaires and finding out, you know, what those areas are? What do even those concepts start with our interest? And how you go about doing that. And why it’s important to do very specifically for disease areas or whether it’s children or adults and things like that.

It is very important that we actually really get those concepts really separated. It’s very big, you know, it’s very psychological. It’s very much about all those sorts of aspects of being understood, understanding those very different nuances about what it is about. It wouldn’t just be a pain, for example, it could be something a lot more specific, interference is how much interferes with daily life? Or how much you can’t do? Or where the pain is? What types of pain? Or how long your pain is? For example, there are some linked concepts and things like that in terms of really thinking about that concept elicitation phase is really getting to the bottom of those important things through those interviews and although it’s qualitative and less familiar to a Statistician, you know, there’s a lot of science behind how that qualitative research is done and applied and you know, it’s a very rigorous, scientific discipline and it’s in its own right. And you know, sort of really underpins that what we might be used to seeing is a questionnaire turning up in clinical trials. There has been a lot of thought behind that originally about how that was developed. 

So, yeah, I think it’s still an active feeling that it still is worked on because of maybe the growing disease areas that we work in, the different areas making sure that contact is appropriate across different demographics of aspects. 

Alexander: Okay, Guidance 3, what is Guidance 3 all about?

Rachael:  So, Guidance 3 is a lot more about how you refine those concepts and the impacts that the patients have told you about. But how do you make that into a questionnaire? Which determines an instrument? So, in terms of how you will make sure that those questions and instruments that you give to patients are doing what they hope they do, that they’re reliable, they’re accurate, if you ask the same patient, the same question, lots of times and nothing has changed for them. That you’re getting consistent results and equally that you are seeing a change where you would be expecting to see change. It is less sensitive enough to those relevant contexts.

And I think this particular sort of guidance doesn’t serve to use a term like validation per se of all questionnaires and outcome assessments, that they’re more fit for purpose, but that doesn’t just mean that the sort has gone. Oh, I do think that there’s quite a lot of guidance and is quite a lot of rigorous about how you have to demonstrate it. How it is fit for purpose. So whatever purpose that is, where that is. You know, for a patient-physician interaction or whether it’s used in a clinical trial that can be lots and lots of different areas where Clinical Outcome Assessments are used. It’s not just in clinical trials. So these holes, if you know, the FDA is still interested, could be used to help ascertain how you were going to use a medical device in terms of walking aid or something like that. It’s not all just about drug treatments as well.

So lots of different aspects can come into play though. So yeah, making them fit for purpose. I think Guidance 3 really, you know, gives that kind of perspective in terms of those really important things that would enable you to say that and demonstrate. 

Alexander: Okay. Well, you just mentioned a word or a phrase that is kind of related to PROs and that is COAs, can you talk a little bit to see how this relates to each other?

Rachael: Yeah, so I think that’s when you’re new to it. You know, the guidance is so suddenly swaying COAs and you’re like, hang on, I thought you’re talking about PROs. So COAs stand for clinical outcome assessment, and that’s the general term for all these sorts of measures. So a PRO is a patient-reported outcome. So that the act does mean, so what if it sounds like the patient themselves report, but you could have caregivers helping, you know, caregivers filling out the form or parents, for example, filling out a form on behalf of patients. So there will still be a clinical outcome assessment in terminology, but there are different acronyms depending on the exact type of questions. 

So, yeah, you’ll kind of find the guidance document generally uses COAs in the headings and those terms, but you know, PRO is a type of a COA.

Alexander: Very good. So, in terms of these 4 guidelines, what is Guidance 4 that we know about? 

Rachael: Yes, Guidance 4 is intended to, it is titled that they will, address them methodologies and standards and touch on maybe Technologies to really think about how you’re going to collect and analyze the COA data in clinical trials or at least to incorporate those into endpoints that could be considered to be robust but regulatory decision-making. 

And also to do that in Guidance 4, there is also quite a big section on being clear about how you can identify a meaningful change in the clinical outcome assessment because, you know, not necessarily a statistical significance is even an important change for a patient. So, you know, there’s guidance that covers these bases, to some sections. 

Alexander: Yeah. I think the important difference is a really interesting topic in itself, isn’t it?

Rachael: Yes. 

Alexander: Because when you think about it, an important difference can depend on so many different factors. Yeah, so if you maybe start with a very severe disease, a small change might not be relevant. But if you start with a very low symptom set, a small change means that you get rid of all the symptoms which are probably relevant. Yeah, so this is one thing. The other thing is a kind of change of time. Well if it happens in a day, may be really relevant, if it happens over a month, may not be relevant. And this has nothing to do with P values because what it depicts shouldn’t depend on sample sizes and things like this. So that’s a really interesting aspect. What are your takes on these clinically relevant differences, relevant for patient differences? 

Alexander: Yeah. I know. It’s a big topic and I think the FDA is very clear that they’re very precise about the importance of identifying what’s important within-patient change. So almost as you’re saying, you know, for those patients, how much can they change? What’s an important change for them? And that’s some of the research that we do earlier on potentially in the validation and looking at those sorts of aspects. You can really do qualitative research, you can help understand some of these aspects as well in terms of what would you say is an important change in these outcomes. 

So there’s a focus on what’s really meaningful and important for a patient’s individual response, and that’s particularly what you want to focus on in the FDA guidance. And then I think there is also the aspect that you’re almost touching on. There are different considerations for what’s meaningful as a change over time if you’re looking at a group. So if a group of patients and that meaning changes over time. What does that mean in terms of interpretation of that mean scores over time, and also how big should the difference be between the means between patient groups? 

So it kind of starts to sound all a little bit confusing because you’re like, there are three different types of changes that we might want to consider. I think that’s exactly why some good guidance is there. And, you know, it’s not just a simple, you know, 01 number it is just this time point you need to consider these aspects in interrelatedness and because of the depth of the patient, that information that there’s a lot to know how many symptoms are there when it started? How much have they gone down? Or is it a floor and a ceiling effect because they can’t improve anymore. But that doesn’t mean that there’s not been some degree of treatment effect. 

So yeah, they kind of all come together, but I think the FDA, particularly in this document, is very clear about saying what’s important for individual changes and that can be useful. So just put into contacts about could be, you know, in PROs and you would like to say what proportion of patients improved that obviously depends on the threshold using the group agreed for that change. But it also, you know, becomes a quite interpretable number once everything underneath that is understood. So, how many improved, how many decreased and, you know, we’re basing that on saying we had to exchange by 10 points, 15 points, or on the scale that looks meaningful on that scale. And I think the thing that’s important with meaningful change, I think people do recognize, but it isn’t just a blanket 10% change on a scale everything. You know, questions are asked in different ways.

Alexander: Not like with a quick approach. 

Rachael: Well, you know, if you have to start somewhere, you know, it creates like 15%. But use it for everything. But I kind of see some if you’re not sure you’ve got to start somewhere, haven’t you? But I think that there’s more recognition now that that’s important. And I think this sort of guidance from the FDA from being a bit more precise, and some a bit clearer that those individuals thresholds and really key will help.

Alexander: That’s good. And the other thing is it talks about Technologies. So when I hear the word Technologies for me, you know, that looks like we are talking about variables or we are talking about apps or what’s these kinds of things? Is that what is in mind? Because technology doesn’t look like you. You kind of fill out a questionnaire on paper. 

Rachael: Yeah, a little bit strange to be grouped, I found in this guidance document and it didn’t have the draft, didn’t have a whole lot on this sort of aspect that it was thinking about things like the impact of the variables to collect some of that information or is it that it’s, you know, written down, questionnaire data, and things like that. 

So there’s a small section on reflecting, maybe some aspects of different sets of Technologies do you use and even by technologies that sort of seems to have to mean like paper copies for questionnaire versus electronic PROs and those sort of differences how you ask those, how you ask the questions can be important to understand the types of response you’re getting and it’s not really technology. But so, you know, sort of how that study design could impact it, or when you’re collecting those. So I think that’s sort of what it was kind of getting at when it says it’s going to address Technologies. The draft didn’t have an awful lot that was completely clear in that section. So I don’t know how that’s going to pan out in their final documents and whether they are really going to detail a lot more about technology because technology has changed quite quickly.

Alexander: Yes. 

Rachael: You say, it feels a challenge in a guidance document that might sit around for 10 

years. So I think it might just be a bit more of an outline of those sorts of considerations of the sorts of Technologies and what concerns they might have on how that impacts their decision making. Because, you know, how robust or validated those are not just the concepts that have been asked about. 

Alexander: Yeah. The next point I would like to talk about basically combines two hot topics in statistics so that is really a super hot topic. So to say it’s a hot topic of PROs and the hot topic of Estimands. So when we think about these two things that go together, what’s the FDA’s point of view on that, or what is the kind of draft guidance, kind of going into? 

Rachael: Yeah. I think that the draft guidance did sort of have quite a big section in a way it was a range was really track taking the estimand framework and splitting it and thinking about population and really, particularly a lot about endpoints. And then, it links very obviously in my mind, sort of, because it’s about what is the research question and fundamentally that’s what the FDA guidance is about, is setting a lot clearer research questions. So that as a decision-maker, regulate decision-maker, you can then know what question has been asked, and if it’s been evaluated in a suitable way. 

So the draft guidance document is pretty much based around the estimand and topics in terms of the way it was organized. There were quite a lot of things in different sections, and of course, I think the estimands are using the estimand framework a lot. It can sometimes feel like a little bit almost circular because it is something if a treatment, or is it an into a current, or is it part of an into current events strategy, you know, if you’re talking about say comments, you know, is that the treatment allows you to have comments or not, or is it something that is then taking those comments that, you know, it was sort of effectively disallowed. And I think things like that are really kind of key with a PRO type endpoint, you know, where you could really see that taking, and the pain medication would potentially really influence when you are asking how somebody’s pain was so. It’s maybe more than some of the other endpoints, you really need to just think about it, hang on there. They’re allowed to take paracetamol in a study. That’s no problem, but, is that balance between the arms, is it going to create a buyer in your treatment effect when you’re looking at a particular PRO I think that’s where it comes from. 

So yeah, I think that the FDA definitely is using the estimand framework for framing this. And, you know, they had some examples in the draft guidance, but it was also before the estimands were finalized as well. So I think the section might just be changing in languages, everybody has been developing it in the last couple of years and some of the examples, maybe I think might get amended or altered or made more relevant in the final guidance 

Alexander: So it really talks about treatment population endpoint and these kinds of things.

Rachael: Yup.

Alexander: Okay.

Rachael: But not so much the treatment in the current heading. 

Alexander: Before that.

Rachael: And that certainly says a lot about the endpoint. 

Alexander: Yeah. Let’s talk a little bit more heavily about one aspect of it, that is how we actually then analyze the data. There are a couple of let’s say analyses that maybe are especially suitable for PROs and let’s kind of go through these a little bit. So first that is mentioned is Landmark analysis. So for those not familiar with this term, what does that actually mean?

Rachael: Yes, it’s interesting in the draft guidance, that was almost mentioned first or interesting to me because I don’t know how applicable it is in every setting. But I mean generally in terms of looking at the data up to a certain fixed time point. So I think I’m used to using Landmark analysis more in a kind of survival oncology setting whereas they’re kind of using the same language to say, look, they want to look up to a certain time point. So the FDA quite uses an example, of a sort of up to 12 weeks. I don’t believe that they, you know, 12 weeks as any sort of global standard. That’s the only time we can look at your own data, but it’s always good to have an example. So they’re kind of saying that it’s important to specify the time point up to, which you’re going to look at your analysis

Alexander: Yes, okay.

Rachael: So whether that is them for survival type analysis, where we usually typically think of a landmark or, but more generally they’re using it to say, looking up to a certain time point, which then could be applied with different, you know, whatever sort of method you’re looking up at that point. So I think that’s all guidance will stay that just being a little bit clearer for a regulatory endpoint like just being clear about what time frame you’re looking at. It is useful for everybody and makes it a lot more comparable perhaps to other treatments and things like that. 

Alexander: Okay. The second part is analyzing Ordinal Data, which I love personally because I worked on ordinal data, already when I worked on my Master’s Thesis which at that time it was called Diploma Thesis. Ordinal Data is, of course, something that you very easily get this kind of patient-reported outcomes, how do you feel on a scale from 1 to 6 with one being, you know, super happy and six being kind of want to kill yourself. So, where are you? And then, of course, this kind of different label, 1, 2, 3, 4, 5, and 6 in a way, where 1 is better than 2. And 2 is better than 3, but the difference between 1 and 2 is not comparable, really different between 2 and 3. And what do they say about that in the guidance?

Rachael: Yeah. I think it’s exactly that the FDA is pointing out that many of these domains are Ordinal and the model should be ordinal. We shouldn’t necessarily just be assuming they’re reasonably normally distributed, you know continuous. So I think it’s more thinking about, should we be presenting a little bit more descriptively in terms of percentiles or more sort of stacked bar chart rather than min values? And I think it’s about understanding the PRO arrangement that you’ve used. So some can have quite a range of scale. Although a single question might have just been scored on, you know, 1 to 5, 1 to 7 scale and quite often some of the things we report, for example, pain or even just the global health status, you know, it’s a couple of questions that have been used and so the actual range of the values possible and for a patient, it isn’t just 3 or 5, you know, actually, it can range because it’s usually a sort of multiple, or addition. 

Alexander: Some of the items, you know, if you have 10 items each range from 0 to 10, then you probably have it’s called from 0 to 100. 

Rachael: Yeah, and I think that it’s maybe understanding from any given sort of instrument thing that is used to the area. Is it just one kind of question? And not that’s usually that common in a purpose questionnaire, that’s used in this way. They are usually a little bit more detailed. So you can use a continuous sort of endpoint analysis. I think the point in the document is where it’s not though. And for it, one example might be in the RTC which is commonly used in cancer,  is using things like the GP5 and which is just sort of how impacted are you on the side effect of treatments and you know, that is just one question and we did to be looking at that. No, I’m not impacted. Yes, I am a lot, but lots of other things like pain, for example, can be a bit more continuous.

Being clear, which you’ve got, I think is what they’re really saying, do an appropriate analysis for the data you actually have, not just assume it’s all the same. 

Alexander: It’s the same also if you think about lots of the endpoints, they look into the covid studies like this OSCI I think stands for where you kind of have to go from being dead to being kind of completely free of symptoms and discharge from hospital. And so that’s a typical ordinal scale. And where’s this big difference between dying or you’re being discharged from a hospital. 

Rachael: Yeah. 

Alexander: So yes, by the way, if you’re really interested in learning more about the analysis that takes these kinds of ordinal structures into account. I highly recommend getting back to one of the most downloaded episodes with Konietschke about non-parametric analysis, because that is an approach that really just takes into account the altering of the outcomes. And not just kind of assumes some kind of other metrics on it. 

Okay, we already talked a little bit about the time for event analysis or the Landmark analysis, which I think is a pretty clear thing to do. But what would be the typical events in PROs we would look into? Because it’s not mortality or something like this. 

Rachael: Yeah. So, I mean, I think we did tackle it, so what we talked about earlier is that we would usually think about a decline in a score, a decline in quality of life, or an increase in symptoms that are meaningful to the patient. So it’s a within-patient change threshold. So you would be looking at saying there, the patients had an increase in symptoms that meets that threshold. So that time, until it was that, that was very quick, was that delayed and, maybe more commonly, you know, really of more General sort of physical functioning or quality of life, that might be something that you’re really interested in to look for, that might be someone with a disease that you might be thinking. Oh, my physical function is okay. It’s maintained, but it’s going to decline as it as a treatment effect declines. So you might be thinking that but equally, time to the event is also for improvement. Yes, how do you know when maybe there’s tolerability early on with the study but actually, then once you’ve been taking it for a while, your symptoms overall are improving and staying maintained, at that improved level, so time for improvement is equal. 

So you need that threshold of change of individual trains that is key, in a deteriorating setting death sometimes also would be, you know, regarded as an event depending maybe on the diseased area or stage of disease as well. So I think the FDA in a document is also depending on where you are, consider death as an event as well. 

So the other thing that is sort of mentioned as well is the censoring roles in a time to event in our system, you know, a very important crosshair in any analysis and it should be made much clearer. There’s always concern about missing data sometimes with PRO data, and particularly perhaps sort of variations is a change you want to be a definitive change unless you got worse, stayed worse for at least example, two visits and depending on how frequently you’re measuring your PRO instrument and how long you’re collecting post, any other things that happen post-treatment discontinuation or anything, you know, that might influence the understanding of how you apply censoring in your study as well. There’s a new answer to those definitions of the event, but I think the point really in the FDA guidance has been clear what the event is and what your censoring rules are sounds straightforward.

Alexander: I think one of the key things for me in these considerations is really kind of how dense or how vast you collect the data. Yeah, if you collected only a baseline at the endpoint, well, there’s not a lot of time to study. Yeah, and if you collected, maybe only every four weeks. Yeah, but most of the differentiation happens within the first four weeks. Yeah, then you will not be able to see anything. So this kind of thing really plays a role. 

Rachael: I think it’s being clear, depending on your disease area, are you expecting to see a decline in quality of life in that disease area in that you’re measuring in that sort of time frame. Now if it is a sort of injection treatment, you know, is that really something that you’re really interested in, time to something happening. So I think it is used, but I think it’s about the time to event, the thing with your data, what message does it give? What question do you really ask? 

Alexander: Yeah. The next question is about Respondent Analysis. I think that there have probably been piles of paper on Respondent Analysis and lots of arguments about it. Oh, why should you take your nice data when you have all the data that is throwing away information? Why is that here? 

Rachael: I think it is because it has been very commonly used in PRO data and because for an individual patient, it’s useful to sort of thing, did they change goal? Was it meaningful to them? So they are responders. But as I say that that’s all very relies on the one you’ve picked that one responded threshold and then, you know, hopefully, research that and you’ve got that but you’ve already mentioned as we’ve talked, you know, that could be different at one point change at either end of the scale might not really be the same thing. So I think that there’s that aspect and in the guidance. 

They are basically saying in general, they don’t really recommend a responder analysis per se because of that kind of concern of that collected, richness of the data, but I think that what they do want to know, is if you’re then using this more continuous data that we do analysis that shows that the changes that we are seeing after relating to the ranges that are meaningful. So it is a funny statement in the document, they kind of talked about it and said, you know, it could be appropriate, but we don’t really recommend it. They also have a section saying, it’s important to think about individual change thresholds. So, I think it is that getting those together, they aren’t really recommended as primary but they are interested in understanding that changes are meaningful. 

Alexander: Yeah, and that goes to the last part that is mentioned, is the percent change from the baseline because that takes some power into account that if you’re starting with fewer symptoms and kind of immediately have a smaller change, smaller absolute change to have something meaningful. Whereas if you have stopped, there are a lot of symptoms in the beginning and have some change. So some improvement and I’m just thinking about improvements in the area. So what’s your take on the percent change analysis? 

Rachael: Yeah. I think the guide really sort of suggests that they’re not keen on it either, extreme caution should be exercised if you’re going to use percent change from baseline analysis which kind of possibly puts you off the idea of that really good idea. But I mean that they’re highlighting this, you know, all this concern you’re saying in the floor and the ceiling, you know effects then becoming more a problem because you can’t improve over 100% So yeah, they’re kind of suggesting that maybe not a great thing to do, and that’s my take. I think a change from baseline is fine and commonly used, there isn’t a section in the guidance documents that have been more positive on other approaches. So I think this section was more in the draft guidance. It is highlighting some of the pitfalls, particularly for PRO types data and why, and it’s important to appreciate those in the PRO analysis.

Alexander: Okay, let’s go to the last part and that is about additional considerations, for me that sounds like all the other things that we didn’t know exactly where it fits. These are mentioned here. That’s how I would set it up. So what kind of specific areas are mentioned there? 

Rachael: Yeah, I mean maybe when the draft document becomes more final that they all find a home because we’ve touched on some of those topics already. But they actually really do highlight things, a couple of things we haven’t talked about are that they really are suggesting that for COA to be really considered for a label claim. It needs to be clearly defined and also included in your sort of endpoint testing strategy, and hierarchy. So that concept was touched on elsewhere here. But there’s not any real lots of detail per se in the draft-up document, other than making it clear that we should be able to be defining a PRO endpoint in such a way that it can be part of that statistical to a hierarchy testing. Although of course, we’ve touched on the fact that it’s got to not just be a statistical treatment effect that we’re looking for. 

It’s got to be a meaningful difference is, you know, maybe more challenging to add to a sort of statistical hierarchy testing, but I think they’re just pulling out, you know, don’t just sort of expect PRO to put in and not do something. That’s a robust test effect. 

Alexander: Okay, it sounds pretty obvious to me.

Rachael: Exactly. I’m sure that it might come out earlier in the document because it’s kind of you need to do that. And how do you get it in the endpoint strategy being clearer? I think that’s what all say they suggest in this section of the document that it should be clear and know what the objective is. 

Alexander: And the other point that also is clear and protocol and SAP, for me that’s also pretty kind of straightforward thing. Is there some kind of history that this hasn’t always been the case? How have PROs been analyzed?

Rachael: Yeah, exactly. I think it differs depending on the therapy area. So we’re essentially a PROs has been kind of a key part the primary endpoint has been done a lot better. I think, you know, it’s not more widely and has lots of labels, because it’s been a secondary endpoint. We’ll just look at the quality of life or in the quality of life and symptoms and I think what I mean is the estimand framework really helps. It is just being like, that’s not good enough. I don’t know what you mean by. I’m going to look at the quality of life and all the things we’ve just spoken about think that you know, there are those complexities and the sort of tease and it’s not just, oh, just going to summarize and then I just across everything every time point and then hope that somebody can conclude something from that. 

You actually do want us to be specific about your time point, which endpoint is it physical functioning? For example, that’s you know, really going to support your label claim and other aspects that you’ve collected in the PRO more supplementary and support that. But for a label, they can’t really be expected to think we’re going to cover every single aspect. 

Alexander: Okay, very good. One last thing, there is currently a tick list in this draft guidance. Can you speak a little bit about this one? 

Rachael: It’s kind of a list of when you’re planning a study, confirm lots of items. So I think it’s quite reasonable. It’s quite nice to have that and in there it kind of touches on most of the things we’ve put before. Being very clear and point suggests making it clear that what you’re using is meaningful, collecting meaningful outcomes, and that it’s fit for purpose, and that’s for validation. Your study design is adequately equipped for collecting, you know, whatever it is set out to achieve if you’re looking at an immediate effect or an impact of adverse events that are going to happen quite early on in your study. Collect the data, the time, the frequency, and the timing of collecting those. I mean, they kind of seem obvious, some of these seem obvious points, but they come together, that’s what makes its guidance documents and everybody can see that you need all these factors. And I wouldn’t say some of these are clearly not new that they are what happened, but I think that I guess in putting this together in their draft guidance document, they’re pulling things out a little clearer that maybe wasn’t as clear for every sort of regulation in terms of study. 

They’re clear that the instance needs to be clear, how they’re going to be administered, you know, and that people have been trained or the instructions for them are clear. And that everybody is going to be able to complete them. They should be together with it being a well-known instrument. There are tons of how you’re going to score the questionnaires, how they as I mentioned before sometimes the questions are combined. So, that’s going to happen in line with how that’s all been going to be done. And I think it also mentions being very clear on as we said about, putting endpoints in the endpoint hierarchy testing and any sort of plans for multiplicity adjustment if needed. I think that might reflect on what types of points are relevant to your study, what you’re labeling, terms of what should be tested, and how to make clear plans on handling missing data. 

Alexander: Pretty straightforward. 

Rachael: Yeah, I think this is why it’s in there, but that’s the sort of thing that hasn’t been clear as it ought to be, you know, the estimand framework perhaps will help identify current events and separate out them from just missing data aspects. So once you know what your question is, that’s a lot easier to write down, what you’re expecting to be there. And I think this is one of the big blockers from a regulatory perspective. When I’ve been involved in some of the debates, it still seems to be the biggest concern missing data handling. Oh, we’re not sure about using missing data because of PROs because it has a lot of missing data, and I don’t know if most of the studies that I’ve worked on in a clinical study setting. We tried very hard to enable patients to be able to complete everything we’ve asked them to do and it doesn’t seem as much as maybe as a concern as it used to be.

And in terms of missing that information, it’s still massive. Yeah, sounds obvious, but it’s a really big thing but still that we are really clear about what’s missing. Because, unlike any other endpoint, you can’t just go and collect it afterward. They didn’t get collected if the patient was too ill to complete their patient-reported outcomes, and that’s important to know. It’s not just that it was missing. So there are nuances around that. 

And yeah, the other things in the tick list of things like, you know, just being clear of how you’re going to portray those treatment effects between great differences and really how you’re going to collect and store that data. And from there, you say there’s a tick list, I hope it will become a little bit more of an organized tick list in a later document that I think, you know, making sure you’ve covered all these sorts of things. All these Concepts even if they’re in a different order in my view. It’s really useful. 

Alexander: Yep. That’s good. And it says it was actually a really nice summary of what we all discussed about. There’s one last thing that I wanted to talk with you about. And I notice that this is already a pretty long podcast episode. But there’s a new face-to-face event, coming up at the PSI conference in Gothenburg. We have actually met at PSI conferences in the past and I think we’re both really big fans of PSI in general. So, in terms of the PSI conference for you personally, what are the best things about it? 

Rachael: Ah, yeah. Well, I’m really looking forward to going in person, I really hope in Sweden this year because it’s an interaction with other people, which I think is what I’m most looking forward to. Maybe I came into PSI a little later in my own career than someone who has been there a long time, but everybody’s very welcoming and keen to meet new people. 

Honestly, I’ve been on the scientific committee and part of the organizer conference for a number of years now, so it’s really good to always meet the speakers. The conversations that you’ve been planning or reviewing abstracts for and you get to go and listen to the talks, which is much better than, you know, trying to just gleam things from abstracts and understanding the details, so yeah, I really like that combination of meeting people, and listening to talks and then being able to discuss some of these topics as well. And the variety of topics is just really nice. 

It’s an opportunity to go and to listen to something and listen to great presenters on a topic that maybe is outside your day-to-day work. And I feel you can always learn from how other people present stuff. And how, how they explain things. It’s a really good learning opportunity, and lots of levels and, and great fun. 

Alexander: Yeah, you know what? My biggest pain was that a conference always has so much great content that I always have pain choosing which session I go to. Yeah, if you look at these parallel sessions, I think like I want to go to all four of them at the same time, but well, unfortunately, I don’t have these magic things like the golden Harry Potter. What’s her name?

Rachael: Hermione. yeah. 

Alexander: That would be really good to say, but I also completely agree that meeting different people. If you’ve been there a couple of times, it feels a little bit like meeting friends from school again and talking to each other. So I highly recommend going there and arriving on Sunday because Sunday evening is already really nice to get together.

Rachael: Yeah, and I do know, just to say apparently it is a really really nice, sweet City in Gothenburg, this year. So everybody has visited the scientific cause because it’s a really good city. So it’s a really nice place to go. 

Alexander: Yeah, awesome. So, this is your first business trip after the pandemic? Maybe a really good choice. Thanks so much, Rachael, for this awesome discussion about the FDA, PROS, and at the end of the conference, and I’m really, really looking forward to meeting you in Gothenburg, and all the best until then. 

Rachael: Thank you very much. 

Alexander: This show was created in association with PSI, don’t forget to register for the Gothenburg PSI conference. Thanks to Reine who helps the show in the background, and thank you for listening. Reach your potential, lead great science, and serve patients. Just Be an Effective Statistician.