In this episode, I sit down with Rachael Lawrance to dive into a topic that has become absolutely central to clinical research: patient-centered outcomes.
When I first started as a statistician, I knew these measures existed—but I didn’t really understand how they were developed, analyzed, or used in decision-making. That has changed dramatically over the years. Today, patient-reported outcomes (PROs) and the broader framework of clinical outcome assessments (COAs) play a key role in regulatory approvals, payer decisions, and how we demonstrate treatment value.
Rachael brings deep expertise from her work at Adelphi Values and shares how these endpoints are developed, validated, and applied in practice. We also discuss how statisticians can contribute more effectively by understanding the science behind these measures—not just treating them as “just another scale.”
Why you should listen
Listen to this episode to:
- Understand why patient perspectives are now central—not optional
- Learn the difference between PROs and COAs, and when each matters
- See how questionnaires are scientifically developed and validated
- Improve how you select, analyze, and interpret endpoints
- Avoid common pitfalls when using patient-reported data
Episode highlights with timestamps
- 00:00 – Introduction and background to the episode
- 01:31 – Rachael’s journey into patient-centered outcomes
- 03:24 – Why PROs have become central to drug development
- 05:18 – What COAs are and how they extend beyond PROs
- 05:46 – The four types of COAs explained
- 08:58 – Generic vs disease-specific questionnaires
- 13:16 – How PRO instruments are developed and validated
- 15:06 – What “meaningful change” really means
- 17:02 – Using Phase II data to inform Phase III endpoints
- 18:24 – FDA expectations and guidance on PROs
- 21:47 – Using the estimand framework for patient-centered endpoints
- 23:15 – What “psychometrically sound” actually means
- 31:16 – How I would choose the right PRO or COA for a trial
- 34:01 – Avoiding missing data and reducing patient burden
- 37:18 – Key takeaways on interpreting patient-reported data
Links and Resources:
🔗 FDA Patient-Focused Drug Development (PFDD) Guidance Series
🔗 Learn more about PSI and become a member
🔗 The Effective Statistician Academy – I offer free and premium resources to help you become a more effective statistician.
🔗 My New Book: How to Be an Effective Statistician – Volume 1 – It’s packed with insights to help statisticians, data scientists, and quantitative professionals excel as leaders, collaborators, and change-makers in healthcare and medicine.
Join the Conversation:
Did you find this episode helpful? Share it with your colleagues and let me know your thoughts! Connect with me on LinkedIn and be part of the discussion.
Subscribe & Stay Updated:
Never miss an episode! Subscribe to The Effective Statistician on your favorite podcast platform and continue growing your influence as a statistician.
Never miss an episode!
Join thousends of your peers and subscribe to get our latest updates by email!
Get the





Learn on demand
Click on the button to see our Teachble Inc. cources.
Featured courses
Click on the button to see our Teachble Inc. cources.

Rachael Lawrance
Senior Director & Functional Lead Statistics, Patient Centered Outcomes (PCO), Adelphi Values
Rachael is a statistician specializing in patient-reported outcomes (PROs) and clinical outcome assessments (COAs). She works in the Patient-Centered Outcomes team at Adelphi Values, where she focuses on the development, validation, analysis, and interpretation of patient-centered measures. With previous experience at AstraZeneca, Rachael combines her pharmaceutical statistics background with expertise in psychometrics to help clinical researchers generate meaningful evidence that reflects patients’ experiences and supports better treatment decisions.
Transcript
00:00
You are listening to the Effective Statistician Podcast, the weekly podcast with Alexander Schacht and Benjamin Piske designed to help you reach your potential, great science and serve patients while having a great work-life balance.
00:21
In addition to our premium courses on the Effective Statistician Academy, we also have lots of free resources for you across all kind of different topics within that academy. Head over to theeffectivestatistician.com and find the academy and much more for you to become an effective statistician.
00:49
I’m producing this podcast in association with PSI, a community dedicated to leading and promoting user statistics within the healthcare industry for the benefit of patients. Join PSI today to further develop your statistical capabilities with access to the ever-growing video-on-demand content library, free registration to all PSI webinars, and much, more.
01:13
over to the PSI website at psireb.org to learn more about PSI activities and become a PSI member too.
01:31
Welcome to another episode of the Effective Data Station. Today I’m talking with Rachel Lawrence. And Rachel has been on this show before, something like four or five years ago. We have both worked together on PSI for quite some time. And that’s how we know each other already for quite a long time. I have met at lots of different conferences. And Rachel works in an area that have…
01:58
find really fascinating and said when I started to work as a statistician, I had honestly no clue about. So I knew that these kinds of things exist, but I had no clue how you get to them, how you analyze them, what’s important for them, what are different guidances. And these are patient-reported outcomes. Or there’s another acronym to which we will come later. So Rachel, welcome to the podcast.
02:27
Thank you very much. Great to be back. For those who haven’t listened to the first episode, maybe you can quickly introduce yourself. Yeah, certainly. So I work at Adelphi Values in our patient-centered outcomes team. So we’re a specialist consultancy company that focus on everything to do with the PROs, including the development from the interviews and the qualitative side, all the way across to analysis and interpretation.
02:56
and all sorts of different disease areas, everything you can think of really. So yeah, I’ve been with Adelphi for eight years now, really focusing on those patient reported outcomes, but from a statistical point of view. I brought my previous background in pharma, AstraZeneca into this world as a statistician. So I’m really passionate about thinking about things like the estamens being really defined right for those PROM points across the ground and the interface between.
03:24
statistical world in the pharma side of things and all this PRO language and the way we have all these questioner data and how they can be used and interpreted. Let’s talk a little bit about patient reported outcomes, PROs, and what role do they play in clinical research and how has that developed over the last years? Patient reported outcomes used to be a bit more periphery in the drug development.
03:54
side, but they really are much more recognized to be really central. They can be primary endpoints that determine drug approval and labeling, even as secondary endpoints. they’re really recognized by regulators and particularly payers as well as really important to understand that patient viewpoint of the outcome of treatments. And I think over since the early 2000s,
04:21
Those guidance documents from the FDA, EMA, payers have really increased and helped people understand a lot more what the FDA and other regulators are really looking for in terms of putting patient outcomes at the heart of all aspects of the drug development. Yeah, it’s definitely important for payers. If you, for example, look into the European JCA documentations that are heavily influenced by the German HTA.
04:49
documentation, so the eQUIC methods paper, you can find a lot about patient reported outcomes there. And you can demonstrate your added medical benefit or value through these, sometimes much easier than through other clinical endpoints, especially if these are lab results or anything like that. There’s a new term that has developed and this is clinical outcomes.
05:18
assessments. Where is that coming from and why do we need to speak about co-hours now instead of PROs? COA is the umbrella term if you like. And so I think it has always been there, but it’s helpful because then it reflects different types of these ways of measuring the patient’s experience. So that’s four main co-hours if you like. So PRO that we already spoken about. So that’s
05:46
stands for patient reported outcomes. So that would be something a questionnaire of patients sits and fills in. It might be on their phone, of course, on a tablet, not necessarily on paper very often. And it could be measuring something like questionnaires around fatigue in Crohn’s disease, for example. So you’re really directly asking questions for the patient. Then we’ve got one called OBSRO. So Observer Reported Endpoints. So that might be something like a
06:16
count of an event that a caregiver can describe. So the number of seizures that somebody has experienced. So it’s very relevant, but it’s reported by that observation, if you like. We also have clin rows. So they’re endpoints that are reported by the clinicians. An example of that would be something like the PSAI, the PASI index in psoriasis. And that’s
06:40
severity and extent of your psoriasis. So there are quite a lot of few elements to that and they come together and make this confined score that the clinician is scoring about those patients. And finally, the fourth category of a coerbio curve row. So this is a sort of performance-based measure. There’s something like measuring that might directly measure your mobility. They have been all put together to make the terms
07:05
COA and different ones are more applicable in different disease areas, maybe different things in pediatrics. And you might have a combination somewhere, sometimes of both aspects as well. yeah, we could have it. So what is then not a core? There’s also this sort of line in terms of digital endpoints as well and wearables, where do they sit? So they’re also related.
07:31
If you’re using a wearable to measure your steps, it’s something that kind of has come from that data from a patient without additional sort of interpretation by somebody else, but it’s come from this wearable device. So those are wearables and digital health technologies are also included in this sort of framework, I think of things. But the main definition is really something that is not translated by something else. So it’s a fairly direct measure of something that’s applicable to that.
08:00
patient in that way. So for example, a rating of the physician of some kind of PET scan or something like this, that would be definitely not a core. Yeah, something that the patient or sometimes a caregiver is very direct with the patient reports. Yeah. But not exactly your examples, your results of your scan, your test results that, you know, been interpreted, although relevant to you as a patient is not like from your point when you’re a patient, your
08:30
personal, patient or subject point of view. Okay. And then you mentioned, for example, Parsi, which is a specific endpoint for psoriasis. But then we also have these very generic questionnaires like EQ5D on SF36, these type of topics. Also in oncology, we have one set of oncology specific, but not indication specific. Why do we have
08:58
both to the generic questionnaires and the indication specific ones. Yeah, so you first mentioned this more general like things that might measure quality of life as you might think of it, as you said, the SR36, the Q5D, and that might have a very broad question that says today or in the last few days, how would you rate your own quality of life? And that’s a very broad question and it could be answered in lots of different ways. And you can also understand that for any individual,
09:26
Lots of things could influence that going on. The idea of also having more indication or disease specific questionnaires, and they show that kind of can able to give that detail about what really matters to that disease. So it’s useful to have both types in a lot of cases, depending on the questions you’re ultimately trying to answer. You might want these broad questionnaires if you’re
09:55
thinking of comparisons more broadly outside of your disease area. For example, EQ5D is used a lot in the utility and the economic modeling side of work. So it’s still taking values from patients about what that quality of life they experience with that treatment is. But because it’s a questionnaire that can be asked across all sorts of disease areas, it provides a measure that’s a lot more comparable if you’re looking at
10:25
comparisons of costings, for example. Whereas the disease specific ones really matter for that disease. So yeah, for complete evidence, they both are complementary, but if you’re looking more detailed or perhaps regulatory approval, you really do want to be looking at that disease. What are those really symptoms that really matter for that disease and for that kind of really clear concepts that you’re measuring with the questionnaires?
10:55
I think it’s not just the costing where the generic questionnaires come in. It’s also when you have these stories about burden of disease. Let’s think about a disease that is not very common. don’t know, psoriasis or COPD or something like this, but it’s much more rare and where maybe the payers don’t have direct understanding of what that actually is. Hear about it first time when they get your dossier.
11:22
Then it’s really helpful to give an understanding, okay, how severe is this disease compared with other diseases that are known to the different people, the two payers. And therefore from just understanding the overall burden of disease and how much it is reduced is definitely helpful and makes things a little bit more easy to understand for people that don’t deal with such patients all the time. Yeah, certainly. think in rare diseases as well.
11:52
think that as you almost say, there’s a lot of use for potentially a slightly more broader questionnaire rather than it necessary. Lots and lots of development are very targeted questionnaires because it gives you that more different understanding. But every rare disease is very different to have very different important symptoms and things like that. So that’s where that combination of yes, some general questionnaires can really help even in the regulatory setting.
12:19
And if it goes together with those concepts that are really important in that disease area as well. Can that also help to better understand, for example, clinically relevant changes? So I know, for example, when we think about these overall changes, patient global impression and severity or change, and you have this seven point like that scale things that is very often used as a kind of generic question.
12:47
And then you can look into this and basically look at it as an anchor to look into your more specific endpoints to understand what does a five point difference, let’s say on a specific scale, what does that actually mean? Yeah, definitely. And I think even before the sort of meaningfulness of that interpretation point, I think it’s worth appreciating that there’s quite a lot of development to get to a questionnaire that’s even at that point. So there’s.
13:16
You really need to think about what is that conceptual grounding of the questionnaire, first of all, and that most of that might come from qualitative research where interviews, finding out what is important to patients so that you’re really targeting a sort of something that’s a meaningful aspect of health. And then you’re taking that forward to identify what’s a distinct concept that you can be measuring with that questionnaire. So that’s a really fundamental part in this sort of
13:47
Developing a fit for purpose is the terminology, sort of validated but fit for purpose questionnaire at the outset. And I think when you’ve got that kind of initial idea and the concept of interest, part of that is thinking about how the properties of that, how reliable is it? How valid is it? If you ask it one day and you ask the next day when nothing should have changed, what happens? And things like anchors, measures, other questionnaires that you just mentioned.
14:16
It can be really useful with that part of the development of the questionnaire because you can say, I’ve got something that I know hasn’t changed between these time points. What is going on with this new question I want to do? And equally, you can have a global item. So I know patient scores improved on this, what happened with this concept that I’m looking at.
14:37
So yeah, I think that you can use these global items in all sorts of ways as well. And fundamentally getting those validation psychometric side of the analysis first is really important. And then if clinical meaningful piece, as you say, the next pillar, if you like, developing that good PRA measure, because absolutely you’ve got to be able to interpret it does a change of three points, five points, 10 points mean something useful to those patients? So
15:06
Yeah, combining it to make sure that you’ve got a really clear questionnaire first that’s robust and validated and then can be used in the right context. If you’re, sometimes we might have a questionnaire that’s been used in adults, but if you’re going to use it in a pediatric population, can it be, does it have to be modified to be asked in questions, have to be asked in different way? Something like a really basic sort of example, but it’s still got to be fit for purpose in that patient population, if you like.
15:36
I know, for example, in oncology, it’s not all the same, right? Yes, you might use a kind of fairly standard questionnaire. ERT-CEC-30 is a sort of generic across all cancers measure, but within disease areas, you might want to be making sure that if in non-small cell, how much changes are we expecting to see what’s meaningful to patients in physical functioning? Is that the same in breast cancer as it is in pancreatic cancer patient populations, for example?
16:06
So that’s why we quite often end up seeing that we want to do some of that additional validation and think about meaningful change really targeted in that population. And that’s what the FDA want to see as well. That’s what the regulators want to see, because they want to see that really robust data about how these PROs are able to be used and able to be interpreted in that way. So you’ve got a really solid PRO endpoint at the end that’s really based and really grounded in that area that you want to use.
16:35
So even if you have, let’s say, took PRL for your study and you say, excuse, it was accepted by regulators, that doesn’t mean your work is done in that regard. You basically need to do more research specifically to show that it’s robust, that the properties remain the same, that you, and how far you can transfer from other areas. That sounds pretty interesting.
17:02
If you would have, let’s say a phase two study, would that give you enough data to inform everything for a phase three study or would you usually need some kind of specific studies for that? Yeah, ideally using data from a phase two study is brilliant because you can use that, you use that to see if those ranges of meaningfulness seem to be appropriate in that study population. And you can use that to inform phase threes, I think.
17:30
depending on the disease area or the time of when you’re thinking about this or your program design, you know, it might be, you don’t necessarily have to do this in every study. If there’s similarities, it is quite consistencies that if you’re thinking of really primary endpoints for regulatory, the FDA really do want to see the evidence that’s behind them. Some they, the FDA do have some
17:57
PRO instruments that they call have been through the FDA validation process. One recently is in non-small cell lung cancer, they call the NS-CLC-SAQ. It did take a long time to go through this validation process from the FDA. So they don’t expect every PRO you use in any one study to have been like, oh yes, I’ve validated this. You could just go ahead and use it absolutely everywhere. There’s a pragmatism in that, but.
18:24
As you say, yes, I think it does vary with the disease area. ranking of, you if your PRO is being used as part of your primary endpoint, then it really needs to be sure that you can interpret that and it’s right for the population you’re using it in. But the FDA have also got a lot of guidance on this. The FDA has got this PFDD guidance series with four guidance documents that they started a number of years ago now.
18:54
a few years ago because I must have talked to them about them last time. finally they’ve got guidance one and two and three are all finalized. So that’s really useful. That’s a lot of the background. Guidance four is probably the most interesting to statisticians because it is really focused on that’s more final hurdle, that more end point part rather than the development. It’s still sitting there as draft, but it
19:21
half of that whole document is about interpretability or the meaningfulness of, of your PRO scores and the sorts of analysis that they like to see for that. That is definitely a really good go-to document and really recognize that having these thresholds of meaningfulness, whether that’s between within a patient for an individual, how much they change, whether that’s how much a mean changes over time to be meaningful.
19:51
or whether between two treatment arms, mean, difference between groups is meaningful. So there’s lots of different nuances to that meaningfulness and that interpretation piece. But yeah, being able to provide evidence that support those different aspects is really key and makes sense. Yeah, because it’s also this meaningfulness is, I think, associated with different decisions. Yeah. So let’s say you look.
20:20
into an individual change from baseline after let’s say two weeks. Yeah. And it’s, it’s for longer term treatment. The decision might just be, do I continue or do I stop? And so I just want to see whether there’s any response. And so you want to see whether that is big enough. Then when you have a placebo comparison, yeah, of course, that does this benefit outweigh the side effects.
20:50
There’s another question. And if you have your active comparator, then I would say, is one treatment really better than the other? Or are they pretty much the same? Would they feel the same for the patient? Which are all very different questions from my point of view. so the answers should be very different. And also, by the way, I also mixed up here kind of group differences and individual differences, that’s yet another topic. Yeah. And it just gets a bit…
21:19
complicated when you start laying it on like that. And I think that’s why you have to really be clear about where you’re going with what your Piro based endpoint is. So, you know, what is important, what timeframe is important. And I know for me as a statistician that can use that nice estimate framework now and the variable to name this, but the variable, it’s not just that you want to look at your change or improvement in a score.
21:47
You need to know what timeframe that’s relevant in and that’s got to be something that’s relevant to the patient in that population and that disease area. So yeah, for me, the Estimand Framework there gives you some language to help you with defining that end point part. So you might know that your meaningful aspect of health, your concept of interest from the patient interview really early on, you might know that you’re wanting to look at fatigue and then you even might then have done your work and know that
22:16
fatigue and we’re going to use it with this particular PRO because that’s relevant in my really particular patient population and it’s been used extensively. So we’ve got a lot of evidence that it’s psychometrically very sound and validated if you like. And then that final piece in terms of them, you might then know what that change in score and how to interpret it, but you still need that time point as well, that time frame of interest to make it all the way to what we would know as a statistician in terms of those sort of aspects to
22:46
be able to think about how we then do the best analysis and help with that interpretation. You just mentioned a term that not everybody is used to, psychometrically sound. What does that actually mean? What is psychometrics? I would have to say that until I started working at Adelphi here, I hadn’t really come across this concept either.
23:15
And what does it mean? exactly. What does it mean? But I think it was things I was talking about before earlier in terms of to do with those properties of a questionnaire. So not thinking about a treatment effect in the way that a clinical trials statistician might come into this, but thinking about how we take those concepts that are out there that may or may not overlap. How do we ask different questions to come together?
23:42
and make this domain, how do we work out how we should score them? How do we work out that we might ask 10 different questions, but some of them are redundant or they overlap a bit. want to have really, you only need to ask five questions because they cover, they mean different things reliably to different people. we might do psychometrists, might do things like a sort of factor analysis approach in your data to make sure that
24:10
You’re really getting this right kind of questionnaire. And then you might be thinking about how should that be scored? Should it be a total score across all these things? Should it be an average score? And what are the, what are the properties around that given the concepts that were there? Can you have a total score that adds up lots of really distinct concepts? You’re combining physical functioning with something that’s a symptom, right? You can’t really.
24:38
add them together and get a useful score because no one’s going to be able to interpret that, right? But sometimes things are more linked together and you can cluster them and you can say, okay, there are seven really important disease symptoms here and then we can make a meaningful total score. Or you can say there are six things and there’s another question that’s important to patients and that might be separate. So I view it like that, everything around that kind of properties of the questionnaire, we sort of term psychometrics.
25:07
aspect and some of it, as we said, is that borderline, how do you interpret it, what’s meaningful and how do we assess that? So we might still use statistical technique, what we might think of to do that, but we’re not comparing potentially treatment arms in that way. And when we do a psychometric analysis, say we mentioned before in a phase two data, it’s always a blinded analysis. We don’t use
25:32
the treatment that anybody’s been assigned to, we’re just saying, okay, these patients should an improvement on this anchor. These ones declined. How do we evaluate meaningfulness? So yeah, that’s another way of looking at it. And there’s a lot of, think that’s it. One of the things to say, really appreciate there’s actually quite a lot of science behind getting to an endpoint that we just see as a naught to 10 in a clinical trial.
25:59
SAP, right? We’re like, oh great, it’s 0 to 10, how tricky is that? But it’s really been bought through and it really comes together. You might get things like the fact questionnaire and it, you know, how does this end up being scored from one, not 132, but it’s got all these different components, but it’s got these components because they’re really important on their own to make up this, know, fat G total score, for example. So key aspects there is looking, for example, from a statistical point of view and the
26:29
whole correlation or covariance metrics of all the different items or questions that you’re using to better understand how they relate to each other and the factor analysis or PCA principle component analysis would look more into understanding how these go together, there subscores? Are they going actually in the same direction always? Yes, if the patient improves, do they all improve or do they go in the other direction or don’t they change at all?
26:58
As you mentioned, the redundancy, another aspect is of course, are they stable over time? If there’s no change, then there shouldn’t be so much variability because then you might interpret changes being as real when in fact there’s not. They need to be somehow stable. If you have something from the physician point of view, and so not the patient itself, but another rater is that more or stable across different raters.
27:26
this inter-rater variability is another topic. Are they ceiling effects or flow effects? So, it items always or nearly always is a top category or the lowest category, then they don’t carry any variability and therefore no information. So, these are all kinds of different things that you need to look into. And I find it really fascinating how you come up with these set of questions at the beginning. And also not just a set of questions, but also the anchors.
27:56
you have within these questions, from never to all the time and there’s so many different anchors, it’s really fascinating. And in terms of when you talked about concepts, for me this is kind of, I want to give an example. I worked a lot in ADHD and there were usually three concepts, inattentiveness, hyperactivity and impulsivity.
28:22
And inattentiveness was largely uncorrelated with the other two, but the other two usually go together because they’re highly correlated with each other. And so that’s why I have a usually a total score and then you have two subscores. If you want, you can break one even further down, but hardly anybody does that. But this is when we talk about, about concepts and one concept is usually represented by various questions or items. Yep.
28:52
Exactly right. Yeah. And I think that’s all part of this term knowledge, co-work development. And you really summarized all those sort of steps that go together and psychometrics is the broader term for those sort of type of quantitative analysis. But we often potentially do sort of mixed methods or mixed studies where you’re using the patient’s interviews to say qualitative, what did they see as a meaningful change? they see?
29:19
given this score is this many points, what would they, what does that mean to them? And that can help go together with what you can see on a kind of quantitative level as well. They’re very different. get different, you can get different answers from these approaches, but you can put these, stack them together and it gives you a really rich picture in terms of what’s going on and what’s behind that. And sometimes in like, if you’ve got using it as a primary or secondary endpoint, a COA,
29:49
It might even be that even though you’ve got a well-developed co-oral period instrument, you might also have some qualitative interviews included in your clinical trial to give a little bit more richness, if you like, to the patient experience in that trial and not just a number of what a mean change happened in 200 patients. You can have these small numbers of interviews that really flesh out what fatigue means to them and what a change.
30:19
means to them in terms of their life and what they’re able to do in the morning and have energy and get out. And it might vary for different people and massively varies in the disease area, for example, as well. But one person being able to go for a run, being able to see their grandkids, being able to get out of bed without pain, all sorts of different concepts can come together and not come together, can be measured. And then you can really use those qualitative side to support that more rigorous numerical.
30:48
end point that we see and we have to see in an evaluation of treatment effect, right? We want to see that in our clinical trial designs and understand that as well. Okay, very good. Then the last question, which is really important for all our project or trial statisticians. How do I choose the right course or PROs for my clinical trial? I don’t think there’s a
31:16
quick one answer, I’m afraid, but you’ve got to start with the patients and the clinicians. Most disease areas have some precedent. Really what is a meaning, what’s meaningful, what’s really key in aspect of the health. There’s the literature, there is what everybody asks us out there. You need to find something that’s fit for purpose. So as I mentioned before, this sort of concept of validated, pure is not quite strictly true.
31:44
But yeah, there is a concept, it’s got to be fit for purpose. It’s got to be reliable to use. It’s got to be consistent, provide you with those sort of results. And you do want to have a pure strategy with regulatory engagement early. know, these are, is what we think we’re going to measure. is why and get that alignment. We mentioned payers as well. We know they like to see that patient.
32:09
sort of endpoint as well. But yeah, regulate your alignment really clear and really document all that evidence in terms of where you’re going. I think different. If you look in any one disease area, there’ll be newer questionnaires that coming along to really establish ones that everybody uses at all time points. Do I need a new one? What’s wrong with the old? I don’t think we always need a new one. That’s not really the game we’re in. We all want to be efficient and that’s perfectly fine. But is it suiting the purpose? And then in terms of thinking more forward.
32:39
know, practical implications. We don’t want a big burden on patients. The FDA and everybody I think, if you’re running a trial and your regulators, they really don’t want us to be asking memes and memes of questions. So there is also that part of, yes, we might want to know about disease specific symptoms and more general quality of life and adverse events from the patient’s viewpoint.
33:03
that getting a little bit of a balance between what we want to ask, how often you need to ask in your clinical trial as well. And so it’s thinking again about what’s that strategic kind of evidence base that you’re going to need to meet and how can you do that in a sort of robust way that’s not burdensome because that’s not great for anybody. Yeah. I’ve worked a lot in psychiatry and says, course it’s very questionnaire heavy and you don’t want to have hundreds of items. Yeah.
33:33
Especially not when you deal with, let’s say, older patients or pediatric patients. The end points that you collect at the end of that kind of is probably not that valid anymore when people are completely exhausted from ticking boxes. Yeah. And I think that often one of the things that really comes up with PROs is that, you know, missing data sometimes is, the biggest concern with PROs. And obviously the best way is to avoid missing data.
34:01
design a schedule of assessments that’s going to work. Don’t ask too many questions, but are focused on the right things. And statistically there are things you can do with missing data when you’re then looking at treatment effect. But really what you want to do is be have something that’s fit for purpose for collecting what you need and really to think about not how many questions you ask the world, also how often you ask it. Do you need a daily diary data really, or could you ask a question just
34:30
once every seven days and would that give you what you need? Maybe it wouldn’t if something’s very fluctuates, right? There’s definitely a place for some sorts of diaries, data collections. They could be really intense, but equally in other areas, no, you could ask for seven days. Maybe you ask something every six weeks or very occasionally and it still gives you a long-term input. So really is quite context dependent and that yeah, strategic answer to how to choose them. But there’s quite a few of those.
34:59
components and yeah, probably the final one on that is yeah, really thinking about that interpretability piece. It’s got to be relevant with what they need, but where you get to the end, is that going to be interpretable to your different audiences? You might have different opinions from your patients, your clinicians, your regulators, your payers. Always a bit of a juggle about what the best approach is there. Completely agree. It needs to fit into your overall evidence strategies that you have. And here,
35:29
It’s not just about your clinical trials. It’s also how do these fit into the bigger picture. And if you are the only one with this new or different questionnaire and all the others have used another questionnaire, you make your life harder because it’s much more difficult for payers, for regulators, for patients to put your data into perspective. I’ve seen that once where new compound used.
35:56
One anxiety questionnaire and all the others used another anxiety questionnaire and it’s really bad from doing any indirect comparisons or any of these kinds of things. Because the difference in the questionnaires is bigger than the difference in the treatment effects. Yeah. The other side isn’t there as well, like in some times things were developed a long time ago, say, for example, total oncology ones are probably in other disease areas where the treatments have moved on, the concerns of patients have moved on somewhat.
36:26
It doesn’t mean a disease has changed, but maybe we’re diagnosing things at different stages. And the questionnaire that was once used is now potentially being used perhaps much earlier and asked about questions that are not relevant. So there is that balance between, you can’t always just use whatever you used in the past. There is a space for advancing sometimes. So yeah, bit of a balance there between the two, but yeah, being really clear with.
36:49
what that concept is and why that’s important, I think is really helpful. And it flow through to everything you then do as a statistician at the end in terms of interpretation. Thanks so much, Rachel. That was an awesome discussion about all kinds of different things, Eros and also Co-Oth. We touched on where they’re coming from, why they have evolved a lot and play now a much more central role than, let’s say, 20 years ago.
37:18
I think it’s really the standard things that we are now using as it was much more controversial 20 years ago. I for younger statisticians, they may not realize how much of a change we have actually seen. And there’s a couple of different reasons why you might want to have both generic and indication specific questionnaires or cores and what a good core actually look likes and what are strategic.
37:47
thoughts you need to have to include it into your clinical trials. Anything that is really important that you want to see listeners take away from this episode? Yeah, thank you. Yeah. Hopefully we’ve got over in our chat really. think it’s that appreciation of that development of a PRO score. It’s quite a long process and it involves a lot of really good research, whether that’s interviews with patients and also as we were talking
38:16
about the kind of properties, exactly how it’s scored. It hasn’t just been, oh, really easy. It’s really thought out, but really measures something that’s meaningful. And from that, you can get to a really good concept and you can get to really good PRM point. And I think just continuing to be able to analyze those in a rigorous manner and being able to interpret them really well and appreciating that there’s lots around that.
38:44
nuances interpretation are complicated, but it’s important to understand the bit of that background and that will really help you really interpret it, really make sure that you’ve got a meaningful endpoint where we can really help use this in these pivotal phase three trials to provide evidence around that treatment benefit decision-making process, I think. So yeah, I think I really appreciate a lot more than I did for I worked in this field that all the work that goes into them and then.
39:15
the things we can really champion and start our sessions to, we now understand some of that background. So that why the score is, what it is and how we can really use that really advantageously. And patients can be more confident that they’ve had a direct view and that’s really accounted for in all sorts of disease areas that as you say, maybe didn’t used to be as common previously. Yeah, hopefully that.
39:44
Thanks so much. Last notice as we are recording this, we are a couple of months away from the PSI conference in 2026, which will take place in Belfast. Rachel, are you attending? Yes. I haven’t even booked a ferry over. I’m ready to travel there. So yeah, I’ve got a poster and my patient report outcomes. Sig, I’ve got a talk. So yeah, excited to be there and see everybody you catch up with.
40:13
All the statisticians are going hear some wonderful talks because, a big fan, of course, of the PSI conference. So we’ll put a link to the PSI conference, also to the PRO CIC and the documents that Rachel mentioned into the show notes so you can find all of them there. And maybe if you come to the PSI conference, if you listen to this later than the PSI conference, 2026, and you come to the next one in 2027, don’t know yet where that is.
40:42
We talk to Rachel and myself will likely hang around there. Thank you a lot. Bye. Thank you, Rachel. Bye.
40:54
This show was created in association with PSI. Thanks to Rain and her team at VVS for help with the show in the background and thank you for listening. Reach your potential, leave great signs and serve the patients. Just be an effective statistician.
Join The Effective Statistician LinkedIn group
This group was set up to help each other to become more effective statisticians. We’ll run challenges in this group, e.g. around writing abstracts for conferences or other projects. I’ll also post into this group further content.
I want to help the community of statisticians, data scientists, programmers and other quantitative scientists to be more influential, innovative, and effective. I believe that as a community we can help our research, our regulatory and payer systems, and ultimately physicians and patients take better decisions based on better evidence.
I work to achieve a future in which everyone can access the right evidence in the right format at the right time to make sound decisions.
When my kids are sick, I want to have good evidence to discuss with the physician about the different therapy choices.
When my mother is sick, I want her to understand the evidence and being able to understand it.
When I get sick, I want to find evidence that I can trust and that helps me to have meaningful discussions with my healthcare professionals.
I want to live in a world, where the media reports correctly about medical evidence and in which society distinguishes between fake evidence and real evidence.
Let’s work together to achieve this.




