Understand and master multiplicity in practical situations

Understand and master multiplicity in practical situations – Interview with Alex Dmitrienko

00:01
You’re listening to episode number 25 of The Effective Statistician. Today we are talking all about multiplicity and as a guest we have a world-class expert, actually one of THE world-class experts in this topic, Alex Dimitrenko. So keep on listening.

00:28
Welcome to another episode of the effective statistician with Alexander Schacht and Benjamin Piske, the weekly podcast for statisticians in the health sector designed to improve your leadership skills, widen your business acumen and enhance your efficiency. In today’s episode we’ll chat with Alex Dimitrenko about multiplicity and especially all the practical aspects around it.

00:55
We’ll of course go a little bit into the theory, but we make it really, really a high level and very, very actionable for you. This podcast is created in association with PSI, a global member organization dedicated to leading and promoting best practice and industry initiatives. Join PSI today to further develop your statistical capabilities with access to

01:23
Special interest groups, video on demand content library, free registration to all PSIs, webinars and much much more. Just visit the PSI website, psiweb.org today to learn more about it and become a PSI member.

01:45
Welcome to another episode of the Effective Statistician. And today we have another Alexander on the call. Hi, Alex, how are you doing? It’s actually not Alexander, it’s Alexei, as I just learned. Hi, Alexei, how are you doing? I’m doing great. Thank you so much for inviting me. I appreciate this opportunity. And then we have, as usual, our co-host, hi, Benjamin. Hi, Alex. Hi, Benjamin.

02:13
Alex and Alex. Alex and Alex, yeah. So, Alex, we have worked together at Lilly a couple of years ago, but maybe you can start a little bit with an introduction of yourself, how you came to be really a world-class expert on multiplicity. And…

02:40
how your career has been up to now where you actually have your own company? Yes, yes, of course, I’ll be happy to do that. I think this would be a natural starting point. So by way of outline, as we typically say at the beginning of our presentations, as Alexander pointed out, we all started at Lilly. In my case, my career as a pharmaceutical statistician started back in 98.

03:08
just a little over 20 years ago. So I moved to Indianapolis after receiving a doctorate degree in statistics and I joined at that time a very rapidly growing statistical organization at Eli Lilly and I was very pleased to discover lots of opportunities. I was really thinking that you know, this would be a great chance for me to help develop and apply innovative approaches to improve

03:35
the design and analysis of Lilly’s clinical trials. I remember that it was only as a statistician at Lilly that I learned about adaptive designs and I immediately embraced that concept. I wanted to work on company-wide initiatives to help facilitate the use, application of adaptive designs. I did a little bit of work on analyzing QGC interval prolongation, I remember. And it was at that time, probably about 15,

04:04
years ago, 15, 16 years ago, that I found a new setting area of research, novel methods for addressing multiplicity issues in confirmatory clinical trials. So that’s when I discovered the joy of multiplicity and also adaptive designs. And I started contributing fairly actively to multiplicity research, ended up writing quite a few.

04:30
papers on this topic and was very happy to have an opportunity to collaborate with multiplicity experts across the industry in academia, at regulatory agencies. Back in 2009 we published what I think was a very nice summary of multiplicity methods used in clinical trials. It was a book published in the Chaplin and Hall CRC Press series which was co-edited by

04:58
Professor Ajit Amhani, Dr. Frank Bretz and myself. So we collaborated with multiple experts in the field of multiple comparisons. That’s how I pretty much got started and continue to be actively involved in multiplicity research and publishing books or book chapters that are generally related to this topic. I would like to mention very quickly that last year we published a book on

05:25
clinical trial optimization using R. And one of the most important chapters of that was related to how we should use available information and the tools, software tools that we have in our hands to help us find best, ideally optimal multiplicity adjustments in clinical trials. But I think we’re probably going to return to this topic later on in this interview.

05:55
step one, step back, how would you describe multiplicity? So what actually is it? I mean, I know many people work with it in different ways, but how would you describe it in a rather short summary and saying, you know, what are the topics for beginner statistician? Oh, sure. Yeah, I can give you maybe a short 10-minute summary. I mean, this is a topic

06:24
It would normally take me a couple of hours, which is the length of my short course on this topic. Just imagine you’re doing a 60 seconds poster presentation. In that case, probably the best point to start is go back to a famous biblical story of David and Goliath. I’m sure that you have heard the story multiple times, the Israeli army versus the Philistines and how David managed to defeat Goliath.

06:54
by throwing a stone at him. Everybody knows this part of the story, but very few people realize that when before that that bad lecture took place that David actually walked over to a brook and he picked up not a single stone, he actually picked up five stones. Because he realized the power of multiplicity. And the Bible is actually surprisingly unclear on this point. Was that

07:18
when he finally brought down goal ice, was that the first shot? Maybe this was actually the fifth shot, because he wanted to have, being a practical person, he wanted to have multiple shots on goal. So that’s what multiplicity about. That is something that creates additional opportunities for us to show in the context of clinical trials now, we’re going to, we’re done with the biblical story. In the context of clinical trials, it’s something that gives us more opportunities

07:48
formally speaking, the efficacy profile of novel treatments, but there’s a price of course that we’ll need to pay because there are multiple shots, you know, and some of those shots could lead to incorrect decisions. So that’s what this whole topic of multiplicity is about. We need to find a way to efficiently then control that multiplicity, realizing all the pros and potential cons of employing multiple objectives in clinical trials. Yeah.

08:15
I think the key is of course then to control the probabilities that we make false positive claims if we test too many times. Of course, lots of people know the Bonferroni test where you just divide by the number of tests you have or similar approaches. And there’s, you know…

08:43
Also very, very common way is these gating approaches where you basically kind of order your tests and then go through it step by step. When I’m discussing these with people, then there’s very, very often the question, okay, if I build this kind of gating list, do I need kind of, you know, should I…

09:12
put as many objectives in it as possible, because very often you have 10, 20 endpoints in your trial and you can of course all sort them in some kind of way.

09:26
how to best do this in practical situations, right? Yep. So should we always test as many objectives and put as many objectives as possible into this, you know, basket set where we control the primary, the alpha level, or is it better to kind of, you know, restrict it to very, very few? Mm-hmm, mm-hmm. That’s a great question. I assume that here we’re now discussing

09:56
confirmatory or pivotal clinical trials where we have to make sure that we satisfy appropriate regulatory requirements. Based on my experience doing multiplicity consulting over the past at least 15 years, you know, whenever I discuss this topic with the statisticians or clinicians, for example, they’ll realize, of course, you know, that one needs to use their best judgment to choose probably the most meaningful endpoints, but

10:25
Whenever marketing people enter those discussions, they typically get overly excited to say, yeah, let’s go for 20, let’s go to 30 different endpoints. The reality, of course, you know, regulatory agencies encourages us to reduce the number of secondary objectives. For example, secondary endpoints to be able to support, I guess, meaningful inferences. When you look at the recently published regulatory guidance documents,

10:55
First of all, the draft guidance by the FDA that you, as you may know, was published in early 2017 on Multiply and Points. And as you know, the European Medicines Agency released their draft guideline in, I believe this was April or May of last year. This topic is not explicitly discussed because obviously our colleagues, the regulatory agencies, do not want to give us those strict rules. In certain cases,

11:24
perhaps a larger number of endpoints could be justified from a clinical perspective. The reality if you try to average over multiple clinical trials, clinical development programs that have employed several endpoints, several clinical objectives, you will see that this number is typically does not exceed five. Just for example, one of the first, probably one of the first successful

11:55
was the Lurazidone development program. I remember it very well because again this is the program where we were able to successfully show the benefits of using some of those multiplicity adjustments, gatekeeping procedures that were reviewed by some people to be kind of overly complex, but everything worked out very nicely. At that time, this was over 10 years ago, they had only two key secondary endpoints.

12:22
And looking at some of the recent trials that I was involved with, for example, the famous Compass trial, the study was stopped early to interim analysis. It’s a very, very large study in cardiovascular population with over 25,000 patients. They had three key secondary endpoints. So my recommendation, of course, would be to work with your colleagues, I’m talking to statisticians.

12:52
most meaningful secondary endpoints that will help you provide additional meaningful supportive evidence of efficacy relative to what you can infer based on the primary endpoint. And maybe I can quickly also mention this very interesting and confusing topic of pseudo specificity. I don’t know how much you have heard about this, but this is something that maybe not to that extent applies to European

13:21
regulatory negotiations and communications but in the United States topics of pseudo-specificity plays a fairly important role when it comes to when you begin working with certain divisions at the FDA. Basically the idea is that when certain secondary endpoints are very closely related to the primary endpoint they are considered pseudo-specific and they should be removed from consideration. Unfortunately there is no

13:48
definition. But it’s kind of when there’s from a statistical side a pretty high correlation in terms of for example. No, unfortunately no. It’s more of a clinical concept and I believe it was developed if you will at the FDA mostly in the context of neuroscience trials. Some of the FDA divisions would not really…

14:13
talk much about pseudospecificity when it comes to neuroscience trials, you would probably be expected to say something and address this concept of pseudospecificity and make sure that your secondary endpoints are not clinically related. Obviously, this is what makes this concept somewhat mysterious and vague, because what exactly is clinical research?

14:40
studies in, for example, depression and your primary endpoint is a pretty broad scale in terms of measuring all the different aspects of depression and then you have further endpoints that measure more specific subdomains of the depression, then of course there’s quite a high clinical correlation between your overall depression endpoint and…

15:08
certain subdomains of depression. That’s right. Yeah. But you’re making a very, very, it’s a very good example, Alexander, you know, but, and this is something that FDA’s Division on Psychiatry products would point out to you. If you were to present a very similar strategy to a different division, for example, recently I worked on a series of migraine trials, you know, and those are…

15:34
reviewed those applications reviewed by FDA’s division of neurology projects products, you know Then they would go ahead and allow you to use Components essentially of the primary endpoints as a second endpoints for them. See the specificity is not that of a concern Okay, so it’s it’s really a case case by case Decision. Okay. That’s right. That’s right. Yeah, unfortunately the FDA’s guidance the multiple endpoints does not mention this topic

16:04
Yeah, and also to say the clinical correlation may not be that obvious, but sometimes you wonder why the statistical correlation is that obvious. It’s all kind of the beauty is in the eye of the beholder. It’s subjective. Yeah, but just we mentioned that for the rules that we do have objectives, several, let’s say five objectives.

16:34
then once you test them and using multiplicity, so how do you communicate the objectives that are non-significant on multiple testing, just that case, but are significant on the local level, so by itself? Is there any guidance or? At this point, my interpretation of the FDAs, for example, draft guidance, it’s also based on my experience with multiple

17:04
development programs that have employed increasingly more complex multiplicity adjustments is that findings that were significant before a multiplicity adjustment, that’s what you’re referring to, but after an appropriate multiplicity adjustment they are no longer significant. Those may still be presented in the product label but they would simply not be identified as significant as you as you’re kind of trying to visualize the contact

17:34
the content of a typical clinical study section of a product label. Again, I’m talking in this case specifically about the FDA’s review process. They would present certain findings for let’s say key secondary endpoints and it will be one or two multiple asterisks. And the footnote would say those findings are significant. And when they’re not significant, I guess the best way to communicate is to use the data.

18:02
those findings would be as descriptive, perhaps they’re still clinically meaningful, but the sponsor cannot obviously claim statistical significance. And I would say there’s always a chance to at least discuss the clinical relevance of those findings in appropriate clinical trial. Yes, that is kind of the other part. So I think we have different other people that would be very, very interested in these topics.

18:30
And if you write, for example, a clinical manuscript, I think I would always still mention that and just present maybe the adjusted and the unadjusted p-value or if you can’t adjust the p-value, it doesn’t make sense based on your multiplicity adjustment, at least kind of say, okay, on a local level, that still is significant.

19:00
because I think other people might have a completely different view on this topic than kind of the FDA or the sponsor, in terms of what’s important for them.

19:16
That is right. And actually, maybe to make a very quick comment related to this, it sounds like it looks like the concept of multiplicity adjusted p-value is too confusing for product labels. So most of the time, the actual original p-values are presented and then they’re identified as either being significant or not significant. Wow, that’s maybe even more confusing. Yes.

19:44
Yeah, exactly. You’re looking at a significant p-value, but there is no asterisk. So it’s not significant after a multiple list adjustment. But at least the fact that before an adjustment, the p-value was significant. That is communicated to some extent. Yeah, I think. How do you kind of if you wrote a writer publication, what would be your kind of… have you any kind of best practice how to kind of communicate it there?

20:15
How to communicate the general, I guess, the general question is how to communicate multiplicity, right? How do we communicate multiplicity adjustments to non-statisticians? So that would be the first part, and the second part would be how to kind of communicate these, in the best way, these findings that are on itself significant, but are not significant after the multiplicity adjustment.

20:45
Ideally, maybe stepping back, I think that all of this information that would be important and all the potential outcomes would need to be ideally incorporated into the testing strategy and maybe in a few minutes we’ll get to talk about how clinical trial simulations help us kind of take care of those potential outcomes and those potential inconsistencies in the future.

21:14
Because when it comes to performing multiplicity adjustments in pubertal clinical trials, as you know, everything must be prospectively defined, which means that everything is set in stone and signed in blood. There is no way back. So if something is not significant before an adjustment, something, I’m sorry, is significant before an adjustment, but after an adjustment is no longer significant, that is something that would be ideal for a project team to be prepared for.

21:43
for that discussion. But maybe if I may again step back and kind of think about this general topic of how to best communicate some of those more complex concepts related to multiplicity to our colleagues. It’s obviously a bit of a philosophical question. How do we facilitate general communication in this case? And it’s probably something that would apply to many other advanced statistical methods including adaptive designs.

22:13
quite a few people who share their thoughts about how to best communicate some of those challenging statistical concepts to non-statisticians. And I think the general solution here is to try to focus on benefits rather than technical features. And this would be conceptual similar to how physicians communicate with patients. To me, I’ve made this analogy on multiple occasions.

22:42
When we think about the challenges that physicians face, when they talk to patients who may not really know much about the underlying science, instead of going to technical details and saying, you know, I would like you to try this treatment option because let’s say it’s a TNF-alpha inhibitor, they would never say that. They would try to present the available options with emphasis on

23:08
benefits on how this will help address specific symptoms, how to achieve certain desirable outcomes. They would never talk about the mechanism of action. Therefore, when I get into those discussions with non-statisticians, I always try to emphasize the fact that the proposed solution, be it a multiplicity adjustment, adaptive design, helps address important regulatory requirements, but also does it in a