Alex Dmitrienko has researched and published so much, that without a doubt, he can be called one of the world-class experts in the field of multiplicity. E.g. his book Multiple Testing Problems in Pharmaceutical Statistics contains a wealth of practical information on this topic.

His career has brought him from pharma over CROs to his own company Mediana. Via his company, he provides also online courses on his favorite topic. Please access these here!

As a listener, you will get the following benefits from this episode:

  • Understand the multiplicity of challenges in practical situations
  • Learn how to implement it
  • Learn where to get further information
We cover the following topics during the interview:
  • What is multiplicity and why does it matter?
  • Should we always test as many objectives as possible or is it better to restrict the list of objectives?
  • How do we communicate best those objectives, that are non-significant on the multiple testing adjusted case but are significant on the local level?
  • Non-regulatory stakeholders like payers and physicians may have completely different views on the priority or importance of variables. How should we manage this?
  • There are many different ways how setting up systems for multiplicity adjustments. What are the best steps to agree with a cross-functional team on this?
  • How do we deal with multiplicity, if the study has different components?
  • How do we deal with multiplicity, if we have 2 or more studies and analyze them combined as well as individually?
  • What are good resources to learn about multiplicity for beginners?

Alex Dmitrienko

Alex Dmitrienko, PhD, is the Founder and President of Mediana Inc.  Dr. Dmitrienko has been involved in pharmaceutical statistics for 20 years and, before founding Mediana, worked at Quintiles (Vice President, Innovation Unit) and Lilly (Research Advisor, Center for Applied Statistical Expertise).

He has been actively involved in biostatistical research and has published over 100 papers on key topics in clinical trial statistics, including multiple comparisons, subgroup analysis, clinical trial optimization, and adaptive designs.  He has authored/edited two SAS Press books (Analysis of Clinical Trials Using SASPharmaceutical Statistics Using SAS) and two Chapman and Hall/CRC Press books (Multiple Testing Problems in Pharmaceutical Statistics and Clinical Trial Optimization Using R).  Dr. Dmitrienko has served as an Associate Editor for The American StatisticianBiometrics and Statistics in Medicine, and is a Fellow of the American Statistical Association.

Helpful links

References

  • Alosh, M., Bretz, F., Huque, M. (2014). Advanced multiplicity adjustment methods in clinical trials. Statistics in Medicine. 33, 693-713.
  • Dmitrienko, A., Tamhane, A.C., Bretz, F. (editors) (2009). Multiple Testing Problems in Pharmaceutical Statistics. Chapman and Hall/CRC Press, New York.
  • Dmitrienko, A., D’Agostino, R.B., Huque, M.F. (2013). Key multiplicity issues in clinical drug development. Statistics in Medicine. 32, 1079-1111.
  • Dmitrienko, A., D’Agostino, R.B. (2013). Tutorial in Biostatistics: Traditional Multiplicity Adjustment Methods in Clinical Trials. Statistics in Medicine. 32, 5172-5218.
  • Dmitrienko, A., Pulkstenis, E. (editors). (2017). Clinical Trial Optimization Using R. Chapman and Hall/CRC Press, New York.
  • Dmitrienko, A., D’Agostino, R.B. (2018). Multiplicity considerations in clinical trials. New England Journal of Medicine. 378, 2115-2122.
  • European Medicines Agency. Guideline on multiplicity issues in clinical trials. 2017.
  • U.S. Food and Drug Administration. Multiple endpoints in clinical trials: guidance for industry. 2017.
  • Huque, M.F., Dmitrienko, A., D’Agostino, R.B. (2013). Multiplicity issues in clinical trials with multiple objectives. Statistics in Biopharmaceutical Research. 5, 321-337.

Transcript

Understand and master multiplicity in practical situations – Interview with Alex Dmitrienko

00:01
You’re listening to episode number 25 of The Effective Statistician. Today we are talking all about multiplicity and as a guest we have a world-class expert, actually one of THE world-class experts in this topic, Alex Dimitrenko. So keep on listening.

00:28
Welcome to another episode of the effective statistician with Alexander Schacht and Benjamin Piske, the weekly podcast for statisticians in the health sector designed to improve your leadership skills, widen your business acumen and enhance your efficiency. In today’s episode we’ll chat with Alex Dimitrenko about multiplicity and especially all the practical aspects around it.

00:55
We’ll of course go a little bit into the theory, but we make it really, really a high level and very, very actionable for you. This podcast is created in association with PSI, a global member organization dedicated to leading and promoting best practice and industry initiatives. Join PSI today to further develop your statistical capabilities with access to

01:23
Special interest groups, video on demand content library, free registration to all PSIs, webinars and much much more. Just visit the PSI website, psiweb.org today to learn more about it and become a PSI member.

01:45
Welcome to another episode of the Effective Statistician. And today we have another Alexander on the call. Hi, Alex, how are you doing? It’s actually not Alexander, it’s Alexei, as I just learned. Hi, Alexei, how are you doing? I’m doing great. Thank you so much for inviting me. I appreciate this opportunity. And then we have, as usual, our co-host, hi, Benjamin. Hi, Alex. Hi, Benjamin.

02:13
Alex and Alex. Alex and Alex, yeah. So, Alex, we have worked together at Lilly a couple of years ago, but maybe you can start a little bit with an introduction of yourself, how you came to be really a world-class expert on multiplicity. And…

02:40
how your career has been up to now where you actually have your own company? Yes, yes, of course, I’ll be happy to do that. I think this would be a natural starting point. So by way of outline, as we typically say at the beginning of our presentations, as Alexander pointed out, we all started at Lilly. In my case, my career as a pharmaceutical statistician started back in 98.

03:08
just a little over 20 years ago. So I moved to Indianapolis after receiving a doctorate degree in statistics and I joined at that time a very rapidly growing statistical organization at Eli Lilly and I was very pleased to discover lots of opportunities. I was really thinking that you know, this would be a great chance for me to help develop and apply innovative approaches to improve

03:35
the design and analysis of Lilly’s clinical trials. I remember that it was only as a statistician at Lilly that I learned about adaptive designs and I immediately embraced that concept. I wanted to work on company-wide initiatives to help facilitate the use, application of adaptive designs. I did a little bit of work on analyzing QGC interval prolongation, I remember. And it was at that time, probably about 15,

04:04
years ago, 15, 16 years ago, that I found a new setting area of research, novel methods for addressing multiplicity issues in confirmatory clinical trials. So that’s when I discovered the joy of multiplicity and also adaptive designs. And I started contributing fairly actively to multiplicity research, ended up writing quite a few.

04:30
papers on this topic and was very happy to have an opportunity to collaborate with multiplicity experts across the industry in academia, at regulatory agencies. Back in 2009 we published what I think was a very nice summary of multiplicity methods used in clinical trials. It was a book published in the Chaplin and Hall CRC Press series which was co-edited by

04:58
Professor Ajit Amhani, Dr. Frank Bretz and myself. So we collaborated with multiple experts in the field of multiple comparisons. That’s how I pretty much got started and continue to be actively involved in multiplicity research and publishing books or book chapters that are generally related to this topic. I would like to mention very quickly that last year we published a book on

05:25
clinical trial optimization using R. And one of the most important chapters of that was related to how we should use available information and the tools, software tools that we have in our hands to help us find best, ideally optimal multiplicity adjustments in clinical trials. But I think we’re probably going to return to this topic later on in this interview.

05:55
step one, step back, how would you describe multiplicity? So what actually is it? I mean, I know many people work with it in different ways, but how would you describe it in a rather short summary and saying, you know, what are the topics for beginner statistician? Oh, sure. Yeah, I can give you maybe a short 10-minute summary. I mean, this is a topic

06:24
It would normally take me a couple of hours, which is the length of my short course on this topic. Just imagine you’re doing a 60 seconds poster presentation. In that case, probably the best point to start is go back to a famous biblical story of David and Goliath. I’m sure that you have heard the story multiple times, the Israeli army versus the Philistines and how David managed to defeat Goliath.

06:54
by throwing a stone at him. Everybody knows this part of the story, but very few people realize that when before that that bad lecture took place that David actually walked over to a brook and he picked up not a single stone, he actually picked up five stones. Because he realized the power of multiplicity. And the Bible is actually surprisingly unclear on this point. Was that

07:18
when he finally brought down goal ice, was that the first shot? Maybe this was actually the fifth shot, because he wanted to have, being a practical person, he wanted to have multiple shots on goal. So that’s what multiplicity about. That is something that creates additional opportunities for us to show in the context of clinical trials now, we’re going to, we’re done with the biblical story. In the context of clinical trials, it’s something that gives us more opportunities

07:48
formally speaking, the efficacy profile of novel treatments, but there’s a price of course that we’ll need to pay because there are multiple shots, you know, and some of those shots could lead to incorrect decisions. So that’s what this whole topic of multiplicity is about. We need to find a way to efficiently then control that multiplicity, realizing all the pros and potential cons of employing multiple objectives in clinical trials. Yeah.

08:15
I think the key is of course then to control the probabilities that we make false positive claims if we test too many times. Of course, lots of people know the Bonferroni test where you just divide by the number of tests you have or similar approaches. And there’s, you know…

08:43
Also very, very common way is these gating approaches where you basically kind of order your tests and then go through it step by step. When I’m discussing these with people, then there’s very, very often the question, okay, if I build this kind of gating list, do I need kind of, you know, should I…

09:12
put as many objectives in it as possible, because very often you have 10, 20 endpoints in your trial and you can of course all sort them in some kind of way.

09:26
how to best do this in practical situations, right? Yep. So should we always test as many objectives and put as many objectives as possible into this, you know, basket set where we control the primary, the alpha level, or is it better to kind of, you know, restrict it to very, very few? Mm-hmm, mm-hmm. That’s a great question. I assume that here we’re now discussing

09:56
confirmatory or pivotal clinical trials where we have to make sure that we satisfy appropriate regulatory requirements. Based on my experience doing multiplicity consulting over the past at least 15 years, you know, whenever I discuss this topic with the statisticians or clinicians, for example, they’ll realize, of course, you know, that one needs to use their best judgment to choose probably the most meaningful endpoints, but

10:25
Whenever marketing people enter those discussions, they typically get overly excited to say, yeah, let’s go for 20, let’s go to 30 different endpoints. The reality, of course, you know, regulatory agencies encourages us to reduce the number of secondary objectives. For example, secondary endpoints to be able to support, I guess, meaningful inferences. When you look at the recently published regulatory guidance documents,

10:55
First of all, the draft guidance by the FDA that you, as you may know, was published in early 2017 on Multiply and Points. And as you know, the European Medicines Agency released their draft guideline in, I believe this was April or May of last year. This topic is not explicitly discussed because obviously our colleagues, the regulatory agencies, do not want to give us those strict rules. In certain cases,

11:24
perhaps a larger number of endpoints could be justified from a clinical perspective. The reality if you try to average over multiple clinical trials, clinical development programs that have employed several endpoints, several clinical objectives, you will see that this number is typically does not exceed five. Just for example, one of the first, probably one of the first successful

11:55
was the Lurazidone development program. I remember it very well because again this is the program where we were able to successfully show the benefits of using some of those multiplicity adjustments, gatekeeping procedures that were reviewed by some people to be kind of overly complex, but everything worked out very nicely. At that time, this was over 10 years ago, they had only two key secondary endpoints.

12:22
And looking at some of the recent trials that I was involved with, for example, the famous Compass trial, the study was stopped early to interim analysis. It’s a very, very large study in cardiovascular population with over 25,000 patients. They had three key secondary endpoints. So my recommendation, of course, would be to work with your colleagues, I’m talking to statisticians.

12:52
most meaningful secondary endpoints that will help you provide additional meaningful supportive evidence of efficacy relative to what you can infer based on the primary endpoint. And maybe I can quickly also mention this very interesting and confusing topic of pseudo specificity. I don’t know how much you have heard about this, but this is something that maybe not to that extent applies to European

13:21
regulatory negotiations and communications but in the United States topics of pseudo-specificity plays a fairly important role when it comes to when you begin working with certain divisions at the FDA. Basically the idea is that when certain secondary endpoints are very closely related to the primary endpoint they are considered pseudo-specific and they should be removed from consideration. Unfortunately there is no

13:48
definition. But it’s kind of when there’s from a statistical side a pretty high correlation in terms of for example. No, unfortunately no. It’s more of a clinical concept and I believe it was developed if you will at the FDA mostly in the context of neuroscience trials. Some of the FDA divisions would not really…

14:13
talk much about pseudospecificity when it comes to neuroscience trials, you would probably be expected to say something and address this concept of pseudospecificity and make sure that your secondary endpoints are not clinically related. Obviously, this is what makes this concept somewhat mysterious and vague, because what exactly is clinical research?

14:40
studies in, for example, depression and your primary endpoint is a pretty broad scale in terms of measuring all the different aspects of depression and then you have further endpoints that measure more specific subdomains of the depression, then of course there’s quite a high clinical correlation between your overall depression endpoint and…

15:08
certain subdomains of depression. That’s right. Yeah. But you’re making a very, very, it’s a very good example, Alexander, you know, but, and this is something that FDA’s Division on Psychiatry products would point out to you. If you were to present a very similar strategy to a different division, for example, recently I worked on a series of migraine trials, you know, and those are…

15:34
reviewed those applications reviewed by FDA’s division of neurology projects products, you know Then they would go ahead and allow you to use Components essentially of the primary endpoints as a second endpoints for them. See the specificity is not that of a concern Okay, so it’s it’s really a case case by case Decision. Okay. That’s right. That’s right. Yeah, unfortunately the FDA’s guidance the multiple endpoints does not mention this topic

16:04
Yeah, and also to say the clinical correlation may not be that obvious, but sometimes you wonder why the statistical correlation is that obvious. It’s all kind of the beauty is in the eye of the beholder. It’s subjective. Yeah, but just we mentioned that for the rules that we do have objectives, several, let’s say five objectives.

16:34
then once you test them and using multiplicity, so how do you communicate the objectives that are non-significant on multiple testing, just that case, but are significant on the local level, so by itself? Is there any guidance or? At this point, my interpretation of the FDAs, for example, draft guidance, it’s also based on my experience with multiple

17:04
development programs that have employed increasingly more complex multiplicity adjustments is that findings that were significant before a multiplicity adjustment, that’s what you’re referring to, but after an appropriate multiplicity adjustment they are no longer significant. Those may still be presented in the product label but they would simply not be identified as significant as you as you’re kind of trying to visualize the contact

17:34
the content of a typical clinical study section of a product label. Again, I’m talking in this case specifically about the FDA’s review process. They would present certain findings for let’s say key secondary endpoints and it will be one or two multiple asterisks. And the footnote would say those findings are significant. And when they’re not significant, I guess the best way to communicate is to use the data.

18:02
those findings would be as descriptive, perhaps they’re still clinically meaningful, but the sponsor cannot obviously claim statistical significance. And I would say there’s always a chance to at least discuss the clinical relevance of those findings in appropriate clinical trial. Yes, that is kind of the other part. So I think we have different other people that would be very, very interested in these topics.

18:30
And if you write, for example, a clinical manuscript, I think I would always still mention that and just present maybe the adjusted and the unadjusted p-value or if you can’t adjust the p-value, it doesn’t make sense based on your multiplicity adjustment, at least kind of say, okay, on a local level, that still is significant.

19:00
because I think other people might have a completely different view on this topic than kind of the FDA or the sponsor, in terms of what’s important for them.

19:16
That is right. And actually, maybe to make a very quick comment related to this, it sounds like it looks like the concept of multiplicity adjusted p-value is too confusing for product labels. So most of the time, the actual original p-values are presented and then they’re identified as either being significant or not significant. Wow, that’s maybe even more confusing. Yes.

19:44
Yeah, exactly. You’re looking at a significant p-value, but there is no asterisk. So it’s not significant after a multiple list adjustment. But at least the fact that before an adjustment, the p-value was significant. That is communicated to some extent. Yeah, I think. How do you kind of if you wrote a writer publication, what would be your kind of… have you any kind of best practice how to kind of communicate it there?

20:15
How to communicate the general, I guess, the general question is how to communicate multiplicity, right? How do we communicate multiplicity adjustments to non-statisticians? So that would be the first part, and the second part would be how to kind of communicate these, in the best way, these findings that are on itself significant, but are not significant after the multiplicity adjustment.

20:45
Ideally, maybe stepping back, I think that all of this information that would be important and all the potential outcomes would need to be ideally incorporated into the testing strategy and maybe in a few minutes we’ll get to talk about how clinical trial simulations help us kind of take care of those potential outcomes and those potential inconsistencies in the future.

21:14
Because when it comes to performing multiplicity adjustments in pubertal clinical trials, as you know, everything must be prospectively defined, which means that everything is set in stone and signed in blood. There is no way back. So if something is not significant before an adjustment, something, I’m sorry, is significant before an adjustment, but after an adjustment is no longer significant, that is something that would be ideal for a project team to be prepared for.

21:43
for that discussion. But maybe if I may again step back and kind of think about this general topic of how to best communicate some of those more complex concepts related to multiplicity to our colleagues. It’s obviously a bit of a philosophical question. How do we facilitate general communication in this case? And it’s probably something that would apply to many other advanced statistical methods including adaptive designs.

22:13
quite a few people who share their thoughts about how to best communicate some of those challenging statistical concepts to non-statisticians. And I think the general solution here is to try to focus on benefits rather than technical features. And this would be conceptual similar to how physicians communicate with patients. To me, I’ve made this analogy on multiple occasions.

22:42
When we think about the challenges that physicians face, when they talk to patients who may not really know much about the underlying science, instead of going to technical details and saying, you know, I would like you to try this treatment option because let’s say it’s a TNF-alpha inhibitor, they would never say that. They would try to present the available options with emphasis on

23:08
benefits on how this will help address specific symptoms, how to achieve certain desirable outcomes. They would never talk about the mechanism of action. Therefore, when I get into those discussions with non-statisticians, I always try to emphasize the fact that the proposed solution, be it a multiplicity adjustment, adaptive design, helps address important regulatory requirements, but also does it in a

23:37
very meaningful way and something that physicians, non-statisticians, sometimes even investors can relate to. But in general, I can tell you, you should be prepared to be challenged by multiple people because they will have different views. We have recently published, maybe I will talk about this later, we recently published a tutorial on multiplicity in the New England Journal of Medicine and have received…

24:06
quite a few interesting responses from readers. There were physicians who felt very strongly that multiplicity adjustments are counterproductive. They should be just taking out of the picture because it is in fact a penalty. But when I bring up this topic and begin discussing the pros and cons of multiplicity adjustment, maybe that’s why I use that biblical story.

24:34
This is essentially a price we’ll have to pay for having additional flexibility and multiple shares in gold. I think working with different stakeholders is kind of a challenge. I think we had this in many of our podcasts before that we, you know, if you work with different people, different topics, so they have a different view of things, so how to discuss it. And so what is your, for example, I mean, this is leading to the next question, actually. So what we have, you know, the different stakeholders.

25:03
that you’re working with and setting up as a trial using multiplicity, have a different view on the priority and the importance of variables or objectives. And therefore, in the order of how to use the multiplicity or how to go to one or other. How do you manage this? Or is there any guidance or any experience that you can share in how to work with different stakeholders on the use of the prioritization of the variables?

25:33
Yes, that’s one of those very, very important topics in the life of a pharmaceutical clinical trial statistician, but unfortunately there are no simple solutions here or hard and fast rules. This is something that comes mostly with experience. It would be difficult for me to give you a simple set of recipes here.

25:59
it all comes to your ability to essentially be a good communicator. And I pointed out a couple of minutes ago, not to emphasize technical issues, certain things like, for example, why do I, as a statistician, care about certain types of multiplicity adjustments? I really enjoy the fact that they have certain maybe optimal properties, but this is a purely mathematical concept, of course, you know, you need to really speak the language.

26:27
of payers and physicians, as you said, I’ve had to work with investors. Investors speak a completely different language. Maybe not directly with them, but with their advisors. So that’s where you need to put on your consulting hat and really think about how to find the solution, how to explain the benefits, the value of the solution that you’re proposing without getting into too much into technical details, of course.

26:57
where whenever I went into that, I always had that challenge that

27:04
People might even agree on a ranking of, okay, that is the most important endpoint, that’s the second most important endpoint, and so on. And then there was the other question, okay, but what’s the probability that we hit on these different endpoints? Because the power to hit on one endpoint might be very different to the power to hit on another endpoint.

27:31
And of course, they want to take that into account as well and kind of overall optimize that. How would you kind of approach that? I guess I’m actually very glad that you mentioned the topic of optimization. I would love to come back to this later in this interview. But from a practical perspective, if push really comes to shove in you and you’re working with a project team that has very little relevant historical information to be able to order,

28:01
those secondary endpoints, I think it may be a good idea, that’s what we’ve done on multiple occasions, is to simply step away, walk away from the idea of applying that hierarchical testing approach, which is known as a fixed sequence testing procedure, and simply put not maybe all of those secondary endpoints, but at least some of them into a single bucket. And apply a multiplicity adjustment that does not require

28:30
that you pre-specify the testing sequence. There are lots of those. That’s where you let the data speak for themselves. That includes the Holm, Hulkenberg, Como, and many other multiplicity adjustments that rely on a data-driven hypothesis ordering. And we’ve shown if you apply simulations under some plausible sets of treatment effect assumptions, you make sure to show that in many cases, this kind of approach would give you more power.

29:00
than following a predefined sequence. How do you define power? I would guess you would have multiple different ways to define power if you have many, many different endpoints. So basically, you have a basic power for each of these endpoints. And depending on how you set up your kind of testing strategy, you have for one endpoint,

29:30
more power than for another endpoint because an endpoint that is maybe coming later in the testing strategy has overall less power because there’s much higher chances that previous fail and you never come to that area. You’re absolutely right. Exactly. When it comes to multiple hypothesis testing problems with a…

29:57
multiple hypotheses, the concept of power is not really defined or maybe you should say it’s not uniquely defined and there are actually quite a few papers that have been written and published that simply deal with the topic of available power definitions or success criteria definitions in those cases. Probably the most basic one would be to ensure that at least one of those endpoints

30:28
something that’s been used in multiple clinical studies. I personally do not really like it that much because this definition, it’s known as disjunctive power, by the way, it kind of does not really draw a line between outcomes that would be clinically completely distinct. You can have five end points, all of them would be statistically significant. And from the perspective of disjunctive power, it would be a successful outcome, or it can have…

30:55
single significant point and four non-significant ones. It would still be countered as a successful outcome. Yeah, so it’s better actually. My personal preference recently has been to work with weighted power when you compute marginal power of each individual endpoint test and then you add them up with appropriate weights that help you quantify the relative importance of each endpoint.

31:22
This is a way to first of all account for relative importance which end point you know and then you also account for the number of successful outcomes and those definitions that’s the good news you know they’re relatively robust to the choice of the weights so that’s something that we’ve actually done in several recent studies. Okay it’s good to hear because otherwise my question would be so how do you define the weights then I mean this is probably the same same as for you know defining the priority of the objectives itself.

31:50
us then how to define the weights in order to give them enough power for each of the end for each of the comparisons. So that’s right. Yes, this the weight weighted power is is more forgiving. It’s less sensitive to the choice of those end point specific weights. And in certain cases, those weights are actually can be derived quite naturally. When we look at multiplicity, for example,

32:17
problems that arise in clinical trials with multiple patient populations. So you have the oral patient population and you can have one sometimes even two subsets predefined subsets of the patient population then the importance of those subsets can be set to be proportional to the relative size of those subsets. So that’s where weighted power can be defined in a very objective way.

32:44
Or you can kind of define it based on the anticipated value it has in the label. Or you do even kind of a short preference survey with your study team. That would be ideal, yeah.

33:13
unique metric for which you can overall optimize your design and your multiplicity adjustment. In a multifunctional team. Yeah, absolutely. I mean, that’s a challenging topic, really. I have one question as we talk about that was maybe more also with the internal stakeholders. If we look into the external stakeholders, would it be possible, for example, to have

33:43
different approaches for different stakeholders. So let’s say that you have a different one for the email versus for US regulatory.

33:55
That is actually, that is now I would say is becoming more common. We all, we, I wouldn’t say always talk, you know, but we talk often enough about the fact that different regulatory regions, if you will, different regulatory authorities may have different requirements for the choice of primary and secondary endpoints. That’s true for diabetes, for example. That’s true for many other areas such as rheumatoid arthritis.

34:25
And therefore, you may see a protocol in which two different sets of primary analysis are essentially defined. One is for the purposes of the US submission, the other is for the purposes of European Union submission. Likewise, region-specific multiplicity adjustments can also be defined. It’s actually, I guess, just even more common than we may realize.

34:49
So when we have these submissions, in many areas, these submissions, these regulatory submissions consist of a package of studies. So you have maybe two studies or even more studies and they, you know, maybe share the same objectives. So how do we deal with these kinds of things?

35:19
or more studies? Could you kind of have different alphas for, let’s say, integrated analysis as well as for alphas for the individual studies? Or how do you deal in this kind of way?

35:38
In that case, I guess we’ll have to draw a line between the multiple list adjustment and general testing criteria for the primary endpoints as opposed to key secondary endpoints. Because when we look at development programs that include several phase 3 studies, which is once again very, very common in diabetes, for example.

36:06
then there’s a regulatory requirement that when we analyze the significance of the treatment effect based on the primary endpoint, then the results must be significant across the trials that were included in the development program. So there’s this very strict consistency criterion applied. And most of the time, as I understand, when we look at combined pooled analysis across the studies,

36:36
that those may be helpful to, of course, to perform additional sensitivity analysis, consistency, heterogeneity, those kinds of analysis, but those may be also used to potentially support pulled analysis of key secondary endpoints. And in that case, in fact, it would be ideal, and that is something that I’ve discussed on multiple occasions, and I still feel very strongly that

37:06
when it comes to applying multiplicity adjustments to key secondary endpoints, those tend to be underpowered to begin with. And therefore, and then we apply a multiplicity adjustment, which is also a penalty. So there is this trend that the probability of success for secondary endpoints tend to be lower compared to the primary endpoint or endpoints. And one way to address that would be to

37:35
first pull two or maybe more phase three studies within the same development program and then apply a multiple list adjustment for the secondary endpoints. That is something that has been discussed at multiple conferences. In fact, there was even a white paper published on this topic, but as far as I understand, this is not still a recommended approach. If you look at the

38:01
draft guidance, the multi-plan points that was released by the Food and Drug Administration in the United States. This approach is not explicitly described there, but I think this would be the way to go. So this is how I would approach in general multiplicity issues in clinical development programs with two or more studies.

38:22
Okay, I think a question will come up after people listen to you and the questions and the answers that we had is, but where do people get good resources to learn about multiplicity? I mean, obviously they can take courses and you only read your books, but I mean, this is the obvious, but is there any good source that you can recommend where people can look up and get…

38:51
get a first thoughts about it or read good books? Yeah, I’ll be happy to. Maybe I should begin with the obvious. Talk about my own papers. Go on Amazon and ask people, do you track this? Everybody does it. This is going to be a shameless plug from my own books.

39:19
it’s a it’s a time-honored tradition. I would say you know that there actually have been because of the general interest in the in the topic of multiplicity and there’s been so much work done in this area over the past 10-15 years which is actually a lot more compared to the decade before that. There have been quite a few nice review papers. I’m happy to have been able to contribute to some of them. In fact, I have provided a list of

39:47
recent tutorials or review papers that were published, for example, in Statistics and Medicine, and they will be available on the web page that will be set up for this podcast. Just check out the effe and the podcast. Yeah there have been papers, I’ve been happy to, I’ve been lucky to collaborate with Professor Ralph de Agostino and there have been papers.

40:15
published by again review papers by by Dr. Frank Bratz and FDA statisticians Dr. Alok and Dr. Hawk for example and in fact I would say you know that as far as statisticians are concerned we do have a lot of resources available right right now and that’s why we have started paying a bit more attention to developing writing tutorials for now for non statisticians that’s why I’m so happy that

40:42
our tutorial on multiplicity has finally been published. It came out in the New England Journal of Medicine just a couple of weeks ago, and at the end of May, I should clarify. And I would also recommend, in addition to those resources, I would mention a variety of traditional online courses that have become available. I have become a big fan and supporter of online training,

41:13
relatively recently decided to go ahead and record several online training courses. I think it’s a great option and because they’re available 24 hours a day, 7 days a week, you can watch them in your spare time. You can skip a certain topic, which is not an option, which is available when you’re taking a real training course because you have to power through sometimes. But in this case, you can basically choose the modules that would make most sense to you and

41:40
Maybe I can mention very quickly that the two online courses that I have recently recorded, they have been actually included in an online training program sponsored by the Biopharmaceutical Section of the American Statistical Association. And this program provides a 50% discount on online training courses. I think it’s a great deal. And you will again find information on this online training program on the podcast’s website. I know that…

42:10
Quite a few US and European pharmaceutical companies have taken advantage of this inexpensive and convenient option and hope that the listeners could also find those online courses useful. Yes, for sure. As Alex said, you’ll find also links in the show notes. So actually, beyond these kind of practical things,

42:37
Where can people find you if they want some specific coaching from you or consulting from you on their specific submission?

42:50
Oh, thank you. They can almost find me here in my office in Kansas City. If you happen to live in Europe, if you’re willing to fly over the Atlantic Ocean. And online? And online, exactly. I mean, again, I’ve mentioned those online training courses. Maybe, actually, as you said at the beginning of this interview, my work right now…

43:17
is very similar to the work I used to do at Lilly and at QueenTiles. I like to refer to it as strategic buy statistical consulting. I’ve been spending a lot more time on the development of statistical software. Maybe I can tell you a little bit more about the free software tools that we have developed. So I’m very, very still excited about doing consulting work.

43:44
on topics such as adaptive designs and of course multiplicity. This is my favorite topic. As I’ve said, this is something that’s been near and dear to my heart for a number of years. So if the listeners have any multiplicity related questions, I would encourage them to contact me. Maybe you can share again my contact information. Yeah, we’ll do that. Very good. So I think that was a great, great interview.

44:14
starting from why multiplicity matters. I think the public story will be really in my mind from the heart with a very different view on it. Absolutely. I think what also was very clear for me is that as a statistician, you need to be a really good communicator, a really good teacher on these kinds of things.

44:42
Because this is not a very easy topic and actually applying it and making it happen for real life scenario is really difficult. And having a clear understanding of what is the business background, managing all these different stakeholders is really important. But of course, also it’s important to be, you know…

45:08
up to date with all the different methods and techniques that are out there. There’s quite a lot developed over the last years and it’s really nice that there’s also this New England Journal of Medicine paper from you that we can help. It’s quite nice if you have such a paper in hand to give it to your physicians as well. And yeah, just to sum up.

45:37
You’ll find all these details also in the show notes. So thanks a lot, Alex. That was great. Well, thank you so much. I’m once again, congratulations on your successful podcast. And I wish you I wish you luck and appreciate this opportunity to talk to you about my favorite topic in clinical trial statistics. Thank you so much.

46:03
Thanks Alex, it was a pleasure.

46:23
Please tell your colleagues about it. That’s the most effective way that we can help more statisticians benefiting from this podcast. Thanks a lot and stay tuned for next week’s episode.

Join The Effective Statistician LinkedIn group

I want to help the community of statisticians, data scientists, programmers and other quantitative scientists to be more influential, innovative, and effective. I believe that as a community we can help our research, our regulatory and payer systems, and ultimately physicians and patients take better decisions based on better evidence.

I work to achieve a future in which everyone can access the right evidence in the right format at the right time to make sound decisions.

When my kids are sick, I want to have good evidence to discuss with the physician about the different therapy choices.

When my mother is sick, I want her to understand the evidence and being able to understand it.

When I get sick, I want to find evidence that I can trust and that helps me to have meaningful discussions with my healthcare professionals.

I want to live in a world, where the media reports correctly about medical evidence and in which society distinguishes between fake evidence and real evidence.

Let’s work together to achieve this.