What makes a great secondary paper?

Transcript:

Alexander: You’re listening to The Effective Statistician podcasts, a weekly show with Alexander Schacht and Benjamin Piske, designed to help you reach your potential, lead great sciences and serve patients without becoming overwhelmed by work. Today, we are talking about secondary papers. Yes, these papers that you do once you’re done with your primary paper and what makes a good one, so stay tuned for this discussion and now some music.

I’ve been working on such secondary papers for a very, very long time and I think they are so much fun and you can make them really extraordinary and you can do lots of great things about it and so stay tuned for this one.

I am producing this podcast in association with PSI, a community dedicated to leading and promoting the use of statistics in the healthcare industry for the benefit of patients. This association has a special interest group, the launch and lifecycle special interest group, that speaks a lot about these kind things. So, if you’re working in this space and you are working on a lot of these kinds of secondary publications, then check out the SIG that might really help you. Head over to psiweb.org that talks more about PSI activities and becomes a PSI member today. 

Welcome to another episode of The Effective Statistician today, again with Benjamin. Hi, how are you doing? 

Benjamin: Very well. How are you, Alexander? 

Alexander: Very good. It’s a nice spring day here. So yeah, good mood. 

Benjamin: Lucky you. It’s a nice winter day over here, gray and windy. 

Alexander: But we’re not talking about the weather today. Today we are talking about secondary papers. First, maybe we talk about what actually is a secondary paper. 

Benjamin: Actually, today, I’m very interested in the topic as well because, to be honest, I’ve never come across a secondary paper myself. It’s quite an interesting topic. I’ll try to interview you, Alexander, because from my side I have been working at zero for my whole working life. I supported more publications in the past but this has not been on secondary papers, this has been on primary papers meaning that for the original contents of it So my question, maybe I can make a start and ask you what actually is the secondary paper, Alexander?

Alexander: The typical primary paper contains all the primary analysis, the primary endpoint, analysis, key secondary, the safety analysis, the key facts. It really contains all these kinds of typical, most important pre-specified analysis so that paper of course is a kind of bright largely by itself. Maybe medical writers would disagree, but content wise, it’s largely written by itself based on the pre-specified analysis. You have lots of the content or in the protocol or in the study report and, of course, it always needs a bit of fine-tuning here. There’s, of course, always the typical things that will come from external authors in terms of additional questions and from the peer review process, you will have additional things. But largely it is why you did this study. So the key topics, why did you study?

Benjamin: When I just think about the process and how the analysis at the end is to develop. It usually starts with protocol, so you say what is the focus then on the primary paper? But when it gets to SAP writing, for example, isn’t it that there are already like a lot of ideas being added? So if it’s reviewed by other parties, usually, just remember things like subgroups or mass, or other things just out of interest, which I usually at least mentioned in the clinical study report as well but not dived into. So is this the more typical context of the secondary paper? 

Alexander: Yes, that is very often where secondary paper stops. And of course, depending on the study, you might have many, many secondary papers. Imagine you have, let’s say, a three-year study that runs over three years, but your primary analysis is maybe after three months. Then, of course, everything that comes after these are first read out are all kinds of secondary papers. Or maybe you’re working in the disease area where there’s lots of different questionnaires and assessments done and it’s far too much to put all into one paper.You focus your primary paper on the most important ones and there’s still lots of data available for secondary papers. As you said, there might be specific subgroups that are of interest, or there might be specific topics, like there’s maybe a specific safety aspect that you want to dive deeper into. Or there is maybe you want to speak specifically about let’s say how fast the drug works. Or you want to explore further into certain endpoints. Like maybe, you have a specific endpoint that you have first used after it got validated in the study and you want to kind of dive deeper into these kinds of things. Depending on the disease areas, there may be many topics.

Benjamin: And it’s not necessarily that we write the idea down prior to the analysis. It’s also things that you do afterwards like Ad hoc, the ideas to brainstorm together with others and then find out interesting findings in the results, etc. 

Alexander: Usually it’s a kind of combination of certain analysis that you already have done. Usually you already have some kind of summary statistics or overall analysis, but not specifically all the analysis that you need. So there is usually much more kind of going backwards and forwards from it. Also, ideas come from all kinds of different places during the strategic publication planning. Maybe there’s an idea coming for all. We would like to have this specific subgroup paperboard, because that is a subgroup that’s important for reimbursement. Or maybe there’s a question like I said about specific side effects that are coming up. Or some things that you haven’t expected like maybe there’s some skin side effect and you want to dive deeper into it, kind of, what are the patients that are most affected? When does it happen? How is it treated? What kind of cool medications do these patients have? All kinds of different things that you could never put into a primary paper. Just to the sheer volume of all the different analyses that you put in a specific one. So that’s often kind of content for these typical secondary papers. 

Benjamin: When brainstorming about where this comes from. It’s quite obvious that the approach that we have and working and driving a primary paper doesn’t work. So I mean, consorts, why is that not fitting? 

Alexander: Well, consort is for those who don’t know, is basically a guideline for how to report papers and it’s really targeted towards these primary papers. It talks about all the mentions, the pre-specified endpoints, and all these kinds of different things. You need to talk about sample size and all these different things. This is not so relevant for secondary papers, you can very easily refer to lots of the content of the primary paper. You don’t need to rewrite the complete study description and everything. There’s a lot of stuff that you can just cite and say. 

Benjamin: I mean also refer to the primary paper.

Alexander: Refer to the primary paper. That’s why it’s sometimes a little tricky to submit these unpublished primary papers. Then at least maybe you can cite a poster or cite clinicaltrials.gov or cite something else so that you don’t need to have too much space wasted for describing lots of stuff that is actually not so relevant to this paper.

Benjamin: But if you design a study and you have the primary idea behind it and put this in a primary paper. There must be a lot of audiences waiting for it for whatever reason. It could be the success of a drug, could be submission. What is the challenge then in actually writing a secondary paper? And if it’s not as obvious and pre-planned as we have for the primary endpoints or the key endpoints? 

Alexander: The first challenge is to develop a great story. For the primary paper, it’s easy because the story is basically the story of the study itself. It’s written in the protocol why you need the study and all these kinds of different things. It’s pretty straightforward. But here you need to come up with a new story. Let’s say you want to speak about how fast your drug works? And then you can talk about it in terms of, what does that mean for the patient? What does it mean for the physician if the drug works fast rather than slow? What additional benefits will that have? Then you can further expand on the story, ‘okay, we can see with this new treatment that it is specifically fast, that also reflects on maybe less drop out due to lack of efficacy, it leads to better overall symptom response over the first couple of months. They conclude, ‘this is a specific feature of this new treatment and then you can compare it to other existing compounds’. And say, ‘it looks better than this one and that one, could be the usual strengths and limitations’, and then you have your paper. And that helps you to answer specific questions that you have. 

Benjamin: I think the challenge really starts, if this is like an unexpected or maybe like a controversial story to the primary paper, like the overall protocol. So did that happen to you when you found out that this is your secondary paper suddenly? I wouldn’t say that it cannot be a success, but something that is completely unexpected.

Alexander: Both. Sometimes you can expect quite a lot, like the topics that I mentioned with the patient reported outcome that is used the first time. You know that this will be an interesting topic from a scientific perspective, it will also be an interesting topic from reimbursement perspective to have sufficient data to show that actually is a valid endpoint for these kinds of things. These are pretty well defined such kinds of things. But there are others with the side effect of you haven’t talked so much about. And now this becomes much more of a topic and you need to dig deeper into it. The key here is also for the more difficult ones. You need to make sure that we align them with your overall strategy. So in terms of overall strategy, if you do good strategic publication planning, that follows a strategy that is based on your overall compound strategy. And this compound strategy speaks about what are the key differentiators of this new treatment? What are the key patient populations you want to go for and the key aspects you want to highlight? What are specific differences versus the competition and the needs that you need that you have from reimbursement from a marketing perspective, from a regulatory perspective? All these things are part of the overall strategy and then, of course, you need to make sure that your papers align with this overall strategy and you first work on all the things that help support this overall strategy. For example, the speed of onset is important, then don’t shift that to the tens paper, have it early on, these kinds of things. It’s also that you’re consistent in terms of how you phrase things across different papers. So, for example, the terminology you use in the primary patterns and secondary paper is consistent to some extent.

Benjamin:  That might be tricky. I mean, to monitor this because often there are many parties involved and it might take this back to, let’s say, academic centers or, companies or different groups within the company because of the focus of it. So that is something then they’re working in parallel. Maybe they’re involving you, maybe not. So this is indeed like a challenge if you don’t control terminations, over the stories, also over the frequency or maybe the order or when things are being published and where. Because sometimes it’s more like a statistical side, sometimes it is more on oncology paper or publications, whatever it is.

Alexander: One of the main things, for example, is that you’re consistent in terms of certain definitions, for example. So you shouldn’t have in one paper, let’s say, 151 patients that you are analyzing and the next one 149. Or you look into a certain group of side effects and you find them based on certain sources or PTs. And you have one set up in one paper and the differences are in the other paper, but you call it the same. There you need to be consistent, but in terms of other things, you need to be flexible.

Benjamin: It could be consistent that you very well explain this in the paper, what you did, so why did the number change? I’ve been through these discussions not for the papers but for the decision of maybe how to group adverse events of special interest. So, I mean, it’s a wide range. Some people say, ‘well, yes, you take this PT in combination with the onset of the adverse event’, and others say, ‘well, only if it’s like this and then so, it’s very strong. But that’s probably very important to have especially a statistician looking over this and make sure that there’s a consistency throughout the other different papers. 

Alexander: There are these not trial-specific but maybe compound specific analysis plans, come into place so that you have certain definitions for really consistent across all the different studies, across all the different publications and things like this. 

Benjamin: But that requires that the authors read the sap. 

Alexander: Well, it requires collaboration. I know it’s easier said than done, especially if big organizations and lots of different departments that do not work easily together. I know. 

Benjamin: And also I notice from medics like, if they have a long history of defining things differently than you do in the company and then they run and they drive its key publication, they may or may not agree through the SAP. 

Alexander: That is a really important thing. Working with the coercers, especially the ones that are outside of the company, is really, really crucial here. It’s very important to have, let’s say you have five external coerces. It’s very important to have some kind of strategy for how you work with them. One could be, for example, that the five get some hierarchy and they agree with each other. Who is the primary coerce, the least coerced, and takes responsibility for the different things and then also, in terms of thought, deciding? Because otherwise you can get lots of conflicting feedback and never get it resolved.

Benjamin: If you have five papers and five different authors, while the others are coercers, everyone is the primary author taking the decisions, then you have five different definitions.

Alexander: So there’s a kind of flexibility coming into place. If there’s a need for distinct things, then at least highlight it, that you said, ‘okay here we define skin irritations this way and in this paper we have to define it in a different way.’ Or, ‘here, we think about pretreatment specifically in terms of all these and in the other paper, for whatever reason, you have pretreatment defined in other ways. And of course there’s always certain reasons for this, but you need to be at least transparent with these. 

Benjamin: Okay. Now we talked a lot about what the differences are and maybe the challenges, especially in the planning. What are some ideas of what to consider when you actually write the secondary papers? Is there any guideline? Is there anything that you learnt that was a good way to apply? 

Alexander: So one structure that I often use and that has helped quite a lot is the 3×3 structures. And you have three points that you want to bring across in your paper. You mention these three times. In the introduction you say, ’we will talk about A, B and C. Then you show, here are a, b and c into discussion. And to summarize, this paper shows A, B and C. That is a really, really nice and easy structure that helps you to focus the paper. When you do presentations, for example, there’s very often the rule of three. Brings three key takeaways, mentioning three ideas. In all kinds of different stories, there’s always these three. There’s three nephews of Donald Duck, it’s not two or four, it’s three. In the same way, you can use your paper and that helps you to structure, it works very well.

Benjamin: And I think it’s also easier to read because I remember reading papers and at the end you just ask what it actually showed here. That’s good. And I think three is a magic number here because three is something which you can probably repeat yourself. I mean, it’s easy to get three things and you keep it and you memorize it. And if you are less than that, it’s probably you need to put in more emphasis on the two items, so 3 is something like a number to take away. And then when it gets higher than three, it’s probably difficult. Because then what is the message? It’s splitting apart. So I think three is a good number for this.

Alexander: And please remember these 13 points. That’s also why we have three letter acronyms usually? The other really important thing about it is good graphics are really important. Having three graphics that support your three major results is nice. Graphics are especially important because you want to stand out from the crowd. You want to bring your message across very crisp and fast. These visualizations that you put into the paper will carry forward quite a lot beyond the pit. You can actually take some from let’s say earlier posters that you did because very often, these secondary papers will have associated posters with it. For the poster, if you have a great data visualization, that helps you to get your message across very quickly. Same way, you can adapt its end for the paper and then thereafter, used for presentations by speakers at all different conferences, at internal events. They will be part of, let’s say, HTA submissions, they will become part of promotional material. The better you make your data visualizations in this paper, the more likely all the other subsequent data visualizations, or adaptations of it, will also be right. 

Benjamin: Well, I think this is a point which is probably not only true for secondary papers, but it actually is more important than for primary papers. Simply because, as I mentioned before, I said the interest for primary papers is higher even before you start so that is something that there’s required that people are waiting for to get the message from the primary but they prepared very nicely, through protocol, SAP. So the complete story of the study. But when it gets to the secondary paper you have to have something which is eye-catching, right? So, which is something that puts the interest even more because you have to create the interest more than for primary papers. And that’s why even though good graphics are always important, the importance is even higher for secondary people. 

Alexander: And often the concepts are not as easy as for the primary paper. So let’s say the primary papers, very often about just the treatment difference. But if you want to display a kind of how,for example, an ease develops over time, that’s much more tricky I think to have a look into. Or if you want to look into lots of different subgroups, that’s the trickiest thing. So, invest more time in these kinds of things to make it very, very crisp. 

Benjamin: Actually, that brings me to another point. If you work in treatment groups and plan the complete study on comparing A versus B, you can live in p-value and it’s good or not. But that probably means that for secondary papers, you are often not successful in using just standard analysis that your pre-plan because there isn’t often anything prepared. 

Alexander: Yes. There are some other nice things about it, whereas your primary papers are often very much driven by regulatory needs, while for these secondary papers, you have much more freedom. So you can use more advanced things like, for example, doing different machine learning algorithms. Maybe you want to understand what are the patients that respond best? And you can do lots of different things around this. Or you want to understand, are there certain patterns in the data, cluster analysis, or unsupervised learning techniques? Or you want to understand other certain underlying concepts in your PRO and you do some latent class analysis. There’s lots of really interesting things you can do here and that’s the fun part about these papers. In my experience, these more interesting analyses as well as really, really nice graphics, help you get it easier published. Once a medical writer told me ‌she really loves to work with me on the papers because it’s much easier from the submission process to the journey. Instead of having this ping-pong to the journal getting rejected, send it to the next journal which gets rejected again, sending to the next journal and over the period have lots of kinds of backwards and forwards and peer reviews and so on. You get it accepted faster. And that is, of course, really important, not just from a timing perspective and from a workload perspective, but also, usually target the higher-ranked journals than you go to the lower impact sector ones. And so it also helps you to have higher credibility of the papers.

Benjamin: It’s true. That’s why I haven’t gone through the process yet. What about any other thoughts? I mean, we touched the combination with posters and upcoming publications, presentations. What else would you recommend considering and thinking about before writing a secondary paper? Any other cross-connection?

Alexander: One thing is that all these papers are electronically. With these electronic versions, there are a couple of additional features that you can consider. Of course, that is also for primary paper, but I think especially for secondary paper it just makes it really nice. You have the electronic appendixes where you can put additional data that doesn’t fit into the original paper. Things like additional sensitivity analysis, additional data visualizations, animations, videos, all kinds of different things. Lots of journals are going in that direction that maybe allow something like a brief presentation to be attached to it. Someone that is the primary also talks for 45 minutes about the paper in this presentation. And so these kinds of things can help you make the publication more beneficial overall, more impactful overall. Everything that is there is published and then you can use it for further efforts, like in promotional material, in HTA submissions. Of course, that always depends on the local regulations but there’s a lot of benefits to having things published. 

Benjamin: I think that is a good point because of old school papers, PDFs. it changed. Meaning that this advantage of electronic papers is something you need to consider.

Alexander: I’ve seen some interactive data visualization on these. So that you can check, especially when you show other different subgroups or things like that. Having some things that it’s interactive is really really nice. Or you have these more complex graphics that have a video that explains these kinds of things in a better way. It also helps with being really transparent about all your data. Therefore, it helps to build trust. These are a couple of additional considerations to have for the paper.

Benjamin: It makes sense. Any other advice or anything else?

Alexander: I think one of the main things is to be very clear and crisp about what is your paper about and what is not your paper about. I’ve seen lots of discussions going in circles because you weren’t really crisp and clear about what is in the paper and what is the opposite of the paper. Managing these expectations among all those different authors in the study team and the compound team is really important. Not just have these kinds of titles, this is a paper about functioning or this is a paper about safety or this is a paper about rapid onset? Things like where you can have thousands of different ideas about that and then you go in circles. Because different people had different expectations about it and that can delay the process quite a lot, and can lead to lots of lots of frustration. So, having a very, very crisp and simple plan at the beginning, such as agreed by everybody, helps a lot to move easily through the overall process and also to get to excellent results early on. Then, of course, comes back to the overall strategic planning that you have other papers where you can put all the different other ideas. People will always come up with, ‘this is a nice additional thing’. If you have good publication plans and you can say, ‘yeah, but it covered this somewhere else’. 

Benjamin: It makes sense. What is the role of a statistician there? 

Alexander: The role of the statistician works together with the medical writers, the Physician, the other people to get to this crisp definition but also to come up with good ideas. If you see there’s an opportunity, here is a very, very nice feature of our data that makes our track stand out from the crowd and we can best show it that way through this analysis, these graphics, and bring these ideas forward. Statisticians, if they see themselves as track developers, being statisticians and not statisticians that are just stats functional representatives on the team. Then you can influence things and drive things, and make sure that you have a consistent overall story. 

Benjamin: I fully agree. It is just a challenge because I mean, not number crunchers, but at least it’s kind of where we come from originally saying that the function of a statistician is more the interlinking. I mean, it sounds a little like we are talking now for statisticians, primarily statisticians, about writing papers. Actually, we are usually not writing papers. So we’re working together with somebody who makes medical writing and medics and that’s a kind of combination of it. It’s more the mindset. What we agree upon is all the facts, all the advice that you just gave in terms of these three times three rules, etc. That is something we should keep in mind and bring in, even though we are not necessarily the primary author of the study. The input that we need to work with an entire group and now it gets back to the communication, on argumentation and bringing it for the topics we touched before on our podcasts are coming in and bringing it all together. So I think it’s a challenge, especially since this is not like our key responsibility in writing the paper. 

Alexander: To write papers, there are a couple of different ways you can do this. What I found really helpful is if you have some medical writer, the primary physician and the statistician together in a room and you outline your paper. You don’t need to have full sentences there, but at least you have bullet points. What are the key things that we want to say here? We take the typical description from the protocol here. There is a data visualization added here. Here goes the stats parts that are written by you. Here are the key conclusions that we want to have. Have a couple of bullet points for the strengths and limitations. And if you have that outline done together and then just planning sentences made by a medical writer, you really write paper together instead of just throwing over the analysis to someone and then get them to pluck them together in some shape or form for you to critique it. I think putting more effort at the beginning helps a lot to save time later on. 

Benjamin: As always, probably. 

Alexander: As usual. Good planning is everything. Not necessarily that a plan always works out, sometimes you see something that has some data aspect that you need to put some additional thoughts on it or these additional challenges from coerces or peer review comments, of course always. Well, there’s one paper my team submitted that went through without a single comment and they accepted it.

Benjamin: Probably just over saw it.

Alexander: In 20 years. Okay, we have talked now for quite a long time about secondary papers and just let me summarize a couple of things. Secondary papers are really important, but they are very specific from the primary papers. Don’t just have the typical concept following paper, you need to be very, very crisp and clear about the story that you want to tell. And we mentioned this three times, three rules that I think are really nice to structure your content, at least with freedom to do some amazing things in terms of data analysis, in terms of data visualization, and it can be quite a lot of fun to write these papers. I’ve written dozens of these. I think I stopped counting beyond 70, so you can actually boost your CV ‌ nicely with it. Thanks so much. 

Benjamin: Very good. Have a good day.

Alexander: Have a nice day and talk to you soon. 

I created this show in association with PSI. Thanks to Reine and Kacey who help with the show in the background and thank you for listening. Reach your potential, lead great sciences and serve patients. Just be an effective statistician.