In order to make new drugs available to patients in Australia, a successful interaction with the PBAC – the Australian HTA body is needed. Alan Brnabic lives in Australia and collected a lot of experience with these submissions over the years.

In this episode, we cover:

  • the basics of understanding how the PBAC works,
  • why a successful PBAC submission is important,
  • what evidence is needed to have a successful PBAC submission,
  • how much work is needed from the statistician’s side for the PBAC interactions,
  • what are the typical challenges from the PBAC,
  • when is a good start to get prepared for the PBAC work, and
  • how is the data submitted to PBAC publicly available

Finally, we’ll also cover an upcoming one-day PSI workshop on RWE in which such evidence needs are discussed besides other approaches, that are available and emerging?

You can register for the one-day event at

Alan J. M. Brnabic

Mr. Brnabic is currently Principal Research Scientist at Eli Lilly working in Real World evidence (RWE) with a focus on specialized analysis that supports this area. Prior to this he was the Asia Pacific Director of the Health Outcomes and Health Economics, Life Sciences for OPTUM.

Whilst at Eli Lilly he has been the Health Outcomes and Statistics Asia Pacific statistical sciences group leader and manager. His work has included: leading and managing staff, designing/reviewing concepts and studies (Phase IIIb & IV observational studies), analyzing/reviewing studies, presenting & writing/reviewing results in peer review publications, leading and reviewing external methodologies/ guidelines for use within the company as well as consulting/coordinating strategy for analysis on Reimbursement dossiers & other related Health Outcome activities for countries like Australia, Canada & Korea.

Mr. Brnabic received a BA Dip Ed in mathematics and statistics and a MA in applied statistics from Macquarie University, Sydney. He has worked for SPSS Australasia as a Technical Support and Training Consultant; at Macquarie University Practical Demonstrator/Tutor in statistics.

In 1994 he worked as a Consultant Biostatistician for 5 years in Public Health NSW Health Department. Following that he was a Senior Biostatistician at the George Institute which is affiliated with UNSW where he worked on epidemiological studies and RCTs. Before joining Eli Lilly in 2002 he also took a position at the NSW Department of Corrective Services as Deputy Director of the Research & Statistics, Sydney.

Mr. Brnabic’s interests are in the design and analysis of observational studies with a focus on methodologies related to subgroup identification as well as selection bias adjustment tools including matching, propensity score analysis and local control. He is also interested in Health Outcomes and statistical approaches used to help support the reimbursement of medicines like matched adjusted indirect comparisons as well as mixed treatment comparisons.

He has A-STAT Professional Accreditation with the Statistical Society of Australia (SSAI). He is an active member and previous Chair for the Australian Pharmaceutical Biostatistics Group (APBG) and on the Australian Bipolar Comprehensive Outcomes Study Steering Committee. He is also a member of ISPOR and the SSAI.


How to successfully submit a reimbursement dossier in Australia? Interview with Alan Brnabic on the statistician’s perspective

Welcome to the Effective Statistician with Alexander Schacht and Benjamin Piske, the weekly podcast for statisticians in the health sector designed to improve your leadership skills, widen your business acumen and enhance your efficiency. In today’s episode number 16, we’ll talk about how to successfully submit a reimbursement dossier in Australia. An interview with Alan Brenevich on the statistician’s perspective on this submission.

This podcast is sponsored by PSI, a global member organization dedicated to leading and promoting best practice and industry initiatives for statisticians. Learn more about upcoming events at

Hello, this is another episode of the Effective Statistician. I’m very happy today that we have a dear friend and colleague of mine, Alan Branovich, from Australia calling into this interview today. Hi, Alan. Hello, how are you going? Hi, Alan. And of course, we have my co-host, Benjamin Piske, here as well. And my name is Alexander Schacht.

Topics that we will talk today about are PBAC submissions. And if you don’t know what that is, just hang in a moment and we’ll clarify that. But first, let’s introduce our guest. I’m working with him for quite a long time and he’s a well-known expert on real-world evidence, HTA submissions, onto all these kind of different topics within our company.

So, and maybe you want to talk a little bit about your history and your work experience and whether you have any kind of special interest topics with regard to statistics. Sure. I don’t want to reveal too much about my job history because it might reveal my age, but I’ve been working as a statistician for over 20 years in various places, including the health department here in Australia, a research institute called the George Institute.

as well as the corrective services in the research and statistics unit before I joined Eli Lilly. So each of those jobs were interesting in their own right and involved a mix of clinical trials as well as observational real-world studies. I suppose my special interests of late are really working in statistics in the late phase, so observational studies, real-world evidence,

trying to get our medicines reimbursed through health technology assessment. So I’m involved a lot in that sort of work, not just in Australia, but around the Asia Pacific and now a little bit in Europe.

So good. And by the way, Aaron is working from home today. And so you might hear some kids in the background. So don’t be disturbed by that. So I mentioned in the beginnings that today we will talk about PBAC. Can you explain a little bit what that is and why that’s important? Sure. So PBAC stands for Pharmaceutical Benefits Advisory Committee. And it’s a

independent expert body, which has been appointed by the Australian government. The members on this body include doctors, health professionals, health economists, and consumer representatives. And its primary role is really to recommend new medicines for listing on the PBS, which is the Pharmaceutical Benefits Scheme, which is the subsidy scheme for medicines. Now, no new medicine can be listed unless the committee makes a positive recommendation.

And the PBC, PBAC sorry, meets three times a year, usually in March to July, November, which means if pharmaceutical companies want to try and get their products listed, they have three opportunities to do that. Now, the PBAC is a little bit complex and it has two subcommittees to assist with analysis and advice when making recommendations. There’s a…

drug utilization subcommittee as well as an economics subcommittee. So it’s very important to make sure the right documents are put in at the right time. And why, just answer your second question, why PBAC is important. Well, in Australia without successful reimbursement of medicines, access to medicine for Australian patients can be limited or even impossible to access.

especially for those that can’t afford the medicines. And also without the PBAC recommendation, it’s also important that the medicine gets accessed quickly. So timely access is the other important reason for PBAC.

So PBA-C basically is very similar to what we have in Germany, the EQVIC or in UK NICE or the French Transparency. Yeah, similar sort of idea. So it’s a health technology assessment body. So that’s correct.

And it both looks into the medical value, but also into the costs of the new. Correct. So it looks at what’s called the value for money of the treatment, as well as the cost effectiveness. So it looks at what the actual medicine was registered for in Australia. It looks at its clinical effectiveness, it looks at its safety and its cost effectiveness.

and in particular compared to other treatments that are currently available in the market in Australia.

When we talk about comparing to other treatments, so lots of our studies that we usually do from a phase three perspective are placebo controlled and sometimes only placebo controlled. What evidence is actually needed to have a successful PBAC submission? Can you just hand in these placebo controlled studies? Good question, very good question. Unfortunately, you can’t. So as you’ve pointed out, usually the head-to-head.

evidence that’s available is against placebo, but in the Australian market, they’re interested in an active comparator. So usually based on a couple of criteria, either market share or similar sort of mode or mechanism of action. So the head-to-head evidence, unless that is against your comparator of interest, that’s

before submitting, you have to use indirect evidence. And usually, the indirect evidence is in the form of adjusted indirect comparisons. Occasionally, methods like mixed treatment comparisons slash network meta-analysis are used, but primarily, the Australian reimbursement bodies, the PBAC are interested in indirect comparisons.

So applying the Bucher method would be the usual way forward to come up with the… Correct. And they have a guidance document that actually spells that out quite specifically to use the Bucher method. And in fact, they always want you to present the random effects models for the meta-analysis and using the Decemion and LED estimator for tau. So they’re very specific.

what they want. They also specify that they want risk differences, relative risks, odds ratios and number needed to treat. So they make it very clear what they want, which is good.

Okay, and then for each of these approaches, the relative risk difference, you would need to have a p-value for each of them or do you do one p-value kind of overall? How do you come up with an indirect? Yeah, so the obviously, as you know, it’s a two-step process to do the indirect comparison. It’s the meta-analysis first, then the indirect comparison. So all measures of risk.

need their associated confidence intervals and p-values within the meta-analysis as well as the indirect comparison. You know, you need the confidence interval plus the p-values for that as well. Plus they ask for all heterogeneity measures as well as the I-squared and its limits of uncertainty as well. So they’re very specific and of course they want to see

If you do see some heterogeneity in your meta analysis, if you have more than two studies, what you will do with that, how you address that and what that actually means.

So you really need to have a pretty good understanding of meta-analysis and all these kind of different problems around heterogeneity within sets, the different heterogeneity approaches, how to explain heterogeneity to actually be successful. Correct. And I think you also need to know your studies inside out, the inclusion-exclusion criteria, and really understand what patients are going in that meta-analysis.

So if you do see observed differences between the studies and the heterogeneity is present, then what could be driving that? Is it different baseline risk factors? Is it effect modifiers? Is it the time that the study was conducted? We see a lot with some of our immunology studies that some of them are conducted 10 years ago and the treatment.

of these patients have changed dramatically. So you see differences in the common comparator, which anchors the indirect comparison across these studies. And that is an issue of course, because if the common comparator response rates are similar across the trials, then you have an issue doing the indirect comparison.

Okay, so in terms of, you mentioned kind of you need to take care of the patient population that goes into this. Actually would you have sometimes kind of a situation where you basically look at only a subsection of your or subpopulation of your phase three studies in order to come up with a treatment effect that you then can plug in into the indirect? That’s a good question, yes.

The listing for the medicine in Australia may be a little bit different to the clinical trial phase three program. So sometimes subgroups or subsets of the population are needed and hopefully the competitors information is presented in that same way for those same subgroups or subpopulations.

If they’re not, of course, it’s almost impossible to do that analysis.

Does the PBAC sometimes come back with requests to extend, for example, the population? So what are the requirements in general that the PBAC is then asking you, or how to interact when they interact with you? Yeah, so that’s another good question. So it is possible that there could be line extensions, but it’s discussed and negotiated up front before the dossier is submitted.

what that population will be. And PBAC are also interested in the budget aspects of the treatment. So they really want to know what sort of patients, patient numbers are expected for this treatment and what sort of uptake there’ll be and how much your new treatment that you’re proposing to get reimbursed will, you know,

I suppose supersede other treatments that are on the market. So it’s rare that the PBAC will ask for line extensions unless it’s some breakthrough medicine like a new oncology drug that’s first in class, best in class. Okay. So how much of interaction is needed from your side? Anyway, obviously you are assistant physician.

in a way responsible to clarify questions upfront, as you said, or before. For how long are you involved usually in such a submission and how much of interaction is needed? Yeah. So usually the statistician in Australia doesn’t go to the PBAC. It’s usually the pricing reimbursement access or health economists that actually have the discussions with the PBAC.

Because they’re a government body, they can’t really give opinions of what we should be doing. So we have to come in with a strategy and be confident with that strategy. It’s also helpful if other companies have actually already listed their drugs on the

are available to everyone, which we can look and see what sort of strategy and approach that they took. But in terms of preparation and timing.

As early as possible, the statistician should be involved right at the beginning, should be involved discussing at the global level what the strategy is for each of the countries, what sort of data is needed because there’s a combination of actual clinical trial data, so from the phase three program as well as the indirect evidence which comes from systematic literature reviews.

there’s a lot of requests that are needed. Once you’ve submitted that dossier, it then gets reviewed and it comes back after a period of time with a whole lot of questions that you have a week to address. So that’s another opportunity for the statistician to be involved and try and address the questions that come back. Well, one week is really like a, there’s a need actually to be available.

don’t take vacations. That’s right. It’s very well planned. Yes. So we actually have the March submission coming up as you know and so I’m trying to take my vacation as soon as I can before they call me up.

Yeah, one week is a great opportunity but I would probably call it a challenge. The thing is you do prepare and anticipate. So we do a lot of extra analyses, indirect comparisons that we don’t put in the submission but anticipate they may ask us just in case they do and we have it ready. So that has happened in the past.

Yeah, so basically you check your submissions, think of all the different questions that might come up in order to be able to respond within a week. Sorry, I was just going to say, we also call on our global counterparts for assistance during that period. Yeah, I think it’s really important to have a very close collaboration here between those that do these submissions on the local HTA levels.

together with those that work on the global phase three program. If that’s not there, then everything is, I think, far more difficult. I think that also what is needed there is a very good mutual understanding of what are the needs of the different countries or what are the needs of the different requirements from a regulatory and from an HTA.

point of view and the look on the data is just very, very different if you talk to the email or the FDA or if you talk to the PBC or another HTA buddy. So what are the typical challenges? Yes, there are many. Often they’re concerned about the quality or uncertainty of the evidence.

assessing bodies, they follow the hierarchy of evidence. So if you’ve got a randomized control trial, phase three, then when that shows convincing superiority, then you’re in a good position. But as we discussed earlier, that’s often not the case. So when you start using methods like adjusted indirect comparisons by the Booker method, that evidence is a lot

they call that evidence observational, post hoc observational evidence. So that’s a challenge, making sure you have the highest quality evidence. The other challenges include, and this is not uncommon to Australia. I believe that this is an issue around the world, the generalizability of the results from the phase three trials. So often…

phase three trials have a certain inclusion exclusion criteria with a certain population in mind. However, it doesn’t actually reflect the true patient population that the drug will be listed for. So in Australia, for example, for psoriasis, the clinical trials were, I think, for a PASI baseline score of greater than 12, is that right, Alexander?

Whereas in Australia, the patient population is for those that have a PASI baseline score of greater than 15. So already you have that discrepancy in the patient base. However, this is something that all the HTA bodies around the world are facing and there are methodologies that our group and some of the universities are working on to try and tackle this issue.

challenge is what comparative do you choose? As I said before, sometimes it’s based on share of market or sometimes the mechanism of action. I mean, you can choose a comparator that you’re likely to win against, but it may not be appropriate. So how do you choose that comparator and how do you agree and what sort of evidence you present? Like if you have 10, you know,

medicines in the market that are indicated for the same condition as yours. Do you do indirect comparisons against all 10 or do you choose the most expensive or the least expensive? There are a lot of issues there and we get challenged for that. Also another common one is, excuse me, if there are any safety issues that are flagged by the FDA or the EMEA, they’ll pick up on that.

and question you about that. So you need to make sure you’ve present all the evidence relating to safety. And of course, the uncertainty around the patient uptake of the drug and the impact of that on the overall budget is a big concern as well, because if you underestimate the patients that will take the drug, then that has huge implications on the budget. So these are some of the things

at typical challenges.

Yeah, I think some of these are similar to other HTA bodies, but I think what seems to be very special for PBAC is that you don’t actually have upfront kind of pre-submission discussion with the PBAC like what’s possible in other markets, where you can actually get some guidance on what the comparator might be. What’s the, you know…

endpoints of interests are and where you possibly can get some further information on any other Yeah, that’s correct. And I think in the last, I’m not sure, I think it was five years that there is a fee for putting in a submission by the sponsored company. I’m not sure if that’s the same in other countries, but there’s a set fee that you have to pay if you put a submission in.

Yeah, I think in Germany, there’s also a fee, but I remember it’s much more kind of a nominal fee. It’s not really covering the complete expenses. At least not what I think is based on the overall…

amount of data that is actually put in. So to have that very well understood, I think that the nominal fee is not covering that.

So in terms of you talked that some of this data is actually later available to the public, so companies that come later can actually learn from it and make better judgments in terms of how to submit their dossier. How does that work? Is everything in there or what’s in this? Sure. Good question. Yes, sir.

As I mentioned before, these are called public summary documents and they’re released after the PBAC makes their recommendations. So usually it’s a few months after the decision. So there was a recent Attanacept public summary document that was released in December after the August 2017 PBAC meeting. It’s not the entire dossier. It’s an abridged version.

and there are key bits of data and key decisions in there. However, if something is commercial in confidence, the sponsor can request to have that blacked out or redacted. But it does give you an idea of what information has been presented by other companies, what approach, what comparators they’ve chosen, and how successful they were at that. So I suppose that’s our way of the discussion

you would have with your HTA body, you know, we use these public summary documents. Of course, it’s not as good as having a direct conversation, but it’s better than not having anything. Alan, you mentioned that obviously that there’s a strong need for methodology background that you need to do this task. What exactly or how would you kind of summarize?

what is needed for a statistician to take the role that you are currently doing. So what is the requirements or what would need to be the key tasks or interactions of the statistician to do a PBAC submission? Sure. Good question. Well, firstly, an understanding of the clinical data.

and sort of a creative approach of presenting information. I think there is so much information to present you in the dossiers, you have to, there are like five sections in a dossier. One of them is the clinical section which requires you to present all available evidence from not only from your trials, but all the trials from other companies.

So presenting that information, then when you’re doing the indirect comparison or head to head comparisons, what’s a good way to present that information? So knowledge of meta analysis is vital, knowledge of indirect comparisons, network meta analyses, understanding a little bit about economic models.

you know, all crucial skills. Are you involved in directly in the programming task as well to prepare the dossier or is it that you’re working then together with the medics and the programming team to get a combination? So for the clinical data, so from the randomized control trials, often working with programmers and the

with that sort of data. As far as the indirect comparisons or network meta-analysis, we do that within our team. So I do a little bit of programming, but we also have other programmers and we have tools that help us do that programming. But the nice thing is for the indirect comparisons, it’s very simple methodology.

And for the network meta-analysis, all the code is available up on the nice website. So if statisticians want to start from scratch, that’s a good place to go. Another good place to go is the Canadian health site, the CADATH website. That also has some good documentation and tools as well. And of course, tools like R and

RevMan which is used through the Cochrane collaboration are also good tools to use.

Yeah, another good resource is actually the upcoming one day PSI workshop on real-world evidence, where we will touch base actually on lots of these different topics like generalizability, indirect comparisons, what are the new approaches that we have there. So there’s a couple of new methodologies that are emerging.

also adjust for baseline differences between different studies. And so if you want to learn more about these kind of things that we just discussed, to be better prepared, I think that workshop will be a very, very good way to learn this year about this and to grow your expertise in this level. And we will actually have lots of different people from…

universities from different companies there that will work together. It will be much more like a workshop style than a series of presentations. So there will be lots of possibilities to interact and to learn and to ask the questions that you are not familiar with. Just look onto the psi.web.

29:15 homepage and check out the timing of this one-day event register. And it will be in Bad Homburg in Germany, a nice little city, very easy to reach via Frankfurt Airport. So thanks a lot, Alan, for the discussion today. Thanks a lot, Alan. Thank you. Thank you for having me and putting up with this Australian accent.

They are accents that are really worth it. That’s good to hear. And we actually didn’t hear so much your kids in the background. That was also nice. Thank you. Thank you a lot, Alan. Bye.

We thank PSI for sponsoring this show. Thanks for listening. Please visit thee to find the show notes and learn more about our podcast to boost your career as a statistician in the health sector. If you enjoyed the show, please tell your colleagues about it.

Join The Effective Statistician LinkedIn group

I want to help the community of statisticians, data scientists, programmers and other quantitative scientists to be more influential, innovative, and effective. I believe that as a community we can help our research, our regulatory and payer systems, and ultimately physicians and patients take better decisions based on better evidence.

I work to achieve a future in which everyone can access the right evidence in the right format at the right time to make sound decisions.

When my kids are sick, I want to have good evidence to discuss with the physician about the different therapy choices.

When my mother is sick, I want her to understand the evidence and being able to understand it.

When I get sick, I want to find evidence that I can trust and that helps me to have meaningful discussions with my healthcare professionals.

I want to live in a world, where the media reports correctly about medical evidence and in which society distinguishes between fake evidence and real evidence.

Let’s work together to achieve this.