When should we move beyond Excel for health technology assessment?
I sit down with Dr. Robert Smith to explore why increasingly complex health economic models call for a different approach—and how R can help.
We discuss the challenges of reviewing large spreadsheets, R’s strengths for simulation modelling, and how the R for HTA community shares tools and knowledge. Robert also explains how generative AI speeds up coding while making domain expertise, clear problem definition, and human judgement even more valuable. We close with practical ways to learn, contribute, and get involved in open-source work.
Why you should listen
If you work in statistics, health economics, or HTA, this conversation will help you assess where R could support your work. Together, we explore how to:
- Recognise when a model’s complexity calls for a move beyond spreadsheets.
- Connect statistical analyses and economic modelling within R.
- Learn from an open-source community across academia, consultancies, and industry.
- Use generative AI while keeping human judgement and domain expertise central.
- Support junior colleagues as AI takes over more routine coding.
Episode highlights with timestamps
- 02:25 — How Robert found his way into R and health economics
- 03:54 — Why move beyond Excel?
- 06:15 — The challenge of reviewing and maintaining spreadsheet models
- 08:40 — Why R rather than Python?
- 11:29 — What the R for HTA consortium brings together
- 13:25 — Where the modelling work goes deeper
- 15:47 — How to learn and get involved
- 17:39 — Why industry supports open-source methods
- 19:29 — How consultancies adapt their services
- 20:51 — How generative AI changes model development
- 23:17 — Why human judgement still drives the work
- 25:08 — Start with a small contribution
Links and Resources:
🔗 R for Health Economics courses
🔗 Structured Expert Elicitation Platform
🔗 Learn more about PSI and become a member
🔗 The Effective Statistician Academy – I offer free and premium resources to help you become a more effective statistician.
🔗 My New Book: How to Be an Effective Statistician – Volume 1 – It’s packed with insights to help statisticians, data scientists, and quantitative professionals excel as leaders, collaborators, and change-makers in healthcare and medicine.
Join the Conversation:
Did you find this episode helpful? Share it with your colleagues and let me know your thoughts! Connect with me on LinkedIn and be part of the discussion.
Subscribe & Stay Updated:
Never miss an episode! Subscribe to The Effective Statistician on your favorite podcast platform and continue growing your influence as a statistician.
Never miss an episode!
Join thousends of your peers and subscribe to get our latest updates by email!
Get the





Learn on demand
Click on the button to see our Teachble Inc. cources.
Featured courses
Click on the button to see our Teachble Inc. cources.

Dr. Robert Smith
Director at Dark Peak Analytics | Chief Scientific Officer at Initiate
Dr Robert Smith is a health economist, data scientist, and the founder of Dark Peak Analytics. He holds a PhD from the University of Sheffield, and has previously held roles at the Joint Biosecurity Center and the UK Health Security Center, as well as advisory positions for the WHO and as a consultant to a range of pharmaceutical companies, universities and governments. As a co-director of the R for HTA consortium, Rob prominently advocates for using open-source statistical software. His company builds bespoke health economics models, primarily in R and C++, and industry software, including the specialized SEEplatform which elicits subjective probability distributions from experts. Rob also leads Dark Peak’s online R for Health Economics Courses which help analysts build robust, reproducible cost-effectiveness models. In his spare time—a rarity with three children—he likes trail running in the Peak District.
Transcript
00:00
You are listening to the Effective Statistician Podcast. The weekly podcast with Alexander Schacht and Benjamin Piske designed to help you reach your potential, great science and serve patients while having a great work-life balance.
00:21
In addition to our premium courses on the Effective Statistician Academy, we also have lots of free resources for you across all kind of different topics within that Academy. Head over to theeffectivestatistician.com and find the Academy and much more for you to become an effective statistician.
00:49
I’m producing this podcast in association with PSI, a community dedicated to leading and promoting user statistics within the healthcare industry with the benefit of patients. Join PSI today to further develop your statistical capabilities with access to the ever-growing video-on-demand content library, free registration for all PSI webinars, much, more. Head over to the PSI website at PSIweb.org.
01:18
to learn more about PSI
01:30
Welcome to another episode of the Effective Statistician. It’s in the middle of the summer here and it’s going to be really, really hot. And we are talking about two topics that are really hot as well. And combined they are even hotter. All has been quite a big topic for a long time in the industry and continues to be one. And of course, the other topic is HTA, which…
01:59
all the changes in terms of the European JCA process. There is so many things going on. So we’ll talk about exactly that. And for this, am very, very happy to have Robert Smith here on the line. He has his own company and you can tell a little bit more about this in a moment, but he is very deeply into this topic.
02:25
So Robert, before we go into R and HTA, maybe you can introduce yourself a little bit and how you came to get connected with this topic. Yeah, sure. And thanks for having me. great to be on the podcast. So I first got involved in the application of our Health Economics during a PhD at the Shahr at the University of Sheffield. I’d previously worked in health consulting in universities and got very involved in the modeling of public health interventions.
02:55
at a shah. They’re kind of renowned for alcohol, smoking, physical activity, modelling. And because of the nature of these models, they tend to be substantially more complex than a lot of the pharmaceutical based models that we see for submissions to HTA agencies. And so the modelling structures tend to be slightly more complex with capturing things like human behaviour. And there’s a lot of kind of feedback loops and individual level effects, which means that the models end up being
03:24
substantially more computationally burdensome and so tend to be built in programming languages. And so during my PhD I specialised in building out these types of models and realised that there were lot of other benefits other than just computational time that could be applied to models for HTA and health economics more specifically. And so got very involved with a group called the Arthur HTA Consortium and the work that they’re doing to try and make a lot of these methods more open, transparent and widely used.
03:54
and found that group to be incredibly welcoming and a great sort of industry group to be involved in. So that’s how I’ve come to be involved in that. And the interesting piece is when I think about health economics models, I primarily think about Excel. Yeah. When all the models that I have seen were usually Excel based, why are instead of Excel? So I think what’s happened over time is that
04:19
We’ve moved from a system or a situation where a lot of the models were very simple. So if we think back 10, 20 years, there were some very simple calculations being made with some assumptions around quality of life and longevity from these various interventions. And these are originally almost borderline back of an envelope calculations that then became one or two tabs of an Excel spreadsheet with some calculations and fits to curves.
04:47
And then they gradually introduced within the models things like one-way sensitivity analysis and various other components, which meant that you suddenly had lots of different tabs and sub-analyses running within the model. And then we introduced probabilistic sensitivity analysis and then expected value of information. And so we’ve had this kind of gradual shift from what was a very simple set of calculations towards things like now for submissions to CDA, one-way sensitivity analysis on the probabilistic model, which changed the kind of
05:17
level of computational burden and complexity of the models from being a simple calculation and we’ve slap-walked with the same piece of software that isn’t really quite fit for purpose and people are having to hack solutions to get models to run in Excel. But that’s not to say Excel doesn’t have a place and I think we’ve tried to be open in the group that there are lots of times where Excel is absolutely fine. It’s just that as you start moving towards models which are trying to capture
05:45
individual level effects or interactions between groups of people or any other level of complexity. It just becomes almost, it becomes very frustrating to work with a tool that is absolutely fantastic for analysis of financial data or spreadsheet software for accountancy. It’s fantastic, but it’s not really designed to be a piece of software to run a micro simulation model. It’s not really what Excel is built for. Yeah. Honestly, when I think about that.
06:15
Excel has so many different features that make me go crazy. If you have just a simpler thing, moving from a US setting to a German setting and all kinds of different things break down because of the dot and the comma. Or you maybe accidentally changed something somewhere, a cell, and it’s impossible to debug that or to put things somehow in a development, in a production form. That’s also not possible.
06:45
So I always, in terms of Excel, I’m always super anxious about these kinds of things. Yeah. And I think when you end up working now or inherit a model that’s built in Excel from somebody else, say you’re doing a country adaptation or you’re developing on some element of the model, you inherit this thing, which is just a huge number of tabs, the sheer volume of cells, most of which do exactly the same thing as each other with the same equation, just with a drag down, but hundreds of thousands of cells.
07:15
that nobody could possibly go through and review the whole thing. It takes absolutely forever. And each cell often will contain quite a large equation that is also almost impossible to decipher what that equation does. And so there’s a huge amount of time being spent by health economists kind of gradually working through cells in Excel. And this is not how other industries work. So it’s an incredibly manual process and it’s not like Uber doesn’t send out.
07:42
million different Excel files for the location of its drivers or the routes to certain places. It’s just not a modern approach to this problem. so really quite often people are using Excel because it’s the status quo as opposed to the best tool for the job. And where we really see the shift is where suddenly it completely falls flat. I think people are very willing to go through the pain because it’s the status quo and they know that other people are used to receiving models in that way. And so they’ll just continue along because
08:12
We’re all very averse to change unless you absolutely have to. But where we’re seeing the switches initially is for things like discrete event simulations, micro simulations, anything where you need to do value information analysis. That’s the first stuff to shift over to our infectious disease modeling is again, anywhere where there’s interactions between individuals or groups of individuals or resource sharing is shifted over to our, and I’m sure we’ll touch on gen AI because it’s the big topic at the moment, but that’s also.
08:40
really accelerated the shift in the last probably two years, three years, we’ve really noticed a ramp up in engagement with script-based programming. Let’s talk a little bit about why did we go for R and not another open source language. That’s a really good question and one we get a lot in the group. So it’s why not Python? And the first thing to say is I think the R for HGA group, John Luca might be angry at me for saying, I think it was a mistake when it was first named.
09:08
I think ideally it would be open source modeling group with the idea being that we should shift from Excel from spreadsheets to some script based approach. And I don’t think it really matters too much what the language is. So sometimes people email saying, can I submit this talk? It’s in Python. We’re like, yeah, go ahead. Let’s like, we’re all very interested because really the jump is really from Excel from spreadsheets to a script based approach. And anyone who’s very well versed and can code.
09:37
an advanced user of R or Python will find it quite easy to review the other. So we could pick up a model in Python and be able to work through it very quickly. So why R when the original kind of choice is made, why is the industry gravitating towards R and not Python? I think a lot of that is because historically a lot of the software packages that have been built and validated for survival analysis and the earlier, if you think of the pipeline from the clinical side through to the health economics, a lot of that
10:06
that survival analysis and analyses that done on trial data, there’s some fantastic packages, FlexServe and others that know kind of the R ecosystem will know that those are widely accepted and if you use those R packages no one really questions them, the quality of the software. And so because of that there’s been a natural tendency if you’re doing your survival analysis in R, it kind of makes sense just to continue through to build the model in R as well.
10:32
And so that I think is the reason. And I think I will continue to dominate just because of that legacy in the same way that Excel has remained around. So I suppose that our kind of group will benefit from that in much the same way as the Excel kind of camp have benefited from the legacy effect. We’re seeing certainly on the clinical space with things like Adam data sets and the FDA approving submissions in R. So we’re seeing that earlier on in the clinical pipeline and it’ll be interesting to see how that flows through.
11:01
because if there’s a movement from SAS to R, then perhaps that benefits further down the pipeline, a movement to R for health economic components as well. think it’s also something to do with how many people in our industry already use R. Having validated environments for R, all these kind of different things makes things easier. In terms of this whole group for HTA, maybe you can talk a little bit to words.
11:29
what’s happening in this group and especially also to what’s the goal of this group. Yes, I think the overall goal of the group is to make or propagate the use of R for HTA. So ranging from kind of survival analysis to model builds to submissions and the practical implications of submissions in R to HTA bodies. And it’s primarily an academic endeavor with many of the key individuals being based at universities either.
11:57
as academic positions, but also in ERGs and there’s involvement from key individuals in HTA bodies and across industry. So it’s quite a nice mix. think we did a poll in the last workshop that we ran. We run an annual workshop and we did a poll and I think we were operating around a third being kind of academic ERG, HTA kind of space, a third consultancies and a third industry. So it’s actually a really nice mix between the different stakeholders. The group.
12:27
really is coming together to try and push the use of open source programming languages in health economics. And it’s gone a long way to providing a kind of platform for people to publicize the work that they’re doing, any open source work they’re doing and any learning that they have around how they’ve navigated working with industry or within HTA bodies or ERGs in the R or as I say, Python programming languages. And I think the thing I’ve been most impressed with is how
12:57
how helpful the group have been to each other. So the willingness to publish things open source, the willingness to provide tutorial papers to others to really push these methods along is very impressive relative to other industries. So I think there’s a really good spirit of, I guess, that open source spirit of being very transparent about what you’re doing in the methods. And I guess they’re the type of people that are interested in that. And so then they embody that as well in what they’re doing.
13:25
being concentrated on so far. So if I understand it, most of your aspects has been around economic models and these kinds of things. A real mix. there’s been presentations and I’d really encourage people to go and have a look at the website, ArthurHTA website. I’m sure we can link in the comments. So it was originally, I believe, set up by Gianluca Bello and Howard Thong from UCL in Bristol, and now has a wide pool of the scientific committee who all have their own expertise. So ranging from
13:54
NMAs and more of the stats analysis through to modeling, through to value of information. So everybody I suppose comes at it with a slightly different background and expertise. I’d say the area which has the most focus has probably been the modeling side. So there’s a lot of talks at this year’s event around discrete event simulation, microsimulation modeling, and also a lot around kind of statistical analysis, makes, NMAs, and then some around value of information.
14:23
But think the area that a lot of people are particularly interested in is the modelling side. I think that goes back to the question where Excel is just not very well suited for running discrete event simulations and individual level simulations. And as I mentioned before, as often people will build a model and it’s quite simple and then they want that model to answer a question that is slightly more complex, that maybe incorporates individual treatment adherence or something.
14:48
And then there’s this shift towards individual level model, a micro simulation or a discrete simulation. And I think that’s where we’re seeing a huge amount of interest. And a lot of the presentations this year were presentations of an interesting kind of R package to do DES. So the Warden package was presented and then a number of different talks where people presented models that they’d built. This is so cool. And as you say, network meta-analysis, matching and direct comparisons, these kinds of things are of course not just helpful for an HTA submission.
15:18
They’re also helpful for all kinds of other areas. If you want to design a study or if you want to do submissions to FDA, email, there’s so many other aspects. Have you ever worked on something related to more the German submissions? I’ve not. No, unfortunately I don’t have that experience. I know that we’ve had some interaction with Equik on developing our skills. And so it’s something that I think they’re interested in, but I don’t personally work in the German market.
15:47
I’d be really interested to know what the scene is now. So if you are as a listener now interested, that sounds fancy and my company would really love to get engaged or maybe I personally want to get engaged in that work. How does that work? Yeah. So think if you log in and look at the RFA HTA website, it’s probably the best protocol and on there’s lots of training materials. So there’s lots of links to upcoming open source, open access, academic papers to read.
16:17
There’s the annual workshop that we invite people to, and there’s generally another hackathon session as well in the year at some point. And that’s really fun. I really enjoy that. think people that come to that find that really engaging. So they split off into small groups and try and solve some interesting problem and learn from each other. And then there’s also kind of bespoke training sessions that are run for groups from often from HTA bodies or ERGs. So there’s the Arthur HTA kind of consortium group.
16:44
And then I also run a consulting company, Darkpeak Analytics, that specialises in these methods. And we do a lot of consulting work into industry and, and governments on these methods. And we have our own training courses as well, some of which are linked from the R4HTA site as well. So definitely worth checking those out, but it’s obviously great to promote my own stuff where I can. I would say there’s also just a lot of material out there. So we also signpost to free resources and free materials and…
17:11
generally suggest people look at those and have a look first and see whether the courses that we have online are worth it for them. Because often, might not be worth it for you. Or you might find that you read some of those open access papers and then you get even more interested and then you’d like to come on a course and that’s fantastic. yeah. This is so cool. Whenever you want to build a model or you want to have some kind of R-based things built for you, it’s a great way to reach out to
17:39
roller dentists, colleagues at DarkPixl Analytics to help you with these kinds of things. And if you’re not in the HUR area, I recommend them to your friends in that space, especially the economic people. I’m sure there’s a lot of things you can benefit from. And the nice thing is you only pay once and then it’s public and then you can reuse it again and again. Yeah, exactly. There’s a lot of interesting work out there. I think what I’ve been
18:08
really impressed by is the willingness of industry to sponsor a lot of work that gets made open source. And I think some people are often surprised by how much open source work is sponsored from industry. Sometimes there’s a kind of belief that the industry are keeping everything closed and hidden. And actually often a lot of the methodological work is in the industry’s interest to make that open source, to improve the quality of it and improve the robustness.
18:34
because they’re not trying to protect the IP of statistical analysis. They’re trying to protect the IP of their product. And industry is often actually really pro making some things open source, which is great. And in the end, if you want to have something that is understood and believed by your stakeholders, so by the payers and the decision makers on the HTA side, and having something that is open source and that can be critiqued by everybody.
19:00
is much easier than something that is kind of black box somewhere. Yeah, I couldn’t agree more. And this is so important. And it’s really nice to see this kind of happening gradually. I think from my understanding anyways, from an industry perspective, the least amount of friction going through the process, the better. And I think that open source and transparency on the methods reduces that friction and just makes that process smoother. And so I think it’s in everybody’s interest to make that smoother. I think the IP
19:29
on the modeling side probably sits more with the consultancies. And I think that the kind of industries may be slightly less concerned with that. Yeah, I think for consultancies, it’s really kind of a mind shift thing. Certain consultancies are completely outbased and they embrace that. And then there’s others that are more coming from the SaaS experience. That’s a completely different business model. Yeah, definitely. And we’re definitely seeing a shift in consultancies getting ready for this move. So…
19:58
Yeah, we’ve seen ERGs, obviously the universities are quite well set up and some of them are teaching a lot of our own masters programs. So they were quite quick to move. And now, yeah, we’ve definitely seen consultancies taking interest because they want to make sure that they’re ready in case there’s more demand from industry for these types of models. So we’ve seen a lot around in particular Shiny apps because they look very nice and they’re fancy. So there’s a lot of kind of development of Shiny apps within consultancy for industry.
20:23
and a lot of starting to build out kind of internal open source packaging. So we do a lot of review work for other consultancies. So we’re reviewing some of these internal packages that have been built, which is really good to see. Yeah. So there’s definitely progress there. But it’s very mixed. As you say, some consultancies are just not really engaging with it at all. And others are starting to jump in because it’s a nice kind of additional service they can offer and a value add, especially around some of the sort of bolt on stuff around automating our reports and Shiny apps. And now with the use of AI, I think they’re seeing that.
20:51
the development time on the model build is much lower in R than Excel. Just because of the way that the generative AI works, it can write code very efficiently. Whereas a generative AI tool understanding the relationship between a cell and then the header above or to the side, or it’s very complicated and actually surprisingly tricky for a generative AI tool to really understand what the value in the cell actually corresponds to. Whereas as a human, you look at it and go, oh yeah, that’s the cost for this. But often if there’s missing
21:20
There’s gaps between the cells and there’s other text included and as a human, yeah, I need to ignore that. That doesn’t relate to this. Whereas obviously a Gentivere tool, it’s much easier for it to misinterpret something or not have the full context that you just know implicitly. Yeah. Gen.ai tools were written by people who are really good on programming. That’s true. They will never touch Excel. They are trained on essentially GitHub, which is basically a load of open source coding. they’re for tasks like…
21:49
building out websites or doing analyses and visualizations from data. They are incredibly good because that’s what they’re trained on. They’re not trained on, there’s not that many health economic model Excel models that are in the public literature that they’re being trained on. Like it’s just not something they’re used for. And probably the developers said like, why would I code something in Excel? is why. Yeah, exactly. Exactly. But yeah, so we’ve already seen and we’ve noticed over the last couple of years.
22:18
we run these courses as I mentioned and we have a large number of delegates going through those courses every year and when we first started running them a lot of the questions were very simple questions like I’m learning to code, I can’t get this line of code to run and the response was ah you’re using the wrong type of bracket or you’ve missed a comma or something. These basic errors we get none of those questions now because the first thing you do is just ask chatgbt or people I’ve now clawed built into their IDs for coding and so nobody makes any of those mistakes.
22:48
The type of questions now we have are very specific, nuanced questions for health economics, for which these LLMs are not, they’re not trained to be specifically health economics modeling experts. So. The whole thing moves much more to the specific knowledge, domain knowledge, understanding, communication, and it’s not so much kind of in the pure execution of programming anymore, I think. Yeah, exactly.
23:17
So if you think of the steps for conceptualizing the problem with a client and then understanding how do you take that conceptualization of that conceptual model, move it into a mathematical model, and then move it into a piece of code that replicates that model or builds that model out into a programming language. That step can now be done very well by an NLM with supervision. But really the scale is how do you take the understanding of the problem and convey that into such a way that a computer or an NLM can build it out.
23:47
And I think there’s people that are trying to make the jump to take sort of the client’s ideas and map them directly into a model. And that’s a big ask. Whereas I think having a human be the interaction between the need for the model and the conceptualization of the model and then work with an LLM to code it. I think that’s very possible. And that’s what we’re seeing development time on models for quite rapidly, but you still need that human. And my concern, that is a very high end skill.
24:16
And that’s the high end skill that’s generally done by more senior members of the team who are running projects. And so it may be some of the day-to-day coding that was done by juniors is now being automated away. But we need those junior team members to develop the skills to become seniors. And so it’s a real risk that we’re not necessarily training the next generation or the next generation is kind of very much reduced because we’re automating out part of the work that they would have otherwise done while building up skills. Thanks so much.
24:43
We had a really nice ride here through all kinds of different topics. Open source, HTA, what’s happening in that space, the whole kind of group of R and in HTA, what’s happening on that side, all the trainings that are developed, the regular yearly conferences, catch-ups, and also how the overall field is developing.
25:08
which I think is really, really interesting up to even how generative AI is now included in the overall workspace and where most of the value is probably at. If you want to leave the listener with one key lesson, what would be it? I think it depends on their background, which I think being willing to get involved in some of the open source work that is going on and coming to say hi.
25:35
And just saying, you know, what could I get involved in as a small side project? I think is, goes a long way to some kind of moving into a new interesting field and people are incredibly friendly and generous with their time in our group. So I think being willing to put your neck out there and say, I, and Kenneth, what can I get involved in? This looks really cool. think that’s really appreciated. And people have been incredible at doing that. please feel free. Awesome. Very good. Thanks so much, Robert. You’ll find all the links.
26:03
to what we talked about in the show notes, so R and HTA and his company and so on. So just check out the effectivestatistician.com.
26:19
This show was created in association with Kielcei. Thanks to Rain, well positioned in the background and thank you for listening. Read your potential, leap rate signs and certifications. Just be an effective statistician.
Join The Effective Statistician LinkedIn group
This group was set up to help each other to become more effective statisticians. We’ll run challenges in this group, e.g. around writing abstracts for conferences or other projects. I’ll also post into this group further content.
I want to help the community of statisticians, data scientists, programmers and other quantitative scientists to be more influential, innovative, and effective. I believe that as a community we can help our research, our regulatory and payer systems, and ultimately physicians and patients take better decisions based on better evidence.
I work to achieve a future in which everyone can access the right evidence in the right format at the right time to make sound decisions.
When my kids are sick, I want to have good evidence to discuss with the physician about the different therapy choices.
When my mother is sick, I want her to understand the evidence and being able to understand it.
When I get sick, I want to find evidence that I can trust and that helps me to have meaningful discussions with my healthcare professionals.
I want to live in a world, where the media reports correctly about medical evidence and in which society distinguishes between fake evidence and real evidence.
Let’s work together to achieve this.




