What does “quality” really mean in statistical programming?

In this episode of The Effective Statistician, I speak with Andrew (Andy) York about the evolving world of programming validation, traceability, and quality assurance in clinical trials. Andy has decades of experience in statistical programming, leadership roles across pharma and CROs, and now works with AI-driven solutions focused on improving validation and traceability.

We discuss why traditional approaches to validation are becoming increasingly difficult to sustain, how expectations from regulators continue to grow, and why traceability is far more than just linking programs and datasets.

Andy also shares how modern AI-powered tools can automatically map programming workflows, connect datasets and outputs, and create end-to-end traceability from raw data to final tables, figures, and listings.

If you work with statistical programming, clinical data workflows, submissions, or validation processes, this episode will challenge some long-held assumptions and introduce you to where the future may be heading.

Why you should listen

  • Learn what “quality” in programming really means beyond simply writing working code
  • Understand the challenges of maintaining traceability across complex clinical trial workflows
  • Discover why manual validation processes are becoming harder to scale
  • Hear how AI is starting to transform validation and traceability in programming
  • Explore the balance between regulatory expectations, efficiency, and confidence in outputs
  • Gain insights from someone who has seen the evolution of statistical programming from the very beginning

Episode highlights with timestamps

  • 00:01:30 — Andy York’s journey into statistical programming
    Andy shares how he started programming during the early days of SAS in pharma and how the role of programmers evolved over the decades.
  • 00:04:41 — What does programming quality actually mean?
    We discuss confidence in outputs, customer expectations, regulatory requirements, and creating programs that your future self can still understand years later.
  • 00:06:46 — The regulator’s perspective on validation and traceability
    Andy explains why full traceability from raw data to final outputs is essential for regulatory confidence.
  • 00:08:15 — The limitations of traditional traceability approaches
    We reflect on the common experience of manually navigating folders, programs, and datasets to reconstruct programming logic.
  • 00:09:45 — How automated traceability changes the game
    Andy explains how modern tools can automatically create end-to-end traceability matrices across programs, datasets, and outputs.
  • 00:10:45 — Forward traceability vs. backward traceability
    A fascinating discussion about not only tracing outputs back to source data, but also understanding where every data point flows forward through the analysis process.

Links and Resources:

🔗 Verisian

🔗 Bayer case study

🔗 Learn more about PSI and become a member

🔗 The Effective Statistician Academy – I offer free and premium resources to help you become a more effective statistician.

🔗 My New Book: How to Be an Effective Statistician – Volume 1 – It’s packed with insights to help statisticians, data scientists, and quantitative professionals excel as leaders, collaborators, and change-makers in healthcare and medicine.

Join the Conversation:
Did you find this episode helpful? Share it with your colleagues and let me know your thoughts! Connect with me on LinkedIn and be part of the discussion.

Subscribe & Stay Updated:
Never miss an episode! Subscribe to The Effective Statistician on your favorite podcast platform and continue growing your influence as a statistician.

Never miss an episode!

Join thousends of your peers and subscribe to get our latest updates by email!

Get the shownotes of our podcast episodes plus tips and tricks to increase your impact at work to boost your career!

We won’t send you spam. Unsubscribe at any time. Powered by Kit

Learn on demand

Click on the button to see our Teachble Inc. cources.

Load content

Andrew (Andy) York

Strategic Advisor in Statistical Programming, Clinical Data Science, etc. for Clinical Research in the Pharmaceutical Industry

Andrew is an experienced SAS professional and strategic advisor specializing in statistical programming and clinical data science within the pharmaceutical industry. With 40 years of SAS experience and a background in IT and biostatistics, he brings extensive expertise in supporting clinical research, statistical programming, and data-driven processes in pharmaceutical development.

Transcript

00:00
You are listening to the Effective Statistician Podcast, the weekly podcast with Alexander Schacht and Benjamin Piske designed to help you reach your potential, great science and serve patients while having a great work-life balance.

00:21
In addition to our premium courses on the Effective Statistician Academy, we also have lots of free resources for you across all different topics within that Academy. Head over to theeffectivestatistician.com and find the Academy and much more for you to become an effective statistician.

00:49
I’m producing this podcast in association with PSI, a community dedicated to leading and promoting user statistics within the healthcare industry as a benefit of patients. Join PSI today to further develop your fantastical capabilities with access to the ever-growing video on demand content library, free registration to all PSI webinars and much, more. Head over to the PSI website at PSIweb.org.

01:17
To learn more about PSI activities, you become a PSI member too.

01:31
Welcome to another episode of the Effective Statistician. Today I’m super happy to talk to Andy York about programming. And I’m not a programmer, but of course, statisticians do program quite a lot. And when you ever worked in the pharmaceutical industry, you will soon learn about quality of programming and validation of programming. And when I first started in the industry, had

02:00
no clue about it. But since then, a couple of decades later, there’s actually quite some interesting things moving forward and some well-known assumptions are getting challenged and so there’s new ways of doing things and so this is something that we will talk about. maybe you can introduce yourself shortly where you’re coming from and what led you to where you are now.

02:27
Hi Alexander. Yeah. No great to be talking to you today. And yeah, I totally agree. I’m a statistician also by training, but I’m a statistical programmer by a specialism, if you like. So I fell into that quite quickly as a placement student when I worked for a company called Smith Klein and French, which gives you an idea of how far back I go. That’s now obviously black, so Smith Klein. And I was probably one of the first programmers in the industry because they really didn’t exist. And mostly the statisticians doing programming.

02:55
And obviously they had mainframe SaaS and they just got PC SaaS and it piqued my interest if you like in terms of that. And I knew Fortran and I knew Basic and all these old languages and yeah, I fell quite naturally into doing that. But it’s grown and grown ever since. And I think it’s grown to the point where the amount of work we have to do exceeds the number of programmers actually that we have in the world for statistical programming in the pharmaceutical industry. And that’s been a real problem, I think, over the years.

03:23
We’ve introduced standards like Cdisk STTM and Cdisk Adam, and various other things that have come along. And now we’re starting to look at my solutions to help us with what we’re doing. My career went from being hands-on up to management levels for various companies, including CEROS, as well as pharma companies. So I worked for different size companies throughout my career. Until I joined Verisian, most recently I was the vice president of clinical data science at Nova Nordisk.

03:53
which was a great role. was like an SME type role, if you like, for the business because interestingly, their programmers were fully integrated within their statistics department rather than being separate as they are in a lot of companies. there was no head of programming there as such, but I was the senior programmer, if you like, at that company. I went on FAM. I decided, well, slow down a little bit and go part-time. I set myself up as a consultant and I’m currently working as a specialist advisor for Verusian.

04:23
AI technology company in the pharmaceutical area. And it was, it’s been going for about three years and it’s set up a number of really interesting tools to do with validating statistical programming, the output and the data that goes into that and creating a traceability matrix of everything that you do. The first question I want to start with is when we talk about quality and quality of programming, what is that for you?

04:51
So for me, it’s creating your own product, the table, the figure, the listing, the statistical analysis, if you like, with absolute confidence that what you’ve created is correct to the best of your knowledge. It’s less so about the steps in between, but by inference, they also have to be a very high quality and very high standard as you go through. And it’s about having the right knowledge to craft that from a very disparate set of clinical data.

05:21
that you may have from study to study, because obviously there are a lot of indications that we work on. New indications come along all the time and a lot of different ways that we collect and record data for our clinical trials. So understanding that and how to manage that is extremely important. I think that speaks to a couple of different points. When I think about quality, it basically meets the expectations of the customer. Yeah. And now customers, you can have many different things.

05:50
customer could be your future self. So when you next time look into a program and usually a program is always changed, you can easily change that and it’s fast and easy. It meets the expectation of the statistician that you work with so that what you have actually produced is what was in the specifications. It meets also the demands of a regulator.

06:18
And the regulators have of course all different demands in terms of how they ensure that what they get is actually correct. And you can, a pharma company or as a CRO, you can demonstrate that you have a good quality in terms of you have a lot of insurance that this program does what it’s supposed to do. So

06:47
What are typical approaches to reach to this insurance for the regulator? The regulator is obviously the king of all of this in terms of who is the ultimate customer. And I think it’s just so important to have a full control and traceability from end to end of what you’ve done so that when you’ve taken a data point from raw.

07:09
Maybe you’ve mapped it into your STTM or created your analysis variable within Adam, for example, that you can trace that all the way back with full confidence, including all of the assumptions around subgroups, demographics, stratification, for example, that done within that analysis to know that it is correct. And your point about your future self is that it is really important too, because how often do regulators come back and ask questions five, 10 years down the road and say, Oh, let’s have a look at this particular adverse event. you do some?

07:37
tables based around that so that we understand what’s going on there. So you’re having to revisit that code that you wrote maybe five, 10 years ago. And the ability to understand that process is critical. And that carps back into traceability to make sure, because the chances are that programmer who wrote that originally may well have moved on in that time period. So that intellectual property around how the program was written needs to be transferable at somehow and somebody recorded somebody. So incredibly important that we can prove what we’ve done.

08:06
And that’s obviously just in the coding. can talk about validated systems as well. I can fall into all of the regulations, but that’s the given that the piece we’ve just talked about is the big must have. So let’s focus on the programming, not the system part, but yet another completely different level, which I also don’t know a lot about. When I think back of my time in terms of when I did programming, the key for traceability would be around things like footnotes.

08:35
So I would have a table and then I would look into, okay, so this footnote, that link gives me a link to some folder system somewhere. First problem that very often would be, I have no clue how to get there. Yeah, I maybe I don’t have access or maybe that’s a zero or whatever, but theoretically it’s at least linked. But if I have access and I get to a program, yeah, and then I need to open the program. And in the program, there’s hopefully,

09:04
also links into further programs. So, okay. These datasets were pulled from this one and that from that one. And there’s three different datasets pulled in from other programs. And then you go from there. And that can create quite a big tree. Yeah. In terms of where things are coming from. And it’s a pretty manual process. Not something that you do five minutes after lunch can take quite a lot of time.

09:32
So is that what we think about traceability, how it should be? Yeah, absolutely not. We need far more than that. I think the core elements of what you’re looking through are absolutely correct. This is how you do manual traceability. But one of the reasons why I loved the Verusian software and one of the reasons I joined them is because their software actually completely automates that traceability. So it builds a huge matrix of all the related components of your clinical programming.

10:02
from raw data through to table figure or listing at the end. And you could pick any point in that graph and trace backwards or forwards to see exactly how your data was derived. It will show you the source code that actually was used to derive it. It doesn’t matter which program it was in. It will run across all of your programs and understand the connectivity between the different elements in the programs. And it’ll display this pictorial so you can literally step back piece by piece to see.

10:31
The full traceability matrix of how your data got from raw to the table. And it’s just really is to me, it’s the future of understanding full traceability of code. Sorry, just to interrupt, because you just made a distinction. I just talked about backwards traceability. Yeah. You have a table and you go back to something. But you now also talked about forward traceability. So from a data point, what is actually happening from that?

11:01
And I don’t know that I have ever seen something like this. I always only have seen backwards traceability. This is absolutely correct. You can go forwards, you go backwards, you can start in the middle if you really want to and go in either direction. It’s just really so flexible in understanding. So it works almost for data managers as well as for statistical programs and even medical writers can really dig into how a particular piece of data arrived in a table.

11:28
and how if there was a derivation involved, how that derivation was actually done. It does require knowledge of programming language to understand the code that was there, but that’s actually embedded really deeply within the tool. So LessDeep is actually understanding did this particular program or this particular piece of data conform to your study level specifications, as well as the CIDAS standards as it flows through that process. And the system will automate that and say, yeah, it actually follows the rules with a Tecmo.

11:58
to it or else it needs investigation by somebody who understands what’s going on to say, yeah, actually that is correct or not. Maybe there’s a problem with the specifications or there’s a problem with the coding that was done to get there. So for that, that’s going a little bit beyond traceability, more to validation of course, but it’s, yeah, it’s something we’ve never had. I totally agree. And it’s something that’s been a very manual process. It’s quite nice use cases for that. Yeah. So just thinking about there’s some change in the data and you want to know which tables are affected.

12:27
And for certain types of data, maybe it’s pretty clear which labels are affected, but for others, maybe not on first glance. And that would make things so much easier. And you can directly show it and prove it and document it. Yeah. And it’s absolutely right that the system will do that. It will also generate what you need for submissions. So it’ll define generate the defined XML for you automatically now. And it will also generate the

12:56
things like reviewer notes as well, all based upon the traceability of your code. So you have these things, not only that you can track it back yourself manually, but actually it’s now starting to create the submission level documents that programs always create and always take a lot of time to get done. It can do it in days to minutes. I would say the goal is to be able to do this in five minutes at the moment on the test. Well, not testing the pilot study that we ran in. It took two days, I think, rather than several weeks to do.

13:24
Very good. So that helps us a lot with the traceability part, which I think is really, really important. think in the future it would be awesome to have extensive traceability beyond the clinical trial reports and the tables for that also into the more the medical affairs area. Because as soon as you have some work product that goes beyond the clinical space,

13:52
Yeah, I’m more into the marketing, medical affairs space. Very often all the traceability is completely lost. And specifically, the forward traceability was never the thing there, but even backwards traceability is really, really difficult in these big organizations. Yeah, I totally agree. That’s not an application that Marician has so far considered, but I think it’s a really good point that once your information reaches the marketing guys, you just don’t know what they’re going to do with it.

14:21
having that stamp of quality that goes forward to marketing is really critical, think, for companies because it would be quite costly to make incorrect claims about your drug and what it can do. as we’ve seen in Indian Street as a whole, think super suggestion for the future. Yeah. And there’s a lot of opportunity for cost savings, for quality improvements, for speed improvements. So let’s talk a little bit more about the validation step.

14:46
So I learned there’s a couple of different validation things you can have. Basically, he review your own code, someone else reviews your code, double programming. I’ve seen people doing triple and quadruple programming even. The CRO did double programming, but we didn’t trust them, and therefore we programmed that as well. then, because for some other things, we as a senior statistician still wanted to program it himself.

15:13
I’ve experienced that directly myself at a company I worked for in my past life. Yeah. Quadruple programming is just why. So what do you think about all that work? Is that really necessary? Yeah, we’re going full circle. I think these days we’re forced to be and go full circle because when I first started out as a programmer, there was no double programming. I actually just had a table of contents that I programmed everything from. And it was up to me how I actually designed the table. So I never even had.

15:42
mock table, or maybe a few, but not many mock tables to go with. And so I programmed it. I gave it to the status system and said, yeah, that’s good. Although this needs to change. And that was it. And then we started to work in a more structured fashion where we had a lot of specifications that needed to be produced in order to produce our outputs. And you essentially had to work power at fashion from those specifications to produce your output. And that was then with a manual.

16:07
check by another independent programmer who would go through all your SAS logs and check each data step and prop prints from every data step to make sure that the data had been moved through that process correctly. And of course, it went on from there, double independent programming came along and we started to get these different options. As you say, do you have an independent program and do you self-check, do you double program things? So it’s all adding layers and layers of inefficiency, if you like, in what we do. It’s been essential to check quality and particularly if you work for a CRO.

16:37
quality is everything because the last thing you want is a client coming back and complaining at you that you’ve done something wrong. And particularly important is the industry moved towards low cost centers in India, where you had to spend a lot of time training people who didn’t know clinical trials very well to get the quality that you needed out of those people. And of course the industry’s moved a long way in that direction since, but it’s extremely inefficient. And I think we’re the only industry in the world that would do something like this. So if you go to finance or anything like that, any bank,

17:06
You can’t tell me that they double program all of the financial reports that they do using SAS or whatever language they’re using. Even in audio, if they’re developing software, they’re not going to double program Microsoft Word or something like that to make sure that it runs correctly. So we’re really a bit of an anomaly in that sense. Coming back to the validation that’s built into what Marissa has done is it takes away that need to have two people program a piece of output because of the validation reports that it throws up.

17:36
You can go down, you can shrink maybe from two FTEs, one to program the original and one to do the QC programming to maybe one and a bit FTEs. So one to do the programming and one to check the outcome of the validation that’s been pushed through the RISC and AI to tell you whether things look correct or not. And that’s just a huge saving in terms of resource when you look at where the industry is now. And it is full circle. So we’re going back to a more efficient way of validating what you’ve done.

18:06
from a very inefficient process that’s grown up over time, if you like. And this is where AI, I think, truly starts to come into the equation and can truly help us. That removes the mundane work or the busy work that we’re forced to do to guarantee quality. But it’s doing it in a really efficient and sensible way that is still controlled by humans ultimately. And I think that’s so important that we have that degree of control or say over what the outcome is.

18:35
And regulators are basically fine with that approach. So I think, yeah, I don’t think regulators mandate that you have to do double programming. think all their mandating is that you have to have quality. So I don’t see any reason why that should be an issue to a regulator because there’s still a human in the loop and they’re still doing all of the checks that they need to do. And I think even now, if you were to do a submission, I think the FDA might require you to say, if anything’s been generated by AI, then you have to declare that.

19:06
But with this tool, you’re not actually generating anything with AI. You’re just using it to check what you’ve done. There still is that assumption that an original programmer has done the initial work to get the output in the first place. But of course that’s another step along the AI path that may be taken by some companies. Yeah, I can see that saves a lot of time, a lot of budget, and for the programmers, a lot of mundane work. don’t know. I think tricking someone else’s work is not probably the most

19:35
fancy thing. think it’s actually, it’s going to change our roles somewhat as programmers. So I think the days where you can just sit in the corner and follow your specification and turn out programs to do your tables are probably over. I think what we tend to become now more is data scientists and people who understand the data, understand the quality of what you’re producing. And really we’re not interested in just frantically following the specifications to get to an output.

20:05
That may or may not be correct because there’s no guarantee with that will program in a course that it’s correct. You may both have followed the specific coaches incorrectly. So yeah, I think it really does change the landscape and I think it takes a lot of pressure off of statistical programmers who spend an awful lot of time doing other things as well. They’re not just programming. They’re in meetings discussing the data and how to meet the timelines and things like that these days, as well as considering how to.

20:32
things like TSURs, PSURs, what you name it. There’s so many different things that we do today. And this starts to alleviate some of the resource issues that you have in the industry. The other thing is about removing jobs or replacing with AI. It’s actually using what you have better in a more efficient way as you go forward. Awesome. Thanks so much. That was a very, very insightful episode about where we have came from in terms of programming, how our demand for quality has

21:00
evolved into a lot of specifications, a lot of additional work. And what we now can do in terms of technology to make things much more transparent, traceable, higher quality with removing all the mundane work and what that has implications for the future of programming. Any last idea, thought that you would like the listener to leave with?

21:29
I think the main thing I’ve been doing this for almost 40 years, I like to look in a crystal ball and think about where we’re going in terms of how we present our results to the regulators. you know, just would love, instead of presenting them with this realm of static information that we start to think more about, giving them more dynamic tools so that we can say, well, you know, here’s our interpretation of the data. Here’s a tool with everything in it. See if you agree and move to a very different style of approval of what we’ve done.

21:58
And I hope that comes in for the next five, 10 years. With technology, think that the advancements that we see. Yes. Basically our biggest transfer was that we don’t ship piles of paper folders, but we do the same in a PDF, but with hyperlinks. That’s the biggest advancement that we had is sometimes my feeling. Of course there’s a lot of things going on behind it, but yeah, there’s more interactive things. There’s so much opportunity. Exactly. Thanks so much, Andy.

22:27
Great to have you and I have the feeling this will not be the last time that we talked. And if you want to talk to Andy, you can check him out on LinkedIn. And there you can also find all the details about the company, Ferrisian, that we talked about. Thank you, Alexander. It’s been a pleasure talking to you as ever. So, bye.

22:52
This show was created in association with GSI. Thanks to Rain and her team, Afeevias, who helped position the background and see you from the knee. Richard Potential lead great summons to self-features. Just be an effective statistician.

Join The Effective Statistician LinkedIn group

I want to help the community of statisticians, data scientists, programmers and other quantitative scientists to be more influential, innovative, and effective. I believe that as a community we can help our research, our regulatory and payer systems, and ultimately physicians and patients take better decisions based on better evidence.

I work to achieve a future in which everyone can access the right evidence in the right format at the right time to make sound decisions.

When my kids are sick, I want to have good evidence to discuss with the physician about the different therapy choices.

When my mother is sick, I want her to understand the evidence and being able to understand it.

When I get sick, I want to find evidence that I can trust and that helps me to have meaningful discussions with my healthcare professionals.

I want to live in a world, where the media reports correctly about medical evidence and in which society distinguishes between fake evidence and real evidence.

Let’s work together to achieve this.