Have you ever wondered how we make crucial decisions in the early phases of clinical trials? 

How can a Bayesian framework enhance these decisions?

Today, I talk with Audrey Yeo from Roche to answer these questions. Fresh from the PSI conference, Audrey, a seasoned statistical software engineer, introduces us to an innovative R package designed for early-phase clinical trials.

This tool promises to revolutionize decision-making with its Bayesian approach.

Join us as we explore the development, features, and impact of this groundbreaking tool, and discover the collaborative efforts that drive its evolution.

Key points:

  • Bayesian Framework
  • PSI Conference with Audrey
  • Early-Phase Trials on binary endpoints for decision-making.
  • Commitment to open-source development and collaboration.
  • Clinical Decision Process
  • Binary Endpoints
  • Oncology Focus
  • User Experience
  • Collaboration
  • Efficiency Gain
  • Software Engineering
  • User Feedback
  • Community Involvement

As we wrap up, it’s clear that better decision-making in early-phase clinical trials is not just a necessity but a transformative step for the industry. Audrey’s innovative R package, grounded in a Bayesian framework, exemplifies how advanced statistical tools can drive efficiency and accuracy. 

Don’t miss out on this deep dive into cutting-edge statistical software. Listen to the full episode to learn more, and be sure to share it with your colleagues who are passionate about improving clinical trial methodologies. Together, we can drive the future of effective statistics.

Transform Your Career at The Effective Statistician Conference 2024!

  • Exceptional Speakers: Insights from leaders in statistics.
  • Networking: Connect with peers and experts.
  • Interactive Workshops: Hands-on learning experiences with Q&A.
  • Free Access: Selected presentations and networking.
  • All Access Pass: Comprehensive experience with recordings and workshops.
Register now! Register now!

Never miss an episode!

Join thousends of your peers and subscribe to get our latest updates by email!

Get the shownotes of our podcast episodes plus tips and tricks to increase your impact at work to boost your career!

We won’t send you spam. Unsubscribe at any time. Powered by ConvertKit

Learn on demand

Click on the button to see our Teachble Inc. cources.

Load content

Audrey Yeo

Statistician | Software Engineering | Life Sciences

Audrey Yeo is a Biostatistician and Statistical Software Engineer in Pharma. Together with the statistician engineering team at F. Hoffman La-Roche, they have created a state of art engineering tool to enhance decision making for early development. Audrey’s background in pharma spans oncology, hematology and neuroscience rare disease areas and enjoys creating software and contributing to conversations about the development of early phase trials. Audrey graduated from the University of Zurich with a MSc Biostatistics in 2020 and is also a Registered Nurse with specializations in pediatrics and oncology and has worked in the pediatric COVID ICU in her home country of Australia. Personal website and phase1b slides can be found here : https://audreyyeoch.github.io/ 


R-Tool For Bayesian Statistics in Early Phase

[00:00:00] Alexander: Welcome to another episode of the Effective Statistician. Today I’m talking with Audrey from Roche. How are you doing, Audrey? 

[00:00:10] Audrey: I’m doing really well. Thank you. How are you, Alexander? 

[00:00:13] Alexander: Very good. We are recording this right after the PSI conference, which we both attended. [00:00:20] Awesome event. How did you feel about that conference?

[00:00:24] Audrey: Very smoothly run. I think the topics were so interesting, and I think it progressed a lot of conversations we would not have had because it would just be offline or online if you like so productive as well. I’m very impressed. [00:00:40] 

[00:00:40] Alexander: Okay, cool. So, before we dive a little bit deeper into the topic of today, maybe people who don’t know you, you can introduce yourself.

[00:00:50] Audrey: Oh, absolutely. I’m Audrey Yeo. I’ve got three years experience in pharma. I am also a statistical software engineer. I’ve made a [00:01:00] package which we’ll talk about today as a background. I’m a registered nurse from Australia. I’ve worked in oncology, adult care pediatrics, and four years ago between my master’s and starting at Roche, I worked in the COVID ICU in my old hospital because they let me.

[00:01:18] And I’ve, I’ve had a [00:01:20] ball so far both in my nursing career and my pharma career as a statistician and a statistical software engineer. 

[00:01:29] Alexander: Wow. Having worked in a COVID ICU, that must be no. interesting experience. 

[00:01:36] Audrey: I think I can admire the comradery [00:01:40] and the staff there. I pick up a lot of good skills and good experience of what great teams look like.

[00:01:49] So I’m looking forward to it. Yeah. 

[00:01:55] Alexander: Great teams make a huge difference and pretty much [00:02:00] all that we do is teamwork. And I just talked recently to some people in my leadership program about teamwork and what creates great teamwork. And one of the common denominators has always said. It’s very satisfying, very rewarding and often fun [00:02:20] to work in great teams.

[00:02:22] Audrey: Absolutely. I can attest to that. 

[00:02:24] Alexander: Yeah. So let’s talk a little bit more about your tool and how that came about. So it is for better decision making in early development. So. [00:02:40] What are the biggest challenges that we face in terms of decision making in early development? 

[00:02:46] Audrey: Yeah, that’s a really great question.

[00:02:49] This tool uses a Bayesian framework and it looks only at binary endpoints. What’s really great about the Bayesian framework is that it mimics a clinical [00:03:00] decision process. By using prior data. So as a clinician myself, I can attest to, you know, updating myself on the clinical decisions I make.

[00:03:09] And that’s what I like about this Brazian approach. 

[00:03:12] Alexander: So what kind of decisions. Are we talking about here? 

[00:03:18] Audrey: Yeah, that’s a good [00:03:20] question. So we, I think it’s very specific to early development statisticians where we are monitoring the data continuously. And at any point, the program or the molecule can be terminated, because it’s doing so well and it can go straight to phase three expanded cohorts or it should be [00:03:40] terminated early because of utility reasons.

[00:03:42] Thank you. 

[00:03:43] Alexander: Is that only for the oncology area or is that for other areas as well? 

[00:03:48] Audrey: Yeah. So this, this package looks at binary endpoints and we will use, we use response rates, but we also use the beta prior. [00:04:00] And if that suits any other therapeutic area the Phase 1B package, that’s what it’s called, is not restricted to oncology examples.

[00:04:09] I do not have any other examples for other therapeutic areas. And since there are so many oncology drugs on trials, on investigation in the early [00:04:20] phase, It’s not just at the Russian Genentech organization. I feel this could be very useful to other organizations as well. And it is open source, so we’re very happy to share, share that.

[00:04:32] Yeah, that 

[00:04:33] Alexander: is actually one of the great things about Roche. That’s a really committed to developing [00:04:40] everything open source. So people can benefit from it. Outside of Roche as well. Kasper Rufibach just talked about this at the PSI conference as well, and mentioned that this offers a lot of opportunities for the community.

[00:04:59] But also [00:05:00] for Roche itself. So that’s really, really great. Now let’s talk about the properties that we need for better decision making. What, what do we care about within this area? 

[00:05:14] Audrey: Definitely. Good question. I also want to acknowledge that this package [00:05:20] has been used by the Russian Genentech organization for the last 10 years, and it has changed a bit due to the questions that was raised internally by two large organizations.

[00:05:32] And it was led primarily by Daniel Sabin and Bovet. And currently, he is my [00:05:40] engineering lead. I started last year to refactor, rename, and add unit and integration testing. The last two are features of today’s state of the art software engineering practice, which means not just, you know, It’s not, it’s not only state of art, it’s also open source and, and it can always get better. And [00:06:00] that’s what I want to talk to your point about open source, if I may. 

[00:06:03] Alexander: Yeah, sure. 

[00:06:04] Audrey: It attracts, well, I, I’m hoping it attracts a diverse audience. To contribute the issues list is is open. People can add issues of Oh, I wish this feature was there. Please make that [00:06:20] and also collaborate. So I hope that with this openness to iteration that it will only get better, including including other endpoints.

[00:06:30] And so how, how can we make better decisions? Or what does this tool package this tool offers? I, I like that it’s at your fingertips of the [00:06:40] study statistician. who in the early oncology study, the dynamic is very fast. Things can change very fast. Molecules can be terminated just the next day. So there’s a lot of efficiency gain to be had.

[00:06:54] And what I try to do there is make documentation as accessible as [00:07:00] possible. We know statisticians are diverse. We are working in a neurodiverse atmosphere. So I try to make the documentation Readable and come from the user perspective. And I also have peers who helped me do that. For example, I discussed design [00:07:20] questions with my engineering lead and also from peers to understand if they would see it the same way.

[00:07:29] And how do we make this too? Sorry, you want to go ahead. 

[00:07:31] Alexander: No, no, go ahead. 

[00:07:33] Audrey: One more thing I want to add to that is how do we make this a better decision making tool is that [00:07:40] I might be the lead developer of this tool, but I don’t have all the clinical trial experiences in early development that others have and can add value to this package by saying, well, it doesn’t speak to me like this. That’s what I wanted to say. 

[00:07:55] Alexander: Yeah, I think. Developing these tools always [00:08:00] connecting with a user is really, really, really important to make sure you have a great user experience. What, what do you do to, or what actually is a great user experience? 

[00:08:17] Audrey: Yeah, wow. I’m so happy to get [00:08:20] this question as a statistician slash, you know, I’m making statistical software, but what makes it a good user experience?

[00:08:28] I think, like you said, feeling like the creators had known what you were looking for, being very connected to the user. And I think that is a [00:08:40] relationship over time. For example, engaging the user in small seminars having, having a possibility to To interact with the users through, I guess, now with it being open source, getting feedback the user, our conference in Salzburg is going to help with that then [00:09:00] understanding the product and us understanding their problem.

[00:09:05] So I’m a clinical early clinical trial statistician myself. So I hope I can lend that experience to, to the future user. 

[00:09:14] Alexander: You mentioned you can submit issues to basically make [00:09:20] suggestions, improvements, things like that. How does that exactly work? 

[00:09:25] Audrey: I think anyone with a GitHub account can log in, go to the phase 1b GitHub repository, go on the issues tab, and submit an issue.

[00:09:39] It is [00:09:40] really that simple. I guess the hurdle could be getting a GitHub account. I would hope that with this open source movement that we would all have one, even if it was the one attached to our organization. So I have a private one, which I work on 

[00:09:59] Alexander: with 

[00:09:59] Audrey: this Phase [00:10:00] 1 BA. 

[00:10:01] Alexander: How do you recognize these kinds of issues? So, in terms of how can you make sure that you understand the issue correctly? 

[00:10:12] Audrey: Oh, so I also want to add that I guess the users can contact us via LinkedIn as well. [00:10:20] It’s always an open, open Open door there. It’s important to prioritize the issues when it becomes a business continuity issue that becomes urgent.

[00:10:31] So a sense of prioritization on the issues is currently on the issues list. If you see it, we have things like the next [00:10:40] function we have to refactor, Oh, do this plot this way, kind of a thing. So, so those are secondary at the moment, but important in completing the package. 

[00:10:50] Alexander: Okay. Yep. Yes. So and of course, as I understand it, it’s not just you who can work on these issues. It’s also [00:11:00] others who can work on these issues, isn’t it? 

[00:11:02] Audrey: Yeah, so that lends us to this. Developing a product the dependent developing a software kind of workflow where we can have different branches so someone can work on it. It’s like an app. Someone can work on this page of the app and the other [00:11:20] person who work on the other page and they don’t conflict or they shouldn’t anyway.

[00:11:24] And conflicts are very common and can be resolved. So, so we work on a feature branch or a, or issue branch. One one issue per per branch and and make that issue as as manageable as as, you know, bite sizable as small [00:11:40] as possible. So we can have that. That effect of checking that box off later, we have to keep ourselves happy.

[00:11:48] Alexander: Yeah, I, I love this approach. It’s, it’s it’s kind of a Wikipedia approach of of working on topics and helping with the community. And I would love [00:12:00] if more and more people dive into this because ultimately our business model is to develop therapies. And don’t spend time with kind of reinventing the wheel from a programming perspective again and again and again.

[00:12:16] Audrey: Yeah, very good point there. What I do want [00:12:20] to bring the point home, there are feature branches, and then people can, you know, do a pull request to say, okay, it’s ready to be be integrated into the final product. And that gets reviewed by by one of us. 

[00:12:35] Alexander: Okay, cool. Yeah. And we will by the way, give [00:12:40] references to all these kinds of different things in the show notes.

[00:12:43] Also links to the LinkedIn pages that you mentioned. So that it’s super easy for you to. Yeah, connect with people. And so just head over to zeffectivestatistician. com, search for Audrey, and then you will easily [00:13:00] find that. So let’s dive a little bit into your experience of creating such an R package.

[00:13:08] And not just from a, you know, academic point of view, but you did it within a, within a pharma company. How does that, how does it look like? 

[00:13:19] Audrey: I think the [00:13:20] word is agile. I think we had very frequent stand up meetings of 25 minutes, sometimes daily, sometimes two or three times per week. And initially that was very important.

[00:13:36] And it still is very important. [00:13:40] Because I think this process is doing software engineering has made me rethink whether I know some basic things in R and things that I’ve forgotten. So I think it strengthens me as an R user. And My experience has been very positive anything, any ideas that I wanted to have [00:14:00] on making this product more accessible.

[00:14:01] In my point of view, we’re very welcome. And if there was something we couldn’t address at the time, we could put it on the ideas of what I issue list And I always had time to to reflect with my my mentor, my engineering [00:14:20] lead on on design questions. And also borrow his clinical trial experience.

[00:14:25] That’s that’s greater than mine. What I also like in this organization is that we have our own stack overflow where we can receive questions. And those are very valuable. Those are little gems because they tell us what kind of questions people have, [00:14:40] you know, like from their perspective, what haven’t we made clear, etc.

[00:14:44] So that only makes the package and the tool better for them. So it’s been very positive. I, I, I have, I guess, I’ve learned so much and I hope that I can make more packages and, you know, spread this kind [00:15:00] of work style where the mentee is very much empowered to, to make an impact on the user experience, especially.

[00:15:09] Alexander: Okay, cool. So how does, you mentioned Stack Overflow, an internal Stack Overflow. How does that look like? 

[00:15:17] Audrey: It looks like the external one where you can [00:15:20] tag people, you can write questions, you can get alerts on topics that come up with the tags of your interest. So it looks, it looks just like the one externally.

[00:15:33] And I think people get a lot of hearts and points for asking questions. So I think it works better than the one externally.

[00:15:39] [00:15:40] Awesome. I’d like a little bit of praise for asking your question. 

[00:15:43] Alexander: Oh, cool, cool. That’s, that’s great. Yeah. So having such kind of Internal tools that reflect the external kind of community tools that we have for open source development is, of course, super, super helpful. And yes, this. [00:16:00] incentivizing good behavior, I think is another really, really important thing.

[00:16:06] Audrey: What you probably didn’t know about GitHub is that when you, when I write code Daniel looks and reviews it. And on top of his his comment that he can make is in gray. Write a review, please be [00:16:20] kind that’s, I, I’m not coaching it exactly, but those little things help remind the reviewer that this is an atmosphere where I guess we’re all learning.

[00:16:31] And this is what I really like about software engineering is that iteration is forever. And the best idea wins. I mean, that is the principle [00:16:40] anyway of engineering. 

[00:16:41] Alexander: Yeah, yeah, that that’s a good kind of segue into one of my last questions I have. So what is the, how does a journey continue from here?

[00:16:53] Audrey: Oh so we, as I mentioned before, we have some more issues that I’m going to get to. [00:17:00] We are bringing this package to the User Aura Conference to get more feedback and to present it. 

[00:17:06] Alexander: That’s in Salzburg in 2024, yeah? 

[00:17:09] Audrey: Yeah, it’s only next month, so it’s coming really soon. And this is probably a more superficial point to others, but you know, we are going to put some hex stickers there just like any other package [00:17:20] to distribute.

[00:17:21] And I know it’s a very specific topic, but I hope that it can spread more of the enjoyment in, in package development and also probability theory coming from a nerd like me. 

[00:17:33] Alexander: Yeah, that’s awesome. Yeah. I love the good combination of [00:17:40] practical implementation, as well as some nice statistics theory.

[00:17:44] Awesome. Yeah. And you will also speak about your experiences at the upcoming conference of the effective statistician in fall 2024. What will people expect from you there? [00:18:00] 

[00:18:00] Audrey: I will, I will wait if I get accepted, but it sounds like I am. So I, I want to touch, I want to expand the point more on my journey as a statistician in software engineering.

[00:18:15] I don’t feel a huge shift within me, but what I [00:18:20] understand is well, I do feel a shift in that. I, I, it is so much fun and it expands your, your coding experiences. I think I want to borrow on the point that this kind of work, just like a statistician requires creativity, autonomy, and how to go about.

[00:18:39] [00:18:40] gaining that there are conditions that we as people who practice this art can create for ourselves to set us up for success. And also, when I say that, create a team environment that sets us all up for success and brings more opportunities to each and every one of us. 

[00:18:57] Alexander: Yeah, I love that. [00:19:00] Yeah, that exactly talks to the to, you know, motivational aspects.

[00:19:05] Yeah, you have autonomy, you have creativity, you can improve. Your skills all the time, so that speaks to mastery. And of course, that really contributes to the bigger picture of [00:19:20] making drug development more effective and therefore bringing new therapies to patients in a much more efficient way.

[00:19:28] So Yeah, it speaks to exactly these points that Daniel Pink talks about in in his book, Drive, and that is backed up by lots of, lots of research. So this is what [00:19:40] motivates us. And if you combine that with a good experience in terms of teamwork, it’s absolutely brilliant. 

[00:19:51] Audrey: Yeah, you said it very well.

[00:19:53] Alexander: Yeah. So for those who might think like, Oh, can I also submit something to the [00:20:00] conference? Yes, you can say are basically three different ways. You can submit something to the effective statistician conference. So first is you can supplement abstract about a poster. You want to do because we will have a virtual networking space.

[00:20:16] And in this virtual networking space, you can then present [00:20:20] your poster. The second is you can also present an abstract for an 18 minute long pre recorded Ted talk kind of thing. And that’s the second opportunity. The last opportunity is if you want to work together with some others, you can also [00:20:40] submit actually a complete session.

[00:20:43] And then that session can be pre recorded or can also be live. Just make that clear in the proposals that you have for the session and there are more guidelines on the conference page of the effective [00:21:00] statistician. And by the way, if you do so, you get actually the complete conference free. The price for the conference increases over time.

[00:21:10] And so have a check on the conference page. What is the current price? Because some increase the price because the closer we get to the [00:21:20] conference, the more you will know about the conference. So I basically reward if you bet on being at a great conference and you register early. Thanks so much, Audrey, for this great discussion about creating this R tool for early Bayesian decision [00:21:40] making and your insights into software development and statistical engineering, which is pretty, pretty cool area.

[00:21:48] Is there any kind of Final thought, recommendation, inspiration you want to leave with the listener. 

[00:21:59] Audrey: Yeah. [00:22:00] So there are many opportunities to contribute to packages in R and it doesn’t have to be statistics per se. There is if you go to the R Foundation website, I believe, or the interview I did with the fireside chats I did in the user R there are very many links on how you [00:22:20] can contribute even to R and making it a great language that serves us.

[00:22:25] So I encourage everyone to come as they are, actually. And, and see what they can do for the community. 

[00:22:33] Alexander: Thanks so much for that great discussion. 

[00:22:35] Audrey: Thank you.

Join The Effective Statistician LinkedIn group

I want to help the community of statisticians, data scientists, programmers and other quantitative scientists to be more influential, innovative, and effective. I believe that as a community we can help our research, our regulatory and payer systems, and ultimately physicians and patients take better decisions based on better evidence.

I work to achieve a future in which everyone can access the right evidence in the right format at the right time to make sound decisions.

When my kids are sick, I want to have good evidence to discuss with the physician about the different therapy choices.

When my mother is sick, I want her to understand the evidence and being able to understand it.

When I get sick, I want to find evidence that I can trust and that helps me to have meaningful discussions with my healthcare professionals.

I want to live in a world, where the media reports correctly about medical evidence and in which society distinguishes between fake evidence and real evidence.

Let’s work together to achieve this.