From Ethics Guidelines to Implementing Ethical Statistics

Transcript

From Ethics Guidelines to Implementing Ethical Statistics

[00:00:00] Alexander: Welcome to another episode of the Effective Statistician. And this time I’m talking again with Katharina. How are you doing?

[00:00:09] Katharina: I’m doing good, thanks. How are you doing, Alexander?

[00:00:12] Alexander: Very good. So last time we talked a lot about Ethics and how to have an approach to being a statistician and being an ethical statistician and the ISI declaration of Ethics.

So, and today we want to push that a little bit further because it’s of course one thing to have. A great declaration is yet another thing to get it implemented and to make sure that there’s a change in that regard and that we rally people around it and get it implemented and these kind of things. So, what do we do to drive change and get it broader? Adoption of ethical use of statistics and data. What can we do?

[00:01:06] Katharina: Yeah. First thing is we support declarations like those of the ISI that means we support ethical behavior as statisticians. And let me say a few words about ethics itself, because ethics is often confused with moral, at least in my perspective, what I perceive. When we talk about ethics, it’s not about talking about what is good, and what is bad. That’s moral. Moral is what is actually seen as good. Or was actually seen as bad depending on the values that we have in a society and that can differ between different societies in the U. S. I’ve been to Washington a few days ago or in China or in Europe or even in Germany between Germany and Austria and Switzerland that may differ a little but ethics does not tell you exactly what is good or bad.

It tells you a way to decide what is good or bad, what is good behavior. So it’s more about the process to follow. And that can lead you to completely different results, depending on the values that you have as a society. So when we talk about ethics, it’s not telling someone, okay, this is good behavior, this is bad behavior, you should do this or that.

But rather, if you follow a process like this, then you will come to an ethical decision. It’s like following A normal standard like the DIN ISO 9001 for quality management. And in this sense change management is very closely related to quality management. So how can we ensure that change will drive us to a good result?

[00:02:49] Alexander: Yeah, completely. And there’s a very famous statistician actually, who drove a lot of change in terms of quality management early on in the industry in Japan. And he is kind of the father of. Quality management. Yeah. Yeah. And so it’s, it’s really funny. He, he’s a us guy went to Japan, implemented statistics and change management and adoption of quality management there.

And then the Japanese. Car industry really disrupted the US car industry with better products for better prices and higher quality. And so only years later, the US car manufacturers came around and then adopted these kind of things. It’s yeah, maybe it’s a. A prophet in its own country is not listened to if you first need to Japan to get it implemented.

[00:03:55] Katharina: Yeah, if I remember correctly, he defined quality as the degree of deviation from a target, right? And I think this is also really interesting if you see it with respect to ethics, because We also first we should define a goal or a concrete target. What do we want to achieve when we show behavior?

And why? So in the sense of. Why should we behave ethical, and why should someone care, and maybe to say it shortly, nobody cares what you know, unless they know that you care. That means it’s not about us, it’s about the others. It’s about the others that we want to behave ethically, if they don’t understand why. What’s their advantage if they, if they follow ethical behavior and if they change? Well, they won’t just do it. They won’t simply won’t do it.

[00:04:51] Alexander: Yeah. Yeah. That is that’s a really important point. If you want to convince someone that person needs to feel that you care for them in our leadership program, we talk about it all the time.

There’s three components of building trust. The first is care that you care for the other person, your competence and your character. And so the competence, of course, kind of speaks to your technical skills and all these kind of different things. Yeah. That you care show us that you, you know. have the other, the best interest of the other person in mind.

And character, yeah, I think probably has a lot to do with, with ethics and how you behave and these kinds of things. So I think this, these things very well belong together. So if you want to drive change, one of the first things you need to do is you need to build trust with those that you want with.

To get along with you, that you want to get around. Yeah. Whether these are people higher up in your organization, whether these are your colleagues, whether these are your peers around in the industry or whether it’s your general public. It all starts with trust, and so if you communicate about the ISI declaration, yeah, shows that you care, don’t come with this kind of finger pointing and these kind of things. Yeah, sure. What’s in it for them? That is really, really important. So if you would be asked, okay, what’s in it for someone to follow the ISI declaration and, and adopt it and to distribute it further, what’s in it for them?

[00:06:48] Katharina: Well, on the one hand, it’s simply, if you, if you take it in a very broad sense, it’s responsibility for our society and responsibility for our planet because ethical behavior in times of big data and times of artificial intelligence means we have to understand how it can harm society and how it can harm our environment. This is all about fake news, misinformation, disinformation. loss of privacy, even bad legislation that misinformation can lead to.

But also, I mean, using AI means using a lot of energy, using lots of resources and water. So we should use it wisely. This is we have learned as statisticians in the beginning on which just when we simply did not have the resources not only to be effective, but also to be efficient and use the computing resources that we have an efficient way and not.

Run all the algorithms on the complete data set, but, but take a wise sample and use that to test the algorithm. And this is something that maybe has been lost now that simply everyone can get a data scientist, at least if you believe the advertisements for the courses.

[00:08:09] Alexander: Yeah, yeah, I completely agree. I was just this morning; I was thinking back of a colleague who was working on Bayesian statistics. He was one of my first mentors in the industry. So that is more than 20 years ago, pretty long time ago. And computing power wasn’t kind of that readily available at the time. And lots of the things he was talking about, he was finally saying, yeah, theoretically we could do this. But it would take quite a lot of time to actually compute these kind of things.

And at the time, yeah, there was a lot about efficient programming and efficient use of resources. And it still is completely agree. Yeah. Especially for the background of climate change, we need to make sure that we don’t burn our resources just because we, we can easily do. I love your approach to putting it into the bigger picture.

Yeah, that it’s you can convince people by showing that this, if we don’t do it, it leads to really, really poor outcomes. And it’s not just a poor outcome for someone else. It is for everybody and the discussions before we actually started to record you mentioned that even a law in the US that supports the official statistics and, and kind of how they are generated and used. Can you expand a little bit on that one? Because I think this is going in the same direction.

[00:09:47] Katharina: Yes, I recently stumbled about this law because I did some research and in the context of What is evidence and it’s the law is it’s a federal law. It’s the evidence act came into law in 2019, I think, and it’s about the way how federal agencies should support policymakers. By providing data for evidence and the evidence access that evidence is information that is produced by statistical activities for statistical purposes, and this can easily lead to the conclusion that if you just do statistics, and if you have just have numbers, then it’s fine. But it’s not only about data, because the data can be good or bad, and there can be very.

Different degrees of evidence depending on the quality of data that you have and that leads me to the to the idea that the basis for evidence and therefore also the basis for ethical behavior is that you care for excellent data quality and the data really represents it. Thank you. The phenomena in reality that you want to draw conclusions on.

And that is something that’s also in danger at the moment, now that we have the tools for everyone to do surveys online and to play around with data. But most of the people who do that don’t have the slightest idea about how they can evaluate the quality of the data and what the data can be used for and whatnot.

[00:11:27] Alexander: Yeah. Yeah. So that speaks to yet another point that we need to invest on and that is education. We as statisticians, of course, can easily say, well, you don’t know, but that is just the first step. The second step must be, and here’s how I help you know. And so I think it’s a huge opportunity. For us to educate people on case studies where things have gone wrong, you know, and there’s lots about these. Yeah, I’m, I’m just thinking about lots of the things about Watson, yeah, IBM Watson and how they try to predict certain things and completely failed.

Yeah because of, yeah, poor data and poor statistics on the data. I also think about, you know, lots of research in, in healthcare. Yeah, where we made a lot of policies. Yeah. And decided on, you know, different kind of treatments only later to find out. Well, actually, wasn’t as good. Yeah. And so, the history, especially of medicine.

Is full of that. Yeah, of wrong conclusions based on poor data. And so I think we can do much more to educate there to help people understand. Okay, this is what good looks like from a data perspective. This is what is possible. This is what we can’t do with the data. Yeah, I hear all the time, yeah, real world evidence, that is a new thing, we can do all kinds of different things there.

And yes, there is a lot of potential in there, but not everything can be answered with real world data. And very often there’s a lot of effort that first needs to happen so that you can actually use the real world data. And that may take even years to accumulate, you know. And then, of course, the question is, Hmm, wouldn’t it be better actually to run a clinical trial, a randomized clinical trial that gives you much better data in maybe the same time frame?

And yes, it may cost more, but actually collecting all real world evidence data and making sure that it’s also really, really good is also not for free. Yeah, I think people very often underestimate the efforts it needs, to get to that point.

[00:14:18] Katharina: Yes, definitely. Definitely. And I mean, what you say, is good data is expensive, but it’s more expensive to have bad data that you cannot use.

[00:14:30] Alexander: Yes. Yeah. I want, once someone told me, if you buy cheap, you buy twice. You buy first the cheap stuff, then you recognize, ah, you can’t actually do something with it. And then you throw it away. And then you buy expensive stuff and you realize, yes, this is what I really need. But! Of course, he wasted some money on cheap stuff and he wasted a lot of time and opportunities.

[00:15:00] Katharina: May I ask you one question?

[00:15:03] Alexander: Sure.

[00:15:04] Katharina: You mentioned education. If you would have the opportunity to teach only three areas to someone who wants to be a statistician or a data expert or just work in the data world, what would you teach?

[00:15:18] Alexander: So one thing that I would definitely teach Our communication skills. Yeah. And maybe I’m cheating a little bit because this is a pretty broad area, but this is really, really important. First, in communication is about listening. Yeah, you first need to listen to the people that also takes, by the way, goes back to the care that you that we mentioned earlier. If you want to influence someone, you first need to listen to the person, you need to listen where they are, what are their needs, what are their, their goals, what are their constraints, what are their pain points.

And then you can step into these and show that you care and connect your argumentations around these kind of things. And that is then the second point about communication. It’s about how do you move people. Very often we statisticians focus just on the logical arguments. This is the data, therefore we need to do this.

Very often they even say, this is the data they speak for themselves. Don’t say no, they don’t. Yeah. But logics are just one thing to persuade people. Yeah. The others are what’s in it for them. Yeah. Why should they care? And the last is, of course, why should they trust you? W what is your expertise? Why? You know, why on earth are you telling me something and I should listen to you?

Yeah. And then it’s kind of comes full back circle to what we talked earlier, the, the establishing trust. So if I were to say, you know, you want to be a great statistician, yes, of course you need to learn all your statistical stuff. And you also need to learn all the things about communication because you can’t, can be the most brilliant statistician in the world. If you can’t communicate. That will have no impact whatsoever.

[00:17:32] Katharina: Yes, I can totally relate to that and it’s it’s interesting because this is also communication is very important skill that is required if you want to be a certified statistician, according to the new FENSTAT accreditation rule.

FENSTAT is the Federation of European National Statistical Societies, so it’s kind of a head organization for statistical societies. Thank you. And they’ve developed an accreditation for statisticians. It’s not very well known, maybe because they are not so good at communicating, but they require that you prove your communication skills.

So this is absolutely I can totally relate to that. And what you say about communication, I mean, this is the, the basis to be. effective, right? Because if you cannot reach your your client, if you cannot reach those you address to then you can never be effective. I would like to add two more fields.

It’s about efficiency already have discussed. And this is In my perspective, closely related to algorithmic thinking. So because if we, what we now teach people is often, it’s purely coding. So if you do class on Python or a class on R or whatever, I mean, you teach people how to code, how to use the functions how to use arrays and whatever.

But I think we should start with teaching them algorithmic thinking. So how to break down a problem into small pieces, which is what you have as a case distinction in mathematics. What we’ve learned when we did mathematical proofs. I think this is also very important skills because this isn’t about efficiency.

And finally quality can never have quality and the result if you have bad quality, good quality and results, if you had bad quality in the resources and the resources are data. So it means this for me, this breaks down to service statistics. Knowledge and service statistics because service statistics is about how can we get data of good quality, representative data, representative data, and what are the sources of error, the sources of biases that you can have in the data.

And unfortunately, even most statisticians don’t know much about service statistics because it’s not taught when you study statistics. So I did my diploma in statistics in Munich. And I did not learn much about service statistics. I learned it later from, from a good friend of mine who’s also an experienced statistician and, and he’s with me on the board of the German Statistical Society.

So he’s the top expert in service statistics. And when I talked with him, I realized what I don’t know. And I studied. Thousands of pages of books and had hourly long discussions, and that was really an eye opener. And I would strongly advocate for teaching more service statistics to everyone. Also, I think we should teach the basics to our policymakers, because when I observe policymaking in Germany and in Europe, I see how often it’s argued based on bad statistics and bad studies that are So biased because they’re just well study based on an online panel, which is definitely not representative.

But these studies are treated in the same way as if they were official statistics. And there is such a huge gap between quality. And I think this is something we should also teach to policymakers, to journalists, to the general public. But especially to statisticians. And data science as well.

[00:21:32] Alexander: Yeah, I completely agree. I think you definitely need to have a very, very good kind of quality system. And quality is not just about SOPs, yeah? So people, especially in the pharmaceutical industry, very, very often confuse quality with following SOPs. Quality in itself is really about meeting the target. Yeah. What is required to make the right decision?

And of course, if you follow certain SOPs, you can increase that crazy likelihood, but it’s not enough. Yeah. So yeah, completely agree. Understanding was what quality actually means. And this is a key fact. So let’s make a step forward. What would you, if people now kind of want to support these things and want to convince other support to use, what could be kind of easy steps that they could do within their organizations?

[00:22:46] Katharina: Are you meaning within the organizations of statisticians or something?

[00:22:50] Alexander: Well, within their companies, within their associations of statistics. Maybe it’s the American Statistical Organization or maybe it’s the BBS in Basel or maybe you’re listening from India and it’s the Indian Society or wherever in the world you’re listening. Or you’re working in a small or big company. What can you do about this?

[00:23:15] Katharina: I think first it’s, as you mentioned, communication tell others why it’s important, not only for the data experts, but why it’s important for the subject matter experts or the general public for the knowledge workers all around you that use data and algorithmic tools in the daily business.

But not to produce more data or to develop new tools, but just to support their daily work. So this is one thing, tell them why or explain to them why it’s important. And then find, maybe use digitization. To, to spread the word, to spread the news. I mean, we have the opportunity to create online courses, to create small apps, to create podcasts like this one. Yeah. Just to convince people and tell them why it’s important.

[00:24:07] Alexander: Yeah. I think the other thing is find allies. Yeah, people that think ally form a group that wants to move things forward. Everything becomes easier when you work together and it’s not just you. Yeah maybe in the, at the beginning, it’s only two or three people.

Yeah, where you get together regularly and your kind of see. What can you do? So maybe you get, you know, someone more high up in the organization that wants to support you, that wants to sponsor you. Yeah, that says, yeah, this is really important. Let’s, you know, make some kind of initiative project out of it.

Sooner or later, you will need some kind of management buy in so that, you know, you get maybe funding or you get a room or you get, you know, more visibility within your organization. And then look for good case studies. Yeah, for quick wins, where you can how you can drive change and see what you can do to make people aware.

Maybe it’s, you know, an internal webinar or maybe it’s an external webinar to get, you know, with your, with your association where you talk about it, where you bring in people that have talked about this. Maybe you have supported already this declaration within your company, or your country. Yeah. Look out for these people and connect. That is, I think the first, first big step.

[00:25:43] Katharina: Yes. You mentioned allies. And yeah, on the one hand it’s allies in the management. That’s important because otherwise you won’t get the financial resources or the time resources to do so to convince others.

But I also think it’s important to find allies with another expert community. So as I experienced as a statistician that some statisticians, they almost refuse to talk to computer experts. So we have different societies in Germany. We even have, I think we have 12 statistical societies in the Dagstadt.

One of them is a German statistical society, but then we have the German Society of Informatics. We have the Mathematical Society. We have the Computer Society of the IEEE. And we all have the same concerns, but we don’t talk to each other, at least not enough. That’s my experience. And I think there are maybe some, some fears that the other party could maybe get more attention or It’s about losing power or whatever, losing power of interpretation, what data is and how data has to be treated, but I think that’s wrong. I think we need to cooperate more.

[00:27:07] Alexander: Completely agree. It’s not about which association is more important. This is completely the wrong way to go. This There’s enough change that it needs all of us. And the second thing is, as you mentioned, associations. Yeah, look out for allies also there in my area. Why not talk to the medical associations? They need to decide based on data for their guidelines and all these kind of different things. all the time. Yeah. I’m pretty sure you will find allies there that have the same interests that say, yeah, we need to have the best data to make the right decision.

Yeah. There’s another interesting ally in that regard is of course the Cochrane organization. So this is all about having the right evidence. To make good decisions in medicine. So there’s lots of allies. Form a network, get together, help each other.

[00:28:17] Katharina: Yes, I agree. So the army fighting against misinformation and disinformation cannot be strong enough. So that’s the one thing. When you talk about medical associations, one thing comes to my mind because I’m, I was involved as an expert in writing a, an expert opinion for an industry association, and this is in the context of the discussed advertised bans for for food in Germany. And this is a very critical topic because it’s so closely related to moral aspects.

And it easily happens that that one side is, yeah, what we have, we have to protect our children. And then I have the impression you cannot discuss anymore. Because when you say, well, yes, but we don’t have enough evidence that ads for food really are the cause of overweight and obesity. So there is no causal relationship yet.

Then they tell you, well, but you’re not a moral person when you say that. This is what happened to me. I said, okay, the data is not good enough, and it doesn’t matter what my personal opinion is about an advertised ban or not, as a statistician, as an expert, I must say the data, the evidence is not good enough to, to to be the fund, the fundamental of it.

Legislation and one data informed legislation, data driven legislation, evidence based legislation, then we must have the evidence and data will not get better if you use it for a good cause. Yes.

So this is something that we have to make perfectly clear also when we talk to, to medical doctors, because sometimes they, I have the impression, at least some of them, they think, oh, we are on the good side, we’re doing the good thing, we are protecting our children, we are protecting people, we are protecting the weak.

And if statistics is not in our sense, well, then we have to fight against them. Bit exaggerating, but maybe not too much. So it’s, and this is, but this is something which is very common, and this is what Daniel Kahneman, Kahneman said. So he wrote thinking fast and slow. It’s really, really hard for us to accept data and statistics that contradict our opinions. And this also has to do something with ethics, be open towards results that are not what you expected.

[00:31:08] Alexander: Yeah, a good friend of mine, another statistician once told me everybody is entitled to their own opinion, but not to their own facts. So yeah, I think embracing data that Doesn’t fit your belief is, is, yeah, is really important because otherwise you easily get biased.

Yeah. And we know that. Yeah. There’s lots of cases where people, you know, discard the data because that’s an outlier. Or that study is too small, or that study has this flaw, and therefore it is regarded so that I, you know, to date I agree with my my idea of what is right, yeah, and this is, of course, bad quality, and that is bad ethics, yeah, as you said, it’s not about, you know, moral judgment, it’s about making sure that you have the right steps, yeah.

To go to come to a good conclusion. So let’s move it one more step further. Yeah, we talked about what you can do. Initially, we talked about what you can do to change things within your organization to network what you can do to change things, maybe in your statistical association. Now, maybe you’re so, you know, motivated to change things, there’s one more step you can do.

You could actually build a business around it. And I think this is a lot what you are doing at the moment. Yeah. You have a consulting business that does a lot around good use of data. What, what’s your experience with that?

[00:32:56] Katharina: Oh. Well, I’ve been doing this for 20 years now, and I think that already says a lot, so I love it and it’s still every day I go to work and I’m, I feel privileged.

That I can do what I, what I love and that I, that I have tool like statistics that helps me better understand the world and hopefully find out what is true and what is valid and what is not. Yeah, but of course someone needs to pay for it. I mean, I do a lot of volunteering work for the German Statistical Society, for the Stifterverband, for the Ministry of Education. Also internationally for IEEE or FENSTATS and I spend a lot of time for that and it’s not paid. But on the other hand, yeah, I also do, I do consulting for companies. I write expert statements for associations. So and yeah, it’s when I, when I see this discussion around these expert statements I often read that people say, yeah, but it’s paid.

Then it’s biased, then it’s not neutral. Yeah, but someone has to pay, pay for it. And just because it was done, an expert statement was written by a professor at a university does not mean that it was purely from inner motivation or completely neutral. There’s also someone who pays for it. Also, the good people have interests, interests, and they pay for studies.

So, I mean, what would be. The alternative, should we have kind of text? That everyone pays so that studies are reviewed and, and we have expert statements when it comes to legislation. I mean, maybe this is could be another opportunity. A friend of mine recently mentioned, maybe we should have it like the GZ in Germany.

Everyone pays 20 euros per month. And that is used so that policymakers get good data. Well. Interesting idea. I think it’s the same with public media that still there would be people who would just not accept it and say it’s, it’s all in the interest of the governing party. Yeah, but so when you say, okay, this is paid expert opinion, then take a moment and think about what is the alternative. And just because it’s paid does not mean that it’s biased.

[00:35:29] Alexander: Completely agree. If you think about how drugs are developed and approved, all the research, all the data is usually coming from the sponsors and they present the data to the regulators. They interrogate the data, they are involved in the process of how the data is collected, all these kind of different things.

There’s a lot of checking, there’s a lot of transparency, all these kind of different things. So the trust is built by the process. Yeah. Yes. And so still, of course, who pays for these studies? Of course, the sponsors do. Yeah. They put the risk into it. Yeah. And they have developed a trust with the regulators by having, you know, a good process around.

Yeah. Yeah. So I completely agree that you can get to good, good data to good studies. If you have a good transparent process about it, and people can see, okay, yeah, that was done this way, this was done this way, and, you know, everything looks good.

[00:36:43] Katharina: Yes, and I think this is, exactly what we need for other applications of data for other industries, especially when it comes to the public sector and policymaking and the pharmaceutical industry, drug development could be a role model because there are clear criteria, how data has to be collected, how data quality is to be ensured, what processes to follow, how it’s to be documented, etc.

And especially you have to register your study in advance. Yeah. Means. You can avoid this problem of the publication bias that studies are published that lead to the, to the desired result where you have a significant result. So, and this can protect us because it shows, shows us, okay, what is probably not a helpful drug? What is probably not a helpful therapy? And you don’t have to test it twice or three times because you know it’s already documented and we have so many areas where statistics is used and studies are published where you only see those with a significant result.

And they could be false positives. Because maybe you have a hundred studies that did not come out with a significant result, but they were just not published. They’re all lost in the drawers of, of some researchers and yeah. And we just don’t know. And we have a biased view on the reality.

[00:38:13] Alexander: Yeah. Yeah. I completely agree. So I absolutely think. That in this area, there’s a lot of business opportunities for people. Yeah. To earn the living by doing the right things. Thanks so much. Katrina was an awesome discussion about how we can get to see as I declaration of asset kill statistics to actually implementing these, yeah where we talk about building trust with ourselves.

How we can communicate things how we can get others to buy in for find allies, find others who want to support us educate about these kinds of different things, bring case studies, show what good looks like, and all these kinds of different things up to the point where you can even start a business around it and help people make better decisions on data and yeah, have good ethical behavior around data. Thanks so much. What is your final thoughts that you would like to leave the listener with?

[00:39:26] Katharina: I think that we should work towards a certification for data. Certification for data that is used for important things like regulations and policymaking. I would love to work on such a project, even as a volunteer.

[00:39:46] Alexander: Completely agree. Completely agree. Thanks so much for being on the show again.

[00:39:52] Katharina: Thank you. It’s always a pleasure to be your guest.