Right. You're very much for coming, everybody. I'm Richard Simmons.
I'm the coast director of one of the many master's courses on the program.
And we've timed this particular talk to coincide with one particular module of our master's program in Central Medical Statistics.
But like all the talks, it's open to you. Some people here are students on this week's course.
Some of you a staff on the floor. I've got a few visitors.
Either people I know by science or people either by science or people I've just met.
Either way. Thank you. I saw Professor Alexander Byrd speak at a conference last summer.
And ever since then, every time I've heard somebody worried about the replication crisis in medical publishing,
you've got to hear Alexander Byrd speak so very, very close to you.
You've got to hear Professor Alexander both be. And he is from King's College, London, where professor of philosophy and medicine.
That's right. Yes. Great combination. I don't know if anybody else has that job title.
It's very I don't know what's coming.
Okay. Thanks very much indeed, Richard, for the invitation. It's a pleasure to be to be here.
I'm a philosopher, but with a particular interest in in medicine and in teaching medical students, which I do at King's Guys campus.
And I'm a general interested in.
How it is that we know what we know in science and when possibly we don't know what we think we do know.
And the replication crisis or so-called replication crisis is of particular interest to me, therefore.
So what is this so-called replication crisis?
Well, here's a little bit of evidence that a an article in Nature did a survey,
not very scientific survey, it must be said, but a survey of its readers.
And of those 1500 or so scientists who responded, 52 of them,
52% of them said that they thought there was a significant crisis of reproducibility in science.
That is to say, the phenomenon whereby a result is published in the scientific journals.
It seems to be important and interesting results supported by the evidence.
But then others come along and try to reproduce that experiment and then get a result.
That is either another result or is a result which is a of an effect size, much smaller than originally reported.
So we are failing to reproduce much of the the science that we thought was correct.
And this is affecting in particular biomedical research and social psychology and other bits of psychology as well.
But social psychology especially, I will concentrate on the medical side because that's where we will come from.
But afterwards, I'll be happy to talk about the social psychology, because it is actually more amusing than the medical side.
But on the medical side, the biotech company Amgen undertook a large scale replication study of 53 publications in
oncology and got a successful full replication of the original result in only six cases.
And as I say, there's also social psychology, but we'll leave that for some of the after talk matter.
Okay. So what I want to do is to give one angle on understanding what's going going on and to explain why I think that this outcome,
this crisis is entirely predictable.
And then we can talk a bit about how we should feel about it in the light of what I am, what I'm arguing, if it's if it's correct.
And to do that, the first thing I want to do is to talk about the so-called base rate fallacy.
And this will be, I'm sure, familiar to to many of you.
But it's worth repeating just in case there is some who who for whom it's not entirely familiar.
So the you given this this problem was a question you're told about a screening program for a
disease which affects one in a thousand individuals and a screening program is 95% accurate
and screening everybody not not just those with prior indications but but but everybody
of particular age group is being screened and the particular individual tests positive.
We know nothing else about them. We know of no other risk factors.
All we know is they passed the test.
Perhaps your GP and this individual comes into surgery and says, I've taken this test, it tested positive, you don't have this horrible disease.
And then the question is what? What did you say? What should you tell the patient about their chances of having the the disease?
Famously, this question was put to medical students at Harvard 1978, and of those, only 11 out of 60 got the correct answer.
Okay. So what is the correct answer? Well, the incorrect answer is, look, it's a highly reliable test you've very probably got.
The the disease. That's the wrong answer in which we can see this way.
So let us imagine there are a thousand individuals,
one of whom there some dot does indeed have this disease and we run these thousand people through our screening test.
That's 95% accurate. So 95% of the people without the disease are told your sign.
So 99% sent home saying you tested negative because that means leaves 5% who are told falsely that you do have the disease.
And we can assume it's highly probable that the one person with the disease is told they've got the disease.
95% of they will be.
But what we see there is that we've got about 50 individuals who have been told they've got the disease, only one of whom actually has it.
So that means of those told that they have disease or who test positive.
In fact, only 2% and one in 50 actually has the disease.
So that's the right answer. So what we've got is a situation where the base rate of this disease is one in a thousand.
And we can call this if this number pi.
So that is the base rate or the prior probability of an individual actually having this disease and a false positive rate of 5%.
I mean, call this alpha. And what we've shown is that the false positives amongst the 999 people who didn't have the disease,
the false positives amongst them greatly outnumber the one true positive.
And failing to recognise this fact is the fallacy of base rate neglect or of ignoring the the base rate,
I'm willing to argue, is in a sense that's exactly what's going on in the replication crisis.
Now, for those of you who are a bit more into the statistics, I've you will know that I've been he said he said it's accurate, 95% accurate.
Well, that's ambiguous. So what does that exactly mean?
Well, there are two types of accuracy that I could me.
One is the accuracy in the sense of avoiding false positives.
And then there's an accuracy in sense of avoiding false negatives.
And these are two different types of accuracy.
And they can come in, do typically come apart.
I always focusing most of what I say on on this thing alpha which is the type one error rate and the corresponding accuracy one minus alpha.
I will be assuming throughout that the power is quite high for the calculations.
I'll be assuming it's unrealistically high at 0.995, because I want to show that if we get a problem,
even if you've got high powered studies or that the issue arises,
even if all our studies are high, high powered, and we can we can discuss what happens in the in the real world,
where many of our studies are less highly powered than that.
Okay. Okay. So what's the lesson of the fallacy of lowering the base rate down to the sample we had moments ago with the screening program?
Is it we want to avoid mixing up these two kinds of conditional probability, the probability that someone is disease free,
given that they've tested positive for the disease and the probability that they test positive given that they don't have the disease.
This one here is the false positive error.
Right? That's the positive.
That's the probability that they test positive even though they're disease free.
And that's all thing alpha. But this thing up here is a quite a different thing, which is the probability that their disease free.
Even though they've tested positive. And what we've seen is that these two are not only distinct, but they are quite, quite different in value.
I mean, just to get a grip on that difference, it's like the difference between these probabilities,
the probability that the temperature will be below zero, given it's snowing, which is very high.
If it snows, you can be pretty sure that the temperature is below zero or close to it.
But this thing here, the probability it will snow given the temperatures below zero, which is actually that's quite low.
We often get frosts, but without without snow or possibly even more.
Well, medically, what's the probability that someone has sports, given that they've got measles quite high?
What's the probability that they've got measles given they've got spots quite low.
All right. So in the case of diseases, we've found that the false false positive error rate can be pretty quite low at 5%,
even though the false positive report probability the probability that a positive report is in fact,
a false one actually is going to be very high, 98%.
So if you were told by the screening program, you've got the disease, the probability of disease free is very high, much extent.
Okay. Now, now, does this get back to we'll get to the replication crisis.
So I'm going to use the method of philosophers, which is to tell outrageous stories.
You really the little vignettes models that we can use to to to get a grip on what's going on.
So I can tell a story of a mad scientist, Dr. M, who generates crazy hypotheses.
He's wildly creative and imaginative, but he's not so mad that he uses bad methods for testing his hypotheses.
It's just he's got this wild imagination.
But when it comes to testing his hypotheses, he does so really quite stringently.
He uses null hypothesis significance testing quite the proper way you with a significance level of of of 5%.
So now let's imagine that for Dr. M, he's generating all these wild new ideas.
Very creative, but so creative, so imaginative that very few of them turn out to be right.
In fact, only one in a thousand of his new hypotheses is, in fact, true.
The rest are all false. But because he's using null hypothesis significance testing in the proper way, his accuracy his is one minus alpha is 95%.
Right. Okay. So we can ask ourselves the question.
Dr. M has ties to lots of hypotheses. One of them has tested.
He has gone through a randomised controlled trial where it's null hypothesis, significance testing and he's got a p value of less than .05.
So in the sense it's passed our standard test for truth.
What is the probability that it is in fact true?
That's our question. So we've got a publishable result.
Perhaps he sends it off and publishes it top journal because after all he's done a randomised
controlled trial a perfectly proper way and they've got a statistically significant result.
But what's the chance that is in fact true? Well,
I hope you can see that the structure of this question is exactly the same as the structure of the
question I asked about the screening program that we had a disease that was found in one individual.
And every thousand here we've got a person who's generating a hypothesis set of hypotheses, one of which is true in every thousand.
In the screening program case, we had a test for the disease, which was 95% accurate.
Here we've got a. Method of testing these hypotheses, which is 95% accurate.
We've got exactly the same structure of of problem.
And so it will turn out that the chance that Dr. M's positive result that true is just 2%.
But just as in the the screening program case there, the people who the false positives greatly outweigh the one true positive.
The same will be the case for for doctor Dr. M.
Okay. So now let's move from Dr. M to.
Same Professor s time. She generates hypotheses.
She's working in a new field. It's a difficult area of science because it's new.
There's not a lot of indication of where the truth really lies.
She generates some hypotheses and she tests them stringently.
In her case, we are going to imagine that the base rate of truth is 10%.
So she's 100 times better than my Dr. M and she's much better at generating hypotheses.
But still, because of the difficulty of her area, its newness, some other factors.
If she gets things right in her hypothesising one occasion in in ten,
she uses exactly the same hypothesis testing methods as Dr. M and the rest of us.
So high accuracy, one minus alpha is is 95%.
So she gets a positive result.
It's publishable. Same reasons. What is the chance that it is in fact true?
What's the probability? It's. It's true given that it passes the stringent test.
And I think I am. I'm not going to give you the wrong number.
Actually, I think it's one of to this. So the the chances are 68%.
Right. And on the slide, it's just 32%, which is the wrong way round.
That's so slight change.
A slight chance. I should be 68%. This is the chance that it's false, right?
Okay. Okay.
So in this case, we said we don't want to conflate these two two things.
In this case, we've got if you conflate them.
And I think this is part of the this is part of the problem that's going going on.
You'll think we're using null hypothesis, significance testing, alpha of 5%.
That means the previous slide that our hypothesis says will give us a significant result, even though it's false in 5% of cases.
And you might think, oh, that's that's fine. 5% is yeah, it's small enough.
That means we'll be getting things wrong only one time in 20, but getting things wrong only one time in 20 looks.
If you're saying this, I'm saying we're getting things wrong given that we get a successful result only one time in 20.
But that's that's a mistake. So we've seen from all the previous cases that these two things are not the same and this can be quite small,
whereas that can be quite high in the case of Dr. M and in the case of this screening program, that was 5%, but that was 98%.
In the case of saying not so in Professor S, this was 5% and that was 32%.
So, so imagine as it were. Yeah.
Can you conflate these two? But that's wrong because in the case of Professor SS, that's 32%, not 5%.
But say you thought actually what you wanted was 5%, you wanted it to be the case that of all of the successful hypotheses,
the ones that come through our null hypothesis, significance testing, but now for 5% successfully give us a statistically significant result.
You want it only 5 to 10% of them to be wrong. Okay.
You wanted 95% of the stuff you think is publishable to be correct.
Then the question is what alpha would you have to have in order to get that result?
That's the say it. What p value would you have to have is you want to generate only 5% of our positive results being false, 95% being true.
Well, it actually would have to be a whole lot smaller than that 5%.
And we could this is given these assumptions.
So it has to turns out to be almost one ninth of the average of the 5% that we started off with.
Okay. So what I am suggesting is that we would expect to get a high rate of false positives in our research if two things are correct,
if we have a low background rate of truth.
So in the case of Professor, as I was suggesting, that one in ten of her hypotheses is true,
nine out of ten are false, and an alpha that's even though may be low is non-negligible.
So five, 5%. So we regard the P values as a significant publishable and so forth.
It's less than 5%. So that 5% is low, but it's not negligible and it's the combination of these two then I think will generate,
I've argued will generate a high proportion of false results amongst those we think are publishable.
So in the case of Professor Rice, it was that 32% of her results turned out to be mistaken, even though she got a positive outcome.
She got statistically significant results at the 5% level.
So, of course, it's a perfectly good scientist.
SARS is producing results that are mistaken.
Almost one occasion in story.
Then it's hardly surprising when other scientists come along and try to reproduce her work.
They find that they failed to do so in a number of cases.
So but that's to say this is not true, is clearly not to impugn her at all.
She's doing the best science she possibly can in a challenging area.
She's not engaging any questionable research practices.
She's doing everything by the book.
The only problem is, well, the two problems are that in the fields she's working in makes it difficult to know, to guess what the good hypotheses are,
to guess what the truth is, to generate ideas that are likely to be true, combined with the standard value of an author at 5%.
Now. So what I've shown is that this combination will generate a high proportion of false positives in our research.
Next question is, is our science like this?
Does it have these these features? So briefly talk about about that and why I think that the kinds of research that many
of us are engaged in or at least interested in is such that we will have a low pi,
that's to say a low background rate of of truth.
Many of our hypotheses are derived from some underlying lying theory.
Now, let's talk a little bit about the way this works in physics and particle physics.
So if you remember a few years ago at CERN using the Large Hadron Collider, they discovered the Higgs boson.
Absolutely exciting. Why did they spend billions of pounds building a machine to find this Higgs boson, amongst other things?
That was the main thing that they wanted to to find. That's a lot of money to spend on an experiment and unique.
Be pretty confident that you're going to produce some interesting results if you're going to spend that kind of money.
And many people thought that the finding of the Higgs boson vindicated all this expense.
Now it's because the hypothesis that there is a Higgs boson is derived fairly directly from something called the standard model of particle physics.
And that's been around for some decades, you know,
half a cent best part of a century and more than and that theory is one of the the standard model is one of the best confirmed theories in science.
And in fact, the Higgs particle was really one of the last bits of the jigsaw to be to be fitted in all
its other predictions about what kinds of particle there might be had been already confirmed.
So. Scientists were in this position, they were able to say, look, if the standard model is correct, then it's a pretty direct consequence.
Absolutely direct. But with a few very plausible assumptions, we could show that there must be this thing, the Higgs particle.
And furthermore, they could say there is very strong evidence that this standard model is correct.
Put those two together. You think we should be pretty confident?
There is that Higgs particle out there somewhere, and then you can devise the experiment to to detect it.
Okay, but is this. Now, let's turn to medicine.
It is. Are things like that in medicine? The answer, I think, is very rarely.
And that's not because there's something wrong with medicine. This just in some ways a lot more complicated than particle physics.
It doesn't have any physics.
And we they have the if they have it easy, it's you guys involved in in medical research who have have the difficult, difficult jobs.
We medicines suffer from a weak underlying theory, relatively speaking, because, well,
for one reason there can be weak connections between what we discover from basic research,
say, in physiology and what we think might happen if we introduce a drug into someone.
It's we can make plausible connections, but the complexity of the human body tends to have homeostatic mechanisms work or sometimes don't work.
And the fact that things are so complicated means it's very difficult to say with certainty.
With certainty, given what we've discovered from our basic research, this is how we think this drug will affect an individual.
And then secondly, we may often be working with underlying theories that are themselves plausible, have some evidence, but not entirely certain.
So, for example, if you're going to use that of the drug map and using map,
which was developed in order to help patients suffering from Alzheimer's, in this case,
the underlying theory is that it's the beta amyloid plaques that we find in the brains of Alzheimer's sufferers
that are the cause of Alzheimer's and of the cognitive impairment that Alzheimer's patients suffer from.
But and using MAP is a drug based on antibodies to these plaques.
And the hope was that as an antibody that it would prevent the further development of these
plaques and may even cause that reduction and in consequence would help Alzheimer's patients,
help prevent them from getting that condition worsening or possibly even help repair the damage that they had suffered.
A lot of money was put into developing the drug and into trialling it.
Sadly, the outcome was was no, there was no benefit deriving from this drug.
It's entirely plausible to think the idea that it could have helped if we were right,
that the cause of Alzheimer's and the cognitive deterioration that it involves are these plaques,
and it's quite plausible that an antibody to them will be be helpful.
But even if the underlying theory of the amyloid cascade hypothesis was correct,
there's no certainty that the antibody would help seem like a plausible idea, but it's certainly no guarantee.
And furthermore, scientists weren't even sure that the amyloid cascade hypothesis was correct.
So there are others who thought that the so-called tangles that are associated with
Alzheimer's are more significant as a causal factor than the beta amyloid plaques.
So in this case. We can say the following things.
If this hypothesis is correct, then it is conceivable that back rooms that will help Alzheimer's patients.
Yeah, it's not a bad idea. And then we could also say that this amyloid cascade hypothesis might be correct, but the evidence is far from conclusive.
So you can see the contrast with the physics case. If we had no reason to be we had some reason to be hopeful that that would do the job.
But we have no reason to be highly confident. So they hit that's my reason for thinking that in much medical science,
we should start off by thinking that our hypotheses have a low prior chance of being correct just because it's really, really difficult.
And our knowledge is very, very partial.
There are other reasons for thinking that the hypothesis we actually put forward for testing may have a prior probability of being correct.
It's quite low and other things that we can think about the fact that there's pressure to do experiments to try and find out what's going on.
After all, Alzheimer's is a very serious problem for many individuals.
It would be good if we could find something that works.
And so there's quite a lot of pressure on us to think what might help.
And then to test it. But that means, well, we will be putting forward hypotheses and testing them, as it were, relatively early stage.
Episteme Italy speaking the stage when our knowledge and our expectations about that being correct are relatively low compared,
say, to the physics case.
You think about the physics case, they were going to be pretty darn sure that the the Higgs boson existed before they went looking for it.
It would have been rather embarrassing to say we built this billion euro experiment.
And unfortunately, it's not found the thing that we were looking for.
So there the experimentation comes with every motivation to delay experimenting until
they were sure that the experimental experiment would show what they wanted to do.
Medicine, the pressures that are in the other direction.
The pressures to think up hypotheses and test them early, even though we may not be certain of what the outcome is and that may be good,
well, we can just debate whether that's a good or bad thing. It's just different.
So the other feature of the explanation is the Non-negligible Alpha,
the fact that we regard something as a statistically significant outcome if we get a P value of less than 0.05 and well,
this doesn't need much discussion since it's simply the accepted convention.
That is the way much research in clinical medicine and psychology works.
We do we use null hypothesis, significance testing, and we regard an outcome as statistically significant.
If the P value is less than 0.05 and that means that we will get type one error rate of 5%.
Now, of course, this is a convention and we can look into the history of all of this and you can
find that the 5% is the number that was accepted from earlier in the 20th century.
But there's no particular reason why it has to be 5% 2.05.
In physics, things are different.
The convention is five sigma, five standard deviations from the mean.
So instead of having an error rate, a false positive rate of one in 20, there's one error rate of one in 3 million.
Now, I'm yeah, we shouldn't think that's the correct standard.
There will be different standards for four different different scientist sciences,
but it's a reminder that the 5% that we typically work with isn't isn't given to us by by God or by the rules of rationality.
It's, it's, it's a convention that we could decide to change if we so wished.
Now, there are other explanations of what's going on in the replication crisis.
There's discussion, low statistical power,
there's a lot of mention of publication bias and other forms of bias are so-called questionable research practices and even fraud.
And we could talk about those. And I've got something to say about why I think that these are only weak or partial explanations.
I think we'd be nice to give you a chance to to talk. So I'll just say that there are other explanations on the table.
Most of my hypothesis is that we don't have to reach for these.
We don't have to think that something bad is happening in science just because we've got on reproducible research.
We could also ask what's to be done. And I think that if if what I'm saying is correct.
Then there are a number of things we could choose to do, which is to say, live with this is the nature of science.
Who says science is easy? Who says it's always going to produce the right results?
It's just a fact that difficult science, difficult is going to produce false results from time to time.
Just expect that to be the case and learn to live with it.
On the other hand, if you are going to accept that we ought to support replication better than we actually do.
If you're going to accept that our science is going to produce false results,
then you have to be more favourable to scientists who want to try and find out which those are.
And I don't think we live with that is what you think we should do.
Then I don't think we are supporting replication enough.
I mean, some journals simply won't publish replication studies and that just seems to me to be bad practice in the light of what I'm saying.
We could put our effort into increasing the chance that our hypotheses are correct before we test them.
And if we want to be, yeah, we want to move from you don't want to be doctor mad and produce loads of crazy wild but false hypotheses.
You want to move from block to mat professor saying but perhaps we should try and move from professor same to here
you to a better position that she's in by by being more exacting before we put forward hypotheses for research.
And that would require doing it more and more basic science in order to work
out which possible interventions have a decent chance of actually working.
I mean, the other thing we could do is just be more stringent in our testing and say, well, you're perhaps that an alpha of 5% is too, too lax.
Note that there's going to be a trade-off between our value of alpha and the effect size.
So that's to say that. Effect sizes, which are statistically significant at the 5% level,
it won't be statistically significant at the 1% level or lower in many cases, obviously is not a bad thing.
Well, it might or a good thing. It might be a good thing in some cases,
because in a number of studies we find that particularly pharmaceutical companies produce
studies with large numbers of subjects and they find outcome that's statistically significant.
But we look at the effect size and think, well, that's not clinically very important.
Yes, statistically significant, but clinically insignificant.
Now, it might be helpful to that pharmaceutical company because they've shown that their drug
is marginally better than some existing treatment and therefore other things being equal.
If physicians will want to prescribe that new treatment.
So it makes sense for the pharmaceutical company. But do we as a society, if we really benefited from this?
Probably not a lot. And so one outcome of doing this might be that, as it were,
we eliminate those kinds of case that the things that are produced that say if we put Alpha down to 1% or point 5%,
then it might be that more of the statistically significant results are also clinically significant.
But it might also be the case that we miss out on some some useful outcomes.
So I'll finish with one one with my graph simply because I just put a lot of effort into making it.
Just to point out that if we talk about the power of a study, which is the ability to avoid Type two errors.
The that's to say the power is the ability to to detect what's true when it is, in fact, true.
So 8.8 is thought to be a satisfactory level of power.
But there's good evidence that many of our studies have lower power than than that.
And one of the things that I'm interested in is whether we want to put our effort,
whether we should care about power or low alpha or high one minus alpha.
And this side is the positive predictive value is one minus this false positive report probability.
That's to say, if you've got a positive report, what's the probability really is true.
And what we want to get as close to one as possible up there,
we want it to be the case that when we've got a positive test result, that that that really is the case.
What this graph is showing is that so long as your power is reasonably high, then increasing power further doesn't get you much good.
So. So if even if you had maximum power of one but your alpha is still 5% and your and we're
working with a pie of 10% like professor say you're only one in ten of our hypothesis is true.
That may still be the case that almost a third of positive outcomes are false.
So to help her, what we need to do is not increase pounds at maximum.
What we need to do is reduce or reduce alpha. So that's yeah.
On the other hand, if your power starts off really low, then there will be benefit to increasing power.
So some of the debate that goes on in this area is, is it low power?
That's a problem. Well, I think for very low power studies, then that might well be the case.
But the point is that even for high powered studies, there's still a way to go if you have a non-negligible alpha.
Combined with a difficult area where it's difficult to produce two new ideas that's.