So I'll be talking about the replication crisis in psychology. And just to be fair to psychology, I'm not singling it out here.
I could just as well be talking about a replication crisis in science generally, or medicine or biology.
Cancer research is having some problems now.
I'll be speaking about psychology because this is the area that I'm most familiar with and closer to the facts on the ground.
And so I feel most comfortable talking about these examples.
The other thing I want to say about the word crisis here is some people think this term is inappropriate, it's dramatic.
Others think that it's a fair description of what's going on.
So there's serious and substantive disagreement about whether indeed there is a crisis going on and if there is, what the nature of it is.
So I want to be clear about that. In terms of history and philosophy of science, I want you to get your hopes up too much in terms of history.
I'm not going back much further than 1970, and the philosophy of science is going to be sort of 1 to 1 introductory stuff.
So for those of you who have any familiarity at all with philosophy of science, some of these ideas will be comfortable already.
So I'll start with this.
Replication is sometimes talked about in the context of falsification, which is an idea that goes back most notably to Karl Popper.
This is from his 1934 treatise, The Logic of Scientific Discovery.
I've already violated my 1970s rule there, but I won't do that again.
Here's a naive view of what's going on with scientific discovery, and this is something that Popper was criticising in his work.
You might think that you come up with a theory or maybe a hypothesis in your mind,
and you think that it's going to predict some kind of observation, something you should be able to observe.
And you try to design an experiment such that the observation will lead logically.
Sorry, the theory will lead logically to an observation.
And then if you get the observation that you predicted, you might think, well, that now confirms or at least counts in favour of my theory.
That's sort of an eye view of what's going on of science.
The problem with this is that any number of other theories might also just as well have led to the same observation.
And so by finding evidence that's consistent with the theory, that's incredibly weak support for the theory.
Unless you can rule out all of these competing alternative theories that might, as I say, just as well, have predicted predicted the same observation.
So this is related to the fallacy in in logic called affirming the consequent.
So this goes like this. If he then q. Q therefore p.
You can't draw that conclusion because a, b, c, d might just as well have entailed.
Q So getting Q doesn't give you P for free. You have to rule out all the alternatives.
But this points to to a valid argument in logic, which is the modus talon's argument.
And the way that this goes is if. P then. Q Not.
Q Therefore not. P The idea is if P than.
Q I didn't find. Q Therefore it can't have been P in the first place.
So this inspired Popper's notion of falsification.
So the idea here is I have a theory. I predict some sort of very specific observation.
And if I don't get the observation that finds, then again, that's not supposed to seen as falsifying or at least counting against the theory.
Again, this is a highly simplistic view, but that's the general idea.
So Popper's notion was that instead of trying to come up with evidence that seemed to support or confirm our theories,
we should constantly be trying to come up with hypotheses that predict very specific outcomes,
such that if we don't get the outcome that we expected, we should be willing to say to forfeit our theory.
If you constantly have theories that that can't be falsified by any any possible observation rather than being a strength of the theory.
He thought that was a weakness. And so we should be subjecting our theories to what he calls critical tests, trying to falsify them.
And if we can't falsify them, we don't say that they've been proven true. We just say they haven't yet been falsified.
And begrudgingly, well, over time, if we've subjected them to many critical tests, say maybe,
maybe we have something here, we haven't been able to falsify it, even though we tried our best to do so.
Now, as critics of Popper have noted in all the intervening decades and in many different ways, even this is not get us off the hook.
This is still a problematic theory.
So again, the idea is I don't get the observation that I expected on Popper's view that it's at least supposed to count against my theory,
but it might not be that straightforward. So first of all, there might be something wrong with the observation.
Maybe my research assistant wrote down the wrong number or something like that,
so I might not be sure that the observation is really the correct observation.
So evisceration might be what's wrong rather than the theory. It might be that there are various weaknesses in the experiment itself.
Maybe I didn't set it up in such a way that it really would give me the closest test of the theory.
In terms of the observation, there might be various problems between the theory and the experiment as well.
And these are often referred to as auxiliary assumptions.
Basically, these are these are logical links between each step going from theory through the experiment to the observation.
And you'd have to show that all of those were were true in order for the observation not turning out to count against the theory.
Just to show what I mean by this. It could be that if the observation doesn't hold, the theory is not true.
But it could be that the observation is wrong. It could be something's wrong with the experiment.
It could be something's wrong in the link between the theory, the theory in the experiment.
And also I rule out all of those alternative explanations. I actually haven't touched the theory at all.
I certainly can't say that it's not true unless I've ruled out all of the possible links between the observation and the theory.
So this creates something of a pickle for talk of failed replications in the current debate.
You will often find researchers saying, Well, we failed to replicate the initial finding.
This counts against that finding. But again, unless you've accounted for all the possible auxiliary hypotheses, simply getting a negative result,
a result that doesn't appear to be the same as the first one that was reported
isn't going to be enough to show that you somehow falsify the original finding,
much less the original theory. So the opening shots of the current replication crisis were fired around 2012.
I think this is this is a blog post by Ed Yong, a very distinguished science writer in the United States, I believe.
Or maybe he's here. He used to work for discovers and writes for The Atlantic.
And this post came out that caused a lot of controversy.
You can see the title here primed by Expectations Why Classic Psychology Experiment Isn't What It Seemed.
He was writing up this paper here by Doyin and colleagues called behavioural priming.
It's all in the mind. But who's mine now? What was significant about this paper, also published in 2012, is that the researchers ran an experiment,
didn't get evidence supportive of the original published findings that they were trying to replicate.
And then instead of just burying the finding in their file drawer, which is what typically happens, they took that.
They went through the trouble to write it up, which is further trouble of submitting it to a journal.
And they actually got it published in a in a fairly well respected journal.
Now, this used to never happen.
Anthony Greenwald wrote a paper back in the seventies talking about the unintended consequences of what he calls prejudice against.
No. This just means that for the longest time,
journals had a strong disinclination from publishing any kind of negative or, as it were, failed experiments.
So what happened is you would run an experiment to try. I mean, failed replications in some sense happened all the time and continue to happen.
It's just that you would never go through the trouble of writing them up, much less submitting them because you knew you wouldn't get them published.
So this was a significant moment. The only time you would hear about failed replications would be at the bar after a conference.
When you're talking to your colleagues and you say, Yeah, I tried to run that experiment and I couldn't get it to work.
Oh yeah, I tried to run the experiment. I couldn't get it to work. So you had this informal knowledge floating around in scientific communities,
but because all of these failed replications are never published, they just get buried.
And what you end up having the published literature is just a fraction of the attempts at running those experiments.
And this creates systematic problems and also, by the way, increases the likelihood that the published finding is just a false alarm.
So this is the original paper that Duane and colleagues were trying to replicate.
I'm going to call this the elderly walking time study, just the shorthand, because the title is a bit long here.
I just want to check in the room. Do people know about this study or have heard of this study?
If you have, just raise your hand and if you and if you haven't, that's about half.
Okay. So it's it's worth explaining. Oh, I'll go through it in some detail.
Basically, this was a very important finding in the field of social psychology.
It's been cited 3633 times, which is an astronomical number for studies of this kind.
It's been written up in introductory textbooks and so on.
So for researchers to claim that they weren't able to replicate this finding was, as you can imagine, sort of a big deal.
This is basically how the experiment worked. I'll say a little bit about the theory.
So there's this idea that simple cues in the environment activate stereotypes in our mind.
If you see someone walking down the street, you pick out very basic aspects of what you can see,
and all these stereotypical traits just become automatically activated.
The researchers here are trying to test the idea of the further idea that once a stereotype is activated in your mind,
you yourself tend to behave in accordance with that stereotype as a basically a a social lubricant.
It helps you behave in accordance with the way that you think people basically are around you
so that you're not consciously having to try to figure out how to behave in some situation.
So I'm going to be talking about the elderly stereotype. Just imagine that I'm in a room full of elderly people.
Well, I'm going to walk a little slower, maybe speak a little more quietly and so on just instantaneously,
because my mind has a stereotype about what elderly, elderly people are like.
And then it's going to encourage me to sort of just instantaneously act a little
bit more like that so that I can find it easier to engage with my environment.
So the clever idea for how they were going to test this was they gave what's called a linguistic priming task.
Basically, they gave people a series of sentences that were in a scrambled order and you had to unscramble them.
And it was just framed as a linguistic puzzle, basically.
But in a subset of these sentences were words that were meant to activate the elderly stereotype.
So the words are things like wrinkle grey. Bingo. Florida, I think, was in there.
This is conducted in the US and the idea is that not so many of these words should, should be activated, that you notice any sort of connection here.
But your mind is figuring out that there's something going on here.
It's meant to activate the whole stereotype, which means that even traits that you didn't include in your priming materials
should become activated in your mind because they're associated in the culture.
So the trick here is that one of the traits they did not include in the priming materials was one having to do with moving slowly through space.
And then they wanted to see if if that were activated.
And it's true that it influences your behaviour and participants who were primed with the elderly stereotype
should themselves move more slowly through space compared to those who were primed in a controlled condition.
That was the prediction. So they tried to activate the stereotype of the elderly.
And then the very interesting thing they had done here was that an experiment who was sitting
in the hallway with a stopwatch behind a newspaper and just timing participants as they
left the study and went to the elevator and this person was blind to conditions and just
recorded how long it took each of them to walk down the hallway and get to the elevator.
And then you can go back and look at which condition they were in.
And the big finding was that participants who had been primed with the elderly stereotype did indeed walk more.
Well we down the hall than participants who are primed with control.
Now here's the replication attempt by the Belgian researchers.
They administered the same priming task.
The idea was that they're going to activate the elderly stereotype, but they weren't very happy with this measurement device here.
You can see it's a human with a stopwatch and humans with stopwatches are potentially prone to errors.
They might be influenced by their expectations in some way. So they actually made a change.
This is a replication study, but it's not exactly the same and in a way that is important as our highlight in a moment.
What they did is they replaced the students with stopwatches, with infrared sensors.
So these infrared sensors are, strictly speaking, different.
They made a change. But on any reasonable understanding about what's going on here,
this is an improvement to the study design because they replaced a measure that's prone to
human idiosyncrasy with a measure that's going to give an accurate reading in every condition.
So the asterisk I'm raising here is that when you're talking about an exact replication or a direct replication,
it doesn't necessarily mean you use the exact same materials.
It's actually okay, I would argue, to make certain changes so long as they're better tests of the theory on any reasonable view.
And that was true in this case. So unfortunately, they weren't able to find anything looking like the original result.
And to drive the point home, we actually went back to the students with stopwatches and they ran a second study where they told half
of half of the experimenters what the hypothesis was and the other half the experiment experimenters,
what it wasn't. In other words, they were priming them with expectations.
And here they were able to replicate the finding in the group of students with stopwatches who were aware in some way of the study hypothesis.
And so the implication was that perhaps what was going on in the original famous study,
it might have been that the experimenters maybe weren't properly blinded,
that somehow their expectations of what was going on was influencing their effect.
The timing measures. So the way this was written up by Ed Yong referred to what's called the clever Hans effect.
And again, I'll just ask before I don't go into too much detail here. Does anyone not know what the clever Hans effect is?
Just some people don't. So for your benefit, in the early 1900s, there was a show force that supposedly could perform mathematical operations,
and his handler would bring him around and he could do addition and subtraction and so on.
And everyone was amazed at the capacities of this horse. A psychologist at the time was brought along to try to figure out what was going on here.
Did the horse really understand arithmetic? And it turned out, based on a series of careful experiments,
that the horse was just responding to very subtle cues on the part of his handler, maybe that even the handler didn't know.
So when the right answer is hoof is hovering over the right answer, maybe the experimenter perks up a little bit like that or something.
And the horse notices that and says, okay, so I'm going to now give that answer.
Now, that's still an amazing capacity if the horse can read subtle shifts and the psychological state of the handler.
But it's not the the result that was being claimed.
What's important about the clever hand's effect is that this is experimental methods one on one in social psychology,
particularly when you're dealing with these priming experiments where you're doing very
subtle manipulations of the environment to try to alter human thinking or behaviour,
you have to be absolutely,
absolutely certain that the experimenters, everyone who's interacting with subjects, has no idea what the experimental design is,
lest they behave like the horse's handler and subtly give some sort of clue that brings about what's hypothesised,
but not by the mechanism that was hypothesised. So essentially the claim was in such direct terms that this very distinguished researcher
who'd come out with the original finding was guilty of making a Psychology 101 very,
very basic design error. Now, I'm putting an asterisk here because I'm going to come back to this story.
This is not how it ends, but this is this is how it begins.
Something about the timeline here that I think is important to to point out is that if you see the here in November 2011,
this was the year prior, there was this paper, this story, this essay published in The New York Times about a fraud case.
This is you might have heard of Dietrich Stoffel, very important researcher who had published all these findings.
And it turned out that it wasn't the case, that he'd just done sloppy research in some way.
He was just making up the findings.
He would open up an Excel spreadsheet, type in some numbers and pretend he'd run an experiment and then publishes results.
And he made this whole influential career doing explicitly fraudulent behaviour.
And so he was caught and found out there was this big exposé in the New York Times.
And so I think people are thinking fraud now is what's going on in psychology.
And this is this is difficult because if you have ordinary research behaviours that sometimes lead to a result that can't be replicated,
I think a lot of members of the lay public immediately said, oh, it must have been fraud going on there,
rather than seeing it as just a normal part of the scientific process. So that's something that was in the background here.
Another paper that came out around this time was a study by Lesley John and colleagues where they administered a survey to more
than 2000 practising psychologists and they made sure it was carefully anonymized so that the psychologists would be willing,
in a sense, to admit to behaviours that otherwise they might not want to admit to if they knew that they were going to be identified.
And these psychologists admitted to a whole range of what are now called questionable research practices,
which are normal aspects of everyday, ordinary lab practice that are in the grey area between fraud and and perfect methodology.
And when you kind of add them up, they lead to a higher likelihood of having false alarms published in the literature.
And so they would admit to things like stopping and checking their data every
so many participants to see if there was a statistically significant finding.
If there wasn't, they would collect some more participants and check again. And then as soon as it drops below 0.05, they say, yes, we got a result.
They would publish it. This was standard practice until a couple of years ago.
Nobody really was talking about it except for a few methodology journals that maybe this isn't what you should be doing.
So a lot of psychologists didn't know you weren't supposed to be doing this. Another study that came out around this time was this one by Darrell Bam.
There is a very distinguished psychological researcher who has this side interest in what's called parapsychology.
So he thinks that these sorts of phenomena that most, as it were, mainstream psychologists think are not true and couldn't possibly be true.
He thinks that indeed they could exist.
And so he published this study in the top journal in the field purporting to show evidence of what's called precognition,
the idea that some future event can actually influence your current decision making before it's happened.
So Ben presented this as evidence of precognition, whereas others in the field said he's used all the standard methodologies,
he's used all the standard statistical procedures, and he's come up with a result that just cannot be true.
And let's we want to throw out assumptions from physics and basic assumptions about the way the world works.
So maybe something's wrong with our methodology, right?
If he's using the best methods in the field and coming up with a ridiculous result, something's probably wrong with our methods.
This paper is also published around this time. Undisclosed flexibility and data collection and analysis allows presenting anything is significant.
And these researchers famously concluded it is unacceptably easy to publish, quote, statistically significant evidence consistent with any hypothesis.
Now, there's a whole other debate going on right now about what sort of statistical procedure should we be using in psychology and other fields.
The standard now is to use something called null hypothesis significance testing,
which rests on that famous P value when P is less than a five or sometimes less than a one.
Researchers rejoice and submit their their finding to a journal.
For 60 or 70 years, statisticians have been shouting at psychologists that You must not use this procedure.
It's an invalid inference procedure.
It's only gives you helpful information under an extremely rare conditions that almost never hold when you're running a psychology experiment.
And yet, because psychology professors learned these tricks in grad school and teach them to their students,
there's a sort of institutional inertia whereby null hypothesis significance testing is just carried on and on and on,
despite the fact that it's very likely to lead to, as I say, findings that won't be replicated later.
So in response to this, the editor of one journal Basic in Applied Social Psychology, a colleague of mine,
David Crawford, now actually took over the editorship of the journal and banned P values.
He said, You actually can't use this procedure anymore.
And a lot of psychologists about what the [INAUDIBLE] do we do, we don't know anything else to do. So this created a whole furore in the field.
Just to give a personal turn here. I had difficulty replicating a major finding in the field myself.
This was when I was an undergraduate student back in 2009 or eight or something like that.
The study I was trying to replicate, which I just assumed was true,
is published in Science magazine, and at that point it had already been cited hundreds of times.
Now it's about 800 times. So this is a very influential finding. The idea here is part of the so-called embodied cognition literature.
The thought is that if you're made to feel guilty or morally impure in some way, you'll actually be tempted to physically cleanse your body.
And the thought is that moral purity and physical purity are somehow connected in the mind.
There's evolutionary explanations for why this might be so that initially we had discussed towards pathogens,
and then that was sort of accepted by later evolutionary processes to have us feel disgust toward immoral events as well.
And so the finding here was that if you induce a feeling of guilt in the participants,
then they're more likely to want to wash their hands, more likely to select cleansing items on a shopping list compared to other items.
That was that was the finding.
Well, the first time I ran this, I had two times as many participants as the original study reported, and I wasn't able to find anything.
But so I wrote this up with my colleagues.
I thought I would try to not commit myself to the file drawer, but rather make this say, Listen, I tried to find this effect.
I couldn't find it. We sent it to the journal. And what was interesting was that the reviewers said, well, listen,
you know that the effect size that was published in the initial report was probably inflated.
Initial estimates of effect sizes are almost always bigger than they really are once you sort of get more findings into the literature.
So we said you should assume that it's much less than that.
And we eventually had to run with five times as many participants in order to satisfy this reviewer.
And still, even though we were using the exact same methods and materials,
we contacted the original researchers who very graciously walked us through the design.
There was no sort of attempt to hide what was going on. We still just couldn't find any evidence of this effect using standard procedure.
So we wrote that up and published it here. So this whole string of events was, I have to admit, very disheartening.
The area of psychology that I was training at the time, at the time was this priming psychology, which I thought was very interesting and exciting,
that these subtle interventions in the environment could be influencing the thoughts and feelings and behaviour.
And here it looked like this sort of thing was just crumbling before my very eyes.
So rather than continuing on to much further as an experimental psychologist,
I decided to take a detour and study the history and philosophy of science at Cambridge.
And it was there that I wrote a sort of thesis on this replication issue that eventually was edited and published,
as does this paper here with David Crawford. Now, he's the one I mentioned earlier who had its basic complexity psychology.
And well, I was there at Cambridge. I met this visiting scholar named Stuart Firestein.
He was there on some sort of fancy grant to write a book. He's based normally at Columbia.
And we got to talking about this replication issue and this crisis.
And he said to me, Well, what crisis? And I'd just written this whole paper saying, Oh, there's this big old crisis going on.
Have you been paying attention? So we get into these arguments over whisky late at night,
and often what happens when you have one person say there's a crisis and another person says there's not a crisis,
it's that you just have two different senses of crisis in your mind. And that's what I think was going on between Stuart and me.
So I'll tease apart two senses of crisis here. The first is crisis of confidence.
This is basically I'm there's something going on where people have lost their confidence and it's sort of descriptive claim.
It's not saying whether they're justified in having a loss of confidence.
It's just a descriptive claim. The other is a crisis of process.
And the way that I'm kind of summarising this is a crisis of confidence. Since people are freaking out and crisis, the process of science is broken.
They're actually. There's a crisis in the sense that we're doing this wrong and we need to make some serious changes.
So in terms of the first sort of crisis, the people are freaking out.
Sense what's going on. Descriptively, is this true?
Well, here's a paper this published around 2012.
This is a special issue on this. They said, is there currently a crisis of confidence in psychological science,
reflecting an unprecedented level of doubt among practitioners about the reliability of research findings in the field?
It would certainly appear that there is. So this is some evidence that there's a crisis of confidence in the sense I've just described.
Now, when I was in Cambridge and we're studying the history of science,
one thing you learn to be very cautious about is anytime you hear the word unprecedented.
This is another paper from that same journal where Roger Connor Sarala says, well,
crises have been declared regularly since at least the time of William Hunt, one of the sort of founding fathers of psychology.
In fact, to go back to the 1970s, as I promised. Here is one of a number of papers published in that decade.
This is the crisis of confidence in social psychology. I mean, if you can see it's published there October 1975.
So people were bringing up these same problems. Our sample sizes are too small.
What is the problem with the file or the prejudice against? And all that paper was a 1970s paper by Anthony Greenwald that I mentioned,
and everybody kind of acknowledged that there were problems with the standard method methodology and then they would just carry on doing it as before.
I think what's different about this time is the Internet,
this blog post that I mentioned and the response from the researcher and the further articles that were written up.
It was it was all out in the open. The dirty laundry of psychology was suddenly being aired in a very public way in The New York Times and so on.
And I think researchers now, the reason why this is sort of taken hold and persisted in the public consciousness
is because I think researchers are aware that we really have to clean up their act.
There's also a fascinating sort of public communication of science issue going on here as well,
because I think the public generally thinks that if a paper was published in a scientific journal, that just means it's a fact.
And then if somebody says, well, we had a hard time replicating it, then they say, Well, science is just garbage.
I can't trust anything, you know? And there's information coming out about health research or whatever.
You know,
the public needs to know that there should be a certain amount of failure to replicate because that means science is being productive and interesting.
It's trying stuff out that might not work or might not hold up.
If every result replicated, that would mean we'd be doing very boring, slow, piecemeal, bricklaying science.
So that's something I think we should talk about. Now, here's a survey that came out in nature just a little bit ago.
This is, again, the descriptive claim. Is there a reproducibility crisis? Something like 52% say there's a significant, significant crisis.
38 think there's a slight crisis and some small minority think there isn't. This is across scientific fields.
So I think that gives some support for the first sense of crisis. There is indeed a crisis of confidence, but is there a crisis of process?
Is it the case that science is in some sense broken? Well, here's a post from Stephen Porter Gorg, a researcher who says psychology is broken.
And what he's referring to here is a painstaking, years long effort to reproduce.
100 studies published in three leading psychology journals found that more than half of the findings did not hold up when retested.
And what he's referring to is,
is this paper that was published in science called Estimating the Reproducibility of Psychological Science by the Open Science Collaboration.
This is published a couple of years ago, raised a huge splash and it caused a lot of people to think that psychology, anyway, is broken.
So let me bring to the surface what the underlying reasoning is here.
In order to claim that psychology is broken on the basis of these findings.
You need to think something like this. If a field is not broken, most of its results should replicate when independent labs rerun the experiments.
Well, most of psychology's results did not replicate when Independent Labs reran the experiments.
So psychology is broken. That's what I think is going on here under the surface.
Now, I've just indicated here,
I'm not sure we should be very confident about this first claim that most results should replicate when Independent Labs rerun the experiments.
You have to have some prior sense of what you think the correct rate of replication should be.
It shouldn't be 100% because that means that we're not advancing at all.
It should be 0%. That would be pretty disturbing. Maybe it should only be 50%.
So until you know what you think the prior appropriate percentages when you're dealing with trade-offs
between exciting new research and sort of brick lane research to confirm previous results,
unless you know what the correct ratio is between those things, you don't have any grounds for saying that 50% is too big or too small.
But I want to focus on the second issue and just see whether it is the case
that we can conclude on the basis of this open science collaboration paper,
that that it is true that most of psychology's results did not replicate when Independent Labs reran the experiment.
Here's this paper. They looked at 100 studies from three major journals.
They ran each of those studies one more time with independent labs who had sort of they would consult with
the original researchers and try to come up with an agreement about how we should run these five experiments.
And what they found was that the mean effect size went down by about half in the follow up studies.
And the P values, the infamous P values were mostly bigger and mostly over that point five threshold.
They were, you know, point ten or 20 or something like that, which on some sort of naive view of what counts as a replication, was often taken to be.
Failure to replicate? Well, we have to actually dig into this a little bit to understand what we can actually learn from this.
This famous paper that came out a couple of years ago, what counts as replication?
Well, first of all, you have to question whether the materials were exactly the same as the ones that were used in
the original study and whether the number of participants was the right amount of participants.
So I'll just give one example here. I went and looked through those 100 studies to have all the original materials up online,
and in one of them they used about half as many participants as were used in the original report.
Based on this naive expectation that the published effect size was accurate,
which, as I learned from my Macbeth effect paper, that was not that's not true.
You have to have more participants than were published in the original study for most of them.
What's that? I only looked at the one and a handful of others. I can't go into study to usually have a much higher power.
That's what I thought as well, which is why so startled when I saw this particular one.
The reason why I looked at it was it was a former psychology professor of mine had run the study and it didn't seem to turn out.
And I went back and thought, what's going on here?
So this this particular example was a replication of a study by Paul Bloom and a colleague, and they use half as many participants.
So I'd have to go through and look at all the others to make sure whether that's a more general trend.
Also, we have to ask ourselves, what's the effects that we're looking for? Are we what counts as a replication?
Would it be the exact same effect size or the effect size within a certain bound?
So we're looking at the same P value or just any P value less than oh five.
So unless you have some further sense of what you mean by replication,
depending on how the results turn out, you're not sure whether it counts as a failure to replicate.
My hunch is that what we want to know is is not so much is the p value the same or the effect size the same,
but is there a finding here of interest that's comparable to the original finding that was reported?
Is there something theoretically or practically interesting that really does
exist based on our best assessments of evidence that we have available to us?
Now, here's why I think we shouldn't be very surprised about the results of this new study.
This is what we should expect given p b stands for publication Bias Prejudice against the norm when you're only publishing a fraction of studies.
Let's let's imagine you take a study and you run it under ideal conditions 100 times.
And it's the same study over and over and over again. Well, you're going to get a distribution of P values.
So it's going to be the same time.
Every time you run that test, the same p value and your effect size is going to vary as well over the course of those 100 studies.
Well, given that you only typically publish studies that work,
you know that the published findings are going to be on the high end of this distribution.
The effect sizes are going to be the ones that are a little bit bigger, the P values, the ones that are going to be smaller.
And so this means that almost by definition, the second time you run this study,
you're going to get a bigger P value because of regression to the mean you're going to get a smaller effect size.
In other words, you don't have to even run this experiment to know that the second iteration of any
published study is very likely going to give you a smaller effect size and a larger p value.
So this is the problem with repeating a study one time.
The p value, by the way, almost means nothing on one iteration of any study.
It only begins to accrue some inferential value over multiple iterations of a study.
And so, as I say, when you run a study one more time, your informational value from that is is almost zero.
You don't learn almost anything from that.
And this is exactly what you should expect, is that the mean effect size should go down and p value should go up.
So now I want to talk about how do we know we're doing an adequate replication?
I mentioned the smaller sample size which as I, as far as I know, only places that one case AA stands here for auxiliary assumptions.
So I want to go back to this study by Farge and colleagues and the replication attempts by dining colleagues.
So I told you that they gave the same priming task here. They tried to bring to the mind the elderly stereotype.
They replaced the stopwatches with the infrared sensors.
But I didn't tell you actually they didn't do an exact replication here.
They made a change as well in terms of the primary materials. Specifically,
they translated all the materials into French because they're carrying out the
study in Belgium and the participants would understand French rather than English.
Why is this important? Well, here's an still unpublished study.
I saw a previous draft of this sort of circling or circulating around in the grey literature.
I don't know what its fate is,
but Michael Rampart is a very senior linguist who wanted to look at this case of the difficulty to replicate the original finding.
The thing we have to pay attention to here is that this is a linguistic priming effect.
You're calling to mind a stereotype based on the connotations that are associated with certain words in English.
So if you translate just naively translate these words into French without showing that the same properties apply in French as apply in English,
you may very well not be doing an adequate replication of the original study.
One difference between English and French is that in English adjectives can be four nouns, whereas in French, domestic and after.
And so if you're using adjectives to prime and now that's going to be part of the stereotype it's important which all of these things come in,
in the language. And so Michael Ramkumar and his colleagues did a corpus analysis of French and English,
and they found well, generally and with respect to the specific words used in the framing study,
that when it comes to their experience of encountering the adjectives in the prime set in contexts where they.
She served as prime finance.
We can expect that the subjects in the original marginal study would have had something on the order of six times more experience.
In other words, the suggestion here is that the Prime was likely to be six times stronger among English participants than among French participants.
So the failure to find an effect could very well be due to the fact that the Prime wasn't strong enough to begin with.
In other words, it wasn't a faithful replication of the original finding. So there are many more examples of this that could be raised.
But what I'd like to point out is that replication is is hard. You often don't know if you're violating one of those auxiliary assumptions.
You might think that just translating the materials into French. Well, what's the big deal about that?
I have to do it in French because my participants are in French.
Michael has to do this complicated linguistic caucus analysis to show that actually that wasn't an adequate replication of the original finding.
And that happens because my grandma is a very good linguist and had the capacity to do that.
But in how many other replication times are these subtle things not being adequately dealt with?
It's hard to say. So a failed replication doesn't necessarily mean that the original finding was not real.
And therefore I think that replication initiatives like the Open Science Collaboration one
are unlikely to provide the most direct and compelling evidence that science is broken,
if indeed it is. That said, I think we have more direct evidence that something is wrong with the way we're conducting science currently.
We already have more direct evidence of a crisis of process or at least problems of the process.
And that comes from that earlier work by John and colleagues and others who showed that psychologists openly admit to
engaging in research practices that we know are very likely to lead to the publication of Type one errors and false alarms.
And so since we already have evidence of those more directly from the admissions of psychologists, if we want to have an intervention,
it might not be the best way to do that is to have these massive armies of replicating replicators running around,
trying to redo every study that was ever published. You have resource constraints on how to actually do that.
You have questions about which studies are worth replicating or not. It might very well be that in terms of intervention,
we need to focus on the problems upstream that we know are very likely to lead to false alarms rather than trying to count the number of false alarms,
if you see what I'm saying.
So I indicated already, here's the more direct evidence psychologists are admitting to these questionable research practices.
Key Hacking Harking, which stands for hypothesising after the results are known.
This is where you run essentially an exploratory study. You're not sure what you're going to get.
You don't really have a strong sense of hypothesis. You run some statistics and a couple of them pop out.
P is less than five and then you come up with some hypothesis after the fact.
They're going, Gee, what would predict that? I don't know. How about this?
And then you write up the paper presenting what are exploratory statistics as though they were confirmatory statistics, and that's a problem.
There's also quite a lot of work that's been done on the reliability of peer review, which is just not reliable.
You can test peer review by, for example,
embedding a bunch of errors in a manuscript and sending it out to a bunch of peer reviewers and seeing how many of them notice the errors,
and very few of them seem to notice it. There's also just problems with sloppy peer review cronyism.
There's a lot of politics of peer review.
One thing you learn if you ever work as an associate editor for a journal is you get the manuscript and if you have a stance on it,
you can sink it or or floated, depending on who you send it to for review.
You know that so-and-so is going to give it a crappy review or that he's going to give it a good review.
So it's not like there's this beautiful, objective process by which the most dispassionate and qualified reviewers are handling every paper.
So peer review. I think if we're going to count on this as a quality control mechanism,
we should be extremely concerned that peer review is not a quality control mechanism up to the job publication bias we talked about.
This is the issue of failure to publish negative results.
Again, if we have 20 labs running, it's essentially the same experiment and one of them gets it to work.
The the highest likelihood is that that's a false alarm.
And all the other studies, if we never hear about them, we don't have any way to know how much confidence you should place in that published finding.
And I want to share a quick story about this issue of the need to publish negative results.
I'm I was writing some papers on this topic and I was searching around in the literature.
I think I typed in the need for reporting negative results to come up with some examples I might cite.
And I found this paper that goes back to 1927 in the Journal of the American Medical Association.
So this is something that's been written about for ages. I have to tell you what this researcher said.
I'm going to read this thing mostly in because I think it's very clever. So it says, I'm to the editor.
One of the things we practitioners sometimes neglect is the reporting of failures.
In the Journal, Doctor So-and-so reported the treatment of six consecutive cases of warts with a certain injection.
I venture to guess that as a result of this publication, not less than 100 physicians,
perhaps several hundred, injected this substance into the patients.
Supposing that 99% get negative results, what happens? Each of them gives up.
The method is a failure and does not say anything more about it. And the treatment remains on record as an undisputed success.
Maybe 1% who meet with success will communicate with Dr. Sutton so that by and by he will
have quite an impressive series of cases which seem to support the original findings.
To practice what I'm preaching, let me now report that on November 30th I injected the substance into the left buttock of CBM,
a girl age 18, who was at that day complaining of 24 warts.
The present date there are 28 warts and evidence of aggressive changes in the original 20.
Or has not been seen. So now I have this big reveal, which is that the author of this is John Rosenberg, who is my paternal grandfather.
He died in 1949. I never met him. I never knew him.
He died when my dad was seven years old, so my dad didn't know him either.
And here it was that I found this publication where some many number of years ago he was
writing about this exact same issue that I happened to have been writing about at this time.
So that was kind of a fun, personal note I wanted to end on, and I'll just say thank you and leave it there.