Good afternoon, everybody. I don't call him again.
If you don't, I am director of the Centre for Evidence based Medicine, Professor of evidence based Medicine in the weekend and evening of a GP.
And I'm also a dad of two.
I'm going to talk to you about and to take you on a journey around the idea of real world critical appraisal in evidence based medicine.
I know many of you today have been on your staff introduction to IBM on the mythical Earth,
and David will be talking to you and Ball about introduction to media and some of the issues about critical appraisal, study design and so forth.
But what I'm really interested in is the panacea is how do we solve the problem of when you have a piece of evidence at the bedside,
you can apply it to patient care then, and you can make decisions really efficiently and quickly, and you can do that in real time.
So that's what I mean by real world critical appraisal, as opposed to the critical appraisal that I see taught a lot is sometimes I've been
to sessions where it's been like 4 hours long and it's been a checklist of 20 items.
Do come in, sit down, we won't show you anything.
It's a checklist. And by the end of it, everybody think I'm never going to do a critical appraisal again.
Okay. And somewhere in your journey, some of you will have been on that journey.
I've been there. And I found that to be demotivating and a real barrier to evidence based management.
So this is a bit of a philosophical, but where I am now at the 24th years of teaching that I'm thinking slightly differently about some of the issues.
Okay, let's stop. So this is a really unique book and it's worth reading.
Effectiveness and efficiency in biology, cookery.
And because it starts at this point of criticised a lack of reliable evidence behind many of the commonly accepted health care interventions,
I think that's a unique starting point in the evidence based unit.
Not that long ago, though. 30, 40 years ago, yeah.
You know, Juni, so evidence based medicine is relatively new in terms of where we're able to.
Second. Then moving in that journey you'll have got his a David Suchet traditional diagnosis of three.
The traditional definition of evidence based medicine still useful to think about the integration of research,
evidence with patient values and clinical expertise.
And I always ingrain that because where is most of it's gone wrong in the last 20 years is because
people have taken that ham and they've certainly taken that out and they form evidence based medicine.
The truth about research, evidence and all practice can be based on research evidence.
And once we've accumulated all the evidence and we produce all the guidelines, then somehow we'll be in a position where we'll know what to do.
And thinking about where you are. So in terms of the steps in APM, you're asked to learn to ask questions efficiently.
I've been doing that today and search for evidence effectively and you need to be really skilful at this.
So that's the first thing is a skill.
What defines if anybody is any good at a skill is whether they need practice or not and whether they do it regularly.
So a lot of people say to me, Well, how do you know where to look?
And I go, Well, once you've been practising law, you'll get the feeling for it and you'll be really skilful on it.
But once you find some evidence,
the problem is and this is an editorial by Dale Goldman in 1994 that sets out one of the problems with evidence is there are huge
shortcomings and a lot of the evidence what would be ideal if you would ask a question and find some evidence and it was so well done,
it was only indexed in PubMed and then you'd be able to use it because you believe it.
But that's not true, is it? And so there are huge shortcomings in lots of the evidence.
And one of the things is he published this in 1994, and he said, what we need is less evidence, but more higher quality and more better evidence.
And it's interesting. Then you've come in.
That's all right. Nice to see you. We've got presents as well.
Wow. That's never happened to me before. People brought this up.
He said, well, look, 1994 was what's happened.
So actually there has been a three fold increase in the number of randomised controlled trials in that time, but lots of better high quality evidence.
However, when you look at observational research, it's a similar trajectory.
You go from a hundred thousand to nearly 400,000 observational studies.
Well, that's not better quality if you're thinking about therapy.
When you look at Medline overall, you've gone from 450,000 to about 1.2 million current number.
So the first thing is it's really easy to get published.
Because a lot of us are publishing a lot of stuff on a daily basis.
So when you think of all that, even if you just go back to the 32,000 trials on average.
Yeah, bear with me. And if you say let's say on average, they have no effect.
And it's a normal distribution. Now, when I'm in a room of researchers, now they go, What do you mean, no effect?
My research always makes a difference for patient care. But if you say it had no effect and it was a normal distribution,
then you should expect about 800 trials a year to impact on clinical practice because to announce the Senate would be positive,
two and a half percent be negative, and all this stuff in the middle would be sort of sort of nonsignificant.
Wouldn't it balance itself out?
Well, if we could produce a society that produced 800 new trials a year that impacted on clinical practice within about four or five years,
we'd be revolutionary in our approach to medicine. Things would be changing.
That would be about 2 to 3 changes on a daily basis,
and we'd be in a position where you'd have to stay up to date so often because you've got a bad way of doing it.
But everybody in the room who's allied to clinical care would certainly be thinking, But that's not what happened.
When you actually ask people like in clinical settings, they generally notice they do about four or five things differently in a year
and four of them are probably harmful and maybe one thing from beneficial.
And in fact, when you talk to research is you would expect with all what we do that we skew this way to say
actually more of it should be beneficial at the outset because we designed research really well.
We are funded at the outset and so surely then we should be able to say what's the number we expect to see impact from clinical practice?
It should be more than 800 a year. But it certainly isn't.
Now, always when you ask a question like that, somebody is out there.
If you go backwards in time, somebody out there will have done something before.
And this is a study that compared new treatments compared to established treatments in randomised trials.
They said, How often do we find new things make a difference to the chip care?
And in effect, they say new treatments are only slightly superior to established treatments when tested in our teeth.
And this has stayed stable over time.
Over the last 50 years, there been no incremental improvement in the ability to show that we make a difference to patient care.
So if you think about it, slightly more than the number is about 1000 to 1500 trials should impact on clinical practice,
which is a huge number when you think about it.
That means we're in a position to really radically change health care and health care system that every day should be going right.
What we're doing different today. But we all know that's not the case.
So there's a significant problem in disconnecting evidence and evidence based medicine that exists.
And one of the key things is why does that occur? Where does that come from?
And why does so little of that research translate into practice?
It's a really interesting phenomenon. What's going wrong or is it all perfect research?
Why are we not translating that into effective care on a daily basis?
It's a really interesting point. Now, traditionally, when we've looked at the three main problems, we think of it this way.
There's a problem with external validity in the results of the trial.
Just do not apply to the populations we think in practice.
Somehow all that research is in different populations.
The second issue is to say there's internal validity, so there's something wrong with the structure and how the methods are applied.
And there are, although they're positive, there must be huge shortcomings in the methods.
Easy, fixable, but must be that. Or the third thing is to say, okay, we know there are outcomes,
but they may be positive or maybe they're just not significant to improve patient care, and that's called clinical significance.
So although they're statistically significant, they may not be clinically significant.
Okay. So when I have problems like this, I tend to do a lot is I go backwards in time.
And I go and look and read papers about what other people have thought about this.
And this is David Sackett, who, when I first came to Oxford in 1993,
was first director of the centre here, and I worked with him until he died a few years ago.
But he wrote this paper which I found fascinating paper is there's only one formula that you're
ever likely to meet and the whole of evidence based medicine really to understand what's going on.
And then that formula, he said, it's really easy. And I look at it and a formula is ridiculously simple and looks like this.
I have to say, I find that not ridiculous. A simple. And so when I see things like that.
So he's had the confidence in in in in the benefits and harms of her treatment are directly
proportional to signal inversely proportional to the noise and ah factor of the square root,
the sample size. I don't find that simple and very that simple and I'm a simple person.
Okay. I like to simplify things in my life. So I played around to it.
So what we found is a confidence in the balance of the benefits to harm are equal to the effect size, are inversely proportional to the bias.
And that's why we got to and then develop it some more and end up with a very simple equation about how you might first start to think about research.
Yeah, you have to have a simple reference because what happens is when you put in all this critical appraisal and everything to work,
you have to have a visual way of thinking what's going on? So if you find some patient benefit, if you want more patient benefit.
It's directly related to the size of the outcome, isn't it? The bigger the outcome, the more benefit.
Everybody happy with that? But I think that's like proportional to the bias.
The more bias, the less patient benefit. Happy with it?
And then it's food is proportional to the optimal information size.
That's why you do systematic reviews. And it's not about the 32,000 more data.
You have to get to a point where you've got enough data you can stop.
Some people would like to say you patient benefit equals big data, but actually it's not.
So when you're thinking about patient benefit and you do doing a systematic review,
if you have a randomised controlled trial in front of you, you're thinking in your mind, do I have enough information?
Okay. So patient benefit is only directly relevant when you've got the right amount of information in front of you.
So that allows you to think if you can take that on board, you can start to think a bit simpler.
When you see a piece of information, a piece of evidence, you can start to ask some simple questions and I will take you through that unit.
You think about the outcome? To what extent is this outcome important?
You would want to optimise the outcome. You want to minimise the biases, and you want to achieve the optimal information source.
That's what you really want to do in a really simple way. Okay.
So when I think about this for me,
I decided that when I look at pieces of research I actually thought about when I'm asked about confidence in the result, a patient benefit.
I start with the outcome. Doesn't matter how many times I've done this,
I tried to force myself to look at the biases but can never do it because it's too dull and it's too boring.
The immediate thing I do is I say, To what extent does this outcome make a difference to my patient?
It doesn't matter what piece of information, whether I'm looking at an abstract. Do you notice what you do next time you look at a piece of research?
The first thing you do is you go home. What if you have cancer?
You do not say, let me look at the method of allocation concealing what you could do.
I'm okay. You can be quite explain how I think we go back.
So and the reason is I'm going to if the answer is no, you might consider stopping immediately and I'll come back to it.
So that's the first thing. You've got a mind set now of a very simple equation that patient benefit is proportionate to the outcome,
inversely proportional to the biases, and you need to achieve the optimal information.
So if you've got a very simple way of looking at all of research and you can plug that in and go right and then apply that.
Okay. So when I think about when people are engaged with questions that I'm starting to teach and I go from there,
I ask them what groups of people think, does this make a difference?
So you've got about 30 seconds now. The person next to you, you're going to ask them and go, I want you.
Everybody's going to have to either say, do vitamins increase or decrease mortality?
You've got one of three ounces. Yes.
No, I don't know. And I'm going to give you 30 seconds to talk to the person next to you and just decide what your answer is going to be.
And everybody's got to have a vote. But exactly what kind of.
But you know.
Not like that's okay.
Oh, yeah. Now, who said who in your pants?
Who said yes? I'm going to yesterday in the room.
Okay. So we've got one more question. It says increase or decrease?
Okay. Yes. All right. Okay. Well spotted. When did go to them?
Who said? Don't know. So every answer is right in this one, but.
Okay. Yes. Well, actually, it's interesting when you look at the thing.
Look at the evidence. It increases mortality that these patients are consuming.
And dolphins were 1.3 times as likely to die as we the control for slightly more.
But if you said no when all of the trials were combined, antioxidants may or may not have increased mortality,
depending on which statistical combination is important. And if you test that, don't know.
You were thinking the increased risk of mortality was associated with beta carotene,
possibly to many better age, but was not associated with the use of vitamin C or selenium.
Okay. So it's really interesting it when we talk about effects, are we much more engaged in thought processes?
We're interested in the effects of. If I'd have asked you to say let's judge whether this is allocation, concealment or not,
you might have gone, oh, this is very interesting, but we start from this premise.
Okay. Right. Question two. You've got 30 seconds again.
You can answer this one. All right. Uh huh.
All right. I'm going to give you 30 seconds again. Should you do that?
Even people who have had a heart attack. No, no. The like.
No, it's okay.
Now he's voted for?
Yes. People forget about focus or I'm going to come back to you.
He's voted no. No. We left a note in there. Yeah, he voted.
Don't know who about. So the Solomon knows and don't know then.
Yes he's probably coming the last. So the people. He voted yes.
Why did you vote yes. Let's go.
Like guys like. When a person I do not how to talk and this saturation situation is below 95%.
You should give oxygen. Okay. So you're going. Yes. Because you're using specific pieces of information.
So there are it's only use in people below 90 to saturation.
So you're picking out specific pieces of information that you can say yes.
So you're adding that in. If you go back here, what happens is the further back in time you go, it's interesting.
When I was a medical student, we used we felt oxygen was a lifesaver.
That was the drug of choice in heart attack. And in fact, now you say you're not sure you want to put specific limitations in there.
All right. What about the know people in the room now with those who say no as this because it's only recommended one is less than 90%.
Okay. Okay. But it's a yes.
Well, I. Huh. Well, even if he said no, you would still.
Right? Actually, because actually there were similar death rates in both groups suggesting oxygen neither helps no harm in a routine use of oxygen,
people have had a heart attack. So you're right.
So if you said no, you're saying no because you say, well, the routine use in this population is not indicated.
What about the Don't Know group? Why do you say?
Don't know the questions. Okay. You think the questions are clear?
Okay. Okay. So you go nuts. Okay.
And I think if you don't know, you could also say, well, actually,
because we still don't know the answer to the question, but it's also about being specific about the actual scenario.
So everyone has a population that applies to, doesn't it?
You picked out this has to apply to somebody, this research.
It's not just an effect. It has to apply to somebody. I'm applying already.
You've already done it in this small scenario. In fact, I've applied this to a specific population below 90.
That's my peak above 94 different population.
So when we think about the outcome, you have to start to think about who does this apply to her through interest?
So we don't just decide that. In fact, it's about who does it apply to.
So that makes me think really interestingly about when I think about evidence that comes not just about our founding effect.
It has to apply to somebody who doesn't. And that's quite interesting.
And so long. Okay. So when I do this, look, this is where way back again.
Does it make a difference? It's much more engaging when we think about, okay, if the answer is no, consider stopping and I'll come back.
It's not always true. Now, one of the things that we found is the quality, the outcome.
When I talk about optimise outcomes, there are huge problems with an outcome.
So they may be an effect that's appeared.
And we published this piece of research because every time I looked there was a huge number of those surrogate outcomes,
problems with composite endpoint problems with when you analyse the data.
Here's outcome selection patient reported outcome 70% of the outcomes don't mean anything to patients and not the outcome.
They're interested in collective reporting. And so we ended up with a thought system in outcomes.
Try and say, Well, yeah, man, there's loads of problems with outcomes.
Even when you look at the outcome, you've got to be aware and these are my top three surrogate composite and subjective outcomes.
They're so prevalent now that they're an epidemic and people don't even are aware of them.
So they don't say, well, it found a significant effect. And they go, Oh, actually this is a weird composite outcome.
Or I tend to make 10 to 13 won't actually make any difference to patient care down here.
It may be a relative measure, it may be thrown in, might be multiplicity.
So there's so many outcomes, you are bound to get some of that positive. It could be selective reporting.
So we said there's just there's an outcome. It has to be a high quality outcome that's going to affect patient care.
So you can radically start to think when you look at it, does it make a difference if you've already said,
oh, well, it applies to somebody, what's the quality of the outcome?
It's not just a statistical effect.
You're actually starting to think to this, actually make a difference to my patient will in my head do I think this is applicable?
And when you do that, you end up with more than one.
So I've ended up with more for patient benefit, although I'm thinking of outcomes applies to three questions.
Who does this apply to? Does it make a difference? And is the treatment feasible?
Because that's my third point. Don't you see thinking about does it make a difference?
You can't actually do it, isn't it? And you can see this all the time appear.
So you see it in diabetes exercise program, 61 on one sessions.
And I think who's going to pay for that in our CCG? Not a single person, end of my assessment.
So it's a sort of free item that we think can get rid of a significant amount of research really quickly.
So when you're faced with your next piece of research, if you just say, who does this apply to?
Does it make a difference and is it feasible? My current estimate is that get rid of about 95% of all research and that's probably an underestimate.
I'm going to try and do that and say we're going to apply this to a prospective number of studies and say, how do you do this?
There are some caveats. So if the answer is no, consider stopping it.
If there are times when you get an answer that's no and you shouldn't stop.
It may be something where it doesn't make a difference to patient care, but you're actually doing it still.
And that might be an example of too much medicine doing something.
And we just looked at one recently and I wrote a piece about it which was monitoring and type two diabetes self monitoring.
And Type two diabetes doesn't make a difference to patient care, but we still doing it.
That's. The second is sometimes you miss an important outcome.
So you might have a treatment which is potentially less risky, less invasive.
But you're looking at mortality is no different. So a good example of that might be new oral anticoagulants that no better than warfarin.
But actually the patient relevant outcome that might be of interest, which is they don't require monitoring.
So you've missed an important patient outcome. So you often have to say, still check.
Okay, so that's what I do when I see a piece of evidence and I get rid of a load of evidence really quickly and you can't quite see this at the back.
We're going to I'm going to publish it and actually put it plugs in to the PCO.
So that is a population that the outcome there, the comparator and that the intervention.
So it plugs into your pico that you've been taught. So it fits really well with that.
So the first question is when you get your population is, you know, the population of ask the question,
the first question you ask of the evidence in front of you is, does this evidence apply to the same patients I've got in front of me?
Okay. The second question is, does this outcome make a difference?
And you're checking for some important Mexican. You probably got relative measures, but you need an absolute effect.
Poor quality outcomes in my surrogate composite and subjective are the three
most important you should look at for the reporting of a 15 that the top three.
If the answer's no, why should you carry a part from my two caveats if it's too much medicine or you've missed an important outcome?
But you could stop right there. And you get rid of a load of research really quickly.
Now they begin to worry, like before they got to teaching the whole course about critical appraisal.
Frankly, I think it's really important and why really relevant.
I think when you move through and if you yes. Stage, you have to be aware of the other treatment options.
Yeah. And you have to be aware of the comparator.
And if you are where it's an adequate comparator, you should move on and it could be potentially practice changing at that point if you're happy that.
So the reason you need to know about the alternative is often you might be seeing one anti-depressants, a placebo, and that's where the LNP put in.
If you aware of the other treatment effects that are already in place, you might be able to say, well,
this looks like an important benefit, but actually it's far worse than what we're already doing.
Now, why is this important? Wherever I go, I go into a room and I'm going to remove Johnny Depp from the fake.
Tell me about the last treatment you need and they'll go, Can I say?
Tell me, does it make a difference to my patient? And nobody can answer this question.
Most clinicians in their practice don't understand what treatments they use and how much difference they make,
and can they quantify that to the patient, because we should be automatically able to do that.
We should all be able to say, If I take older patients like you and I give you this treatment,
this is how many will benefit and this is how many will be harmed. That's does it make a difference?
So you can revolutionise your approach in the way you teach this stuff.
And think about it. If you say, okay, let me just fix this, really make a difference.
And it's not just a function of the effect size, as I've shown you, and you move through this flow.
Now, what's interesting about this flow is what I'm saying is, is and that's the 20 years of teaching it.
This was never in the ask a clinical question. Acquire some evidence.
There's a drop off there. Isn't that.
Because you ask lots of questions, but you can only acquire the evidence for some of them because can't physically.
There's not enough time in the day for me to find evidence for all the questions I have when I'm in clinical practice.
But when I acquire the best evidence, I can very quickly do these three things in probably 60 seconds.
I can do it mainly from the abstract.
I might have some knowledge gap where I'm like, Oh my God, I'm not aware of what we're currently doing in practice and what the effects are,
but that's a limitation of my clinical practice for this.
That is relatively easy. And that means less should get free to the quality of praise.
Because by the time you're doing this, you're thinking about stuff that really matters.
Do I believe this effect size? And if I do believe it, I should be thinking about applying it in practice.
And you're also this is what's really interesting.
You're also making a judgement about the size of the effect and the importance, and then you're looking at to say, what's the impact of these on this?
Now, if you've got a very small effect, you'll start to think, Wow, this could be overturned by any introduction of bias.
Well, if you've got a much larger effect you think will have an impact on patient care, you may tolerate more both at this approach.
But the problem I see is that most people do appraisal and think about doing that job.
Let's move on a few more great criteria. Moderate, end of story. They're not related to this.
The outcome. What impact does this appraisal have on the outcome of interest?
Is it going to overturn these results? Now, that's really important in your journey of thinking about this,
because there's a real disconnect between people's ability to understand bias and its impact on outcome.
Generally, people disconnect the two in like the two different functions and like bias doesn't affect the outcome in any way.
And in fact, what you can see often is we don't even have to form an appraisal because I often I'm being asked to assess effect.
At this point. I'm saying, well, I'm looking at observational research. I don't need to do much more.
I'm looking at the wrong type of research.
Obviously, there's a huge interruption of bias, so we've only got an observational effect with a small, small effect.
So that's not that important to know. Somebody tells me about an observational study.
I'm like, Well. It's interesting,
but it's not even worth carrying on to do the appraisal at this point because I would want a randomised controlled trial
and I definitely would want to see a systematic review at this point to be able to say we will implement this in care.
And it's really interesting. So here's an example. Okay.
Last week the Science Media Centre debuted on the first time on this list that we send out a study and they go, here's a study from the BMJ.
It's coming out tomorrow, published on the 4th of October.
It's 3rd of October and it's 3:30 p.m.
Could you get back to it by 4:00 with your clothes on?
Mm hmm. And I need a cup of tea because I've just come from one meeting, and I'm like, oh, my God.
And he's going to take me 5 minutes to open up my computer and get dota2.
So literally, if a boil it down, I've got about 10 minutes to give a quote and think about this today.
Okay. And this study is about availability of evidence of benefits, overall survival and quality of life of cancer.
Until I go, Oh, this is stuff I'm really interested in. I could bypass it now.
Plug in a critical appraisal, spend 3 hours and try and write from what I said.
Realistically, I've got about 8 minutes to 10 minutes. So what do I do?
I plug in my formula. I'm basically plugging in my formula.
I think about the outcome. I did write one thing about the bias, but I started with the outcome.
What does this outcome mean when you go to The Guardian and pick up the Guardian from last week?
You'll see that I said, professor of evidence based medicine at University of Oxford to describe the lack of improvement with regard to the Bible.
It's disappointing. It's hard to understand why half the jokes were approved in the first place if they provide no clinically meaningful benefit.
He said, Basically, if you're shown you that in terms of cancer drugs, you could get rid of 50% drugs immediately.
If they do, they make a difference. What's the outcome?
And so you can plug in a system where you go, Oh, I could get rid of half of all cancer research just like that when I'm reading it.
But what tends to happen is we think it's really technically difficult to engage with research, and so we don't bother.
But you could actually start to understand you can read this stuff really quickly and you can plug in something.
You could have started with the biases. But in another quote about the bias is the lack of robust studies.
But nobody enjoyed that. Nobody picked that up. They prefer the outcome just as we do.
So journalists preferred outcomes probably prefer outcomes. We prefer outcomes.
And that's what really switches us on. And when you think about outcomes, most of the things we see are very small, statistical, tiny effects.
So again, there are smart people out there who wrote about this,
and this is Jonny in Odyssey's work who says basically when you're assessing tiny facts or public are more common in the literature,
call should interpretation is warranted, since most of these effects could be eliminated with even minimal biases and their importance is uncertain.
So once you understand the size of the effect you're working with, and if it's particularly tiny, which is highly prevalent at the moment,
you understand that if you detect any biases, which is what you learn about,
you start to think, Oh, if this is going to overturn this effect size really quickly.
So often you might not need to do an exhaustive appraisal to go, Oh my gosh, we just introduced some biases.
I'm already worried.
And what you're trying to do when you're doing that critical appraisal is to see what's the impact of this bias in overturning that effect on.
That's what's really interesting about a system where you can do it really rapidly.
Thank you very much.