So you've been learning about evidence based medicine this week, and most of you are interested in the area.
And I thought I'd talk a little bit about the past way. Where did the term come from?
What's its historical roots? And then I'm going to move on to a snapshot of the present and then particularly
talking about things that I see that need to improve in evidence based medicine,
directions that will go on in the future. So I have to begin with a definition, first of all, of what evidence based medicine is.
This is the standard one that's out of the textbook of IBM,
which says it's the integration of best research evidence with clinical expertise and patient's values.
Then you need the clinical expertise.
You can't practice medicine without it, and you need to communicate with the patient, find out what they're concerned about,
what they would like to know about the treatment options, etc. And that's part of modern medical practice.
The tricky bit is this best research evidence of trying to get hold of that as well.
And I actually see this as being involving two components.
One is having a sceptical attitude because most things in medicine don't work.
We have lots of theories about why things should work, but when we test them in practice,
most things, most treatments don't work, most tests aren't accurate,
and we need the evidence to filter out those things that are actually going to be very useful in making diagnoses and in benefiting patients.
And the other part of that attitude is not only being sceptical,
but also favouring knowledge that comes from experiments rather than theories about things.
So there's this so-called hierarchy of evidence which basically says good experiments go on the top, bad experiments,
etc. further down, and then our theories about things are right on the bottom, which is somewhat of a turnaround.
And it goes contrary to certainly what I was taught at medical school and is
still a controversial thing about how things should fit into the hierarchy.
These ideas, though, are not that new.
You could probably trace them back a very long way,
but one person who comes to mind is Al Razi in 900 A.D. So he was a Persian physician who worked mostly, though, in Baghdad,
and he tried to compile a list of all of the therapies that were being used across the world
and put together this couple of volumes sort of encyclopaedia that was like a pharmacopoeia.
But he was also interested in the evidence. Beyond that, he wanted to know that these things work.
He didn't write down just anything. So, for example, this is a quote from one of the volumes where he says, for once,
I saved one group by the therapy that he was using, while intentionally neglected another group, a controlled trial.
And he says by that I wish to reach a conclusion. So he was interested in he was sceptical, first of all, about whether things worked or not,
one of the full list, but wanted to forget in his volumes what things actually worked for patients.
He didn't have the ideas of randomised trials and blinding and all of those sorts of things,
but he had the clear idea of testing things in practice and at least using a comparative group.
Okay. Are we the first recognised controlled experiment though a very deliberate experiment was James Lynn Swann on board
the the Salisbury where he was treating scurvy and he tried out different things for scurvy and actually allocated
pairs of sailors to six different things and discovered the two that were on the the citrus juices recovered
relatively rapidly and started treating the others he's held responsible for for doing the first controlled trial,
though it took a long time to implement in practice.
The British Navy didn't adopt this for at least 50 years, and other navies around the world took even longer.
So this even back then, there was the evidence practice gap which lasted a long time, and other person around the same period was Pierre Louis,
who, like Al-razi, was interested in things that came from empirical observations, and he was made a major change, particularly with bloodletting.
So he he demonstrated that if you bleed patients later rather than earlier, they actually did better rather than worse.
Published his results. It was rather controversial, but the number of leeches being sold in Paris eventually plummeted over a sort of five or
ten year period after he persuaded people that they needed to use this numerical method,
is what he called it, that is making direct empirical observations.
A real turning point was Bradford Hill who laid down a lot of the principles about medical statistics and how to do experiments correctly,
and in particular wrote the principles of medical statistics.
And was the statistician involved in the MRC trial of streptomycin, which most people take as the first randomised trial,
very explicitly randomised trial though there are if you go to the James Lynn Library,
you'll find a lot of alternate allocations that were very close to Randomisation long before Bradford Hill came along.
But that was a real turning point. Was that particular trial in terms of the, the, the further application of epidemiology to clinical medicine?
A key person was Alvin Fine's team, who published his book on clinical judgement in 1967 and coined probably coined the
term clinical epidemiology and developed a lot of the principles that we now use.
So he was also a seminal figure and was the sort of grandfather of many of the clinical
epidemiologists who have been active in the recent past or who are still active today.
So he was also a very influential figure and helped develop groups in all around the world, including the group at McMaster,
who are probably seen as the fathers of the sort of clinical evidence based medicine movement.
So too interesting things were happening about the same time the McMaster Group at this new university were developing clinical epidemiology,
and this was Sackett and his colleagues. And at the same time,
Archie Cochrane on this side of the Atlantic was writing his book on effectiveness and efficiency from a public health perspective as well.
And I think that had reasonably similar ideas, though this group was closer to a sort of bedside practice of ABM,
but they both had the general principles that everybody else in the past had been
developing and and were trying to push them more to the forefront of medicine.
At this stage. It was still called, as you see, clinical epidemiology.
The term ABM had not been developed as a long history, but prior to the term being developed,
the birth of the term itself time when there was a Use a Guide series that was being developed for JAMA.
The McMaster Group had written a series for the Canadian Medical Association Journal on how to read a paper,
and they decided things had changed and moved on. And I wanted to write a new series.
I persuaded JAMA to use this, to pick this up, and they wanted a name for the series and they said, We can't call it clinical epidemiology.
That's really boring. Nobody gets interested in that. What are we going to do?
And so some suggestions were, Let's scratch out that. What about scientific medicine?
And I said, Well, everyone would claim that that got scientific medicine.
And Gordon Guyot pictured here said, Well, what about evidence based medicine?
That should be pretty irritating, shouldn't it? So I said, Yeah, that's not a bad term actually.
And so that's how this that's how the group so this, first of all,
this evidence based medicine group got formed and how the series got developed and the term was born.
So even though you could mark the beginning of it from this use guide in 1991, actually it had a long history before that.
This was a seminal turning point that because this really attracted attention and people started to take notice both of the series,
but of even more generally, shortly after the series started in 1993, Dave Sackett, one of the four sort of leaders at McMaster,
moved to Oxford and founded the Centre for Evidence based Medicine.
Muir Grey was responsible for getting him over and he was a general physician.
He'd actually retrained who'd been an epidemiologist and retrained in general internal medicine.
And at the John Radcliffe up the road here, he was doing one.
Rounds using this strange vehicle here called the evidence card.
The evidence card was a data projector, a laptop computer with a whole lot of evidence resources on it, including Medline,
the Cochrane Library, their past approaches, topics, past issues of the ICP Journal Club, etc. and had a printer on it as well.
And they would look up two or three questions per patient on the ward round.
It would take them 15 to 90 seconds to find that information.
And it changed importantly one third of decisions that they made in managing this.
So almost every patient, it would actually change a clinical decision.
As you might imagine, mortar rounds took longer. Carl Hennigan was actually a medical student at the time.
Dave Sackett was doing this and says that the ward rounds could take four or 5 hours, which seems ridiculous.
But in a sense what Sackett was doing was moving medical education out of the lecture theatre conferences,
etc., which he didn't go to, he never went to any of those sorts of things.
He said the best place to do medical education is at the bedside.
Why don't we do it where it really matters when we're making patient decisions?
Okay. You're all thinking, this is crazy.
Nobody would do this right or you're never going to do it.
So I put these rings down here to remind me, to tell you these guys were the Olympic champions of evidence based medicine, the gold medallists.
Right? This is thousands of hours of practice, searching, appraising, etc., to be able to get to this point where you could actually do that.
I wouldn't have dreamed of doing anything like this when I first started trying to do IBM.
Answering one question a week was quite hard enough. Thank you very much.
But you gradually get faster at it.
And after your point, I actually felt comfortable in being able to search in front of patients, but that took me a couple of years.
So it's actually a long curve to be able to get to do this sort of thing.
Okay, so things have evolved a lot. I've got some of the the products that have occurred since then here.
So the Cochrane Collaboration in the Cochrane Library started in the early 1990s,
the IBM journal Evidence-Based Nursing Bmj's clinical evidence I think was 1999 or 2000.
There have been these improved filters that I'll show you a little bit later in Medline called The Clinical Queries,
there's a database of physio physiotherapy called Pedro.
I'm developed by Rob Herbert in Sydney.
There's an occupational therapy database called Otsuka Evidence Updates, which are part of the engine behind this,
says all sorts of evidence products that keep developing and they'll keep developing, which is fantastic.
We need better evidence resources to make it easier for people to look up that information.
What David Sackett was doing was actually brilliant in using very primitive tools.
We've got much better tools now and I'm sure they'll continue to get better.
The literature is also improving. We know more about the methods of doing IBM, the Use of Gods series,
etc. Things have evolved to make it easier for us to read the structured abstract, the consort statement to present controlled trials.
The Grey Group has been working on trying to classify evidence more clearly to
get beyond just the simple levels of evidence that we use of randomised trial,
cohort, study, etc. to look at it in a more fine grained way.
There are areas that have been neglected.
We've recently been working on evidence based monitoring, so a lot of tests get done for diagnostic purposes,
but actually probably about one third of all testing is for monitoring purposes and that monitor evidence based monitoring has been a neglected field,
despite being one of the fastest growing areas of testing. And there's a big interest in knowledge translation.
And this is Sharon Strauss's recent book on the area as well.
So things are evolving and it's been specified now as part as a as an appropriate curriculum
to be part of medicine that we should train health care students generally in this.
So there was the Sicily statement. There's a regular meeting in Sicily,
and the curriculum was basically formulated around the four steps of the Abbey and formulate an answerable question,
track down the best evidence, critically appraise that and integrate the decision with your expertise and the patient's values.
Unfortunately, it's not necessarily happening everywhere.
It's very patchy in and in the degree of training around the world.
We did a survey of UK medical schools and got 20 replies from 32 medical schools.
A lot of them didn't have somebody who was who could tell us about what the ABM curriculum was, so we couldn't find a person who was in charge of it.
So this is this is an an optimistic view in a way, because I my guess is that the others are worse.
These are the things like being able to search databases, appraise therapy articles, understand confidence intervals and P values.
As you can see, most of the topics, at least the main ones get covered, but actually there's very little practice in them.
Okay. Most of the medicals, only about half the medical schools do some practice in these, and it's much more rarely assessed.
So it seems like this is sort of superficial coverage, but not actually training in the practical skills of ABM at the moment.
And it's patchy. Some medical schools, I must say, were absolutely superb,
and there are others where the term was a bit foreign to them and it'll be a generation or two.
But I think before we've really moved to it being considered just part of, you know, it's, it's just like having a stethoscope now.
You just wouldn't think of practising medicine without a stethoscope in, in 20 years time.
You won't think of doing it without understanding what evidence is all about.
And McKibben, who works at McMaster, did this interesting study as part of her Ph.D.,
where she looked at people in general practice or in internal medicine.
She gave them a set of questions to answer to What do you think the answer to these are?
And she said, okay, now do a search, see if you can find an answer, some information about that, and then re ask them the questions.
Okay, now what do you think about that to see whether it would improve their answers.
So in 28% of cases, they went from right to right, 13% of cases they went from wrong to right.
They corrected their answer. That's the good news.
The bad news is that right to wrong was 11%.
In other words, people could look up stuff, get misled by information and get the wrong answer because of their you know,
they looked up Google and found some trash website and were misled by it.
And the other disappointing part is the wrong to wrong. That's not quite as disappointing.
I think it's the balance between these two that overall there was a 2% improvement by doing a search.
A couple of similar studies have been done.
One is a group in New South Wales, in Australia and RICO careers group developed a system called Quick Clinical,
which was an interface to try and guide you to the best information to look at and go through guidelines,
systematic reviews to PubMed, using the clinical queries, filters that I'll show you, etc. And this was just with GPS and they did a bit better.
32% went from wrong to right and only 7% went from right to wrong.
Okay, and 40% still went from wrong to wrong. So at least here there was a substantial improvement in the number.
Correct, which is great. So I think quick clinical is actually a very good system.
But are we there yet? No, I still took minutes to do this and 40% of them are still going from wrong to wrong.
So there's still a lot of room for improvement in terms of either training people or improving the interfaces.
And I'm not sure it would be nice if we could just do it through the interfaces.
Somehow we found search engines that could make this easy.
A couple of other studies by Hirsch and Hirsch these were both by Hirsch on medical students and nurses, and again,
a little better than with the with the McKibbin study, but they probably had more room for improvement here.
They were basically getting more wrong in the first place. One of the problems in all of this is just so much research is actually poor.
So for the album Journal, we scanned 140 journals, primary journals,
and get about 60,000 articles per year and ask some simple questions about whether
there's a first screen about whether these articles are valid in these journals.
For an intervention we need the study needs to be randomised and have at least 80% follow up.
That's it. Okay for a prognosis study has to be an inception cohort.
The rules are actually a relatively low bar. This isn't the full scale critical appraisal.
This is a low bar. 5% pass that stage.
And then we do a test on relevance. We asked people about would this be important to change your practice,
etc. And we actually filtered down to a very small number of things that we actually pick up for the BMJ Journal.
But the consequence is that the number needed to read to find one valid article is about 20,
which means most issues of most journals won't contain a single valid article,
and the number needed to redefine a valid and relevant article is about 200.
So if you do a lot of reading, so actually, I mean, one of the purposes of journals like IBM is to help you with that scanning process.
Finding stuff is good. We need to be able to peer review it as well.
And unfortunately, our peer review system isn't particularly good.
So we should you imagine that the good stuff goes into these journals and the rest goes into the bias and confounding trash can?
Unfortunately, that's not true. One of the most cynical studies I've ever seen was Sarah Schroeder's where how this passed the Ethics Committee.
I don't know. They got 607 reviews of the BMJ and they inserted errors into the papers and simply
without telling the reviewers just to see whether they detected the errors or not.
These 14 errors, nine they classified as major and three and five minor errors.
And on average, less than three of the nine major errors were detected.
And unfortunately, our peer reviewers at the moment are not trained properly in critical appraisal.
They don't know how to detect the bad things from the good things. The last area I wanted to mention was the application.
I think this is probably the biggest one that we have to work on over the next couple of decades.
And that's we've got great systematic reviews now that's all happening in Cochrane,
but there's a real problem in the application of the results to individual patients, and there's a couple of areas.
One is taking the average result and finding out what it means for the individual.
But the other is the how to do it, the ways to individualise.
I think you can think of it in several different areas in chronic disease as you
can do a single patient trial of monitoring an adjustment in acute disease.
You've got to try and predict it and adjust for the predictive tools.
And finally, in prevention, you've got to predict the future risk for people at high risk, basically have more to gain than the people at low risk.
I'm not going to go through that one in detail because that's our sort of whole talk by itself.
But I'd recommend this very good book that Peter Rothwell has put together on treating individuals,
which goes through looking at very different slices of this whole problem.
Peter is a neurologist here in Oxford and it's a very good compilation of essays that were originally in The Lancet,
and it's an extended version of them that's in the book.
The other one I said that I wanted to talk a bit more about was the idea of some of what the treatment is.
I'm just going to give you an example of this. This is a paper that featured recently in the BMJ on long term benefits of reduced salt intake.
We've known for a long time that reducing your salt. Leads to lower blood pressure.
But we'd never been out. No one had been able to prove that it actually improved cardiovascular outcomes.
And this trial, the torch trial did well, sorry, the study did.
Here's the description of the patient of the sodium reduction that was in this paper.
Individual and weekly group counselling sessions were offered initially with
less intensive counselling and support thereafter specific to sodium reduction.
So I want you to try and imagine what you would be telling a patient next Monday if you wanted to advise them about salt reduction,
picture, the sorts of things you're going to do with them based on this information.
We'll try to track this down so we track through the references in the papers.
It was one of one of the ways that we tracked this down. And here's the fuller description that's in a journal you have to pay for bottom line.
So this is not free information that BMJ research article is free because they make all their research articles free.
But here it said, this is an individual session followed by ten weekly group 90 minute sessions with a nutritionist,
followed by a transitional stage of some additional sessions.
Topics in the weekly sessions included getting started certain basics, the morning meal, mid-day sources of sodium,
the mind meal, planning ahead, creative cooking, eating out food cues and social support.
The sessions included sampling of foods, discussion of articles on sodium, reduction in problem solving,
and patients kept diaries at least six days a week and urine sodium were measured.
Today, no one imagine all of those like anyone.
Imagine even half of them you probably wouldn't have.
So you couldn't pick this up from that description. And even this I first of all, it sounds impractical.
This is not something I'm going to be doing in general practice next Monday.
But even if I was absolutely dedicated to this,
I couldn't replicate this without getting hold of the manuals and seeing a lot more about how this whole process actually runs.
So this is not a replicable intervention. It's a nice proof of concept, if you like, but it's not actually something you should pick up on.
The editor of the BMJ had actually written an editorial saying All clinicians should be doing this.
And I thought, well, doing what?
So this this sort of thinking has led us to ask about the problem of getting the descriptions from trials or systematic reviews in practice.
So Carl Henning and I did a study a couple of years ago where we looked at the adequacy of the descriptions that were in the journal
things this is so these were studies that we thought were valid and then we thought were important that should change practice.
And we looked at 80 of them, which was a year worth of things from the Journal.
And the question we asked is, could we replicate this tomorrow if we saw a patient with it?
And the overall answer is just just under 50% of them were replicable.
Trials were better than systematic reviews. Drugs were better than non-drug therapies, as you might expect.
So the bad news is a lot of things are not replicable.
The good news is that we tried to fix it.
We wrote to authors, tracked down references, did all sorts of things to try and get the additional information that made it replicable.
And we could fix about half of this. Which is fantastic because it was about a day's work to do this.
I think it's the most cost effective thing I can think of to do in medicine that we could potentially do.
If you think of the costs of a trial and you spend one extra day to get the information that would fix one in four of those,
that is just incredibly cost effective. That's more cost effective than the original trial.
As a follow up to this. Ian and Ian Chalmers and I wrote a paper recently about the avoidable waste in the production of research.
And I've just been talking about the last part here, a usable report.
There are all sorts of problems in the reports that we actually have available.
One of them is the description of the interventions, but others are poor reporting of what was actually the primary outcome.
People swap around things when it actually comes to publication,
so there are bits missing that don't allow you to appraise the paper, etc. So that's poor.
But we decided there are actually four stages that we could break research production down into.
Are the questions relevant to patients and clinicians asking the right questions in the first place?
Did they use appropriate design methods? Did they ever publish it?
And was the report usable? The three of the stages we could actually quantify.
There are various things that says less than 50% of the articles actually have an appropriate method in design.
There are flaws in the way that the trials are set up, and there are flaws in not looking at previous studies.
Most of them didn't access a systematic review even when it was already available at the time of writing the protocol, which is just amazing.
The other appalling thing is that less than 50% of studies never get published in full of abstracts.
For example, presented at oncology, trial meetings, conferences, less than 50% of them,
sorry, about 50% of them have been published after a period of eight years.
And there's been a systematic review of Sally Hopewell that suggested that that's probably about right for the overall 50%.
So if you multiply those things together you can get roughly something like about an 85% loss that occurs in the usability of publications,
not counting this first one, addressing the right questions in the first place.
The world expenditure on research at the moment is about it's over 100 billion USD per year.
So you could say in a rough estimate of the waste is about $85 billion per year.
This is just an amazing thing that we're allowing to happen at the moment when some of it is very difficult to fix.
But other elements of this are actually incredibly easy to fix.
And a small investment could save billions of dollars worth of wasted research that we're not using at the moment.
Okay. I want to end on a brighter note than that.
So I'm going to talk the last couple of slides will be about team based, even digesting the evidence.
When I first started doing evidence based medicine, I thought of it as a sort of solo thing that you did,
basically as a form of continuing medical education. But more and more, I'm becoming convinced that you actually have to do this as a team,
because often the whole team in a practice has to make the appropriate changes.
It's not individual practice,
but that the coordination between different members is important and getting the appropriate training and infrastructure set up in order to do things.
So we run in my practice, a fortnightly journal club and I spoke to several of you last night about this next actions thing.
Not only is do we read the evidence, but we need to agree on what the basic conclusions the clinical bottom line is,
but also to organise what the next actions would be.
And I would like to see that I know what happens in happens very little in primary care.
There are several practices in Oxfordshire now that this is happening in.
Which is great. The other thing that I'd like to see happening is collaboration between these practices then,
because often there's a lot of effort that goes into working out how to implement something in the individual practice.
For example, there are things that have taken me months to get implemented in my practice to sort out the bugs.
And it would be nice that once you've done that, you could share that with another practice.
So we're trying to establish collaborations both here within Oxford, but we've got some funding to do this within Milton Keynes,
where we're getting the practices not only to run the journal clubs,
but also to share ideas across the practice and also work with the primary care trust in order to implement those in practice.
So where there's something where the trust would need to put in more resources to have something happening, they'll actually help out in doing that.
So these are called impact groups. Yeah, that was the best acronym we could come up with so far.
If anyone ever comes up with a really sexy acronym or name for journal clubs, I'd love to know.
And I just wanted to give some examples of some of the projects that they're doing carpal tunnel syndrome,
where you inject patients before you refer them, which is happens in a few places, but now there's specific trained people to be able to do that,
and that's decreasing the number of people that need referral for carpal tunnel syndrome.
One group implemented delayed antibiotic prescriptions and could record it through setting up an E Ms. code.
Smoking cessation has probably been the most successful.
That's been a coordination between both the practices of pharmacists and the C leading to huge reductions
in the number of people taking up at least nicotine replacement therapy for smoking cessation.
And there's a whole series of others that have happened as well. Okay.
So just as a summary slide, I think the gloss for IBM for where we are present and where we need to go in the future is half empty and half full.
There's been this amazing growth in research and trials since that first trial in 1948.
So we're now up to probably close to a million trials now, one and a half thousand per year,
but much of it is poor and synthesised or unusable, as we saw.
Search engines are improving all the time. So can you imagine trying to do ABM in the days when you used to have the Piper Index?
Medicus. It would just be impossible and we've moved on a lot by having it electronic,
but now we're getting improved filters and improved systematic reviews and ways of trying to organise that information.
But it's still very disorganised and little effort goes into that compared with
the the effort to do the primary research in the first place to make it usable.
So half, half empty and half full. And finally, the skills in ABM, I think they're increasing.
So if I'd done that survey that I talked about 20 years ago,
they would have said what's evidence based medicine quite rightly, because the term hadn't been invented.
But if I'd said What was clinical epidemiology, it probably wasn't being done.
It's now happening in many medical schools.
And I see interestingly, when I go around the world, there's a huge interest in this and it's happening in a lot of places around the world,
in the Middle East and Asia, as well as the sort of developed places of the of Canada, the US, etc.
But it's still very patchy and it's still ignored in many medical schools.
So I think the future is bright. But for those interested in doing work in ABM, there's a lot more still to do.
Thank you very much.