Hi. Hi. Welcome, everybody. Just to say who I am.
Some of you I know. Some of you know me. My name's Carl Hennigan.
I'm director of the Centre for Evidence based Medicine and also professor of evidence based medicine here at the University of Oxford.
I'm also a GP, a practising GP and over the weekend, for instance, I was working in the urgent care setting.
But I'm going to talk to you about today is the distillation of my 20 year thinking in evidence based practice and where we are now.
To a group of us who are really interested in solving some of the issues around evidence based medicine, particularly the E in E beam.
So the concept is that I want you to read some of you while I'm talking.
I'm going to talk for about 35 minutes talking about the thinking of what I think the issues are.
And then we're going to have a discussion and you're going to all help me and help the group solve this issue and some of these issues.
And that's part of the mission today. Just to say the for those on Twitter, the hashtag is evidence life, evidence life.
Okay, let's stop. About 20 years ago, a chap called Doug Goldman wrote a piece in the BMJ, an editorial called The Scandal of Poor Medical Research.
And he basically outlined there were huge, significant problems at the time in how research was published and how it with disseminated.
And this was a scandal because much of research was going to waste.
And the problem is, over the 20 years, there's been so many initiatives that have been in different areas that have tried to solve this problem,
and they tend to be isolated and not work together. So, for instance, some of that might be reporting standards,
which are things like consort of try to bring together a system to try and improve the publishing of research.
But one of the problems is that it's all been a bit piecemeal in what it's done today.
So there have been little sporadic issues and little sporadic changes.
There's also a long history of the third of them trying to fix these problems.
But we want to pull them together through a clear a clear set of achievable goals
and a stronger overview of the strategies that work best to help deliver change.
Some of this was precipitated by this piece, which is an editorial which was about two years ago,
which Trish Greenhalgh led, which was talking about evidence based medicine is a movement in crisis.
And just recently with Ben Goldacre wrote an editorial about how medicine is broken and how we can fix it.
So there's been a 20 year problem, if you like, about the quality of evidence.
So some of you hear on the surgical sciences and some of you are on the MFC and IBN.
But remember, the definition of evidence based medicine is the application of the best available
research evidence to clinical practice integrating the values of patients in.
That's one of the important concepts, isn't it, the best available research evidence with your clinical expertise and the patient value.
So that still holds true when anybody comes to me and say that the problem with evidence based medicine,
my first thing is to say there is not a problem with evidence based medicine as an ideology as what we're trying to achieve.
There's a fundamental issue with the high quality evidence that goes into our decision making.
And over the next ten, 20 minutes, I'm going to show you that I think it's actually got worth and still go.
And that is editorial. Things are actually worse than they were 20 years ago in lots of areas.
So those of you are online at the moment, if you want, while you are online,
you can read this manifesto because it's gone up today on the evidence live site.
It's evidence live dot org slash manifesto.
If you just go to evidence live org, you'll be able to click through on this picture which says re DBM 2.0 manifesto.
So if you're online right now, feel free to read because at some point we're all going to read it because I'm
going to bring it up so you can think about some of the issues while I'm talking.
All right. So this piece of work, why we need better evidence.
Why do we need better evidence? I'm going to take you on a little journey of about 5 minutes of one of the most important
pieces of work I've been involved in was with in Tamiflu and the use of antiviral in influenza.
In 2009, I was asked actually by my sister fed my children.
I've gone on the phone line and NHS evidence all NHS 111 said that you drive to Northampton to get some antivirals for my
children and you should give them that because that's important in preventing complications in the swine flu outbreak.
You'll remember that in 2009 when you were all going to die. Yeah, okay.
You shouldn't really be here. You should have all died by now. But remember, this is why we need better evidence.
So at the time I went, Oh, actually, I don't know the answer to that.
And we decided to do the systematic review.
We did the systematic review in children, which showed that with a small benefit in symptom reduction, mainly fever,
which is about equivalent because it was fever about equivalent to what you'd get if you used paracetamol or one of the anti-inflammatory drugs.
And actually there was no effect on any secondary complications whatsoever and there were quite serious harms.
In effect, one in 20 children would have serious vomiting that might precipitate another visit to the doctor.
So I couldn't quite work out why we were using it, why were stockpiling it.
And I thought we would produce some of the evidence to show how this would change practice.
At the same time, the adult team, led by Tom Jefferson, were looking at the adult review.
And one of the key issues about the adult review was that they came to a problem when they tried to find some of the evidence.
And in finding the evidence, it wasn't what what happened is much of the evidence for the adults had never been published,
and they started to try and contact some of the authors and track down the data.
And that's when we combined our forces together thinking, well, this is all this is a really important public health drug.
We've spent millions on this drug and we promoted it in the W.H.O., the essential medicines list.
Surely the evidence should be available. So part of that process, one of the first people they went to was a professor, Fred Hayden.
And you can see here with on one of the biggest ever metre analysis down the bottom here,
what happened is this method analysis done by a chap called Lauren Kizer,
of which Frederick Hayden, with the corresponding author, had published ten trials.
But only eight of them, child. Oh, sorry. Only two of them child were ever published in the journal.
Eight were unpublished. So we went and asked them, Could we have that data?
So you send an email. And his response was, I cannot find the original files related to this 2003 publication.
The questions are not clear to me. They were very clear to me. Could we have the data, please?
But if the original data and probably study reports are required, they will likely need to come from Roche, the sponsor of the studies.
So we moved on. We then went to Professor John Traynor, who was actually the lead author on the largest ever single trial ever done in this trial,
with only ever published as a 300 word abstract in a conference in America, and it's called in 76001.
When asked about this trial that he presented, supposedly he couldn't actually, as far as he could remember,
the trial, published in JAMA was the only large study of death cell of Tamiflu.
He never participated in Channel four News because we went to the media, put it to Roche,
that Professor Turner said that he didn't actually participate in this study and 70 6001 and doesn't remember presented it at the meeting.
He actually wasn't there, didn't present it, didn't participate in this study.
His name's on it. So, you know, start to get worried about the what's going on in the research happening.
So we did go to one of the people who who'd published one of this study. This is a chap called Nicholson, Professor Nicholson.
He was actually on one of the two studies that were published. And his response was, he did not recall seeing the primary data.
He said that the statistical analysis had been conducted by Roche and he analysed the summary data.
So the lead author on a trial, he'd never actually even looked at the data.
He couldn't verify whether it existed. He couldn't even vouch for whether it was correct or not.
Now all of that, anybody in the room, anybody out there should start to have alarm bells going off.
Because when you tell patients this, they get really worried.
But somehow in research, we start to feel that this is just the way that it's done, this is how we do it.
And, you know, we have to keep it quiet from everybody. But when you start to tell people, boy, do they get worried at this point, what did that mean?
It meant that 60% of the data was never published. And the FDA talked about not only that was published,
there is this whole system of research that what they call is above the waterline is journal publications,
and below the waterline is all the unpublished data.
Clinical study reports, individual data, adverse events and all that exist.
But nobody ever gets to look at it for important decision making. Okay.
So what did that do? We had a breakthrough in 2013, Roche agreed to release all the trial data on Tamiflu drug.
We started to make progress and move forward. But look, this is the difference.
Look at the difference. This is m7001, which I showed you the 300 word abstract.
Yep. This is the exact amount of data that exist for that abstract 9809 pages in the Clinical Study Report.
See the difference now? Which one do you think will provide adequate information to even do a critical appraisal?
Never, never mind based World Health Organisation decisions on because that's the one that's been used by everybody until today,
until 2014, when we did the analysis that was used by nice used by FDA is by all these bodies around the world to make decisions.
The largest ever trial, 9809 pages.
So you get the idea there's a fundamental structural problem in how evidence is produced, disseminated, how it's reported.
I have to say, having worked on this, it's almost like if you work in an area for long enough,
you get to a point where you just cannot believe how bad the system is and how much we tolerate it.
But I think we tolerate it because we never really quite get under the bonnet through it.
We never quite know what's in our engine of our car. We just drive it until it goes wrong.
And when you think about it, that's what we do until we start looking under the bonnet and think,
Wow, this is a really old engine, doesn't quite work well.
And you know, by April 2014, the newspapers were reporting that we were wasting millions.
In the UK. We spent 473 million. On the stroke.
In fact, now we spend over half a billion. And in fact, we still stockpile it.
And I can come back to that at the end about the issues, however. That's one of the bits.
And I'm going to show you more examples, brief examples of that.
My own personal thought of journey, when I got to the point where I said, Enough is enough.
This is actually unacceptable. And this has been a recurring theme in many areas of evidence, and it's used in evidence based medicine.
So one of the things we've done, if you go on the manifesto,
you'll see I outline there are 20 reasons why we need better evidence and each one of them you can search.
And if you go through on the website, you can pull out relevant publications, relevant systematic reviews that show you these are real issue.
And when I was doing that, I thought, well, how many problems do Tamiflu trials exhibit of these 20 problems.
And so you can keep going through and they exhibit numerous problems publication bias the research is poor quality report reporting bias.
I showed you the ghost of which they have lack of shared decision making,
regulatory failing surrogate outcomes, too much medicine, prohibitive cost of drug trials to replicate them.
So actually, they have a significant number of fundamental problems that everybody has shown.
This is a real issue. However, irrespective of all them bits in the yellow government still stockpile the drug.
So there's a problem.
If you had a trial or a series of trials with that many problems, surely you would say the results are not valid irrespective of what they show.
You would get to a point where you go the somewhat wrong with the evidence.
So that's my own journey. In terms of Tamiflu, however, this.
Equally worrying is the growth and volume of evidence that has been accompanied by this corrosion in the quality of evidence.
And it's compromised, as I said, our ability to provide affordable, effective and high value care.
That's what's led to the version 1.4.
And I'll explain when we get to version 2.0 how we do that in a minute.
Okay. Here's some other examples that have influenced me on the way.
When I started Medicine in 2000, going back 99,
2000 with a particular Cole Vioxx that was marketed as one of the most effective pain relievers on the market.
And it had added exempt added benefits over anti-inflammatories because the reduction of GI side effect, however,
and within very quickly that drug became one of the leading blockbusters in the world failed in excess of $44 billion a year.
However, only when it got to litigation did we see that the company who knew this had withheld data and they'd only use the
on treatment analysis which minimised the the appearance of any mortality risk using the evidence based treatment.
So that means the only report, the effects in the people who are still on the treatment.
Well, the people who were having the heart attack were stopping treatment and were being removed from the analysis.
And you report not only to the public, to the journals, you report to the FDA, this issue.
And that minimises the risk.
Only when they found out what the it the intention to treat analysis was did they suddenly discover that this drug tripled your risk of heart attack.
And then he subsequently got withdrawn, but they withheld that.
This the editorial at the time from some of the leading editors said no, only with the drug companies.
Part of the problem was that actually many of the doctors involved in the trial were complicit in this failing, if you like.
And they were also to blame. Is a second example.
And anybody here to study 3 to 9. In the room.
David's right. Here's your homework for this week.
It's even got its own Wikipedia. Page 33293329 is now quite famous.
And for the ABM nerds out there, by the end of the week, you will qualify as a nerd if you know everything.
About 3329329 relates to a drug called paroxetine, which is an antidepressant,
one of the SSRI new a type of antidepressant, but it was marketed because it was relevant to adolescents.
Again, the company, when they did the trial, minimised the harms in adolescents.
In what's an effect called reporting bias by saying that some of the events where children had suicides,
they put them down to other potential outcomes and remove them from the analysis again.
Now, it was only in litigation against the company that it was discovered that actually the
rates of suicides in adolescents were twice what we reported in the published paper.
And in fact,
recent reanalysis and restoring of the trial under what's called the Restoring Invisible Trials recap has shown that actually might even be higher.
And that's the BMJ, which was only about a few months ago, another blockbuster used in adolescent Billy Boys.
The failed then actually increased rates of suicide.
In adolescence. So the exact way what it was trying to achieve, it was doing the exact opposite.
But with clinicians didn't know what was going on because there were failings again in the evidence.
So homework study 3 to 9. One of the things, again, here's what we started as a consequence of the Tamiflu with all trials.
All trials was started because half of all research is never published.
So at some point you go, what if half of the research is not published?
We know there's a major structural problem already. So by the time you're thinking about how much research makes a difference, well,
you know, already 50% of what happens is going to be likely never going to be looked at.
So all trials with truth try joints, they all trial should be registered and reported in full.
There's been major breakthrough and you can see even the companies now are starting to understand we need to be involved in this.
And in fact, there's going to be a European Medicines Agency announcement in the next week or so to say.
We're making all clinical study reports available as part of a new drug application.
What old trials showed me was that actually people working together can fix problems.
And this was a group of people of which the Centre for GBM was one, but it included people like things about phones,
included the BMJ Cochrane Library working together to try to make a difference.
It might sound like I'm painting the whole doom and gloom of the world,
but actually I'm just painting a picture of my journey through medicine as a doctor and as an evidence based medicine expert,
if you like, keeping an eye on what's going on. Here's another example, Rosie.
Go to that, another blockbuster drug that used a surrogate outcome of diabetes in death.
So it's it's a it's a drug that lowers your blood sugar.
And apparently, because it's a composite with death, it lowers your risk of death.
It never did lower your risk of death.
Combining the two end points, lower your risk of blood sugar, but you combine the two and point people for lower your risk of death.
In fact, what it did do is increase people's risk of heart failure.
And for every 250 people on the drug,
one extra person would would have heart failure because it causes fluid to be retained and it has all sorts of problems.
Again, not known at the outset, not reported harms, not put out there what happened to this drug?
It took ten years before it was withdrawn in Europe and in America.
It's still on it still is still on the regulatory approval list in America, but it's got restrictions in its use.
One of the things I say about people involved you're up to date with evidence based medicine is
is will get there in the end the truth will get there in the end if you understand the evidence,
you'll probably get there a bit quicker than everybody else because at some point we will understand the truth.
But on the on the on the on the journey there, we can harm a lot of people.
He's so one of the problems is we want to think about independent trials and how you do them.
But the other aspect, while I've been doing this,
the cost of trials have been escalating hugely at the front end, making them almost impossible to do.
So you have to do it in a big trials unit, in a massive team.
You have all this regulation at the front end and then at the back end, nobody cares what you do.
So to get the trial off the ground will be immensely difficult for you.
You probably need hundreds of thousands of pounds. You have hundreds of forms to fill in.
And then when the trial finished, nobody cares where the publisher is disseminating to the people who participated in the trial.
What you do with your vote whatsoever, how that can be ethical is beyond me.
So there's a real problem if we want to replicate trial results.
How do we lower the cost of trials? I put the surgeons in the room.
There are a few surgeons. I thought I'd rather be a surgeon in here.
Because the other thing is, when I come to devices, I get and I'm a world expert on the evidence for devices, by the way,
and that's because there never is hardly any evidence to go with many medical devices that are used,
particularly newly approved devices, because they're often approved with a thought process called equivalence.
And when we do equivalence, that means my device is a bit like yours.
So couldn't have access to the market marketplace in this is we made up a whole device and the
company and got it approved in an Australian notified body to the point where they would have said,
here we go. Not only did it once, we did it twice with a metal hip to the point where it's very easy to get devices that have no evidence approved.
And so not only we talk about drugs, but in medical devices, the problems are just as bad.
However, nobody knows because often there's no evidence to go with any devices.
Here's another one. This is just even recently. And again, this is a drug called Rivaroxaban.
EU approved the round table for secondary prevention after an acute coronary syndrome.
So that's the European Medicines Agency. Okay. You go to Europe.
In in in in Europe. The European Medicines Agency in America.
It's the FDA two approval route. So this is what European Union reports in May 2015.
This is what the FDA put today in 2012.
Approval, not approval. Approval.
Not approval. So we've got these situations where we've got drugs around the world that in America they're on the market.
In Europe they're not or the other way around. In Europe, they're on the market and America they're not.
And in America, they've seen that. Huge problems with missing data here. And I've just showed you.
We've we understood that problem already. Yeah.
Our regulators are going. Yes, there's a real mantra to approve drugs, to get them on the market, to keep business going.
How can we have a system in the world where we have Europe saying yes and America saying no and then not talking to each other?
Are they looking at the same evidence or not? I have no idea what's going on, but there's a huge issue to pursue with that.
And this is Carmelo Anthony with some of you this morning with beseeching wrote an editorial about River Rock,
the band to say we're now where are we but six years into the approval of this drug.
Probably somewhere around there for a year, four years for the ACF that they still under uncertainty because there are key problems with the trial.
In the trial. It's been found out that in the control group who were taking warfarin that the
machine used to monitor the warfarin arm was defective up to one third of the time.
So you compare in a new intervention to a crap comparator, and that's always been shown to be a problem in determining the effect of an intervention.
It's one of them problems that should be alarm bell for regulators and go, Oh my gosh, we can't accept this.
We need a new trial. That's not happened. It's been approved.
And here's another example. This is a one that's led by Ben Goldacre with a group of students.
It's called Compare. It's called the Centre for Evidence based Medicine Monitoring Trial Outcome Project.
And what we did is take the top five journals and started in about this time last year,
about October last year, and said every trial that's published,
every randomised trial that's published in the top five journals, we are going to check to see the outcome that was reported.
Was it same as the outcome that was pre-specified at the outset in the trial registrar or in the protocol that's available?
Okay. Everybody happy with that? Because you should start you put an ethics application, you put it on a trial registrar,
you shouldn't take these outcomes and they should get to the end and measure your outcome.
That good clinical practice that's in the. The purpose of research is to specify an outcome.
Okay. So we tracked 67 trials of them.
Nine were perfect. Okay. That means 58 were of the 58 that weren't.
354 So this is primary and secondary outcome. 354 outcomes were not reported.
That said they would be and 357 new outcomes were silently added.
So do you understand what's happening? You're doing your trial and you're saying, well,
let's pull some of the outcomes and let's put some new ones in its place and maybe we'll put the new ones in place that are positive.
You won't know that because you don't go. And we didn't go until we ask this question and go back and check that.
But what we also did is we sent 58 letters to the journals.
And this was a bit of work actually, because if you look at the journals, a pretty you know, if you think about science should be self-correcting.
They give you a window of about two weeks. Some of them give you three weeks.
But after that, you can't submit the letter anymore. Well, one thing if you see a problem that week four.
Okay. So we met that window of the 58 letters where there was an error.
We were pointing out an error. Only 18 were published. 40 were rejected out of hand.
And we've started republishing all this now. I have to say, the journals in this era, the way they're acting,
are not compatible with the scientific method and not compatible with high quality evidence and are huge problems in what they do.
And this is sort of simple project like you can do with a team are starting to show that.
And finally, here's the one that really gets me. Everybody goes on about shared decision making, if that's the way forward.
We are here to inform patients and this is a systematic review of shared decision
making and patient outcomes that's available in medical decision making.
It's a well-done, systematic review, and that's their conclusion.
Evidence is lacking for the association between empirical measures of shared decision making and patient behavioural and health outcomes.
That's because nobody's investing in the research and nobody's doing it.
There are so few trials in this area. It's unbelievable.
So we all talk about like, we want to do this, but actually nobody's doing it.
And so we have a conclusion that there are no tools. There's no way I can advise you what to do.
If somebody said to me how to inform my patient better or go, I don't know.
I don't know what to do. I'm doing it from opinion. Lowest level of evidence.
Okay. And then. So finally.
So, look, things have got so bad that the Academy of Medical Sciences has had to launch a new project on evaluating evidence.
And they've done this at the request of the Chief Medical Officer in the UK, Professor Dame Sally DAVIES.
There seems to be a view that doctors over medicate so it's difficult to trust them and that clinical
scientists are all beset by conflicts of interest from industry funded under their untrustworthy.
To see what she cited particularly Tamiflu comes into this.
But also the other one,
if you've have seen the huge issues right now around stuff in cholesterol lowering drugs that are causing all sorts of controversy.
I have therefore reluctantly I don't know why she should be reluctant to come to the conclusion that we do need an
authoritative independent report looking at how facilities should judge the safety and efficacy of drugs as an intervention.
Well, we come to a different conclusion. Actually, I don't want to know the ivory tower report that says let's just whitewash it.
If we've come to the conclusion that everybody interested in evidence based medicine should be contributing to the solution,
should be identifying one problem that they're interested in, and let's go and fix it, as opposed to just talking about the problem.
So that's where that evidence live.
Even manifesto to point out the creation of a manifesto that is developed by people engaged at all points in the research ecosystem.
That's everybody in this room, I assume, because you're all here who want to engage in fixing the problem, including above all,
patients in the public who indirectly fund but are directly affected by the outputs of the current system.
Okay, so let me try and bring this up.
Here we go. By the magic. I'm going to.
So if you go online now, you get to the manifesto and it gives you the preamble of what I've done and you can jump.
And there are nine points at the moment. Okay.
I'm just going to take you through I'm not going to say we and we have specific what do we want to achieve there?
Okay. So informed decisions. We want clinical trials.
We want to think about independent clinical trials, reduce the cost of a child.
We want to solve publication and reporting both. We want to ensure that we've got outcomes that matter to patients.
Evidence synthesis.
We want to understand that these are really important tools in decision making, but they should be informed not just by journal publications.
They have to have more informed evidence to underpin decision making.
There's a short bit here, but look, research methods are in a myth right now in clinical trials.
And I was having a talk here at the back with Martha about this earlier is that we are like the head mufflers of clinical trials in effect,
because our job is to ensure that trials are done with rigour and that bias is to a minimum.
And to do that, you have to be really officious, rather like a headmaster,
and you have to come and beat people on the head and say You're doing it wrong. And they go, It's all right.
We've always done it like this. What's the problem?
Well, if you think about it, the problem is, is effect sizes are getting smaller and smaller and smaller and there's less room to play with,
which is what should happen if you adding one drug on a new drug.
It's really important that these trials are done so well that even if the effect is small,
you can believe to believe that in fact, often that's not the case policy.
We'd like to see policy that's based on evidence and that has a strategy to inform and educate policy.
It's really important that our policy makers use rigorous and robust evidence to inform what they do, including routine data.
Everybody believes routine data. The answer everybody wants to go away from trials to observational data because big is better.
There are huge issues with the communication of evidence, whether that's in the media,
whether that's in how we do it from press releases into the journals, what gets in front of the public.
But actually, we've got to do a better job on that. And then our final two is individual accountability.
We perceive that the real problem with conflict of interest in all aspects of medicine in terms of who's paying who to do what.
Particularly when it comes to guideline development. 20 years ago, we used to pay individual doctors who do that round the world in places.
Now they pay them to be on the guideline, to pay them to be involved in policy.
Huge problem. And then there are huge problem within the concept of regulatory evidence and how we inform and do that better.
Okay. That's the manifesto that you all have access to.
What would like to do? If that's the manifesto I've discovered is for you at the end, is to have you effect.
We are going to iterate and develop this manifesto with input from people who give us feedback.
And when you give us feedback, people who do, we're going to write to at the end and say,
Would you like to add your name to this manifesto when it gets published in the BMJ?
So you'll be credited for having input, but you have to give us feedback and you have to say some things like What changes do we want to achieve?
What do you think measureable changes could we achieve? Do you know about actions currently underway that could achieve this?
And when we do that, it's going to be a large collaborative project.
We're going to go out and do many talks with already set and talked about where we're going to do a talk and
we're going to record them and get some of the people who really matter in the room to tell us what they think.
But the most important people that matter if people like you, because if I sit up here and you go away and think,
that was a nice talk, I'm going to go to the pope tonight and I'm going to forget all about it.
We're in trouble because if you guys don't think this is important enough to even warrant a comment, I'm in trouble.
What we're trying to do is it's trouble because this is really important stuff.
If we can spend the next ten years fixing the evidence as opposed to talking about all the problems that exist.
We need to do it in a collaborative way. And we need more people to start to think about how are they going to contribute.
Now, that's the point. Where I go over to you and let you know that next year evidence life we will have a final
version of this manifesto will be published in the BMJ and we will have an action plan.
Will in the main focus of discussion to say right, how do groups of people contribute?
Take this forward? How do we achieve some change in the evidence base?
Thank you.