I've had a busy day, actually, a term. You've seen all these students, brand new students running around.
This morning I started teaching the year four clinical medical students about introduction to evidence based medicine.
And about half of them were nearly asleep with a lecture like that.
So what if you fall asleep? I'm likely to do to you what I did to them.
I walked right up to you and ask you a question. So be careful.
Okay. I'm going to take you on a tour of how I think some of the work we do in the centre and
how it applies to trying to improve the quality of evidence for better health care.
And while I'm talking, if you're online or anybody online between email, you'll be in trouble because I might ask you a question.
But you can go to evidence live dot org and you can see the manifesto there and look at what I'm talking about.
So you can look at the manifesto or you can look at it after and you can see it's all freely available.
Even though the BMJ version is behind a paywall, we published it in full on the Evidence Life site.
So here's a couple of papers. So when I put a paper up like this and basically say, Go and read it, because I've read it and it's really useful.
And this is a paper by Doug Altman, a 1994 editorial called The Scandal of Poor Medical Research.
And in that, he says, there are huge shortcomings in the way that evidence based medicine operates today.
And that's 1994. And here's a second paper that I was involved with.
Chris Greenhalgh wrote this paper, which is called Evidence Based Medicine A Movement in Crisis.
But there are structural problems with even how it's used, how it's applied and its influence on health care.
And that actually is distorting health care in many ways, and much of that is detrimental to patients.
And then finally, I publish this, which I'm going to come back to with a chap called Ben Goldacre that some of you might have heard of.
That says, increasingly, the public is aware of the shortcomings in evidence based medicine.
So although doctors for many years we've been getting a IT in clinical researchers, we know this is going on.
The problem is the public is now finding out and once the public finds out, we have to do something about it.
So just coming back to this paper, it's a really interesting paper.
And this this line has made this cited thousands of times.
We need less research, better research and research done for the right reasons.
And this paper was done in 1994. And so if you look on PubMed and do a search filter into PubMed, you can look at what's happened.
Since that paper was published in about 1994, 95, there were ten enough thousand clinical trials, randomised controlled trials.
And look at the increase here. It's 32,000, it's now about 38,000.
So it's going up. So you could say, well, actually there's been a three fold increase in randomised trials, everything's okay.
However, there's also been this massive increase in observational research, nearly a fourfold increase.
That number is now more than 400,000 articles per year.
So there are 30,000 plus randomised trials.
There are more than 400,000 observational research, about 1.3 million per year in total articles on PubMed.
So amongst all that, there must be something that's making a difference to patient care.
Otherwise, what's all the purpose of it? What's going wrong?
There must be something in there that clinicians can go tomorrow and go.
I can apply this to patient care. Now, I don't know if there are any clinicians in the room.
There are a few. Do you feel like when you get to work tomorrow, you turn up and you go, Oh,
here's all the wonderful evidence that helped me do my job better tomorrow.
Wry smile among the room. In fact, what's happening is we're completely lost in all of this evidence.
And it's even harder now than it's ever been to tell the good from the bad.
So I'm really interested in that, and I'm interested in that.
If you take the 32,000 trials and say on average they have no effect on clinical practice.
Right? So if all the research we spend, the billions of pounds we spend globally is hundreds,
billions, it's probably about maybe a hundred billion on average.
Research makes no difference to clinical practice. Now, if you have a bunch of researchers in the room, they start to go, What do you mean?
Everything I do is beneficial? Surely. What do you think?
If everything. So if it makes no difference and some of it's harmful and a little bit of it is beneficial and you can.
Find out if it's a normal distribution of no effect. So and if you do that.
All of you on the course and all of you statisticians and researchers would go, oh, that's about two and a half percent.
And if it's two an alpha, then there should be 792 trials that impact on clinical practice every year.
Just FYI, just by chance, gentlemen, and saying this on average, no effect.
You would want it skewed one year. So it would be more, would you say, on average, no effect.
However, when you look at new treatments, what we find is in the published literature,
they're only slightly superior to established treatments when tested in trials,
and this small incremental benefit has stayed pretty stable over the last 50 years.
We only make small differences now.
Everybody in this innovation world would like you to believe that round the corner
is this brand new treatment that's going to make a difference to your life.
And if you say that, it's more than half will prove to be better and slightly less than one half to prove it,
then we should expect that actually we get loads of things every day making a difference to practice.
But it's a problem if I feel slightly more than half and there are at least somewhere between 2000 to 1500 per per year.
That means there should be about four or five things differently each day.
We should be doing it for clinicians. That should improve practice. That's not happening.
The reason it's not happening is because the quality of the evidence and what we report has structural issues in the evidence base.
In. The question you want to ask is why did so little research translate into actual clinical care and practice?
It's actually minuscule compared to the 1.4 million articles per year that are published.
It's actually really interesting when you think about it and you can go and talk to clinicians and not to new trainees will tell you everything's new.
But if you talk to Wiley old clinicians, some of them will say, Do you know what?
I've not done very much different for the last ten years. I know the managers are trying to make me do lots of different things,
but I actually do pretty much the same apart from on occasion, one or two treatments that come through.
Now what we do is traditionally we look at research translation in three areas external validity, internal validity and clinical significance.
And this is what we teach the sort of structure,
external validity trials have problems with external validity because they don't apply to the populations we see in practice.
There are systemic differences between the trials and the general population or too internally valid.
There are biases in the trials that prevent us from applying research in practice.
And third, there's an issue that statistical significance, there's loads of that, but actually there's a problem with clinical significance.
Now, it's not only me. You think of the problem because in 2015, Sally Davis is the chief medical officer, released a statement saying,
we need a review to restore public trust in the safety and effectiveness and effectiveness of medicine,
because patients see doctors overmedicating and clinical scientists afflicted by conflicts of interest.
And she based it on two studies, two really important issues cholesterol lowering treatment and antivirals.
And she based on antiviral for the following reasons.
Now, this is work I've been involved in, which is the systematic reviews on neuraminidase in the Cochrane Library,
and I've worked with Tom Jefferson and colleagues and we looked into this and there are problems with tracking down the data.
So I'm going to take you back in time. In 2003, this is Archives of Internal Medicine released the systematic review.
And in that systematic review it says all symptom of it, which is Tamiflu,
reduces your risk of lower respiratory tract complications, antibiotic use and hospitalisation in both healthy and at risk adults.
It doesn't do that.
Okay, because this study included ten studies, included ten randomised trials and one of them was done by this chap, Professor John Treanor,
and one of them was this is the only publication that exists for the largest ever single trial
of Tamiflu and John Treanor were cited as the first author when you can see what he says,
when we tried to track down the data, he basically said, Well, I don't even remember participating in this trial.
I don't even know you should speak to the manufacturers. You can read it in a piece by Deborah Cohen in 2009,
300 word abstract have over a thousand people included in the trial with use to inform what we do now.
What we found out with 60% of the data was never published, was never in the public domain.
Yet we stockpiled this drug and spent nearly £500 million.
And this is a quite an influential find because what it also tells you about is what the FDA called below the water margin.
Does this work? So it didn't want to do that.
Wanted to. Here we go.
Above the waterline. All you see journal publications and conference papers.
Well below the waterline is a whole set of data which are called case report from clinical study reports that
exist on electronic patient level data that exist around a trial and are available if you go asking for it.
So we went asking for it to try and sum up what was going on.
And in 2013 they agreed to release all of the trial on all trial data on Tamiflu.
Now, remember this one I showed you the it's called m76001 300 word abstract.
That's what you base your decision to stockpile on or do you base it on the clinical study report?
We actually have 9809 pages of data on the trial, the benefits and harms.
So I didn't realise all of this information and all this evidence existed and
what we were doing with bathing our decision making on very short abstracts.
It led me to come out with a position to say millions was wasted on Pluto in 2014.
That's my statement. No better than paracetamol. In fact, it's worse than paracetamol.
And I thought that was it. We would change practice and policy based on this.
In fact, when I went to the Health Select Committee on this and I put the arguments for that,
it a week later we spent another 50 million on the basis it was going out of date.
And we still stockpile this drug today despite its lack of effectiveness and all of the issues.
So taking that aside, what was going on for that led me to this position where I realised there were a whole reasons why we need better evidence.
I could find 20 reasons underpinning the need for better evidence and these are the 20 reasons and they're all in the published literature.
And when you when I looked at Tamiflu,
I realised all of these were exhibiting all of these features in the Tamiflu literature were exhibiting quite a number of them.
All of these features of the reporting, both ghost authorship, conflict of interest,
under-reporting of harms you can go on, were all present in the evidence base and are still present today.
So if you want to make good decisions and you've got all these problems and these flaws,
it makes it very difficult to then pass this evidence on to clinicians and then make a very clear decision.
At the same time, going back to 2015, I told you this based on the CMO review of statins and ultimately to where Ben Goldacre wrote this and I wrote
it within this editorial and while realise this and all the time we were talking about problems that were bias,
okay, a lot of these things are just introducing bias all the time into what we do, and there's a lot of it.
This is a tsunami of bias in research, not just financial academic conflicts of interest as well.
I meet a lot of researchers who think research is about using research to back up what they think is the correct thing to do.
What we should do is start from the null hypothesis.
This doesn't work, but much of what we do, much of what we research start from an alternative hypothesis.
This works. The point of the research is to help me show that that's not how it should work.
So another paper you should read why randomised trials fail but needed by a very important person called David Sackett.
If you're interested in evidence based medicine and you're an IBM nerd like me,
you should read everything that data is published in and it'll take you about 20 years.
Now again, because this published a lot. But he published it.
He said, Oh, so simple. When I read it, I thought, Oh, it's a bit complicated.
He said that the confidence of results is proportional to the signal divided by the noise.
Times Square, the sample size. Now bear with me. So I've tried to play with this confidence in the balance.
The benefits of arms is directly, directly proportional to the effect five inversely proportional to the amount of bias.
And then they end up with this very simple way of thinking about it.
So when you think about patient benefit, the bigger the outcome, the bigger the size of the effect, the more likely you are to have patient benefit.
Is that fair comment? However, the bigger the bias, the more by it's present, the less likely you are to have patient benefit.
Get that and if you have the optimal information side, no more data.
The right amount of data. That's why we do systematic reviews is to get the optimal information size.
And if you have the optimum information so you're more confident about patient benefit.
And that helps me think about what it is I'm trying to do and what we're trying to do with optimise the outcomes,
minimise the bias and achieve the optimal information size. Quite simple really when you think about it.
And so often we've looked at these and say, I just take the optimise the outcome, I'll take one example, optimise outcomes.
Well, look, there's loads of people who wrote about outcomes all over the shelf with low the patient papers that tell you about problems
with outcomes because there's one selective reporting bias of how outcomes with study findings from a cohort systematic reviews.
It would just tell you loads of harms are not published. So there's load the paper.
So one of the jobs we do. And your job is to, when you see a lot of information, is try and provide structure to that information.
And here's how my latest technology for providing structure.
Okay. All you tech savvy people, this is how I do it with Post-it notes on my wall and edit So you see.
Edit So it said why most outcomes do not translate into clinical benefits,
and then it edits into why clinical trial outcomes often fail to translate into clinical benefits.
And then you have a paper. That's how you do it.
And in the paper we provide a structure to the problem with outcomes because of our thinking about why are outcomes important?
What's going wrong with the outcome? And they're in four areas the design, the methods, the publication and the interpretation.
And in the design phase so badly chosen, I could spend the rest of this lecture talking about surrogates, composite outcomes,
subjective outcomes, complex scales, and lack of relevant methods that could talk about selective reported or even interpreted.
This is unbelievable, isn't it? I spent this morning with a group of medical students who, three years into their medical education,
did not understand the difference between relative and absolute effects.
And they're in year four and nobody's even bothered to teach them this and how this is used.
Outrageous. But coming back, I could have talked to you about publication bias and reporting bias, because that's what happened in tummy flip.
60% of the data was withheld. But we're going to give you an example here on switched outcomes,
the sort of stuff we do is try to fix the world in in our centre in a completely mad way with people like Ben
Goldacre and a bunch of medical students we set about trying to track switched outcomes in clinical trials.
Now to date has been 27 cohort systematic reviews of cohort and what they've shown is about
a third of trials are discrepancies between the pre-specified and reported primary outcome.
You would think in a randomised controlled trial you set out to report an outcome in a
protocol and then you report what you said you did in the outcome a third of the time.
That doesn't happen.
Yeah, and about one in ten times people introduce outcomes that they never even thought about in the first place, but they'll just put them in anyhow.
Now all the people here, we do a bit of research and understand one about going, Well, that's nonsense.
If you talk to a room of public, they go, That's nonsense.
How can you get away with that? What's the purpose of trial registries and protocols if a third of the time you can just make it up?
That's what's happening. So this is what we do and this is we didn't do.
So we went to trials in the top five medical journals over six week and we looked at the pre specified versus reported outcomes, but we did this.
We thought science is supposed to be self-correcting. Okay, so if you point out the error, you send them a letter, they'll correct the signs,
and then we'll have a journal publication that will have zero problems.
This is actually quite difficult to do. Most journals have a sort of you going to get it published.
You have to do that within two or four weeks. They have word limits.
They have certain things you have to do and achieve. So we did it and this is what we found.
We did 67 trial checked, nine were perfect, 454 outcomes were not reported and 357 were added.
That's a lot of outcomes, isn't it? Not reported or added. We sent 58 letters because now I'm a perfect 18 were published,
eight were unpublished after four weeks and they're still unpublished and 32 letters were rejected by the editor of the Journal.
These are ones you recognise these everybody wants to get published in these.
This is what makes your career. And here's what JAMA rejected all letters.
They said they were too vague and repetition. Between letters they only had 250 words.
And we had to be very clear. You in you reported additional outcomes.
This trial reported insert title because we had a standard template.
So they just said we're not accepting that the annals of Internal Medicine, they were even better.
They responded and said, we rely primarily on the protocol for details about pre-specified primary and secondary outcomes.
They even wrote an editorial about how crap we were and published it and said our methods were crap.
So we what do we do when we get something like that? What do you think we did?
What would. You do at this point? Go away. Sit down. Forget about it.
Project over. What would you do? Try to get more data.
What data would you actually specifically look for? Just do the same exercise again and again.
Improve them. Now, what we did is we said, can we get the protocol because we use a trial registry.
So you're not far off. But we look for the protocols because they said you used the registry entry, which is mandatory by law.
By the way, after one year you do both the report on the registry, all the data,
and use both the primary and secondary outcome measured on the registry at the time of registration.
So it's supposed to be by law, correct? They said we don't use the registry, we use the protocol, we vote for the protocol.
We couldn't find any of them that none of them were available.
They had the statement from Dr. Everson. Yeah.
What do you think we did in the email, Dr. Everson?
This is what he said. I regret to say these protocols are confidential documents,
but we would be able to send you the original involvement if you sign a confidentiality agreement.
That's outrageous, isn't it?
And then went to my favourite journal in all of the world, New England Journal of Medicine, and they basically said, You can get lost.
How can space constraints on an online journal? And then they said this.
Now any interested reader can compare the published article, the trial registration, and the protocol with the reporting with the view discrepancies.
If you have 4 hours available and some friends who know what they're doing, you can do that too.
It's very difficult to do this and it's utter nonsense.
So why does this all matter? So coming back coming back to our thought process, another great paper, this chap John, in all this, okay,
this guy publishes so widely even I can't keep up to date with this guy, but we just have a few in there that are nuggets.
And this one is very interesting because in there he talks about this.
He said this. He says most that I started where it started, most effects are small.
Yeah. And remember I said patient benefit is proportional to size of the outcome.
He backs it up because he's looked at that and said, well, if you introduce any minimal bias, you'll overcome the outcome you'll nullify in effect.
And so he's basically saying you have to be cautious if you introduce any element of bias.
So our job is to eradicate bias. And then if we see a small effect, he could be a really important effect.
We may want to put that into health care and inform people to do this.
But if you've got minimal bias, you're in trouble. And that's what led to this, which is the manifesto.
And in doing that, as part of evidence life, I spent a year going, talking to people on the road, talking about the issues.
What do they think biases are? What are the issues in the evidence base?
What should we think about correcting?
And if we corrected it all and we all got to work, we'd be in a position where we'd have some high quality evidence, and we published that in 2017.
And here, if you can read it, it's available on the website, as I said, available.
And here's what it says. He's the nine recommendations. And so each one specifically is designed as a task where people could look at it and go, Hmm,
I'm going to in my organisation while I'm doing all this stuff that everybody wants me to do for a ref,
and an impact factor could actually get to work and do something useful.
So explain the role of patients, health professionals and policymakers in research.
That's quite an I mean, that's quite an obvious one, but actually it's quite difficult to do really well.
I put these for free, free, because this is what we're focusing on at the moment.
We're focusing on increasing the systematic use of existing evidence.
It's really important at the moment. You might see things like we're going to be doing social prescribing in primary care.
In fact, this morning I got up this morning and I realised we're going to be doing group consultations,
yet people in policy pick these out there and say it's a good idea and they'll often go Right, what's the existing evidence?
Why does it exist? How might I use that to inform what I do?
So that's about why we really big importance about doing systematic reviews.
But actually we may have to do for policy.
A much more rapid approach is to make sure that you've at least a little bit of evidence when the design services make research evidence relevant,
replicable and accessible to end users. I think that's really important and it covers a lot of things.
If it's relevant, that means it has to be designed in a way that the outcomes are important.
At the end of the day, they're not outcomes that are useless.
You have to be able to replicate it in some way. We have to do it more than once.
So at the moment we've got a problem. You might see one single large clinical trial for new drugs and that's it.
There is not enough money now for people to do publicly funded trials for many interventions because we don't see that's worthwhile.
We have to replicate much more of what trials are happening in different settings, and then we have to make it accessible and available, don't we?
And then this would use questionable research practices, bias and conflicts of interest.
That's what I'm really interested in. I think it's really fascinating. Questionable research practices are endemic in university organisations.
In fact, universities train all the people who then go into industry,
and we basically train them to be in some ways pretty poor at what they do because academic
organisations fare much worse in terms of publication bias than pharmaceutical industry.
If you go on online and look at one of the products in the IBM data lab is the trials tracker.
You can look at a paper that came out a couple of weeks on on the European Union CTA trials,
and it will show you that academic organisations fail to publish their results within one year of completion on a registry,
which is the law, by the way, far worse than industry, because the industry now sees it as a quality marker.
It's an easy one they can fix. But academic organisations, we are terrible in this institution.
We do it about two thirds of the time. In many institutions it's less than a third.
So somebody funds research. So my Ethics Committee approved that.
You do the trial, you give people a treatment and a placebo,
and then after one year you fail to make it freely available on a registry which is accessible by all which is the law, by the way.
Now it's the law as of this year and then conflicts of interest.
So this is evidence like I don't know anybody.
No, this lady, Margaret McCartney. Amazing, right.
A GP activist and can make your life a nightmare when she chooses to be and a great speaker and she said this is evidence live.
She said we have to do something about conflicts of interest.
She's really worried in primary care that not only is it in research, she's in clinical practice.
Now, people are involved in developing apps and all sorts of interventions and then getting the federation to take them up with no evidence.
And then people are not even describing they have commercial conflicts of interest.
And I think conflict of interest is a huge problem in the next couple of days.
It's going to be stuff about trans vaginal mesh coming out,
which shows you the huge problem in surgery of conflicts of interest, real problem in how we operate.
So we have decided with next year in mind, by the time we get to evidence log next year,
we're going to try and sort out the mess of conflict of interest so we can get a
statement that makes sense because I think some conflicts of interest are all right.
You have to accept people involved in discovery, have to have conflicts of interest.
So it's okay for you if you're involved in discovery, to take money from industry, to fund the research, to discover things.
And I think that's okay. We can separate that out. The problem comes when people who are involved in discovery want to go.
I want to be involved in effectiveness as well. Can I cross the line?
I want to do the systematic review and be on the guideline and tell people why this is amazing
and not tell you I've got a conflict of interest financially at the end of the day as well.
And to me, that prevents a problem, because by the time you've been involved in discovery,
you are at a position where you're academically and financially involved in the outcome.
And so we've got this problem that is, I think, now so entrenched we can't see a way out of it.
However, what did I say? We showed you that one of the things we do is we try and solve things,
try and make it stir things up quite a bit trouble and see if we can make things shift in a little way.
So we started I sent an email, she sent in email and said, I'd like to write an email to Sarah Wollaston,
who's chair of the Health Select Committee, pointing out this issue with conflicts of interest.
I wrote an email back saying, Let's get to work then. And in the evening we constructed this email that said, Dear Sarah,
we think there's a huge problem with conflict of interests and the government should take this up.
And we think that actually the GMC is a register where I, as a doctor every year have to give them money.
Don't know why I give them money, but I'll give them it. And they put me on a specialist register to say, call your GP, you can look up my number.
It's 4731643 and you can see that I'm allowed to practice as a GP.
So if you come and see me as a GP and you think, Is he really a doctor?
Sure. You might say, God, he can't be. You could go on and you can say, and I'm there.
I paid my money. I'm on the specialist register. I've been appraised.
However, what you can't see is who's paying me.
And people have tried to do it in a sort of nefarious way. We do it in general publications, but it's all a bit of a myth.
But why, at least as a doctor, couldn't we start on the GMC as a register and say, every year you have to declare Who's paid me in the last year?
And over time you'd be able to see where I am. You could look back over ten years and couldn't say I was being paid by GSK either.
It's being paid by Roche. Who was being paid by. Well, that makes sense because you would want to know if you were having a surgical procedure.
Wouldn't you want to know if your surgery was being paid for by Johnson? Johnson shouldn't have the right to know that.
So we wrote to her and only last week she sent us an email back.
Thank you for your email about doctors declarations of interest.
Because of the dissolution of Parliament. I am sending a copy of my letter and I will aim to ensure that the committee is accurate.
Keep you up to date when I receive a response.
So she sent a letter to the GMC saying I want to know why you felt in this out and if not, why we will then need primary legislation.
Either you're going to do it or we'll have to do it.
And so the purpose of that is to say if you start to take a small steps thinking about the quality of research,
thinking about these issues, you start to think about not just the problems, but how might you fix it.
And that was the editorial that said, when Ben said, let's start fixing it.
For 20 years we've been talking about the problems.
In 2015, we started to think about solutions, and if we can solve some of these issues, structural issues in the evidence base,
we'll be in a much better position when we produce evidence to be able to give it to clinicians who can share it with patients.
At the moment, there are too many problems. So I want to finish now and this is what the poor quality of medical research is wider in knowledge,
yet disturbing with the leaders of the medical profession seem only minimally concerned about the problem,
make no apparent efforts to find a solution. So what's interesting, when you look back in time, you see that it was already there 20 odd years ago.
People like Doug Goldman, who unfortunately died earlier this year, were telling you what you should do just before he died to go.
And I met him in the pub and he said, and this is on my left again.
He said, we have to get universities to take this seriously.
And at the moment it is a bit of a joke. It depends on who you are mentored by.
Which group you are in will have a different view of what does quality mean.
I know that a lot of groups in this university don't really matter, but we're probably way ahead of the curve compared to many other universities.
So I think. That's a very important statement in terms of what I should be doing, what you should be thinking about doing next.
Thank you very much.