Thanks for coming, because it is actually terrible out there, isn't it? It's quite nice in here.
Terrible out there. So we're talking about safe and effective drugs.
And thanks for coming forward so I can see your face and you don't fall asleep.
And if you do, I might ask you a question, so be careful. But I want to take you on a bit of a journey.
And this journey starts for myself about 089, but it starts about 40 years ago.
So this is Archie Cochrane, 1979.
He was telling viewers that what we need is a critical overview of all of the evidence
and it should be periodically adapted and all the relevant randomised controlled trials.
So if you really want to know about medicine,
we need to have all the healthy teeth and we need to get them all together on certain topics and understand them.
And generally that led to this important document.
This is effective care in pregnancy and childbirth, which is in trauma out there with other people we brought together in pregnancy and childbirth.
What was the forebearers of the Cochrane Library, Kiran?
But this was an attempt to bring together all the evidence and systematically review it and put it into one volume.
So this is 99 that the state of the are you see here real milestone in the history of our figure on the evaluation of care
not led to the Cochrane Library thinking because that's what went on three years later to set out an editorial that said,
here's the Cochrane Library. But at the same time, what this was starting to show with work that was happening in a number of publications think
Dickinson in controlled clinical trials and Vivien himself with some other people in JAMA in 1990.
And that's about the same time with a worrying concern that although the result was that randomised controlled trial evidence,
there was a particular issue happening called publication bias,
that it seemed that some of the trial data was never published and this was the start of the issues that arose at that time.
So that was 99. Very quickly, we've gone from overview of the evidence into a problem where probably application by fast forwarding 20 years later.
In 2013, I became involved in an organisation setting up the old trials group which basically said
we would like all trials registered and reported in full for a very simple ask if you like.
And it's been signed by 9750 people from people in this room have signed and 748 organisations.
So one of the fundamental principles that led to my involvement,
no trials and why we go trials with the particular research I got involved in on a
drug called Tamiflu and Tamiflu with the drug that was used in influenza outbreaks.
And it had lots of claims. Now, this could to take you back a little bit into the history of thinking about Tamiflu,
the first thing you use is it was evaluated about 2000 and about 2005.
Tamiflu used with minimal. Hardly used.
In fact, the company broached with thinking of giving the license back to Genentech.
But in 2006, Donald Rumsfeld, the manager of the known Unknowns, was part of a report in the U.S. that said,
we're going to look into how many predicted death would be caused by bird flu.
You remember all that hysteria about bird flu.
So let me ask you a question. How many predicted death would bird flu cause worldwide?
I mean the UK. Anything. Hmm.
Very little. Even at that point in time, there've been about 40 to 50.
In the world, you're going to get very little movies and very little.
In fact, the number is 150 million.
And in the UK it's 700,000 were predicted.
And at that point we started stockpiling antiviral because people said we need to understand, we need to have something that helped us them.
And in 2009, at the same time,
we published this review with Matthew showing Shin for the report from the UN had come out saying predict 150 million deaths.
Okay. And that's when we started to produce 14.6 million courses of the job.
We'll have some of the 700,000 deaths that might be expected.
That's a lot of deaths, isn't it? That's about one in every 800 people.
So it's about one of you in this room should be dead, basically. That's a lot of people.
And you think about where we are in reforming health care. Okay.
Now, interestingly, the fourth offence and now in 2009, we've gotten to the point where we've moved on from swine flu to swine flu,
but we've survived the bird flu and now we're into the swine flu problem.
And again, we've got apocalyptic warning from officials.
This is Liam Donaldson, who was the chief medical officer at the time, and he said the 50,000 would be killed in the UK.
But then these provisions and in the World Health Organisation,
this is Margaret Chan saying we have evidence to suggest we are in the first pandemic of the 21st century.
And if you remember again in 2009, there were lots of echoes of the 2005.
But this is a serious crisis. This is an epidemic. It's a pandemic.
And therefore, in this country, what we had was we had situations where you phoned up a telephone line and if you had a fever,
you would be given automatically Tamiflu. With no doctor's appointment.
A very intricate and we did the systematic review in children.
And at the same time there was also another review published.
And this is a timeline of history of the reviews of Cochrane reviews and systematic reviews on some of it.
And here's the children, one which I'm part of, and this is a chap called Tom Jefferson, who was also doing the adult review.
And one of the things was this led to a new collaboration where we worked
together to try and get all the evidence on the data from children and adults.
This is an evolving story, my mind at the time and all of our thinking at the time.
But what happened in 2003 is this paper had been published in Archives Internal Medicine and it wasn't quite a systematic review.
But what he said is we analysed prospectively collected data on lower respiratory tract
infections and antibiotic use from 3500 patients enrolled in ten placebo controlled,
double blinded trials. So some of it. So that was a large dataset at the time.
This review included ten randomised controlled trials and then ten randomised controlled trials had been
included in the adult cocoon review and the data from then trials had been included in the systematic review.
So when you look, if you're randomised controlled trials, sometimes you find these and then you find the data in that overview.
What happened is a chap from Japan submitted a comment to the Cochrane Review.
Anybody who's doing a systematic in Cochrane knows if you've got a comment six months to respond and post the comment.
You have the auditor team come to you and say, Tell me what we found within six months.
And his comment was to say, of the ten trials that were in the kind of review,
only two were probably developed for the German months and the other eight were
preceding the Congress meeting date of follow up and were never published.
And then he went in and said, Well, look, you have publication bias because the two reports, the trials were the only two positive ones.
I knew the right thing. And that meant 60% of the data was never published whatsoever.
And that led to a whole affair of trying to track down the data.
And we use media to try and get to the data. But one of the things that we wrote to the authors of the study is the original data,
and this is what one of the study said, and this is Deborah Cohen, who was working in the BMJ as an investigator.
First, he said he did not recall seeing the primary data.
The statistical analysis had been conducted by Rouch and he analysed the summary data.
But concerning well is the other chap, Fred Hazen,
who then went to the chap who was the corresponding author of the original ten study review and said,
I cannot find the original data to the 2003 publication.
You may need to go to both. And in the proceedings, this is one of the largest ever trials ever done.
If you go into the data, it tells you the number of study patients is about 1400 patients were in this study.
And this is the abstract I've presented in 2000 of the international abstracts of the 2000 annual Infectious Disease Societies in America,
about 300 words, okay. 1459 patients were enrolled.
164 US study finds the chart at the top page on China said you didn't go to the meeting.
It doesn't even remember the abstracts or participated in the writing of the abstract.
I cannot recall the meeting whatsoever. So in 2013, Roche agreed to release all of the trial data on all the Tamiflu drugs.
And in doing that, we had a long contractual issue back and forward about how we get the data.
And you think about information governance and all the issues that come with information governance,
and eventually you just simply load the seeds in the post on secured.
And one day I got all these feeds and that was it, 170,000 pages of document we would see now coming back to 7001.
Remember that study showed you the 300 word study? So in 76001 is the sort of ID you give a through before it gets published,
before it gets into the public domain, and when it probably can become trendy, right?
Write down blah blah blah. 476001 So it's 300 word abstract.
The Clinical Study Report. If if you can see the chart is 95989 pages.
So. There's no provocation in the middle.
So to this date, we've been using that and yet with 9809 pages available.
So the first thing, if you think, well, let's look at the amount of information and what we might get glean from this.
And to this point and probably to about 20,010,
it was deemed acceptable to use these sorts of abstracts in construction, constructing systematic reviews.
Now I could go into the effects of Tamiflu.
We found in the systematic review of the clinical study report, it took us quite a bit of time to evaluate,
but what we found is it reduced the duration of symptoms by 50.
We already knew that no effect on rates of hospitalisation,
but that was different because they claimed effects and no effect on any bacterial infection in children.
And importantly, we managed to say for every hundred people treated free with self-report, they don't have pneumonia.
Self-report. As opposed to the people in the five trials that used an objective diagnosis of pneumonia, there was no effect.
So the clinical study reports allowed you to, instead of, say, eight trials, put them together.
There was an effect on lower respiratory tract complications.
He basically took them apart and settled on the three that positive.
It's just people who self-report pneumonia. Well, anybody who knows anything would know that's impossible to do.
Do you think you've got pneumonia? Well, the answer is always yes in a clinical setting, because that's what drives you both antibiotic free.
It's a ridiculous question, but actually I've used an x ray. There was no effect.
But here's the bit where it gets interesting. Unexpected findings.
The two publications I showed you,
the ones where people had put their name to but didn't actually look at the data or not analyse the data or didn't have access to the data.
It looks like stated there were no drug related serious adverse events.
And this had been replicated by NHS organisations and international organisations around the world.
There are no serious adverse events were noted in the major trials and no significant changes were noted in laboratory terms.
That would be reassuring, wouldn't that? If you went to a meeting and said We're going to use this drug safety problem?
Yeah, immediately we went into the clinical study report in the back.
This is what's in the to 9000 pages of documents.
There are individual table within. I think you can see that that was yet to come up from the back if you can see that serious adverse events.
Listing of serious adverse events. And there were ten of them.
So how do you go from their word, no joke to serious about you, to their attending just one day before?
Again, that's deeply concerning when you think of this is an issue of reporting bias.
Well, give you some other examples which allow you to do so at the time.
People from Professor Legal, Public Health, England defending the strategy to use Tamiflu in to prevent an outbreak.
So one of the arguments is in the very elderly, you would have an outbreak of somebody with flu.
Somebody would do a test and say a person's got flu. Therefore you should you tell me.
So tell me that prophylaxis, if you give it to everybody else, yeah, you will reduce the known exposure by 8%.
Now, I'm not going to go into the that I could show you by that.
I want you to think about the generalisability of this.
If we were going to, you know, really have a situation.
So Akhenaten cow. All right.
So you would expect in research that you would be able to then understand that the research applied to the population, you would use it.
So if you have an idea of somebody in a nursing home, that might generally be somebody who's elderly, might have co-morbidities, lots of a problem.
And that generates a question of do these results apply to my patients?
Now you can then construct because all of this data is missing from the publication and nobody knows about it.
The exclusion criteria. If you think of the type of people you've seen so they can't have cancer,
immunosuppression, chronic liver or renal disease, they can't be taking steroids.
You can't have unstable or uncontrolled renal, cardiopulmonary, vascular neurological or metabolic disease,
diabetes, thyroid adrenal disorders, capital activities about this.
Not only that, if you've got of these around and if you've got cardiac failure, they're all for out.
Now, if anybody knew the person in a nursing home who is now excluded, you then know not to go.
There's a real problem. So when you look at the study that Public Health England is using to be akin to a nursing home,
there are actually only 17 contract with 16 were included in the study.
So the whole of our strategy in public health England for prophylactic exposure is underpinned by 17 people in a study.
However, what it says in the in the short publication across all ages, it didn't even detail any of the breakdown of the people.
So you start to go and when I'm in a briefing with people,
is it acceptable to be able to apply treatment for people who are completely excluded from the study and say that's okay to get on with,
yet we still do that today. Now, the interesting, if you go into the Tamiflu promotional material, it's really quite interesting.
It's going back. What happened is this is the misleading efficacy claim that actually in America Roche submitted
these in America the pill with the power to stop the flu taking the flu with Tamiflu.
Tamiflu, it will reduce duration of the flu by 31%.
Severity of influenza similar, all misleading, the FDA said.
And in fact, what the FDA said, because they'd done a bit more work than in Europe with Tamiflu,
has not been shown to prevent such complications all along.
That was actually new. And in fact, in a recent update in 2010, they even restated this in America.
This is not what it says in America, in Europe and in the UK.
It is the complete opposite in terms of what they may become with the influenza like
symptoms or may co-exist with or occur with complications during the course of influenza.
Tamiflu has not been shown to prevent such complications.
I put to go into the 500 million we spent stockpiling this drug.
At this point, I'm not going to go there. What the message there is, is to start to think of a significant problem.
Reporting by the amount of information around a drug is limited when we talk about journal
publications and also in many systematic reviews and I'll come back to this point,
but going back, what also happened at this point, which is really interesting about 20 1314, all trials that started with all trials,
Ben Goldacre was part of that and he came to Oxford, invented free for the data lab.
And one of the things he was interested in was the publication bias aspect, because he said it's still not gone away, it's still a major problem.
Despite the fact we've been going on about now for nearly 30 years, not much has changed.
And we published a few papers that started with looking at some of the issues.
So we looked here in Oxford at the Biomedical Research Centre.
Oxford is to receive probably the most research funding in the UK.
And in the Biomedical Research Centre they done 286 trials.
We identified and we set out to try and find we couldn't find registration for
44 of the trials that were registered with a completion date pre January 2015.
Just over half were published and half of these were published within 12 months.
Because what happened is the European Union was also saying we should be now having to publish trials within one year of completion.
Well, in 2014, the law changed. Say, if you finish your trial 12, then you should publish your photos with all involved.
And as you can see in Oxford, it's supposed to be the bastion of quality research integrity.
We were only doing that about half the time. And some of the information without the date any more accurate.
And we tried to influence that. Some people worried about that, but at least we were starting to do audits on our own work.
Now this also then led to what's called the EU trial tracker.
Because the DataLab team is so smart they can pull the data from the trial registries and the date and then produce what's called the trial tracker.
And that's because the law did change.
And from December 2016, all trials of the EU clinical trials for EU CPR should post results within 12 months of completion.
So in December 2017, it's the law.
It's not something you want to do. Remembering it been the law in America since 2007 became the law in effect from 2017.
For the first date, you could see the result with with a year later.
Okay, now with the result for the public.
So this is the traditional journal approach of what it looks like in the journal.
So if you this the paper in the BMJ,
which is a top prize and this is normally what researchers do and publish your paper with results and said 49.5%, roughly half reported results.
Yeah. Interestingly though,
even trials with a commercial sponsor with substantially more likely to post results than those with a non-commercial sponsor, 68 versus 11.
Look at the difference. 68 votes is 11.
Don't tell me I can't stand the ratio people. Are you going to learn about why the odds ratio is misleading?
But it's not. It's about 50 times greater and those that conducted large number of trials.
But what happened next is if you go online, you can go on to the EU trial tracker.
You can look at it tonight and you'll see that the numbers got slightly better.
So 62%, 61.8%.
And you can go online and look at that. Now I'm doing a phone with the tool in Manchester in about a month's time at the Manchester University.
And I went into Manchester and they got three free institutions in Manchester and wherever I am, if I do the talk,
I show them and they're shocking and they're so bad they're going to hopefully be embarrassed to do
something about it in Manchester because about 25% of their trials are reporting at the moment.
In fact, they want to do something with one.
You? No, what happens is it will accumulate.
So once you report, you come within the time frame. So you report 14 month, you'll be considered to have completed what you showed.
But what? The thing that you should report by 12 months and it continues to update.
So if a trial fitted that report, it will come out. Yep.
And you can type any of any of your institutions. You can type in where you are in that institution.
And it will show any fact that academic institutions are pretty dire.
Oxford's got better. Not least because they know we're all different all the time.
And also recently, the HRA has said that actually they're going to start and the government has said we're
going to start auditing academic institutions to start to look at their performance.
And if you don't start to meet this requirement,
then I think that's going to be a position where they're going to say, we're going to withdraw your funding,
because the question is,
is it ethically acceptable if you've still got 40% your trials on reported that you should receive more money to do more trials,
to publish less results? So we feel like it. I mean, it doesn't ethically make sense to me.
Surely it should be an ethics should go. How would you check online how you do it?
Because you can look at this by institution. You can look at this by industry sponsorship.
And if you are reporting in full, surely you should say, well, what about the sort of thing we need in the thinking?
So that's where we are. So we can say just a bit more than half of all trials go unreported.
And the thing is, can you make informed decisions about treatment if you're wondering 40% of the data may be missing.
So we're now in two areas where we've got significant problems reporting bias and publication bias.
Now, the thing is, some things have got better in the time in a very short period of time.
And in 2014,
the European Medicines Agency said what we're going to do is provide online access to clinical data for medicinal products that we approve.
And to do this, what they're going to do is make clinical study report data available.
So when they approve a drug, the clinical study report will be made available and you will be able to access it.
So if you're doing a systematic review of a treatment or intervention,
you will be able to get the new drugs now and you'll also be able to access and ask for some about before that date.
Now, the reason I did that is because there are a number of studies that have looked at the type of documentation.
So it's okay me giving you one example of Tamiflu, but actually there are more examples of this.
And this is in Germany from eight quick it quick like the nice in Germany.
The Health Technology Assessment group in Germany is called big week and this is a visa is head of the department of their drug assessment is trying
to investigate to what extent different types of documents full reporting clinical trials provide sufficient information for trial evaluation.
So when you get to the whole aspect of this, you really need the important information in front of you.
When I looked at Tamiflu, they just looked at journal publication.
If you go to this, this is in the BMJ. It is a very interesting table in 2015.
And for instance, here the point here is this work.
So this is this row here, it's study results and here you've got journal publications.
This is posted on the registries like the UTR and this is the clinical study report.
Okay. Our findings information.
And so in the study report information in there complete on 52% of specific outcome in registries is 71%.
And by the time we get to clinical study reports, it's 91% of specific outcomes in study report.
So you can see the shift in information going from general publication through registries to clinical study report.
These are incredibly useful when they populated what I just showed you that only did about 60% of the time.
And they show you the problem to very well we're not publicly available the voluminous
so we've pointed out the disadvantages that really big to get your head around.
But there's been over examples.
Here's an example recently about access to clinical study reports in line with the IMA and it reduced reporting biases for quality of life outcomes.
So important information that's missing in the Journal for that.
But one of the problems here is if pointed out that regulators unfortunately only see a fraction
of the documentation and they frequently that no regulator actually asks for all the evidence.
They only ask for the evidence sufficient to produce a marketing approval so that people can submit to them a little bit of evidence.
They can submit different bits to different regulators around the world, and they all get to see different bits,
but they do not create a table of all of the studies and say, here's all the studies and you're looking at only four.
So that's a huge problem. Well, here's another example.
And then here's again in the old effect trial.
And if people know the study of only stuff, it was a drug that was used to reduce obesity but had significant problems with cardiovascular effects.
And it's one of the things that it shows is when you look at clinical study reports from that journal publication,
the benefits seemed to align where the major differences in understated adverse events, which would make sense, wouldn't it?
If you have a journal publication that you want to accentuate positive,
you are going to put the positive outcomes in and remove all of the adverse events.
And that seemingly is what's happening with the clinical study report.
So last year one of my other roles is editor in chief of BMJ evidence based medicine.
And here's a good example of what's happening right now is a systematic review of HPV vaccine published in the Cochrane Library Juice Publications,
but then trying to use a little bit of registries and a little bit of CFR without being comprehensive and doing it in a systematic way.
And this report caused a lot of controversy because basically in there is the
table that they said there's out of the 20 trials that we think they missed,
and they definitely missed 16 additional trials which were eligible in the review.
And and this is a review by a chap in Denmark, Lath Jorgensen, working with his doctor and Tom Jefferson.
He was doing his Ph.D. on this whole subject. So it's pretty intense.
It's not a few weeks work. I wanted to put my rapid review together and here they trying to match the study ID with the funding
sponsorship database for the trial I.D. in the in the registry and then trying to match them and say,
here's what we think this study is. And so.
The we show this is a bit like you.
You're about you're flying the plane and half the money off makes it.
We don't know which half and you don't quite know when you need that bit the menu but it'll probably be when the really
when when the safety issues happen and you really want to know when it's just about to hit going down with the night,
what do we do now? And all of that is missing.
And and I think it shows a worrying position that if you want to trust the evidence,
because many of these decisions are based on small benefits to harm.
But you really would want all of the evidence in front of you,
and you probably want somebody to look at it in a way that is impartial and puts together the benefits and the harm.
To do that takes a long time. This is not a straightforward job.
You can't do it for all evidence, but you would expect to think we might do it for the public health drugs, the ones where like HPV or Tamiflu,
where it's likely to be used by a wide spectrum of patients and it's important to know the benefits in the home.
Now I'm just going to finish with a few things, which, again, this story carries on.
And it's it's a very interesting story and it's not going away any time soon for this is a new trial.
And the problem is right now, in 2016, they set out they had a delay for about a year, which is for immigrants,
and it doesn't quite work as well, first of all, that it suspended all new activities related to clinical data publication.
And this is the problem of good old Brexit because they've relocated to Amsterdam,
they said it's too much work and we have to continue to do business plan. And so this is peripheral activity right now.
It's not important. And they said by the end of July, but it's not happened yet.
We'll be processed until further notice if we sell them further notice, which is deeply concerning.
The flip side, though, is is is is anybody in Canada here?
Is anybody from Canada here? Well, you guys are leading the field.
You're really aware of this, what you're doing in Canada. So in Canada, Health Canada is within 120 days of a decision.
Health Canada will post clinical study reports on a new government online portal.
And as I say, they set out well, as I understand it, Health Canada is just going to say here,
which is an amazing step forward, because the problem at the moment is even in email,
you've still got to go to an access system that requests them and then they go through a sort of holding pattern where they go,
we're going to we're going to check that they've no data protection issues.
We're going to do some redactions in about year, like you get the data.
So Health Canada have potentially moved the game forward quite considerably.
And if they become available,
we all could get to work in a way that means you're going to be very busy sorting out the problems of publication bias and reporting by.
Thank you very much.