So thanks for coming. I'm going to talk to you about this. The better evidence for a better health care manifesto and why we need it,
why it's so important and why we do it in the centre and how it might relate to you and how you might contribute.
So I'm going to take you back to an article in 1994 that I think you should all read.
It's an editorial by a chap called Doug Altman, who is a professor of medical statistics here in Rome,
something called the Centre for Statistics in Medicine.
And if you remember when in blunt pop you might have heard of that name talked about 10,000 times.
Well, he wrote an article, an editorial called The Scandal of Poor Medical Research.
And he said in that article, there are huge shortcomings in the way that evidence based medicine operates today.
And that's in 1994. Too much research, too much poor quality teacher research, and too much research done for the wrong reasons.
Added to that a few years ago. In 2014, a professor with it in London at the time who's now here, Trish Greenhalgh, had a meeting here in Oxford,
a two day meeting to basically say that evidence based medicine is a movement in crisis.
And there's been a long history of people trying to fix these problems, and it's presenting more problems and solutions for practice.
And then just recently with a colleague of mine, Ben Goldacre, we wrote an editorial in the BMJ again.
So you see there's a theme in the BMJ about how medicine is broken and how we can fix it.
And the important issue is now is that patients and the public are increasingly aware of the shortcomings.
So it's about time we did something about it. Okay.
So one speaking, if you want to go online, you can go online to the evidence, live the org and type in manifesto or type that into Google.
And you can read the manifesto, if any of your online at the moment and have a look while I'm talking.
So wherever I've been, I thought I'd say, You've got to start somewhere.
Why do you get to a point that we need better evidence? What's your story?
Why am I here? What was the seminal moment when I went, Oh, there's a real problem.
So I'm just going to talk you through one area where we've worked, which is in Tamiflu is an antiviral drug, which you may be aware of or not.
And in the 2009 pandemic, remember when we were all going to die?
You shouldn't be here. Some of you that were given no antiviral for the public health measure on a telephone call here in the UK.
I'm a GP myself and we had a situation where you would ring up if you had a temperature,
one of your children had a temperature and then they would send you off to a phone, a phone call to go and pick up the antivirals and take them.
And that was the point where my sister offered me, do these make any difference?
And at that point, I was like, I'm uncertain, not sure.
So one of the basic principles of evidence based medicine is to reduce uncertainty, to use evidence to make better decisions, reduce uncertainty.
And in 2009, we did a systematic review, but we also worked with Tom Jefferson,
who's a colleague of mine, to try and find the evidence transfer and reduce uncertainty.
And what we did is come under significant barriers.
And one of the things at the time is in one of the Cochrane reviews is if you ever do a Cochrane review and publish a Cochrane review,
if somebody put the comment on that review, you have six months to respond to that comment.
And this is a Japanese clinician called Hayashi, who sent a question in saying,
I want to question that you use ten trials to inform the systematic review, and of then ten trials, only two had ever been published and eight were.
How did you not question that? And of the eight that were published, of the three of the two that were published, there was no significant difference.
And of the ones that were unpublished, there was a significant difference.
How did you come to that conclusion? Why didn't you investigate where that evidence was?
And that was based on this review that was published in 2003 in Archives of Internal Medicine.
And that said, it also time of treatment,
reduced lower respiratory tract complications and antibiotic use and hospitalisation in both health and at risk at all.
And then there were the ten trials published and what happened in the Cochrane Review.
I'd taken some of what we used to do with take the abstracts, the non published and then just reproduced them in the Cochrane Review.
And here's an example of one of these abstracts, but what we did is try to ask for the evidence.
So here's an example of one of the abstract. This is the abstract of the largest ever published, largest ever trial of one found.
459 patient published as an abstract 300 words.
When we asked the person who had the name on that review, John Traynor, it was published in a meeting conference.
He said that he didn't actually participate in study 76001 and doesn't remember presenting it at the meeting.
So the largest ever trial, the person who's got the name on the conference but he wasn't at the meeting,
doesn't remember participating in the trial and doesn't know anything about it.
That set the alarm bells off, doesn't it?
And we had the media involved because at first we were being stonewalled and then trying to get the information required to involve the media.
We asked one of the chaps who was involved in one of the published trials, Where's the data?
And he said, Well, unfortunately, I only I only saw the summary data.
All of that you would have to ask Roche because they conducted the trial, only looked at the summary court.
So even the published data we were worried about who's got access to data, who's really doing these trials.
All of this looks really odd. Issues of conflict of interest go growth, authorship.
How do you access the data?
And that meant 60% of the data was never published for a drug that we were stockpiling and using in a public health emergency.
And this is a really helpful slide. What it illustrates is they call it the FDA, call it above the waterline and below the waterline.
And above the waterline is you see journal articles and conference proceedings and very small summaries of what really is out there.
And for any trial below the line is all this over other information completed case report formed Patient Level Dataset Clinical Study Report.
These are documents that are submitted to regulators as part of the regulatory proceeding.
Yet what we rely on is only these small snippets of information to make huge decisions in health care.
And suddenly when I realised this and we realised this, this can't be right.
And what happened in 2013 with that, Roche agreed to release all of the trial data, and that was an important, uh, statement largely as well,
because the publication of a book called Bad Pharma at the time,
which basically said there's a lot of bad practices, and if you haven't read that book, buy a copy, read it.
Some really interesting issues about what we defined by bad trials, all the fundamental problems.
So let's come back when they really object.
So you read a lot about confidentiality, patient confidentiality and the problems that present in pharmaceutical industry and entities.
Law would say to you, Well, actually we can't do this of commercial confidentiality reasons.
Yet Roche and JFK just posted the seeds to me in a package and I got them in the post.
Wasn't even recorded. Deliver it. It was a really odd sensation to say We've got all these issues.
But this, remember in 70 601, that's the largest ever treatment trial.
That was all that was ever published in in an abstract.
This is the actual clinical study report that underpinned and 7001.
So would you rely on that 300 words or would you rely on that for the information about all the benefits and all the harm?
And it's 9809 pages, so you can see that the huge difference between what we currently see and what might be available for drug treatment.
Now, interestingly, what did that lead to?
That led us to say it doesn't reduce hospital complications because we got all the full benefits and all the harms.
It doesn't reduce antibiotic use. It doesn't reduce low respiratory tract complications.
It does still reduce your symptoms because it has an antipyretic effect,
but it increases some adverse events that are important, like renal failure, psychiatric adverse events and so on.
So I haven't clinicians haven't prescribed this drug yet.
I don't see the evidence is there for me to prescribe it. There are still some questions like in people on ICU or in hospital, but in primary care,
the evidence is it's of limited benefit, if any, and the adverse event outweigh the benefit.
Yet we still stockpile with drug. And when I went to the government to present this in the Parliamentary Select Committee,
we still spent another £50 million stockpiling it the month after because it was going out today.
So the huge issue about how we make decisions, so when anybody says to me we haven't got enough money in health care, I.
You can't believe there isn't enough money. £473 million on stockpiling a drug.
And what would happen if we had more money available in health care?
We probably stockpile more youth treatment just in case scenario.
The reason that's there is because influenza is on the government register, so it's on the same level as climate change.
So we have to have a risk register and a risk mitigation plan.
And that's part of it. Tamiflu is our rhythm mitigation plan, not handwashing.
Whether the good evidence is, that's our plan. So thinking about that is how does that think?
So if you go in the thought, you see why we need better evidence.
So my first point say you can see 20 reasons that underpin at least 20 the need for better evidence, for better healthcare,
and that these are structural, systemic problems that have existed in evidence based medicine for the last 20 or 30 years.
And so if you take a drug like what I showed you, neuraminidase inhibitors, like I'll sell some of our Tamiflu zanamivir Relenza.
And I say, how many problems do the trials exhibit? So we just take them 20 problems as we go through them.
I can say that they suffered with pop publication bias.
They had poor quality research. They had research that was more likely to be false and true reporting bias.
They certainly had that go to authorship. They had financial and non-financial conflicts of interest.
They had underreporting of harm.
If you look at journal publications compared to clinical study report, there's about an 8% difference in reporting of harms.
They certainly have a lack of shared decision making strategies.
If you ring up for a year, if you've got a fever, take Tamiflu, where the shared decision making in that trial is lacking external validity.
Who do they apply to regulatory failings? Certainly surrogate outcomes, definitely a manageable volume of evidence.
How can you look at 160,000 pages of evidence?
Get clinical guidelines like Public Health England, no understanding of the evidence and prohibitive cost of doing the actual trials.
And they certainly give us too much medicine. So you can see £500 million worth.
If people have understand the limitations of evidence and the problem that existed,
we might have come to different decisions and said, we won't stockpile this and we'll save that 500 million for something else.
So it's not that people don't want to clinicians don't want to practice evidence based medicine.
It's actually very difficult for them if trials exhibit all of these problems.
So somehow we've got to fix this if we want people to use evidence better.
And if we don't fix, some of these problems will still be in a problem where there's a huge amount of evidence in front of people.
And it's very difficult and it's getting harder in my mind.
Not easier to tell the good evidence from the bad evidence, because if we go back to Goldman's time,
there's been a huge increase in research and there's more. So whenever.
So that's where we start of is the growth and volume of evidence has been accompanied by corrosion
in the quality of evidence which is compromised medicine's ability to provide affordable,
effective, high value care. Health care is either is one of the it is the fastest growing business in the world in some parts of the world,
like America with 20% of the of the GDP. So if you wanted to create new business tomorrow,
would you go into health care or would you go into making cars where it's actually a smaller GDP proportion?
And what I think people, business people in health care is do what they think. They think it should be more than 20% of GDP.
There's more to go for. We can make money out of health care.
Now is a great driver of the economy, but actually that corrosion and uncertainty helps businesses thrive.
When we want to roll back and say, hey, hold on a minute, we need to develop an evidence base, a high quality evidence base.
And there are many examples of this. I could spend the next day giving a huge example, and in any field, wherever I go, people can give me examples.
But here's another example. Vioxx trials. When I first qualified, going back in in 2001,
the fact that evidence was withheld from regulators for a drug that was a global
blockbuster and that simple people who would understand the difference between intention
to treat and per protocol analysis could have come to a completely different
understanding in that what they did is submit to the FDA only the on treatment analysis.
So if you stopped the treatment because you had a heart attack, you were removed from the information sent to the regulator.
So with two years later, before somebody clicked and went, Oh, there's a problem here with something being withheld in the analysis.
Here's another classic example. 3293329.
It's got its own Wikipedia page. They're going to have a read of the study.
329. Everybody should know about this example. Again, paroxetine marketed to adolescents.
Information withheld on important suicidal behaviour in adolescents meant that a number of people a
drug that so billions could actually have been stopped the people to use better quality evidence.
You structural problems. And then here's another example that the flip side is I've work with organisations when you try to fix things,
all trials try to fix things and I'm going to come back to that. Just one specific thing.
We want all trials registered and reported in full. Very simple statement.
That's a commitment. All trials should be registered and should be reported in full.
Doesn't seem nonsensical, but when you realise half of all trials never report,
are never registered and report in full, that half the information is missing.
Now imagine if you was the airline industry and you were in the business like British Airways are buying planes and they said,
we're not going to give you half the information about this plane.
So if it drops out of the sky, because we're not going to tell you the things that have gone wrong, we just would not accept that.
So when we purchase a medicine and we both put it in the chemical compound, or are we purchasing all of the evidence that it's built up?
And at the moment, that seems a ridiculous scenario that actually we can say, but we can hide half of that and we can do that.
No problem. But what it taught me to fix things, it takes a lot of group for people to work together.
And then here's an example.
Recently there are of the rail toll is a drug called Rivaroxaban, which is designed to reduce your risk from both in people with risk of thrombosis,
like with atrial fibrillation or risk of having a deep vein thrombosis.
But here's it's in the secondary prevention after an acute coronary syndrome.
And here the European Union approved the rail to rivaroxaban for secondary prevention of acute coronary syndrome.
Two, that's the European Medicines Agency. The European Union approved these.
However, in America, the regulatory position is to say we're actually going to vote against the same drug for acute coronary syndrome.
So how can you have a system, global system,
where the two major regulators come to completely opposite conclusion about the evidence in front of them?
One of them saying, well, we've got like a few papers in front of them, looks quite nice, will trust it.
And the other one is going, Oh, we see some of the issues that are emerging in the evidence base.
We understand the problems of missing data because we've been here before and I do look at this and think that is ridiculous.
This could be an amazing treatment. However, it's equally clear it's not animated and it's a harmful treatment.
We need to reduce uncertainty. How can we have two European agents in FDA coming to opposite conclusion that we've got
a legitimate regulatory system and they're looking at the same bits of evidence things?
Absolute nonsense to me. But that's the problem, because we've got an issue with the evidence,
and that's why we end up writing all sorts of editorials all the time about the same issue.
That's too much uncertainty to make a decision. Could we reduce the uncertainty?
One of the things we said is it was 2010 when these drugs became available.
First off, we could have run our own independent trial in the meantime,
which would have been far less money than what we've already spent on the drugs and come to our own independent conclusion.
Getting rid of all the problems that exist in the evidence.
But we don't want to do that. So we have a problem of replication and independent replication, and that's what we find here.
Independent scrutiny of key trial data. Everybody involved in the trials have a commercial conflict of interest.
How would we expect them to come to an independent position?
Because we just know that even if you want to act in the best well of the world, you're not going to be able to do that.
If you have a financial tower. And then we have a world where what we keep saying to everybody is we should be informing the public.
You should be sharing decisions. We should work clinicians and patients together.
And then I have clinicians who work with me saying, So, Karl, what can we do to operationalise this tomorrow?
Where are the tools that are going to help us do this?
And I say to them, Well, actually, if you go and look at the evidence, there's actually this is an area where there's so little research,
it's impossible to inform anybody about what you might do, how you might go about it.
There are no tools that can help us.
When you say to somebody, how might you help somebody with cardiovascular risk make a better decision about whether to take treatment or not?
And it's a really difficult area to operationalise in and work in.
But I can tell you that the funders and people are not taking it seriously.
And then finally we get invited into areas to work with people where we see this huge problem.
And we work with Panorama, BBC before Christmas to look at the evidence for IVF treatment and add on itself.
And it astounds me that you go to an area like this and we say the most patient relevant outcome here has to be live,
birth and all of the people in this field go, No, that's not the important outcome.
And we're like, Well, what is it? They go, No, no, no. You need to come back.
You need to come back, and you need to do so at that for surrogate or maybe the six week pregnancy implantation rate.
And which is what? No, that cannot be anything, but it's too difficult to measure.
That would be good. How can it be difficult to measure live birth or not?
Surely that's what you want to know. When you decided to have some additional treatment that could cost you up to 7
to £10000 extra because this is outside of the NHS and this is the best world.
If you want to look at what would the NHS look like in a private sector, you'd have this mish mash of thumb treatments we'd offer you and then others.
We would operate in a world of uncertainty where we'd go. We're not quite sure, but we think it's a good idea.
And give us 5000 and we'll let you know and you'll find out.
And people spend up to 60 to 70000 extra in this.
And all we said it very clearly is actually there were 42 different treatments
available that you could be offered in IVF beyond standard treatment of them.
42 only three had some form of evidence where we would say they show some benefit of improvement.
But within Cochrane Review however, that evidence in them free with limited by real issues that need meant they needed a large a bigger trial.
So to be honest, it was zero that we would recommend above and beyond the full placebo effect that you might get from parties.
But the key that some of them were actually harmful and we couldn't understand why they were still available and being offered in the private world.
They were. But what it taught us that there are huge issues across all different sectors.
Wherever you are, there are issues in the quality of evidence, which is probably not that ridiculous, to be honest,
because you can think we've only about 25 years into the idea of high quality developing evidence,
and we're only at the beginning of the journey of evidence based medicine.
And while this has been going on, not least, the Academy of Medical Sciences has launched a new project on evaluating evidence.
So there are other people coming to the fore. And not least Sally DAVIES, who is the CMO.
Some of you may be aware, Professor Dame Sally DAVIES.
There seems to be a view that doctors over medicate so it is difficult to trust them and that clinical
scientists are all beset by conflict of interest from industry funding and therefore untrustworthy too.
And she picked out one of our favourite treatments in the world to say this is often a huge problem in
understanding evidence because what they don't want to do is be putting out a policy where some people are going.
Hold on a minute. There's a problem with the evidence. The problem with the evidence, they don't like it.
We need an authoritative independent report looking at how society should judge the safety and efficacy of drugs of an intervention.
How can anybody come up with an authoritative, independent report?
That's a really bold statement, isn't it, by very nature in a shared decision making world.
Nothing can be authoritative when we're operating in a world of uncertainty.
So that's how we ended up with that sort of editorial.
How can we how can we go from this? Because we wrote this in response to that.
How can we fix it? Not just say, okay, we're going to do more of the same?
And what realised when we were doing that, often what we were pointing out is we want to reduce the biases.
That's what we really talking about. There's so much bias out there.
How do we reduce bias? It was interesting last week of us talking about this from wherever I go.
And I thought I'd leave the thing. And with this couple of flight with at the Centre for Evidence-Based Dermatology in Nottingham and
we have the director of the Centre for evidence based Veterinary Medicine in Nottingham here.
So to go with him one way and there's a chap there how.
William I don't know if you've ever met how he's now also not only head of the Centre for Evidence based Dermatology,
he's head of the Natural History Programme, not the Trials program.
So really influential person knows a lot about evidence, about the problems in evidence and also is really interested.
And we had a lively discussion about some of the affected, but when I went there I picked out this article.
He wasn't aware of it, he was in the audience. So I said, Look, I'm here today.
And I said, Look, strengthen and limitations of evidence based dermatology.
And look what we think. Evidence based dermatology can have its limitations.
For example, monolith pooling together of data are still susceptible to some degree of dishonesty and bias.
But in its article, he makes this point, if possible,
those with a vested interest to twist the methods and conclusion of a systematic review to their own advantage.
One could conveniently fail to include one crucial study, for instance,
or not publish a study that showed you unpublished data in a way that favours a particular product.
And I like this readers need to develop a good nose for what constitutes a good systematic review in a clinical trial.
So you all need to. So when you teaching tomorrow on the MFC you have to be talking about do you have a good nose or not?
What does that mean? That's really ridiculous.
And so what we have to do is to base our decisions, our most important policy decisions right now of whether you've got a good nose.
And he's right because you have to be able to detect the subtleties, the biases, the problems that exist.
And if you're not aware of them or you don't understand them or you conflicted in some way,
you can easily dismiss them and come to an alternative viewpoint if you do.
SMITH All the biases dismiss the poor quality outcomes.
You can still come to an end. To that said, we should use this treatment.
Why don't the problem is we can't afford to do that anymore, can we?
We just have not got enough money in the pool.
Can we invest 2.5 billion in health services extra tomorrow and of that, spend another 50 million on Tamiflu?
Is that what we should do or should we stop and go?
Actually, let's work out what's worth investing in and what do.
So we've tried to fix some things.
And when we've tried to fix some things, it's been fun but actually provoked a lot of force in our unit and trying to think, what do we do?
So I'm going to talk to you about clinical outcomes.
Failed to translate into benefit for patients.
We published this article last week on Monday, so it's available in trial so you can go and read it freely in open access.
But one of them is we think there's a huge problem in outcomes. And if you go to outcomes, here's what I'm brainstorming.
I use the latest technology to brainstorm. And as you can see,
it's like downloadable on the iPhone is Post-it notes and a large Post-it note that sticks on your wall and you put them on the wall and you think,
what does that look like? And then you think, oh, inappropriately reported, selectively reported, poorly chosen, they're very thin.
You see failure to work and then you end up with a nice little diagram like this that's in the report.
That's what it looks like. And I'm going to talk to you one thing about that, you switched outcomes.
So we think there are all these problems. There are there are 15 problems, I think 911, 1215 for fixing problems in the design,
badly chosen outcomes, badly collected, selectively reported and inappropriately interpreted.
And you can pick any one of them and say, we need to optimise and we need to improve what we did.
And if we don't do that, that's why you have so many trial, 35,000 a year and so many of them don't improve outcomes.
So Compare is a project led by Ben Goldacre and a group of medical students and a couple of us in the centre.
So there's been a long history of reporting of the problem between what people pre
specify at the outset of a trial and what they report at the end of the trial.
Because if you go to the members of the general public and take anybody aside who knows nothing about research and being in a university,
an academic setting, they think it's absolutely obvious that you should say, this is what I'm going to do, this is how I'm going to measure it.
And then you should report that. And if you change from that, they go, Well, isn't that like cheating?
And that's what they consider it cheating. They don't say, it's like, Oh, slightly.
And what's been shown over time is that there's been a systematic review of 27 out.
About a third of trials have discrepancies between pre-specified and reported primary outcomes, and 15% of trials introduce none previously.
So they add brand new in. And somebody said to me,
it's a bit like if you were putting a bet on a horse and you could put your bet on after the horse before the race had finished, wouldn't you?
And if you could do that, we'd all be very rich and we wouldn't be still. There would be it.
So imagine like the Grand National and you go at the end and go, Not me, but now please, I want to put it on that and you go over your £10,000.
So it said it's obvious. So we work in things like other arenas that actually pre-specified means of the time stamp
when you put your time only betting slip and it has to come in before the race starts,
the end of the race is started, all bets are off and that's what should happen in research.
But it doesn't. I've I've showed you third of the time you can sort of say I didn't pay the money or I'm going to have a new winner.
So what did we do? We did all the top five medical journals over six weeks and we checked the pre-specified both reported outcomes.
But what we also did differently, instead of doing repeating the 27 study that had already the problem,
we said we're going to try and fix it, so we're going to send.
An email to the journal, which is actually really difficult to do because the first thing you realise is that rule like you've only got so many words,
250 words for one, 400 for the other, 400 for the other someone.
Only two weeks, three weeks and four weeks. But if you don't take 15, you can't submit a response.
Even if there's a glaring error, you're out of time. So you've got to get your act together.
You both have got to be right. So we had a system where two students would do independently and then a clinician would check it.
That's quite a lot of work actually in a short period of time. So we had to be organised and this is what we did.
This is what we found. We checked 67 trials.
Nine were perfect as many outcomes were not reported as was silently added, which was shocking to us.
We sent 58 letters, 18 were published and after four weeks, eight letters were unpublished and 32 were rejected outright by the editors.
So the idea that science is correcting itself correct itself is automatically ridiculed by this.
So this is what the different journals did here.
The BMJ, you can read online, you can go online and access the women who told me that all the revolt there freely available.
You can look at all our data sheets. You can read them online. So interestingly, they publish all their letters within 24 hours, which is good.
And here's one Henry Drysdale. And it had a correction.
But interestingly, for another letter, which you can see, which includes my name with Henry Jagdale, there was no correction.
So you've got journals operating in this really odd way.
And in fact, I was presented this at the BMJ editors meeting with all the editors from the big journals in the room.
And I said, What the [INAUDIBLE] is going on? This is supposed to be the scientific record and you seemingly make it up as you go.
You are the bastions of top five. You people believe these are the upholders of the scientific record.
The good thing is I recently. But it's so ridiculous. I've got to have a back into conversation to show this in public to the journal,
the editor in chief who was so embarrassed that they actually had to no faith me come to him and say, can you send it?
This will thought it. But he got to that level to so the Lancet.
They published the letters up to 180 days it took.
So it's ridiculously long time. By then, people have forgot what the trial was.
They would publish an awful reply, but there'd be no correction, no response in the record.
JAMA rejected all letters. They said they were too vague and there were repetition between letters.
That's because we only had 250 words. And you can imagine you need to be very specific about where you insert the primary outcome.
So we would have exactly the same and say the primary outcome measure with the reported, correct?
Yes. No secondary outcomes. There were seven of which five were incorrect.
Two weren't. Again, we're trying to embarrass JAMA into thought in this out,
but it's ridiculous that they said we have outcomes which you know the problem and we're doing nothing about it and also of internal medicine.
They published some letters they didn't publish, others they did all sorts of thing,
published a 1000 editorial trashing our methods without letting us respond to that.
And we keep going back and forward.
But we know they've changed their policy, but they don't want to admit why.
But they have changed their policy. But what if they say this?
We relied primarily on the protocol details about pre-specified primary and secondary outcome.
What they did is go to chairman and say, your methods are wrong because we rely on the protocol.
And what we said is we will use the protocol if it's dated pre-specified, but if it's published two years after the event,
which often happens a month before the trial finished, and there's no data on it all the time.
Stamp How are you supposed to call that pre-specified? And they try to say that the protocol is the document.
And we said, Yes, I understand that, but if you understand good clinical practice,
every protocol should have a version number on it with version control and a time stamp.
Yeah. If you go to an event and look in there, they've got no version control, no protocol, no dates on them.
So somebody could have wrote it a week before and make it out though.
This is the pre-specified outcome. We said that's utterly nonsense and anybody who's been trained in GCP would know that.
And it doesn't make sense because every document I've been involved in a trial has the time stamp on it and the version on it.
This is version 1.9. And all we're saying is if you've made a change, then you will highlight that in the protocol.
We would have then accepted it.
If you say this is two years after there have been no changes in C section, version one, we would accept it, but that's not the case.
But look, so we went looking for that protocol as well, were available for none of the Annals trial.
And even more worrying if statements like this reproducible research story, doctor,
everything we wrote to doctor everything and it said this is what we've got a study
protocol available from Dr. Aveton statistical code and data set not available.
Okay. But he said that's what he said when we emailed him.
This is his response. That's not what that says, is it?
This is outrageous. So if you go back, I don't say we rely on the protocol.
You ask somebody for the protocol and they say you have to sign a commercial confidentiality agreement.
This sort of stuff is just outrageous. Then we said, Can we trust the evidence base?
So New England Journal Medicine, one of my favourite journals in the in the world.
How can space control not allow for all outcomes to be reported?
And any interesting reader can compare the published article, the trial registration and the protocol with the reported results to view discrepancies.
That is impossible. Their editors can't do it because they don't know how to do it.
We've just shown you can't get the protocol for some trials.
It took it up to 4 hours to do it well because some of the results are in the discussion.
And when you go checking it, we're astonished to go with the secondary outcome of the primary outcome.
We do it in the discussion. Why is this so?
So they'll do things like they'll try and say, we reported the primary outcome correctly and we'll go, but you had seven timepoint you reported seven.
So they might be say the protocol might say we're going to report HB one C at six weeks and then all of a sudden they introduce six,
12, 18, 24, 30, and anybody will know what happens when you do multiple outcomes.
The potential for inference of that effect,
for the multiplicity argument that you've got too many outcomes by chance alone, one of them will be positive.
That's the type of stuff you see. But this is not possible. And they ready to, if any interested editor cannot compare the published article,
the registry in the protocol takes too long and it needs three of you.
At the moment we're saying, Look, why can't you just have a summary table? Primary, secondary, exploratory outcomes?
Anything changed into an infection? In the footnote, we'd all be able to do it really easy.
Okay, so there's is a junior problems.
He's a junior into the manifesto. So we've got I've got to a part where as a group of us, we've been doing things like even flying for a long time.
We've been teaching a long time about what do we mean by high quality evidence, how do we get there?
And we've thought that actually we now need to provide commitment,
that these commitments are statements that we put together, that people then say, I'm prepared to put an action to.
I'm going to try and do from about one or of an organisation free four five and then look
like so if you go on the manifesto is we're trying to facilitate so it's for everybody.
It's not just a few people to improve the quality of health care.
And so here's the 16 actions right now, the manifesto commitments.
You see, they're all verbs, so they're all about doing something and you can pick anyone.
So these are thought of grown from a small number to a larger number of a spoke to people.
They may collapse again. Not sure where they'll end up, but the idea of having commitment is if you commit to one thing,
I'm going to eradicate publication bias and you then have to say, well, who are the people to work with?
Who are the action? So we are committed.
So when I've done this as an organisation in CPM, we're probably doing too many.
I think we're involved in about eight things.
We're really interested in growing generation of leaders in evidence based methods and cultivating skilled skilled.
We are interested in reducing, don't worry, seed variation and overdiagnosis and unnecessary medicalisation we do do this.
I'll talk to you a bit about these too. And we're interested in these too, but we can't cover everything.
And it might be too many, maybe too many things.
And now I'm interested in how do we put these into into a way that actually communicates to groups like yourself or other people?
Oh, yeah, we should we should be doing something about it.
So if you take something like eradicate publication bias, that's probably one of the things we're best at.
And we've developed all trials, we've wrote about it.
We're working with the World Health Organisation to reach youth to improve the reporting of trials globally.
So increase the number of countries that register trials because that's one of the easiest fixes to do.
And we're trying to get an action plan for all countries to sign up to try to register trials.
We've developed a trial tracker that says Online you can look at how well your organisation is.
We've done audits of the University of Oxford to say we're not as good as we think.
Just to say two simple things Do you register your trial?
Do you publish your summary result within 12 months of the completion of that trial?
But actually it takes a huge number of people.
If you create a commitment like we're going to eradicate publication and reporting both, you need to work with more than one group.
You have to have a plan and you have to have actions. So our key is to get this right and in the right structure, and then from there say, right,
who's going to action it for the next five years of work is going to be about fixing the problem.
No more research about here's the problem.
Apart from in areas like veterinary medicine, where you might say, we still have to say the obvious, but we certainly don't need another cohort study.
That said, here's to to say there's a problem with outcomes which you can do it.
We need to think about how do we fix it. How do we and it will take its time.
So one of the things we are doing is we're not going away.
We recorded in the journal until we fix it and we'll keep trying to make their life a nightmare.
So of these you might think when you go online, you think, okay, I'm really interested in this.
Regulatory decisions who might be in policy.
You might be in clinical patient benefit, you might be in shared decision making, but what are the commitments?
Is there something missing? And then once we've set them out, the key question is what actions do we need to take?
And it keeps coming back. Because when you're in this game of trying to introduce action.
This was last week. Carbon Select Committed Research Integrity Inquiry launched.
So when you're in the job of fixing things, you do things like submit responses to this and say, this is how we should go about fixing it.
My response that there are two ways of looking at this problem. You could say, Well, there are people who are massive fraudsters.
The number of them is small and their actual impact is limited.
Whereas if you consider the questionable research pact practices like not publishing reviews,
the impact of that is huge and we likely affect the former because it looks bigger and it's more interesting news.
But actually we need to definitely have a strategy around the latter.
Anybody we do, we record put the themes up.
People who have a response with about 100 people have responded, Be careful if you do, because then we start quoting you.
And that's really helpful and that's how you do it.
You give us your feedback and be really helpful if you want it to say something or respond, because it's really helpful in our feedback.
And what we're trying to do then is we were going to do that.
We are going to publish a final version of this where everybody who's contributed will be part of the team,
an organisation of people who are saying, I'm an author and we will do that in evidence live.
27 Is we going to be moving to a focus of the action plan?
What does it mean if we've got the commitments? What are the fixes and who are the groups of people who want to talk to each other?
You might say, I'm really interested in working on this or that problem.
How do I go about it? Who do I talk to? Because it's a it's fixing.
It is a lot harder than I first realised. Providing solutions is I tend to think of it in five year for take me five years to fix something of that,
whereas I could do another paper about the problem in the next three months. All right.
Thank you very much.