Great pleasure to introduce Dr. Kamal Bhutani, who some of you know already and some of you will get to know because he's the
deputy director of the Centre for Evidence based Medicine and a practising GP,
and he's very kindly going to talk to us today on why on earth do we waste so much research.
Thanks, Chris. I feel sort of guilty keeping you in when it's such a nice evening, but.
So we'll try. We'll try and just make it so it's it's brief but makes the points.
So definitely I recognise some faces that if not, hopefully I'll get a chance to meet you again soon in one of the courses.
So as I said, I'm a GP, I'm deputy director, so a researcher and a tutor as well.
So I thought we'd just spend a little of the evening on this. Nice evening.
Talk a little bit about research and I know you're doing your study designs or some of your study designs module as well this week.
Okay. So a little quiz.
Some of you may know this already because you're well versed with the literature, but how much per year spent globally on health and medical research?
Roughly globally? This was probably a figure about a couple years.
This is what we've gone up since this year. But have a guess. Global.
3 billion. Okay. 77 million.
Yep. Okay. Good. Thanks. That's a bit higher.
Yep. And you're going to go higher. Lower than 48.
40 billion. Oh, I got it. Okay. High stakes. Okay.
I think I can see where this is going. All right. Well, actually, it's about 200 billion.
And in fact, it's probably more than that. It's probably more. That was that was correct.
As a few as of a few years ago. Okay. So you probably know the answer to the next question.
How much of research produced is wasted? 90%.
90% or 70?
About 70%. About 70%. 90%. Okay.
All right. Well, I'm going to. I'm going to say you're about halfway between that, about 85%.
So we're talking about the money over about 170 billion is waste.
And you can compare that to some of the GDPs of some of certain countries.
And it's it's as you can see, as you find out, it's a lot more.
In fact, I've written it's probably a bit more than this. So a lot of research, a lot of money is spent.
A lot of research is wasted. Okay.
So where does that figure come from? 85%. So, oops, that one.
Right. So if you a starting point, I'll talk about this paper a lot.
Have a look at the launch of paper by Ian Chalmers and Paul GLASSIE.
And this is kind of where they get this from. So 100% of research, this doesn't come out of research.
50% is just not published. You never see it. Of the remaining 50%, about half is just not clear accurate to use in clinical practice.
Of that remaining quarter, half is flawed through an avoidable design that could have been a design that could've been avoided.
Therefore, it limits your application. So that just leaves about 13.
So not even 15%, about 13%. So there's a big problem in science and health research.
So it's a really good paper. And as I said, I will draw on that. And here you are. Here is the paper.
So they set out a sort of stepwise guide to where the sources of research is wasted,
and actually they start discussing where it could be improved and resources targeted to prevent its waste.
So a great paper you haven't read it. Really, really interesting. But I'll sort of drawn, as I said, through the talk today.
Okay. Why does it matter? Well, we end up producing or not producing research that kind of help patients and clinicians.
There are examples, and I'll give you one today of research about potential harms that are not fully revealed.
And of course, we waste valuable resources in ever in a ever dwindling environment of resources.
Okay. So why is research wasted? Okay. Point number one.
Choosing the wrong question for the research. And you want to something you want to study design that's particularly relevant.
Okay. Few years ago in the UK, David Cooksey, Sir David Cooksey was asked to do a review of UK research funding, and it's an exhaustive review.
The Cooksey report very famous, and he identified sort of effectively two gaps in research.
Anybody know this? So if you look at the research pathway, if you like,
from basic research or bench science and working in laboratory to prototype, discovering design and so on, clinical trials,
health technology assessment, health services, research, this it's starting to involve patients,
knowledge management and finally actual health care delivery at the coalface.
This sort of pathway of from basic research to to practice if you like identify two gaps.
First gap in translation from this sort of area.
So the discovery phase, early clinical trials to the next stage of late clinical trials.
And the second gap here, sort of technology assessment actually getting it into practice.
Okay. So there were two gaps that particularly, but this sort of area was fine, basic research.
But if you look, where does the funding actually go to? And again, this draws on Thomson Glacier.
Look, pure basic research receives by far the majority of funding.
So this is pure, basic research.
So if you talk about this sort of research here, when we start thinking about putting it into practice, we think about applied research down this end.
So this and you'll see the gaps are more towards that in this gets the majority of funding.
So they did an audit in 2004 in the mid 2000 and found about 68.3% of funding goes to basic research.
Slightly less later on in 2009.
But when you compare it to pure applied research, although it's going up, it's still a minor proportion of the funding.
And actually, what does it matter? Why do we. I mean, I'm an applied researcher, so it matters to me.
But why does it matter when they review? And this is just one example, and I'll talk through when they review the impact of research.
From basic science. To. To patient care.
You direct patient care. You know what matters? What makes a difference?
This is just one example that it makes. Again, an exhaustive review looking at mental health, tracking, examples of mental health, basic research,
and how it's been transferred, transformed or not into applied research and actual patient care.
And so they come up with a number of conclusions, this being one of them.
Despite significant advances in the biomedical understanding of mental health and brain function,
these are yet to have much practical impact on the diagnosis and treatment of schizophrenia.
So they use schizophrenia as one as an example, and this is actually from 2003.
And so one of the overarching conclusions was through a number of case studies was that clinical research.
So the more the applied nature has a large impact on patient care,
the more basic research, particularly analyses, over 20 years worth of data and research.
So we need to focus on clinical research more than we on biomedical.
And that's not as I used to be a biomedical researcher before I became an applied researcher.
So I understand the field to a certain degree. That's not to say that there isn't value there,
but perhaps one of the reasons is we're not channelling funding into research that really matters to patients.
Okay. Problems with external validity. Validity. So now we are thinking about applied research.
So we're doing clinical trials, we're doing studies on patients, but we've got problems with external validity.
So everybody comes with with the term external ability. Yep.
So did the results of this study apply to my patient or my populations of patient i population of patients?
That's an issue. And when you look through randomised control audits, a randomised controlled trial, this is just one of them.
They look to see areas in which the randomised controlled trials for cardiology, mental health and oncology, they took a sample.
How would the included patients in that in those trials map to real world patients,
the real world populations and in fact they said that some of the conclusions were 71.32% of the studies included in this review.
The individual study authors concluded that RTT samples were not representative of patients encountered in clinical
practice and or that population differences may have relevant impact on the external validity of the RCC findings.
So even with doing applied research, some of the findings,
we're excluding really important patients that they may apply to and here are some of the recommendations.
So they talk about from that paper about ways of managing external validity.
So for example, they say broadening the RC to inclusion exclusion criteria, but we're all a bit edgy.
Well, a lot of people are edgy about doing that because they think, well, it might dilute out my chance of getting a an impact as an effect size.
It's something to think about really carefully when we design research. Okay.
Why else as doing studies are unnecessary or poorly designed?
So this sort of this sort of point lends itself to the fact that why on earth are we doing research when the answer is probably out there already?
And this also highlights and this is what Ian Chalmers and Paul Glass you do is highlight the value of systematic reviews.
And I'm a big advocate of systematic reviews, so everyone know what a systematic review is.
Yeah. So by seeking to search systematic review PGC for all available evidence on a given topic and summarise the result into a usable format.
And there are numerous examples where a failure to conduct timely, systematic reviews, waste resources and harms patients.
Everybody knows the story that you know. Yeah. Oh, no.
Okay. Do you mind if I tell you? So now my son is two and a half.
And one thing I learned when he was born was there are lots of baby books out there, you know, how to be a parent and all this, etc.
But if he was born in the 1940s fifties, we would have got this book.
Okay. Dr. Spock's baby in child care book, he sold over 90 million copies.
In fact, it doesn't quite fit, but it's sold more than that. And I think it's the second bestselling book going into the Bible.
So Dr. Spock got a lot of things right in his advice,
but one of the things he didn't get right was about his advice about how to put your baby to sleep.
So he used mechanistic reasoning, so disadvantages to a baby sleeping on his back.
If he vomits, it's more likely to choke on the vomit. I think it's preferable to a customer baby, to sleeping on his stomach from the start.
So if you think about how many copies this was selling and still sells, in fact you can think about the impact of that sort of advice.
And another if you if you're not familiar with testing treatments, it's available freely as a PDF book or to buy.
But it's a great book with seven examples in it. But if you look at the timeline to what happened to Dr. Spock, he put it edition.
In that edition he took talked about putting babies on their backs.
In this edition, he switched, changed his mind, said, Oh, baby should be put on their front.
And if you look at the timeline to what happened, first, studies just harm.
Second studies just home by the mid 1980s.
Third study. Three further studies suggest harm campaign started to be launched to advise parents to put their babies back on their back.
Back to sleep. Just being one of them. And in fact, it wasn't until the mid 2000 that the first systematic review was published about this.
And in fact, this is what the systematic review.
So I'll talk very briefly, but if you look at these are all the studies that were done exploring this.
These are all observational studies. And actually if you them out on a metal analysis, they would help with the responses.
Yep. So if you look at the summary summary results here over a fourfold increase in cot deaths from putting the babies on their front.
But if you reanalyse that looking at what they call cumulative paralysis.
So what you've got here is the same study, carpenter.
Carpenter. But what they've done is they've effectively said, well, next time this study came about or this study came out in 1970.
Let's add the data to that, to that, and come up with a new summary statistic.
Then people did that. So let's add it's each time your confidence intervals are getting narrower and you're building up a picture.
So my question is, at what point, given that this is the line of no effect,
statistically significant on either side, at what point would you have known the results of that?
Yeah. So by 1970, he should have known that putting babies on their front is bad.
And yet all these studies were done thereafter and the advice continued.
So that's about by 1970.
Another example which the story of violence. Some have.
Okay. So fireworks originally marketed as a drug for is an anti-inflammatory.
So for arthritic pain and basically pain relief.
And hey well hey some of the advertising quite emotive. Sometimes you can improve two lines with a single prescription.
Very nice. Sometimes you can ruin lives with a single prescription.
So but Vioxx is an interesting story because years after,
we understood what cumulative matter analysis is and the value of systematic reviews, we're still having problems.
So this is now in the 2000. Withdrawal from the tooth, from the market in 2004.
Of the concerns about increased cardiovascular risk, particularly heart attacks.
And then after its withdrawal, a systematic review looked back at all the studies published by the manufacturer and said,
actually, at what point should trials of Robert Cox Vioxx have been stopped?
So again, it's a cumulative meta analysis. So these are the this is the year here.
And as you go from 97, each one is a trial. But adding on the results of the previous trial to the next one and so on.
So when would when would you have known Vioxx was harmful?
2000 put me around, too. Yeah, you're right. What about him?
Well, you can see a trend emerging. So all these trials, why on earth what they done?
Okay, so that's a lack of understanding about the value of systematic reviews and not doing them timely.
Okay. But equally, I was going to say that concerning that those were done in showing that.
So all of them were not statistically significant as well. And then what's one word?
Well, yeah, I mean, but they may they were well, there were flaws with some of these designs as well.
So they each try to learn from the next. But the point is, by this point, they should have stopped doing any trials.
All this all this data here was unnecessary randomising people to harm.
That's the question, because it looks like, I guess, that second set of 12,
is it because it's so large why you don't see such a huge relative to the other changes where it's kind of flip flopping back and forth?
Which one? This or this one from there to there? Yep. Yeah.
And it could be the size. So suddenly you get a better powered study and you see a true effect or a reflection of a true effect.
Yeah. Why? They continued. What's. What's the reasoning?
Why do they continue? Market forces.
If you want to be cynical, prove a point.
I guess you could make an argument not knowing what the design flaws of all
these that maybe that that one trial and the second one of 2000 might be right.
It was in that study that caused things to shift in the wrong direction.
And so maybe this one poll or actually worthwhile to show that trend continuing that's accumulated as cumulative studies in 13,000 patients.
No, but I think what you're saying you're saying is there's a fair jump from 51932.
Yes. Yep. Okay.
So remember I told you 50% of trials are not published.
So failure to publish relevant research promptly or at all. So this one and this is talking about bias, one of them in particular reporting bias.
Everybody familiar with the different types of reporting bias, although I'll just talk you through some of them anyway.
So great resource. Cochrane handbook here. Beautiful table here about types of reporting bias.
Publication bias. So publication or non publication of research findings depending on the nature of the direction of the results time like bias.
So a rapid or delayed publication based on the type of result you get and so on.
And so I can go through a couple more.
Multiple publication bias, multiple single publication of research findings depending on the nature and the direction of the results.
And you see this. So I mean, that particular I've seen that recently with some of the some of the trials around anticoagulants,
you know, chopping and slicing data into different publications.
And in fact, this the same trial, it just adds to the confusion. Location, bias, citation, outcome, reporting.
And I want to talk about this later. Selective reporting of some outcomes, but not others, depending on the nature and the direction of the results.
And altogether, it's all very, very common.
I mean, even today, I don't have anybody telling me that report about some new dietary recommendations today in the news.
No, you've got to be sitting in a classroom, which is which is great.
But it's been on the news today about some new some collaboration, talking about new dietary.
They produced a document which they called a guideline. And in fact, it wasn't and it's based example of citation bias.
They just cherry picked various citations and then put it together in a documents and press release.
Did this list the fact. Yeah, yeah. Found carbs.
So it's an example of citation bias. There was no systematic way that they got the data.
Okay. So if you want to know that way about they say you're quoting something in any of your assignments or theses, this is a great paper to know.
And in fact, it's been cut off.
But you're welcome to have the slides with all this heavily cited to sort of basically look systematically at all of the issues around
publication bias and basically found all the all that all the all that examples I gave you found examples empirically in almost all those areas.
So good research points to look at.
Dissemination of research finding is likely to be a biased process. Overall conclusion, massive review.
And when you look at some of the examples,
this is one of the previous systematic reviews when they looked at publication bias in clinical trials due to the direction of fact,
whether they were positive or negative or the results were positive or negative, and published in the Cochrane Library as a Cochrane Review.
Association between publication and statistical significance or direction.
The results were examined.
Trials with positive findings were more likely about four times more likely to be published than trials with negative or no findings.
Positive findings tended to be published after 4 to 5 years, negative findings after 6 to 8 years.
So there was that example of that, that delay. Okay.
Bias or unusable reports. Another reason why research is wasted.
So this is this is about sort of publication of data reports that we simply cannot use in clinical practice.
Okay. Who decided the story of study 3 to 9?
Okay. Well, tell you a story. So this was study 3 to 9 refers to the the study reports.
And this is the publication from that study report. And it was a clinical trial sponsored by GSK about the drug paroxetine.
And it was looking specifically about the use of it as an antidepressant versus placebo in adolescent adolescents with major depression.
So quite an important area. Very important area clinically.
This publication is here and published in 2001.
So, you know, this is not years and years ago. This is this is reasonable things that people can remember.
So 2001, the report was published and it presented four outcomes, all favouring proxies.
In all showing proxy works for this, this, this and this. As I said,
the trial was sponsored by the manufacturer and the report contained the words
that the drug was generally well-tolerated and effective and on face value.
Okay, fine. What will cause you appraise that and yep.
Seems to a reasonable trial done fairly well. And to million.
I mean, this is sort of impact to many prescriptions issued for children and adolescents in 2002 alone.
So just goes to show about the widespread use of this particular drug.
But when we critically appraising and asked, we're only really critically appraising the data that we're seeing.
And if we're appraising, you know, a PDF version of that, we're only praising the data we see.
What about the data that we don't see? And this is a great example or a bad example, if you like.
So this is an example of selective reporting.
So I told you that they reported four outcomes all favouring.
If you go back in time and look at the actual study protocol for that trial and the study was titled 329 and the study data,
they actually said not for outcomes.
They actually stipulated that they'd have two primary outcomes and six secondary outcomes in this study, in this trial.
And what they also did, which they sort of didn't fully eluted, but they said that they were going to measure 19 additional outcomes in the study.
Now, of the two primary outcomes and the six secondary outcomes, none of them showed favourable results.
So they've been. Of the 19 remaining items that they just happened to measure, all but four showed a favourable result.
So the ones that didn't have been a to the fool, they ended up getting into the paper.
So the primary, the pre-specified primary and secondary outcomes disappeared.
And as I said, it was only until 2015 here that the full data analysis was properly put together.
And this article in the BMJ restoring study 3 to 9 the harms the proxy and they
looked at another antidepressant imipramine in the treatment of major depression.
So in concluding the proxy no high dose implement showed efficacy for major
depression in adolescents and there was an increase in harms with both drugs,
clinically significant increase in harms, including suicidal ideation and behaviour and other serious adverse events in the proxy group.
So again, all because of selective outcomes which. Right.
So another thing so we talked about selective outcome switching.
Another issue about the way that studies are designed and the flaws in their in their design.
So this, again, is a really nice paper looking empirically at that.
And again, I'll put the link there if you're interested.
So looking at flaws in design, conduct and analysis in our cities and the systematic reviews that include those asked,
and basically the effect of that has but potentially erroneous conclusions with serious consequences for patients.
And what they found when they did this empirical work. So they search for sources of bias from poor study design within a defined sample of studies.
And they they identified nearly 1300 studies trials.
And of these, about 43% had at least one domain, a high risk of bias, ie from poor methods.
And they went on to explore these reasons and found that all much of this, if not all of it, could have been avoided.
So people are still producing research that is flawed in its design,
which could have been prevented and affects our ability to use that or potentially use that research.
And we should have learned by now from examples like study 3 to 9 about the importance of actually
saying we're going to do something and then actually doing it and presenting what we've done.
Okay. There must be some good news, surely.
So I thought I'd do it. Just talk to you about some of the good news that's out there as well.
And there are changes, and I'm sure you'll know about some of these already as well.
Okay. So I'll give you a few examples of what the NHS is doing.
So a lot of our funding comes from the National Institute for Health Research, which is the the research arm of the NHS.
Some of you may have funding from them as well. Be very familiar with them.
And again, the their aim to improve the health and wealth of the nation through research.
And in fact, they've just celebrated their ten year anniversary.
They've been around for ten years now and it feels like they've been around for a lot
longer because given the amount of development that's come from their their work,
their support. Just to give you a bit of background.
So in in the annual report, 20 1415 spent about a billion, isn't it about billion?
And you can see how they break down. They spend a lot on building capacity through training, individual programmes and infrastructure.
So that's big, big grants for, you know, setting up units, schools, etc., research schools and so on.
And then you can see the sort of areas that they spend their money in.
So cancer, mental health, which is which is important to see that mental well recognised and so on.
So they spend a lot and they respond. They're reflective of needs as well, or they try to be certainly, but they're also reflective.
And take the what we've been talking about, the value and the importance of research and adding value to research.
So building on what Ian Chalmers and Paul Glasgow originally put forward in 2009 in The Lancet,
they sort of had a whole have a whole section of ensuring that every research project that goes through them meets at least most,
if not all, of these pillars for adding value. And I thought that's quite that's quite nice.
So when you're designing,
research yourself either for an assignment or actually when you're when you are principal investigator yourself, if you're not already,
that this is actually a great resource to have a look at, to kind of think about actually,
if I want to go for funding, you know, I've got to make the case for my research.
It needs to be it needs to be of value. So they've got some great resources there as well.
And actually, you can see there they're sort of the mindset we were talking right at the start about
the value of basic research and that translational gap between applied research.
So if you think just at the top line here about what we're talking about in invention,
so we're talking about basic research here to evaluation and then to adoption and then diffusion in health systems.
Right. From the basic research, you can see where the funding is now coming and how much the National Institute for Health Research focuses on this,
bridging that that gap that we talked about right. The start.
So Medical Research Council in the UK we're focussed mostly on basic research and developmental research,
so innovation in basic sciences and then they sort of have this joint funding stream where they both fund this,
where they start to evaluate mechanisms and then and so on.
These are the big units and so on, different, different programs right through, as you see here, to patient care commissioning and patient care.
So this kind of gives you the way that things are changing for the better.
Another important point, and we talked about this, about doing research that really matters.
And this is something that the night Charles picked up on as well. So questions being researched are the most important to patients.
The public and clinicians as well. Haven't heard of the James Lind Alliance?
No. Okay. So it was set up by Ian Chalmers of the Cochrane, who was involved with setting up the Cochrane Library.
But in the. 2000 set up the James de Lions Co, set up with the Sir James Dean Alliance, which has subsequently been adopted by the Night Show,
the national issue of health research as a form of actually getting a more formalised process for setting research
questions to the research community like us that actually stem from issues that really matter to patients.
So one of the main innovations was the idea of setting up what they call priority setting
partnerships space that would specifically focus on bringing patient care as clinicians,
as groups coming together. And they discussed and identified their different.
It's more the discussion. It's the sort of formalised process to identify in certain clinical conditions.
Where are the uncertainties? Where are the uncertainties in practice or in care?
And the aim is to ensure that those who fund health research, like the NIH,
are all those who apply for research actually know what really matters to patients.
And just as an example, so and they're increasingly starting to publish their process.
So if you are interested in in patient, public involvement in research and the value, it's worth looking at some of the work that they're doing.
So here's an example. So this is published in the BMJ Open around setting research priorities for patients undergoing shoulder surgery.
And this sort of talks about the process. So they send out a survey, gather the respondents, put them together,
the rest of the respondents put them together to see which certain certainties could be taken forward to the next stage.
So they identify those that are out of their scope, but those are relevant.
It's quite a systematic process. Remove duplications,
compare the uncertainties taking brought forward to existing evidence and actually identify from that where there are evidence gaps.
Take that to the next level whereby they'll have a bit more of a roundtable meeting,
steering group reviewing the results of the above, and then finalising a set of core research priorities in a particular area.
And that's just for patients undergoing needing shoulder surgery.
And this is an example. So what they then come up with what they call the top ten, are research priorities from this sort of process.
And you can see that I won't read it all for you, but you can just sort of see.
So some of the things where there actually there is an evidence base for we're doing it or it's a question that matters to patients,
but there isn't an evidence base there. So, for example, let's have a look.
Here we go.
All patients, including older age groups with rotator cuff tendencies as a tear in your shoulder, best treated with surgery or physiotherapy.
We don't have the answer. There is no answer out there. And yet both those are in clinical practice, but there's uncertainty.
So it's just highlighting the method that they use to identify these research priorities.
So here's a top tip. If you're a researcher, if you're interested in identifying areas to kind of go and have a look at their website
because the richness of uncertain unanswered questions for you to kind of explore.
And again, highlighted by the fact that they have been through a process that matters to patients and uses things a lot.
Things are still there's still room for improvement.
So in this scoping audit, again, sorry, the reference has been cut off, but they look to see clinical trials for a number of clinical areas published.
They compared sort of what were the the JLA partnership setting priorities compared to the number of the registered trials in that area.
So just to sort of cut that sort of make it simplify it,
basically the long short phase registered trials in a particular area were much more focussed on on drugs, vaccines and biologicals.
But actually what matters to the partnership, the Peas and Patients was more around disease management to do with education,
training, service delivery, psychological therapy.
So it just highlights the fact that a lot of the commercial trials are not necessarily meeting the needs of patient.
Okay. Past few minutes. So everyone had all trials.
Okay. If you haven't, we talked earlier about the fact that around half of all clinical trials have never been reported.
This was a campaign set up by excuse me, a few years ago now to kind of really push this that actually this is just not on.
And there's been Goldacre who works with us at CBM, one of the founders,
just kind of pushing the agenda of actually all trials conducted, should have all trials published in full reports.
There should be no hidden data.
And, you know, there's a catalogue, as I said, and we just looked at a few examples of of where that has caused harm and confusion.
And so they want all trial, all pharmaceutical companies, academic institutions and so on to register trial.
You publish your findings and it must be transparent and accessible. And I think one of the they say on their website,
one of their landmark moments was when they got a massive pharmaceutical company to sign up to all trials to say,
yes, okay, we agree with your with your policy.
And we will ensure that all our trials are published and available freely from all our registered trials.
Now, what's interesting is we're looking at whether that's currently being done at the moment.
So we're auditing that openness and we're helping and supporting it.
So it is a step in the right direction, but there's still much, much more work to do.
Increasing the value of systematic reviews.
Now, I've recently argued that every health researcher should do a systematic review at the start of their training.
Some may disagree, but there's so much value in it.
It's a lot less expensive than doing a clinical trial, and it probably make you do a clinical trial a lot better if that's what you want to do.
But I'm not the only one who's appreciating the value of systematic reviews.
So this from the NIH are about ensuring that every resource, every trial they fund, particularly through the health technology assessment, is funded.
One only when you know about the event, the evidence is available already.
Only when you know what's available already will we look into supporting your your trial.
And so this is one of the again from that website I showed you about adding value to research and they provide guidance notes as well.
So what do we mean by that?
So they explain to you kind of, you know, ideally a systematic review, but if not a systematic assessment, at least of the current evidence as well.
And they sort of produce this guidance note for applicants to make sure that happens.
Is it happening? Well, in fact, people are auditing it now. So looking at just one program, the Health Technology Assessment,
Trial Planning and design from the NIH, are they looked to see of the trial of 47 funded trials,
47 funded trials during this period here, only five didn't reference the systematic review and all those five gave a clear reason why they didn't,
which was justified when they re audited in 2013.
All trials funded in 2013 referenced the systematic review in their proposal.
So things are improving, but not quite, not quite there.
But they're heading in the right direction, making reports more useful, reducing outcomes, switching is everyone is anyone familiar with compare.
So this is work that we're doing at the moment,
led by Ben and five medical students who are putting their time in to support it and two clinical academics, one of which is me.
And what exactly what the idea behind compare is,
is to was to prospectively look at trials as they're being published in the top five major journals and actually
compare hence the name the protocol for that trial to what they published and to see whether there are any outcomes.
We talked about study 3 to 9, see whether there's any outcomes have been switched, lost, changed, altered and so on.
And if I'm given a reason for and justifiable. Okay, fair enough.
But if they've just suddenly mysteriously disappeared into that bin that we showed earlier, why is that?
So the next stage was to write to the journal and say, know, you often have to do that within two weeks of the publication.
This this public, this publication is in your journal, blah, blah, blah. We've noticed that the protocol says this is the reason why this we like to
highlight it to the authors and sometimes even authors replied back and said blah,
blah, blah, you know, this is something that we have considered, but these are the reasons why we switched, you know,
or they've said sometimes you've misunderstood, you've got the trial, you've got the understanding wrong.
But it's actually been really interesting to see what sort of responses and and on the 67 trials checked, only nine were perfect.
And this was all this was this was only done last year. And it's still ongoing.
The correspondence between all these trial authors and the journals who went to the trials,
if the protocol is available for you to look at, it should be it's not always.
And then you've got time to go through and check that it fit in the protocol.
Well, five medical students and two clinical academics.
But. Okay, well, okay. Well, let me flip the question.
You tell me who should be doing it. Should it be the reviews?
Yeah. Funders are going to published in the journal.
You should check that before you publish it in your legitimate basis.
So is that I ask, is that the point of peer review?
And in fact, that's what we put some of the journals.
And that's why some of the most vitriolic, if you like, exchanges have been with the Journal editors, not so much with the authors.
And I'll just give you an example. Have a look at compare.
You'll find there's lots of blogs around what we're doing it, why we're doing it, and so on.
But it has been an enormous resource to do it, but I think very valuable one.
And we talked about that same point. So one of the interesting discussions we've been having is with the Annals of Internal Medicine,
one of the top five journals that we've been discussing with, and they were fairly resistant to to what Ben and us were doing.
And actually there was quite a fairly heated exchange.
We've logged all the heated exchanges in in the blogs, so you can read all about it very transparently,
but actually only recently because of the pressure and the dialogue,
they've actually changed their policy now only in the last few weeks, maybe a month or so.
And this is an income has been reported, in fact. Well, that's few days ago.
So journals publish clinical trial protocol.
So they've now actually changed their their policy about making sure that protocol, as you say, are published next to that their trial.
And I suspect internally they'll make sure that, as you saying as well,
that the peer through the peer review process or editorial process, that the outcomes are much, much more closely scrutinised.
Okay. So a more broader way of reducing waste is through a new initiative called the Reward Program Reward Campaign,
and it's a matter of a number of journals, research that has CBM and it goes,
the list goes on down here talking about talking to each other, partnering up,
but actually kind of making a commitment to ensure they can do whatever they can to reduce research.
So again, a nice resource and all the related content if you if you want and research goes
on for pages and below this about sort of some of the the reasons behind it,
the ways the citations and some of the clinical examples that we've talked about today as well.
Okay. And I'm just like, let's just think it's 45 minutes. So it's the last minute.
I'm going to leave you with one last paper, which I think will be well worth reading.
And a final thought. Written by Devilment is a professor here in Oxford about the scandal of poor medical research.
And this was written in 1994 to 22 years ago now.
But just very one overriding line, which, you know, is something never to forget.
We need less research, better research and research done for the right reasons.
Really good paper. Okay. Thanks very much for listening.