It's great. It's great to be here. And to present one of my hobby horses is about systematic reviews.
I appreciate you sort of nearing the end of your first weekend, you probably a little bit tired.
So I went to I have to tell you now this at the end of my 30 minute talk,
I'm going to ask you to sort of spend a bit of time discussing what I've presented and then to give me some feedback at the end of it is in part,
the reason is that Carl,
myself and Jeremy Herrick are writing a paper around why we think the current system of systematic
reproduction is wrong and we've jotted out some thoughts actually spent a fair bit of time on it.
And in a way, you're the guinea pigs to our ideas. And so we want you to point out all the obvious errors we've made as before we go to print.
But also we think it will be challenging to a lot of the things that are currently thought about systematic reviews,
a bit of background surmise myself. I'm a scientist by background.
I've worked in what was clinical effectiveness in the mid 1990s.
That's my start. And from there I moved to South Wales where I was told to make the GP's more clinically effective.
So I went and spoke to the GP's and they said we don't learn searching skills,
we don't want to learn critical appraisal skills, we haven't got time, just answer our questions.
And so that's what I tried to do using this is pre Google days and when Medline was on CDs or separates or stats
on and it was I basically taught myself to answer clinical questions and as I was answering these questions,
it dawned on me it be so much easier if we could search all these resources of high quality evidence from one place.
And that's how I started the database, and that's grown ever since.
I'm not going to talk specifically about the database. If you want to ask me any questions about it, you can do after this.
But a lot of my thinking has and a lot of the development of the databases come around from answering questions for doctors and nurses.
Me and my various teams over in various services have answered over 10,000 clinical questions, mostly for primary care doctors and nurses.
And it was not too far away from here that I was having a conversation with Aaron Chalmers, who obviously created the Cochrane Collaboration.
And we've got a great relationship now, but I just had to, I, I struggle when I'm answering questions for GP's,
I struggle with the notion on one hand I'm hearing Cochrane's great systematic views are great and on the other hand they don't up too many questions.
That was my experience.
We did an audit and less than 1% of the questions we received could be answered by a single systematic review asking What are they for?
Surely they might be useful. And that was my that was a starting point for my scepticism about systematic reviews.
And now we hear we are sort of sort of think how long it was eight years ago.
And it's just I had that conversation with Ian and and I think he's developed and more
and more information is coming out about what I think are the flaws of systematic views.
I'm sure you've always been told systematic views are great and they can be, but I just think a lot of the flaws are hidden.
You've invariably heard of Archie Cochrane and in 1979 he complained that medicine had not yet managed to produce a
critical summary by speciality or subspecialty adapted periodically of all relevant randomised controlled trials.
He felt it was important at the time and I think most people still think it's important and irrespective of my cynicism of the current methods,
I still think systematic reviews are an important tool. But we're nowhere near that.
We're nowhere near synthesising all randomised controlled trials and many of them are just floating about with,
with, with, with no attempt at synthesising them because the cost per systematic review is huge.
I don't know if anyone actually tries to do a systematic review here and all find it easy.
Know the data I could find is each systematic review takes approximately 1100 person hours.
And if you look at a Cochrane review,
it typically takes 23 months from registration to publication and there's obviously a bit of work going to go and for before it's registered.
So we're looking at well over two years for systematic review from that idea to publication.
And as a result of this work, most of the people I've spoken to feel free to contradict me, those who have done a systematic view.
Most people who do assessment of you don't come it don't come out of it thinking, Oh, I'd love to update it.
I can't wait to start again and update it again, because most people I know seem to look at the thought of updating their system of review, of horror.
It's not easy. I've not done one, but I see enough people near the end of their system out of use,
who are horrified by the work that's needed to do it and to do it properly.
And so most systematic views aren't kept up to date. And well, I'll show you later on the slide.
But about two thirds of Cochrane systematic reviews are currently out of date, which is a huge number.
I've spoken to Paul Glasgow who's called predecessor here,
and they've quote from him They've the methodologies raised the methods and process barriers so high that most are not being updated.
I often view this is the methodologies are running amok and they're trying to squeeze out every single little bit of bias,
but with no reference to the cost. In producing the money.
I feel that they are producing Ferraris where most people will be happy to have a golf or a mini.
And this is a slow decision. I might be I might be seen to be slagging off Cochrane.
That's. I'm not I'm just using Cochran's information because they are transparent, a lot more transparent than most other systematic review producers.
And so they're being picked on in a way because they're transparent.
So I'm saying this is a huge positive and Ian Chalmers is always very happy for
Chris Cochran because he set up a prize for actual overt criticism of Cochran.
So they're big enough, they can take it.
But this is taken from the Cochran Library and Oversight Committee and the annual reports and financial statements.
What we've got here is the funding, the income of Cochran. And as you can see, it's gone up.
And it was a significant increase in 2008, 2009. That's probably when they started working with Wiley.
Now, that was my assumption. And this line here is a number of systematic views in the Cochran lobby that are actually up to date.
So in 2009, it was 39.8, and it's in last time they published information is down to 35.8.
So problems getting worse, not better. That's my reading of it.
I have read that correctly, by the way. I was always surprised. I have to keep rereading it.
Is that out of date or up to date? That's actually 35 days up to date for clarity.
So nearly two thirds are out of date.
And I think one of the biggest problems with Cochran and the Cochran style systematic reviews, they rely on published trials.
There's a paper fairly recently by Cheryl Barron Gertz, a bit of a rubbish reference, but she'll be able to find it from there.
Have you done your searching skills and the problem and basis?
That paper highlighted that the vast majority of Cochran systematic reviews rely on unpublished trials,
and when they try and look at unpublished trials, it's a very unsystematic way of going about this.
And I'll talk more about the difference in published and published trials. I don't know.
We've done that yet, but there's lots of reasons for relying on on published journal articles only.
And two clear ones are reporting bias and publication bias.
Have you done biases yet? There's some weak nods and there's very few enthusiastic party notes.
But anyway, reporting bias when you rely on published trials.
I'll get the exploration out of the way now. What you typically have when you have a trial is you have a huge amount of data.
This data is then typically summarised into something called the Clinical Study Report.
People heard of clinical study reports, clinical study reports, or hundreds of pages of long, long and massive documents.
They are typically unstructured and they're a nightmare to work with.
And what happens is you have a clinical study report and then someone who wants to get a
publication will write that will summarise those 200,000 pages into 810 sounds very full.
And that is the Journal article and then obviously the doing that with some rise to the abstract.
Now most people would say, oh, it's you shouldn't do a systematic review in an abstract.
And I'm certainly not proposing that any of the reasons would probably be because you're missing big chunks of data.
We just rely on the abstract. It's the same argument I see is relying on journal articles compared with clinical study reports.
Clinical study reports is is is summarised and loses about 80% of the data.
And so if it's not acceptable to use an abstract, do a systematic review,
I sometimes wonder why is it acceptable to do a systematic view based on published journal article?
Because you're still missing shared loads of data. It's a pragmatic decision and there was a recent paper the cow was involved with and Tom Jefferson
and Pointer Risk of bias basically is looking at risk of bias of industry funded trials.
And what they did, they compared the core reports of journal articles with the full clinical study reports.
And what they found was, well, I might as well read that we found that his information increased in the documents.
This increased our assessment of bias.
This may mean that risk of bias has been insufficiently assessed in Cochrane reviews based on journal publications,
and that to me is not trivial publication bias.
Certainly if you're familiar with the All Trials initiative they report the 30 to 50% of all clinical trials aren't published and here's a good,
good time to introduce it. Looking at the turn a paper is when a pharma company wants to get a drug approved.
It needs to show the FDA in America all the trials that it that it's done, looking at the efficacy,
the side effects associated with a particular drug, they have to register them before they start the trials for them to be accepted.
But there's no obligation for them to actually publish the trials.
They've got to own the FDA. Sure, but they don't have to publish them in.
Journal articles and turn to looked at turnout are looked at the selective
publication of antidepressant trials and its influence on apparent efficacy.
And what they did they went to the FDA and we've it's not easy working with the FDA or the AMA and the European Medicines Agency,
but eventually they they looked at all the trials that had been registered with the FDA,
and they compare that to those that have been published in journal articles.
And what they found of, of all the studies that were in favour or showed that the antidepressants were working,
37 versus one were published versus unpublished.
And then of a third of those that were found to be negative, I didn't support the use of antidepressants.
Only three were published and 33 were unpublished.
So if you do a systematic review on published trials, you'll find 37 in favour and three against.
It's not fair that reasonable. And I think Turner pointed out the madness of this.
And what what Turner did was they looked at what would the effect be of including the the FDA trial data,
the regulatory data versus the the matter analysis done on published journal articles.
And they found if you just relied on the journal articles, there was a 32% increase in effect size.
This is not trivial, you might say. This is not this is an outlier.
Certainly not good. This is a paper by heart in the BMJ 2011.
And what she did was she looked at an awful lot more,
basically the same principle comparing matter as is done on a on published journal articles
comparing those of those that included regularly regulatory data such as by the FDA.
And what I've done,
I've zeroed in the zero line is the effect size as shown in the best analysis based on published journal articles and the horizontal bars,
or how wide the discrepancy was when you added in regulatory data.
And so the top one, it was out by over 150%.
And down here, if I can get my point to working, is over 50,
50% reduction and then over 50% of the approximately 50% of the cases, the effect size estimate was out by greater than 10%.
And what the biggest problem that's that's that's serious enough.
But one of the things that was actually I predicted before I actually read the article that well,
it's any ones that involve far more will always increase the effect size in favour of the drug.
But I didn't find that there was no pattern. So what makes it even worse is it's unpredictable.
You cannot see if you see a journal or a meta analysis based on journal articles,
whether it's an underestimate or an overestimate, the chances are it's going to be out by 15, 20% and confidence intervals.
Don't touch this. You know, you see little diamond at the bottom and you think lots of that's a very narrow time.
And therefore, we must be pretty certain does not doesn't take into account the unpublished trials misleading.
And so I think part of my concern with with matter and systematic reviews used on published journal articles is that I didn't realise this and
I don't think many people realise this when they look at systematic reviews is the effects and the harms of things such as publication bias?
Well, what's what's even more scary is the amount of trials that are and the amount of data,
not just trials are coming our way in 1976 is about a thousand.
This is often called this. It calls data. You can't criticise it.
In 1976 there was about 1000 trials per year published in 1994, 10,000 in 2000, seven, 25,000.
And by 2018 19 it's estimated to be 50,000.
Because of the success of the All Trials initiative.
More trials are being they'll help contribute to the 50,000 because more trials will actually be published,
but also to be much greater availability of data in the form of clinical study reports and individual patient data and things like that.
Which is. It's great that we've got the data, but my view is the current system can't cope.
So why do we think adding ten times more data is going to make the system work any more efficiently?
It's not. It's going to creak. And I think at the moment the current system is coping really badly.
So my general view is when we add more data, it will break.
I actually think it's broken now, but that's just me. And so we need a new system.
I think, oh, we could we could pray that we're going to get a tenfold increase of funding.
I think in the end that's unlikely, I would say. And so I think it'll be a gross misuse of funds to throw extra money at the current system.
And that's actually how I approached Carl and Jeremy initially, I said.
I felt it an ethical issue around the allocation of funds to should we be spending so much money doing system reviews, which are clearly problematic.
Anyway, that's another point. But I think that if people continue to believe that the current system review system is great, nothing will change.
And I think there's obviously organisations that are very keen on maintaining the status quo and lots of people,
not just organisations, make a lot of a nice career assessment reviews and I suppose change is always difficult.
I suppose I'm saying because you always have the status quo, we'll try and maintain the situation and what the new system needs to be is both quick
and robust and we can see no way of reconciling that unless you have two systems.
One is quick and one is robust. The quick system will be brief and it will be very, very short.
Will you shorten search based? You will take the systematic view method and we will significantly decrease the amounts of the cost at each stage.
Expedited search, simple quality assessment.
And the view is that we could produce and this is a very much at supporting clinical practice and that we will produce very short, brief reports.
If you look at the very first Cochrane systematic reviews, they were quite small documents.
I think they're about 14 pages long and that was with including all the references.
And now I can never count all the pages. I'm always it's always about well,
well over 200 pages by the time we scroll through it all and they would highlight key outcomes and they would highlight likely effect sizes and
they would be easily updated on the various things that we are that are in the pipeline around updating systematic reviews and things like that.
None of which have come online yet. But I think there's that there's a if we can, we've, we can rapidly produce sorry.
If we can significantly reduce the cost per rapid systematic review,
we can do an awful lot more and therefore we can incorporate all randomised controlled trials in that way.
The systematic, the work I've been doing in Public Health Wales.
We did two very quick methods of sort of rapid reviews.
What we did, we blinded one member of staff literally,
and we gave them ten questions based on Cochrane systematic reviews and we got them to answer it using this technique that took 4 hours and 85%.
So is that 85% of the times it was there was we got it the same as Cochrane and got the half is that of the 5% is that because it was ambiguous?
We weren't quite sure. But there's another technique that we do on the trip database which can do rapid reviews in about 5 minutes.
And again, that got 85% agreement with Cochrane.
But that's just my my non-scientific way of doing it.
What I normally do things is just in the office and just mucking around with this, the rapid reviews.
And there's this huge one of the pros of rapid reviews is no standard methodology and people get concerned by that.
But there's been three papers recently which have looked at reducing the time taken to do systematic reviews.
Hammonds and Hammonds, he works with Brian Haines. You may well have heard of McMaster.
They do a wonderful service called Evidence Updates, which if you're not familiar with, you should be evidence updates is a great system.
It's a they look at hundred and 20 of the top journals and they do an internal critical appraisal stage.
And those that fail, a critical appraisal dumped from the system and those that they feel are suitable quality to
inform clinical care are then passed on to a network who clinicians who is a cardiology paper.
It is sent off to at least four. Cardiologist might be three I think it's four and the cardio is rate it on is it news worthy of
its relevance clinical practice and they have to get a certain threshold before they're accepted.
Those two stage processes kick out 96% of the papers and from the top, the top 120 journals,
they get rid of loads of stuff, which is just not pertinent to clinical practice or newsworthy.
Your well done properly using that.
It says here when they updated using the their techniques using the evidence updates they use less than a quarter of the new
studies found in the Cochrane update and they said that most reviews appeared unaffected by the admission of these studies late.
Now, any Italians here before I absolutely murder the pronunciation of this occur and you know, no one can contradict that.
Actually, there's a bit will flair in the sound of it or confident it would have wonder what they did.
If the systematic review was on a cardiology topic.
They would take the top journals in cardiology and just use those to update the systematic review.
And so that's a different way of sampling the randomised controlled trials and they found,
they found able to replicate almost all the clinical recommendations of a formal systematic review.
Again, little difference. And then there was a great paper recently, one answered and they just did the meta analysis based on Medline.
And again, restricting the studies index on Medline did not influence the summary estimates of matter analysis in our sample.
So there's three separate ways of looking at trying to sample the number of trials,
and none of them seem to affect the outcomes of the systematic review.
But what I would say to that is it's very bold to make,
but I think more research needs to be needs to be undertaken to fully appreciate when you can get away with that and when you can't.
But I think if you talk to Tom Jefferson, who is, I would say, more cynical than me, he would say that he mirrors what Richard Smith is.
I don't see what Richard Smith says about published journals. He says they're the publishing arm of pharma.
He says they're a marketing tool for the pharmaceutical industry.
And his view is when I've discussed it with him, well, if you sample the marketing message from pharma, if you take 10%, you take 80%.
You still taking a sample of the marketing message, so you're still going to report the marketing messages they want to see.
That was his cynical view. So to my mind,
there is probably we can probably do an awful lot to get very close to where
Cochrane today and I appreciate the sort of fallacy that I've been saying.
Well Cochrane and Cochrane are systematic views are are flawed so we're getting close to a flawed system.
But my view is, well, if we're going to get to an approximate I call it a ballpark figure,
let's take two weeks doing it or one week doing it as opposed to 23 months.
But in some situations, there's a an acknowledgement that we need a more robust system.
And those, of course, calls spoken to you about his the work is with Tom Jefferson on the Tamiflu systematic review.
Yeah, but but essentially that was a huge amount of work, much more than doing a standard systematic view.
So those of you did a have done systematic reviews that was just a walk in the park relative to what I remember coming here last
year and presenting and then call and and Tom were going off to look at clinical study reports for the rest of the evening.
It was very romantic my friends on up but essentially to move.
I've already said the huge cost in doing a systematic review,
but in some situations a systematic review needs to be more robust in the current Cochrane technique.
So that requires essentially the authorisation of a large amount of cost.
And so it needs to be needs to be a very good reason for saying yes, let's spend even more resource on looking at this question.
And you know, there's various things we can use. I don't know if you've had a look through systematic reviews, but often the outcomes will be bizarre.
Often the topics will seem equally bizarre. My favourite one is music therapy for Ingrown Toenail, which I think I've actually made up.
But some people don't know and I'm, I'm getting confused, but I'm sure there's probably a protocol for that somewhere.
But often you you will get a feel from doing a rapid review about whether the outcomes are actually relevant to patients.
I'm sure you're aware of lots of studies where the outcomes are bizarre.
I saw one the other day for e-cigarettes and it's it was deemed a success if it stops someone craving a real cigarette for more than 15 minutes.
And I just struck me as being a bit strange, but all the outcomes that are being reported in there in the research, are they useful to anyone?
Does anyone care is a likely effect size.
Clinically important because lots of studies will find very marginal benefits, which are probably not going to have a huge impact on clinical care.
And is there a significant clinical or economic reason for doing it?
And the Tamiflu is a great example. But what we believe if and I'll talk about the Tamiflu a little bit more,
but what we think is that it should be the patients and it should be the clinicians who make the decision.
They should be the ones saying are these outcomes relevant in the patient's case of these?
Rather than to me, rather than to my care?
And the clinician can make judgements as well simply as outcomes useful are the clinically important effect sizes.
And if you ever get the chance to see Tom Jefferson talk about Tamiflu, go and see it.
He's a brilliant speaker. He's in the Army.
And I think that gives him a certain press. He was in the Army today. He still is.
No, he's in Italy, but there is a doctor there, but he was in the army.
And you can tell, I think, very, very commands a room.
And when they did the first Tamiflu review, Tom got asked the question, say, in your in your systematic review,
you relied on a a meta analysis on a previous meta analysis, which is using unpublished trials.
How can you trust it? Until my instructions haven't, I can't.
And over the course of the process of him wrestling with that concern, they came up with,
they basically rip of the Cochrane Handbook and they do not publish of in the Cochrane a Cochrane systematic review.
They haven't used a standard Cochrane methodology and they've avoided all journal articles.
They don't touch journal articles when they did the systematic review,
they just relied on the clinical study reports and they had to do a huge amount of work to
identify those clinical study reports and they still keep springing up and I think it is.
Yeah, it is. It's just been a phenomenal amount of work, as I say, much more than the standard systematic review methods.
So you can't just do it willy nilly.
You can't just say, I want to do the systematic review, which is going to take a huge amount of cost is just not feasible.
So if they've got to be reserved for the special cases where it's really important, in the case of the Tamiflu,
the first Cochrane Review was a lot less negative than the subsequent ones,
and it came out about the same time as if the government spent £500 million on Tamiflu.
Which is, as far as I can tell. Not much better than paracetamol, I think is the quote.
So there's a strong economic case of the government thinking of spending £500 million on an intervention.
Well, should we make sure that it's actually worthwhile?
I think that's sort of fairly a good a good example of when a system like this would work and it would
use all the data that's available and it would spend the time trawling through all the regulator data,
knowing what a nightmare that is.
But again, the All Trials Network had been great in getting the European Medicines Agency to be a little more flexible.
There was an agreement to publish all clinical. Well, I don't think it's been ratified yet, but the general views are publish.
All clinical study reports were a bit redacted for commercially sensitive stuff from next year, possibly,
although they're still keeping hold of the clinical study reports that had been prior to prior to, well, 2014.
And this these star reviews will be the proper gold standard.
Not I use that because I think most people's view systematic reviews and especially the Cochrane ones as being the gold standard.
And I think hopefully the evidence I presented will make you slightly sceptical, which is always good.
We're all scientists here. So in conclusion, I have no doubt that the current system is completely broken.
It's. It's on its last legs. And so we need to move to a situation.
And we're not saying this is this is our suggested method. And you can discuss it amongst yourselves in a few minutes.
But we think there's two there's only the way we can think of is do a rapid review for all to incorporate all trials in line with Arches vision.
And then there is a process that you decide is it worthwhile?
All the good reasons why we feel we should spend more money in doing a much more robust review.
But when you do a robust review, you do a proper robust review,
not one that's caught between two between two stools of as I would characterise Cochrane reviews.
That's all I wanted to say for that. But what I'm now going to challenge you guys to do is to help us perfects our paper.
It probably needs a lot of work, and what we'd like you to do is answer these questions because I think my talk,
well, I'd characterise it as being in two parts. One is rubbishing the current methods of doing systematic reviews, and the second one is an ad,
a suggested new approach of doing a rapid and a more robust method where appropriate.
That's how I characterise it.
What I'd be interested to hear is for the first one, were people surprised by you go into groups or talked to your neighbour about this?
Did anything strike you as being particularly surprising and did you feel I overstated anything?
Was the emphasis wrong? Did I not emphasises enough any of the criticisms?
And so I want you to consider those. And also with a suggested new approach.
Have we got it wrong? And I'm really thick skinned.
I really don't mind if you say Johnny to the letter rubbish for the last 30 minutes and let me go I to go and have my dinner.
And also, is there anything we if you think it's broadly correct, is there any way you might improve the way we position our reasoning?
So what's the best way to do this? Is it people comfortable talking with their neighbours or people?
And I'll give you sort of ten, 15 minutes and I'll start asking you to sort of report back.
Joining us to you want to just find the best way of talking with your neighbours and.