I've been working for a long time in systematic reviews.
Began my career as a researcher just up the road from here,
really in the Wycliffe Infirmary at the Cashew Clinical Social Service Unit was employed after
having been a student at Oxford and doing my dphil at Oxford and then not really knowing what to do.
And I saw an advert in daily information and they were trying to recruit someone.
I had no idea about cashew people like Richard Dole and Richard Peter.
I never heard of them. I didn't know that randomised trials,
I didn't know what a systematic review was and at the time we weren't necessarily calling them systematic review sites.
You call them overviews and a clue about any of that and just fell into working in a
place that now makes me feel like part of this process and so many of you will know.
Ian Chalmers Ian has been a sort of someone I've sort of step by step followed and through various things.
Ian set up the UK Cochrane Centre. I became director of the UK Caucus.
Senator Ian. Ian set up the James Dean Library and next year I'll be taking over as editor in chief for the James Lynn Library.
It's about the history of of evaluations. And one of the things that Ian will often say about me is he's not a historian.
I am. Ian's a better historian than I am. He is not here.
So I can say it just happens that I have my my dphil sits in history.
So Ian will often regard me as more of a star.
I think Ian knows a lot more about how historians should act than I do, so I'm going to try and set the scene.
I'm going to take you on a bit of a journey. History of what?
Well, is the evidence synthesis, research, synthesis, matter analysis, systematic reviews,
cockatoos, all sorts of words are used for the sorts of things I'm going to describe.
Some of these words are subsets of other things called reviews are systematic reviews, but they don't all contain a matter analysis,
which is the mathematical best evidence synthesis and research synthesis are words that we might use,
and we don't necessarily mean systematic reviews.
So the first thing is to think the language of it. My aim is to try and give you a framework for thinking about this history.
I cannot stand here and definitively recount the history of systematic abuse.
I think we're going to need to wait decades still for that.
And someone more informed than me is going to really do the history of systematic abuse because there are challenges for me doing it.
I'm building on work done by other people. So others have looked at this.
I'm providing examples, but one of the things I do feel as wearing my historian hat is history is not about who was first.
History is not about the state, their state, their state, their state.
I will show you dates, but they're not to say and this is the first time this ever happened, anyone who says that is a hostage to fortune,
we probably know roughly what was among the first,
but there is no reason to believe that something is the first time that the idea was hard because the idea is not remarkable.
Who was early? That's what I think we can talk about. I think we can talk about things that maybe began something because without that thing,
you know, if the Cochrane Centre hadn't been established in Summertown,
then we wouldn't have a Cochrane collaboration, we wouldn't have Cochrane reviews, it needed that thing.
We might have had something else, but we wouldn't have had that thing.
We wouldn't have matter analysis if someone hadn't coined the phrase matter analysis,
but we probably would have and we know we would have combined analysis of studies because other people
around the same time were writing about how to statistically bring together the results of studies.
They weren't calling it matter. Analysis. History is alive and we're part of it.
I'm looking after the systematic reviews module this week simultaneously.
Some of you are on the matter analysis module. I'm the fortunate position.
Eventually I'm going to figure out when did we actually start that systematic reviews module?
Two weeks from now I'll be running the randomised trial module.
I'm pretty certain we started that in 1998, 20 years ago, who was still at school 20 years ago in the audience.
Anybody is teaching or maybe just you're looking well for your age.
Okay, but just think, you know, some of what I'm going to show you and now those who do teach and we teach undergraduates and you think,
I'm telling you what I was doing 25, 30 years ago, you weren't even born when we were doing this.
How does history film history for some of us is going to feel as though we say, you know, the past.
Now, again, one of the things is we think about the evidence based health care courses here in Oxford.
They began last century. But when they began, let's say.
She was nine years earlier. So we things are moving and I'm going to reflect on it.
As a reviewer, as a researcher, as someone who occasionally gets passed as a historian, so we can think, well, how far back to this whole stuff go?
One of the beauties of the modern age is the Internet.
And one of the things that the BMJ did was they digitised all of the BMJ and Lancet have it, JAMA have it, most of the journals now around it.
They've scanned their stuff and they put it online. So you can go to the BMJ website and you can search for the phrase systematic review.
And this is the earliest 1867. It's in a book review.
They're not using the phrase the way we might use it now for a scientific project called a systematic review.
But they are using it in a way that is part of why we do systematic reviews.
Daunted by the difficulty of any systematic review of these collection of monographs, the monographs are to do with hospital statistics.
In London, there's lots of those monographs. So they're saying, you know, it's too difficult to systematic review them.
So we'll take a flying one through the pages warning our readers that they will
do well to indemnify themselves by procuring the volumes for systematic perusal.
They're saying we ideally should do a systematic review.
That'll be a lot of hard work. That'll take a long time.
So we're going to skim it and you can reflect that now 150 years on and people are skimming the research that's been done and saying,
I'll quickly summarise a couple of studies for you and you should be thinking, well,
hang on a minute, that's a good guide to whether or not I should work harder on those studies to understand them for myself,
that I should indemnify my myself by gathering those studies for myself and systematically perusing them.
So it's a really nice example of the phrases that it's highlighting what we would now be doing, and it's warning against doing things too quickly.
So key source is again the slides a bit. Yeah, we'll figure out a way to make the slides available, but there are other things.
So Ian with Laurie Hedges and Alice Cooper wrote a Brief History in 2002.
Steph Lewis led some work now show on forest plots. 2001 Max Dyer, the originator of Update Software, the first publisher of the Cochrane Library,
and we wrote something with Mark in 2009, and Dan Fox wrote about the impact of reviews on health policy.
So these are some of the sources that I will be using and the James Lane Library.
You are well at the top, so I'm going to take you through it.
So this slide could be a slide that we would use on the course, on the Metro analysis course to say, why are you here?
Why are you learning about evidence synthesis? Why are you learning about systematic review and meta analysis?
Because that I'm going to try and illustrate these points about how have they crack team historically.
So is it about organising evidence, comparing and contrasting similar studies, doing the mathematics, minimising bias, improving access,
appraising quality, building a better study in the future, identifying interventions that we should be using in the health system.
Identifying things that I should buy if I go boots the chemist.
Those of you who are practitioners that you should prescribe surgical procedures you should do if you are a surgeon.
Procedures I would like dawned on me if I needed surgery. So these are this is why we do evidence synthesis today.
So I'm going to try and illustrate where, you know, where did some of these ideas come from?
So I begin with a very long quote, and I'll tell you where it's from at the end.
And it's a sort of you know, it's it's a good rationale for why we do reviews.
If, as is sometimes supposed, science consisted in nothing but a laborious accumulation of facts,
it would soon come to a standstill, crushed, as it were, under its own weight.
So it's a recognition. We've got to recognise that science is moving along and if all we did as scientists was accumulate facts, we'd just be crushed.
The suggestion of a new idea or the detection of a law supersedes much that has
previously been a burden on the memory and by introducing order and coherence,
facilitates the retention and the remainder of in an available form.
And again, that's what we do is we learn things, we assimilate the knowledge and we say,
okay, on the basis of what we know and what we've just found out, here's how we go forward.
Two processes I'll sit work side by side, the reception of new material and the digestion and assimilation of the old.
One way mark, however, should be made.
The work which deserves, but I'm afraid does not always receive the most credit is that in which discovery and explanation go hand in hand,
in which not only a new facts presented, but their relation to old ones is pointed out.
We could almost use that as a rallying call for systematic abuse.
Lord Wiley, 1885. The. Swedish Association for the Advancement of Science.
And I used to think this is. This feels like history.
Now. It's a 19th century. It feels old.
And it's why we do reviews. So I can bring you more up to date, because there's an alternative approach to being not being crushed by facts.
And that is simply saying when a new one comes along.
I throw away an old one. I don't assimilate. I just say I can only I can only cope with ten things at a time.
So when number 11 comes along, number one has to disappear. We don't like that approach, but that is an approach of Homer Simpson.
This is I keep meaning to find the YouTube clip. If you watched The Simpsons, Homer Simpson, this at one point says this.
I know this as a fact because my sons bought me a calendar of The Simpsons sayings, and each day you get a different saying.
And on a day in November 2023, this is what the calendar.
Every time I learn something new, it pushes some old stuff out of my brain.
So Lord Wally in 1885 is saying, we have to assimilate.
We have to synthesise the evidence. We have to do systematic reviews, not using the terminology.
But that's what he's talking about. Homer Simpson, beginning of this century, says, you know, just learn new stuff, throw some old stuff away.
Clearly, that's not what we want to do. But we've got to think about those two approaches.
Is it about organising the evidence? So Homer doesn't want to organise the evidence.
He's got a shelf and he's is going to push one thing off as the next thing comes on board.
Riley is saying, No, no, no, assimilate. So is it about organising?
James Lind, people familiar potentially with this, the treatise on scurvy.
This is that the oranges and lemons for people on ships to try and reduce or prevent scurvy.
And so you several people are aware of that as a thing.
Is it the first control trial? No, it's not the first control trial, but it is an early control trial.
Not many people are aware that he also tested things like sulphuric acid to see whether or not that could keep,
you know, here have some sulphuric acid that might stop you. Getting scurvy didn't work.
Not not quite sure how sure it is. It didn't work, but I don't think it would be something we risk to recommend.
But what's important about this book from the 1750s is not just the trial, the study that was done on the ship,
but it is also together with a critical and chronological view of what being published on the subject 1750s,
trying to bring together and catalogue what has been done on the subject and part of these examples, he said.
It's not new. It has changed the last 20 years on reviews, dramatically different, but it's not new.
Leonard James Leonard is doing that in 1753.
Why is he doing it? And again, look at this and think, oh, you know, that's what we've been learning about on matter analysis and systematic reviews.
We're learning about prejudices and biases.
We're learning that stuff might be rubbish. James Lane 1753, is saying, Before I tell you about my study,
I'll tell you about what's already happened because I need to do that, because I need to set my study in context.
But also there's rubbish out there and I need to tell you what's rubbish and what you might believe and what might help you.
So, you know, think about reviews you've read. Think about those who you work on reviews.
Think about reviews you've done. Part of it is about cataloguing, placing things in order, finding out mistakes, managing prejudice.
We now talk about the risk of bias.
Well, James Leonard is talking about rooting out prejudice, same thing, same ideas, issues around quality assessment.
Then there's other curiosity about this book, because if you look at one of the publishers, one of the publishers is Colin.
And so you think, oh, wow, what? What then is this Nostradamus?
Is this Harry sort of saying, look, I'm going to do this? And actually, one of my publishers is called Cochrane.
And many, many, you know, two centuries from now, there'll be this thing called Cochrane doing, they say these reviews now a history of Cochrane.
There is you know, it's it's a bizarre coincidence that one of the publishers of James Flynn's treatise on Scurvy is Cochrane.
And so let's jump to 1979 and Archie Cochrane, the man who the Cochrane Collaboration is named after.
Archie, I never met Archie Cochrane. Archie Cochrane, who died before the collaboration was formed.
But in 79, it is surely a great criticism of our present profession that we have not organised a critical summary
by speciality or subspecialty adapted periodically of all relevant randomised controlled trials.
And that's a profession book I've come to be.
But of saying, look, you know, 200 years earlier James Lind organised the literature but.
200 years later, Archie Cochrane is talking to the medical profession, primarily talking to obstetrics and gynaecology, saying this is terrible.
We've not organised the stuff. We're doing these trials, we're not organising them, we're not learning from them, we're just doing them.
And as researchers, sometimes we just get enthused by doing our own study and we don't think hard enough about what we're supposed to be doing.
So then we have to say, well, some of the signs were criticised. It's not scientific research.
All you're doing is review and well, that's just like going to a restaurant and telling us how good the food was.
Yeah, that's not science. Well, it is science and it's not new that people have recognised it as science.
Feldman, 71, may be considered a type of research in its own right,
one using a characteristic set of research techniques and methods and getting those of you on the courses.
That's where here on the course as you're learning these methods. It's not.
Go to a restaurant and tell us what you think about the food.
It could theoretically be go to a restaurant, systematically appraise the food in a highly structured way,
write your review of the food and make a recommendation about whether or not other people
should go there in an unbiased way as possible so we can systematically review restaurants.
We're not going to necessarily see it as science, but the process of the system.
It's science. Lyons Smith, 71, noting that it's impossible to address some hypotheses other than by comparing and contrasting the studies.
We still have to show this occasionally because here in Oxford people can do de filles systematic reviews,
but go to some other universities in some other countries and say what a systematic review is it as a PhD?
No, no, that's not that's not research. We still have these discussions.
I still share these quotes with colleagues in other countries to say, well, look,
here's some quotes for you and here's some Oxford DE that are systematic reviews.
And I think you may have heard of the University of Oxford as a place that has been doing research for a while.
And if they think it's worthy of a dphil,
maybe you should be thinking about as a Ph.D. and often defend students in other places who want to do the such research.
And the supervisors are saying, Oh, it's not research. Well, it is research.
It's well-accepted research. Eugene Garfield,
who is responsible for the concept around impact factors and science signs Citation Index suggested there should even be a prize for for this process.
It's 1977. This is not the year to know in the 2000s when we're becoming much more familiar with reviews.
But these are sporadic examples at time of people talking about them.
So let's get some early examples. The place, the source for this is the work that Ian Harris Cooper and Larry Hedges published in 2002.
So they identified so Ian and Harrison.
Am I saying it's not until the early 20th century that the science of research as we know it began to emerge?
So that's a scientific bit. So you might say that James Lind was about cataloguing, looking for bias and so on the scientific bit.
Maybe that's to do with the mathematics and the statistics and the example that they they cite for that.
Carl Pitts A 1904 article in the BMJ pulls the data from five studies of immunity,
six studies of mortality to look at the effects of vaccines, pooling the studies saying none of the studies on their own would be enough.
I am going to find the studies and I am going to pull them. And this journey that we're on is to say there's no it's not new.
The mathematics of pooling studies to boost the power to think about a word that we're all now so familiar with is as we
view as if point to a work heterogeneous heterogeneity corpus and not you know four is talking about that as an issue when
the pooling the studies actually ESP extrasensory perception a book from the 1940s and it's got a systematic view of the of
the research done to see whether or not extrasensory perception is a valid systematically reviewing something into a book.
The comparison of the statistic of more than one experiment suggests the count about the combination of them for an estimate of total significance.
That's the analysis. That's the statistical pooling of results.
It's not new, but then we hit terminology, so then someone has to invent the term.
Jean Glass, 1976 invents the term match analysis.
That's where the term comes from, because it's not a systematic view, is a sort of well, we know linguistically what it means.
It's a review that is systematic meta analysis, you know, so it made up that it made the term a by April 76,
American Educational Research Association presidential address talks about describes the need for better sense.
And introduces the term meta analysis, and the lecture is then published in educational research.
And it's down there and it just I'm not going to read all the quotes out.
You can read them faster than I can. But he saw saying, you know, there's metal mathematics is metal psychology, there's metal evaluation.
I've never heard of those. I think those terms have been and gone.
Metal analysis has stuck. Metal analysis is the analysis of analysis.
Why? Unless we think of the second paragraph, he's working in education, he's working in behavioural things and they're saying,
look, we in five years time research has produced hundreds of studies on IQ.
Again, we said, look, typical with you now. Oh no, I've got 20 studies, 30 studies, four studies.
How am I going to cope? Jane Glass is saying there's going to be hundreds of studies.
And those studies in isolation, that's not going to be reliable.
But actually, what they're really saying is pooling the results will give us something reliable meta analysis,
the integration of research through the issue, as is citing Jane Glass.
So so Mary Smith she's Jane Glass is Mary Smith's husband at 375 studies 833 effect size measures.
Just feel that those of you who are reviewers, you feel the pain of extracting data from 20 or 30 studies, feel 375 studies,
833 results being pooled and then feel the value of that because this is trying to drive to an average whether or not you believe the average.
That's not our job. A systematic review is our job.
A systematic review is to present the evidence and people might say that's an average result.
That's not a good guide to the individual. That that's a valid argument.
That's fine. Fair enough. But maybe you're going to have to use the average because maybe the results for an individual are not a reliable guide.
So again, there there a quote, I'm not going to go through it again, but justifying it.
But the sense being that this is not new, what we now feel and some of the challenges as we do reviews,
the concept has been around for a while, but it didn't explode.
It didn't take off when Jane Glass introduced the term.
It took a little while.
And that's one of that where I think when when the real deep history stuff is done on reviews, people are going to start exploring why?
Why did it take off? We're going to see a graph that shows that we don't know terminology here.
It's cropping up. This is in the BMJ 1982.
The term is cropping up.
It's crossing from psychology, waging class psychology in education, where they're working in the mid seventies, crossing over at 1982.
So this is the first time I've been able to find the phrase systematic review,
meaning what we would now use it to mean in the BMJ go back 800 1867 example this is looking at hypertension,
comparing non-drug treatments and nice little quotes again in I don't know if we still see some of this writing now in the BMJ,
the reviewing the published work ceases to require the judgement of Solomon and becomes a quasi experiment.
We use the meta analysis. So it's half saying this is a more objective way of doing things.
Instead of Solomon or some Supreme being deciding, we're going to let the numbers decide, we're going to get the process decide,
we're going to make sense of what research has already been done, citing, quoting Gene Glass.
So, again, think about the terminology. So I started work here in Oxford.
Richard was, which is still with us and but in 69,
Richard Dole publishes in the BMJ highlighting that it's not what we would now regard as our typical meta analysis review,
but it is what we increasingly are seeing as a review to bring together the evidence in a systematic way.
And Richard Dole is talking about bringing together evidence on adverse effects systematically.
Again, you can see what I'm trying to illustrate is this concept comes we're coming at it from many different angles,
jump back a little bit again on terminology, but also to illustrate how the world maybe needs to reflect on the history and move itself forward.
People were saying things 40,
50 years ago that only maybe people side paying attention to a real a serious attention to over the last five or ten years.
Shaik 1970 628 reports, 29 studies.
And look at the time of that where in my story and look I think we'll overlook they've gone back
to 1922 not studies from 1922 to 1970 because the procedure has been around for a while now.
How many reviews do we see? And they've only gone back, you know, 40 or 50 years because of convenience of searching the electronic literature.
They went back to 1922 that published in Paediatrics.
The purpose of the study is to review the English language issue, and we might now look at that as a bit of bias.
They're just looking at English. Yeah. Shaykh You should have looked at some of the language as well.
Particular emphasis now on the assessment of the scientific merit of the studies.
So now not necessarily on the meta analysis, but are the studies good enough?
And there is some sort of risk of bias quality.
Critical appraisal tool. And you can see again, the point is not working.
The top case in 1922 down to Roadhouse 1970 with a measure of how good was the study design,
how good is the sample, how good is the description of the illness, critical appraisal, risk of bias going on?
And one of the fascinating things that they found when they did that is that the quality and the conclusions weren't necessarily related.
What was well related was, are you the sort of person that does the operation or the sort of person that does not do the operation?
And if you're the sort of person who does the operation, you conclude in your study that the operation is effective.
So the surgeons tended to conclude it worked. The public health people tended to conclude it doesn't work, didn't work.
And again, right now, now suddenly we think, well, that's what we're confronted with now to some extent is is the bias of the researcher.
So 89, this is the, quote, effective care in pregnancy and childbirth.
Two volume work at big hefty books full of systematic reviews.
And there's the foreword from Archie Cochrane where he's moving on and he sort of welcoming this.
Now the systematic review of the randomised trials, which is in this book is a new achievement,
it's a real milestone and that book contains chapter after chapter and they're tiny chapters,
68 individual reviews, nowhere near the many, many pages that we now feel we have to write as a systematic review.
One or two pages, quick, better methods. Quick bit of what studies did we find?
Quick analysis answer for bringing together tons and tons of material to help people working in maternity care.
So we then move to another element of the history.
And again, these are important little things because certain things clicked and certain things have stayed with us.
Forest plots that that picture that you'll be familiar with, if you look at reviews, the meta analysis plot, 78,
we see this thing over on the right here and it's not big because if you make it bigger, it's a it's a screengrab from JAMA.
If you make it bigger, it just pixelated. You can't see anything. You can see it better when it's smaller.
It's not a traditional forest plot because it's not pooling the results, but it's just showing the result of each study and this confidence interval.
And that because they were looking at negative trials, so-called negative trials,
partly to illustrate that some trials are called negative just because they're not statistically significant.
Louis and Alice produced something similar with a meta analysis in 82, and this was their thing.
And this picture is from an article I was pleased to write with Stef Lewis, who is the daughter of the Lewis, who is the first author of of that.
We just we just reproduced it there. And that's what it looked like.
So it doesn't look like a traditional forest plot, but it did.
It's one of the first examples of where down the bottom is the pooled result that all beta blockers pooled.
That's the average of all of the studies.
So it's a pictorial way of showing it up to then you might have a table up to then you might simply have the average result.
This is showing the contribution of each of the studies in 83 and this we haven't nailed down.
Yeah. At the time Steph and I wrote this and we wrote to various people who may have been involved early on.
Stephen Evans, who statisticians some of you may know, Stephen said, Well, actually I think I was the one who suggested that we replace it.
We replaced that Lewis and plot with something that would have a mark for the central point and then a line going through it.
The Lewis and Ellis plot is a rectangle with a mark for their point estimate.
The traditional forest plot now has that square and its size is proportional to the amount of evidence on a horizontal confidence interval.
So Stephen Evans suggests that this would be what the Lewis analysis plot looks like as a regular forest plot.
And so that sort of stuff did is our figure two in the BMJ article was to say, well this is what it would look like now and it looks different.
1998 is probably the first time that what we would now fully recognise as a forest plot hits the literature.
So Steph drew that retrospectively for my father's work, father hadn't drawn a forest plot that looked like that.
But in 1998, the antiplatelet trial is collaboration.
So looking at antiplatelet therapies, drugs, one and that same, and that potentially is the first forest plot in the literature.
And that is not very different to what we would still see now.
30 years on, things have changed slightly, but still the basic principles of the square being proportional,
the horizontal line, the line of no effect going up the middle, sorry, line of no difference going up the diamonds.
To illustrate the average. The dotted line to illustrate.
The average point all the way up. Been there a while.
So are there other examples? And again, I'm just going to illustrate a couple of examples.
Again, just to say that when people say, oh, it's all it's all a feature of the last couple of decades, well,
Stern's fault in 74 did an extremely important review of radiation therapy for breast cancer for women with breast cancer,
concluding that actually it might not work, it might be dangerous, it may not improve survival.
It may have an effect on work, on the cancer coming back around the breast.
But that's not why you're using this therapy. You're using this therapy to try and keep the woman alive.
And maybe this therapy is actually lethal in the long term.
So Stern folded that in 74 subsequent reviews of radiation confirmed it.
But now radiation therapy is better. And those harmful effects of irradiating the heart, those are being minimised.
So now it has a survival advantage. But that's not 1970 475.
Tom Chalmers not related to it at pools ascorbic acid for the common cold concluding there's not a lot going on shake which I've already
illustrated not just looking at the quality of the evidence pools it and makes it makes a statement about cost effectiveness in view of the costs,
financial and human as well as the lack of evidence to support it.
Either get on and do a proper trial or maybe stop it.
And again, how common are those sorts of things?
Now, we need to feel this because we need to feel why are people why are people doing these things now?
Why were they doing them in the past? In International Anticoagulant Review 97, we do the sorts of reviews I'm proud to work on.
Still in Oxford, individual participant data. They gathered the data by building a collaboration.
This type of review is still very rare where people actually share the patient level data, but it's not new.
In 1970, it was pooled, the evidence was pooled.
People joined together in a collaboration to do it early.
This is the review I'm proud to still work on here with the CTS. Do this now up the hill at the Old Road campus in 84 before I joined,
when I was still finishing my my degree that they got together at Heathrow, they formed the Early Breast Cancer Trials Collaborative Group.
That group met in Oxford in the side business school in June of this year.
Is is still rolling every few years. It's still rolling.
Update on the data. New trials are coming along, the idea being that this is a disease that is very common.
The treatments for it might not be great, but if we could pool the evidence, we could keep women alive.
They could survive their breast cancer. And it led to this, which is to some extent, we've always viewed it as an icon, an iconic figure, 1998,
this forest plot in The Lancet, all of the trials of a drug called Tamoxifen versus no Tamoxifen on the left is recurrence.
Does it stop the disease coming back on the right is death does it stop the disease from killing the woman?
Definitive meta analysis published in The Lancet in 1998.
Still remember it very well. Partly because it well, actually because I tell you, my senior colleagues, some of whom you would know well,
we colleagues, Richard Peto here in Oxford, they were the leads on the meteor.
I was one of the juniors. I was responsible for going up to Radio Oxford in Summertown and talking to
in the radio interviews and OP at 6 a.m. in the morning to do the first one.
And what was the first headline on that day? Frank Sinatra has just died.
Who knows who? Frank Sinatra is in the room, hands out.
Because sometimes I say and people who who write very famous person and very sadly the 1:00 BBC News on the
television Richard Peto was supposed to be the live first interview on rang out and said sorry Richard,
we're not dealing with you we've got someone from the Frank Sinatra Appreciation Society.
They're the lead off news on the BBC one news at 1:00 on that Friday.
Well, remember it how it is is a big story and it's a big story until Frank Sinatra dies and then we go on.
But it's the sort of thing you feel sad when you standing up in Radio Oxford at 6:00 in the morning and they don't want to talk to you anymore.
But then you think, well, actually, our legacy, you know, whether we will outlive Frank Sinatra's legacy, we don't know.
But certainly on the day on that Friday in May 1998, Frank Peters collaboration working together.
And again that's so the early breast cancer try this group people share their data.
For a five or 600 trial groups around the world share the data.
Again, it's unusual, but collaboration exists.
The CPC, as it's called, the two volume work that Cochrane introduced, that was a bunch of people collaborating.
Dan Fox in 2011, who's very interested in that policymaking side of things.
History credits it with having the great MOU, being building something, getting something moving.
So we have to think about collaboration. Which brings us to Cochrane finally is the Cochrane collaboration to me.
Really? That's the logo. The logo is of steroids showing the evidence on steroids to keep babies alive.
And, you know, part of the rationale is that until the evidence was pooled, people didn't know how effective these things were once it was pooled.
But it still took a while for people to take notice and start acting on it.
But there will be human beings alive today because that showed that the moms should have steroids.
So reviews are keeping people alive, but we've got to think about why we do them.
So. COCHRANE There's a picture of Archie Cochrane 97. Is he criticising the profession?
And this is really interesting for us to look back on because he says you should bring the results together.
Ten years on and it's got nothing to do with Archie Cochrane. We get electronic publishing and electronic publishing was not the Internet.
Again,
those of you who are the younger ones amongst amongst us electronic publishing was you put it on a desk and the desk survives through your letterbox.
That's how pregnancy and childbirth were. So that's how Cochrane Library was originally circa.
It's on a CD and the CD arrives in your letterbox and you put the CD in your computer.
Now, how many of you have got a CD player that you can connect to a computer?
And how many have got a floppy disk drive? The PC was on the hard, the three and a quarter inch discs.
That's how it came. The virtue being that next time we issue the disk, we kind of update some of its content.
Because when your BMJ applies to your letterbox, there's not a big articulated lorry outside with all of the past issues of the
BMJ scribbled in to update them because new evidence has become available.
But electronic publishing realised that and then a decade on the internet and that transforms it an early internet.
It just means that we can send you the stuff more easily we can.
It's a good dissemination tool. But then the Internet and this is again,
the other thing that needs to be looked at in history has transformed how we do reviews from people
like me going from the Ratcliffe Infirmary to the Radcliffe Science Library to climb a ladder,
to take a volume, to bring it to the photocopier fill in a photocopier form.
See, people smile because they remember filling in those pink forms. The Ratcliffe Science Library, photocopy it, bringing it back to now you sitting.
Click, click, click. Oh, got the PDF job done. Go and have a cup of coffee.
You know, remember us walking across to the Radcliffe Science Library members filling in the forms for an interlibrary loan,
and six weeks later, we might get an article with a big instruction on the back.
Do not photocopy this article. You will breach copyright and just get a copy.
Keep it spare. I got a spare because I'm going to scribble all over it. So.
So that made a massive difference. Cochrane itself.
Cochrane sent hope is likely to update software publish the practice in childbirth
collection 95 the Cochrane Database first CSR comes out 98 Cochrane Library on the Internet.
2013 was the 20th anniversary. We're now at the 25th anniversary.
How has it grown? Well, our first Cochrane Library in 95 got 50 reviews on it.
You can say there was very steady, steady. Now it's almost steady state, you know, 3007, 4009, 5000, 6000, 8000.
It's about 8000 now. But is that it?
Is that all the in systematic reviews now I hope this is an article by Hilda Bastian 2010 and it's an out it's got a title about,
you know, some 70 trials a day, 12 reviews a day.
That was the output. Then this is that graph showing the top one.
There are various different ways you can search for systematic reviews, but the top one is estimating that by from 1990, not a lot, not a lot.
And then we will put maybe 6000 down the bottom one.
The very bottom line, that's Cochrane. So early on, yeah, it's a fair proportion until about 2000 and then the world takes off.
That was Hilda's estimate in 2010. Earlier this year that graph was was in an article.
This is across all disciplines. Their estimate is that we've passed the 200,000 mark now on systematic reviews across all disciplines.
From the very early example onto this graph, they have a very nice timeline for this person,
but also on what was a long few of those early examples aren't even making the graph and then.
It just goes away. And that's what we're going to have to think about what is going on here.
This is a this is a total shift. This is not a gradual growth.
This is a wow. It's taking off. We have a register of systematic abuse called Prospero.
It now has more than 40,000 things in it. It began only five or six years ago with a mindset that we might have one or 2000 a year.
It's now running at 12 or 13 or 14,000 a year being registered as new reviews, and that's only a sample of them.
So something is happening and I can't tell you what some.
But the reality now, one of the really fascinating points from that graph like that is when Cochrane began in 1980,
1994, one of the first things that needed to be done was to find the trials.
At that time in Medline you could find 20,000 randomised trials easily.
We need to sort this out. There's a lot of trials. We need reviews.
Now in Medline you can find about 120,000 systematic reviews.
So what's our next step going to be? Because we needed to sort it out because there were thousands, tens of thousands of trials.
We've now got more reviews than there were trials we could find easily when a lot of this activity began.
And we just don't know where that graph is growth is heading for. And we've got big issues about how high should that graph go.
People just doing reviews because it's easier to do a review than it is to do a new prospective study.
Typically, people just doing reviews to boost their CV is that's what's going on.
Are these reviews legitimately needed? How many more reviews do we need of pet imaging for lung cancer, which is some of these challenges?
And where are these challenges?
Because you have agencies across Europe saying, why do we need to do another review to fit our guideline when there's already a review out there?
Can we please get together and have things more organised?
So I'm going to finish with a quote. Back in 9676 from the Journal that Jean Glass coined the phrase matter analysis,
meta analysis were without doubt be abused as well as use with positive benefit.
And I think is an important point there for us to think about. This is not all happy stuff.
Reviews are used and now we can see bias in reviews. We can see the prejudice that James Lind was trying to root out.
We can see that in some reviews. We can see people doing reviews,
doing things that are not systematic reviews and labelling them as systematic reviews because the label may now mean something.
But then are we going to have a problem in a few years from now where people say, reviews show that, yeah, I don't know what they're doing.
There's just too many of them. Yeah. What I'd rather have is go back to the days of your of a very experienced practitioner.
You've been treating this for a long time. What do you think I should do?
Because I'm overwhelmed. I maybe I can talk to my GP and I can read 15 reviews of the same thing, even though I am an experienced reviewer.
What are we expecting people to do as these reviews proliferate?
So just to actually conclude and this is wearing so sometimes this talk is in the one of the modules on the evidence based health care,
which is the history and philosophy module that Jeremy how it looked after and is to make make them think if you were going to study this history,
how would you do it and how would you do it systematically? So yeah, yeah.
This is a non systematic history of systematic reviews because I haven't searched everywhere where they might be.
I've searched convenience BMJ fantastic that they digitised.
So I can look at I can look back at them, but I bet there'll be examples in other places that use the terms.
But what I'd love to see at some point is someone say, why did they explode the way they did?
And that's probably going to be the anthropologists or sociologists who are going to look at it and say, Why have they taken off?
So thank you.