And thank you so much.
I'm delighted to be here.
The opportunity to address, to talk of participants and people online
to talk about really some of my bugbears when I read papers
about systematic reviews and meta analysis.
of course, these things have become critical
for decision making and chiefly in health through nice and simple
organizations around the world, but across various disciplines.
and so they're really one of the most influential types of research,
which means that articles about systematic reviews
and that analysis, are really highly cited.
I should just permit permeate so often appear at the top of pyramid.
This is one of my favorite pyramids.
so the high five just met.
You just mentioned that I've got a lot of citations, and I
quite often get
well, so sometimes get emails like this.
Congratulations. You're very well cited.
This and I, it's nice, but I don't take it very seriously.
Although I did picture on this spot from silent
that, is a really interesting one because it allows you to rank,
174 million publications
by number of citations across everything and all time.
And if you look at that and the most cited papers of all time
back from 1951, it's got 229,000 about proteins.
But if you look through the list, you find various papers about systematic
reviews and meta Nazis, and these are the ones
that rank high in that list.
these are the number of citations,
and obesity,
I mean, quite a lot in the top 120,
for a relatively young field.
So I thought, I want to look through this list.
I think, oh, gosh, people keep citing that in ways that I don't like.
So I thought I would try to talk around the idea,
what I'm going to do if these are the papers, slightly reordered,
and there are two paths which are essentially about the same thing.
So I'm going to go through these five next time.
and talk about why I think
they are very often misused or misunderstood.
Try and offer some solutions.
And, and
that's it then. Drawing to a close.
so one of the topics first is misuse of reporting guidelines.
The second is misuse of this I squared statistic.
And the third is misuse of random effects meta analysis.
The fourth is misuse of tests for funnel plot asymmetry.
And the fifth is misuse of risk of bias.
Assessment tools.
Before I start,
I'm sure I know you all know this, but just,
for completeness, I just want to have one slide
on each of what is a systematic treatment.
What is a that analysis, at least in my head.
systematic reviews, I mean, are often thought of as reviews
or the method section.
when they first came about, it's my impression
was that they really thought of it as studies of studies and meta analysis.
combined results of studies increasingly,
that the term systematic review and the idea is applied
to what I would consider reviews of literature rather than user studies.
Now that a mixture of the two and some pieces work are actually
a mixture of review, discussion, reviews of a papers,
we can talk about that more if you don't know quite what I mean by that.
But whichever approach one takes,
I think they're characterized by following a protocol
I'm using methods that are systematic, transparent, and rigorous,
really trying to minimize biases.
I systematic errors and random errors by bringing together,
all of the information that is under the tackled.
The question that we're trying to answer
meta analysis.
As I already mentioned,
the statistical combination of results from multiple studies.
This is a probably my favorite definition of a meta analysis for my coffee.
What I like about it,
I like a lot of them, is that it explicitly acknowledges the judgment
that studies should be combined, which is a non statistical issue.
And that's a really important part of the meta.
It's such a pleasure to
be in Oxford because it really is the home of
so many pivotal ideas in this area too.
you know, which Peter and his colleagues and in Thomas and colleagues
copycat versions of it's a,
very special place in this area.
So let me get started.
Paper number one, the Prisma statement,
originally published in 2009.
updated 2020.
What's it all about?
I'm sure you will know.
It's a checklist and a flowchart
for reporting, systematic review and all of that analysis.
I was lucky enough to be at the Prisma meeting
for this one back in June 2005.
so it's something that's quite close to my heart,
what it looks like.
It's a checklist.
A number of things it suggests you should do.
And there's a checkbox to say whether you've done them,
where you've done them in the report.
And the flowchart is a
suggested way of illustrating the numbers.
of all the articles or studies that you've started with
in your research and how they were whittled down
to the number of included studies on the completed reports.
What's the problem there?
Well, how can I introduce these who have decided to do this?
I thought, well, let's pick a pseudo random paper and see
whether they illustrate the problem that I think is right.
So I just looked at the most recent citation of Prisma,
in scope, because if you do that, grab them by date.
And this one was not in English, so I didn't look at that one.
the next one was in English.
It's around birds, urinary continents, in pregnancy.
And if you can jump to the beginning of the method section, you read this text.
The review is conduct it based on a guidelines from Prisma.
Can you
do a review according to Prisma will look at Prisma.
let's just look at one item in more detail.
Specify the methods used to decide whether study
might actually complete two criteria or how many reviews, and so on.
Well, okay.
One reviewer looked at the keywords and drew a decision based on that.
Now, clearly that's really ridiculous thing to do.
but you get full marks in the Prisma
and by telling people that's what you get,
it's really clear that this should not be used.
It is not intended as a guideline for doing systematic reviews.
Despite almost 3.5 to you saying that's
what you did.
I have to say, the prisma elaboration has got
lots of great, ideas and, and,
information about how good reviews are done,
but I think it's, it's still stopped short of being a guide for doing reviews.
And the Prisma paper is very clear about what that is.
What's the solution?
Why? This would be easy.
Don't use Prisma.
Use one of the many documents, books, papers,
standards that actually guide you in doing
a systematic review.
Okay.
Number two,
I squared the
what is I squared.
Well, it's a it's a simple statistic
perhaps why it's an unpopular it.
It measures inconsistency.
That means the extent to which the results of the studies are consistent.
Now when I say the results, I mean the estimates
and the confidence interval that the results,
and you can think of it as an, a measure of the extent to which they line up
for the fact that they don't line up because of the converse consistency.
I should be thought of as inconsistency.
It so happens that it can also be interpreted in a different way.
Perhaps statisticians. Right.
As the proportion of variability in in the point estimates
that is due to heterogeneity rather than chance.
What's heterogeneity heterogeneity variation
in the true effects across the studies.
I've got some math here.
That's not it's not important.
This is a representation.
The top one is a representation of the the second understanding where torsemide
represents the amount of heterogeneity, which I'll talk about more as we go along.
And the bottom is, you can break out the total variation
into components due to heterogeneity, components
due to sampling error.
And I'll come back to that equation.
but I do want to talk about what a bit more about
what it means conceptually and what what we had in mind when we developed this,
what we were trying to do, we think we achieved.
Here's a match analysis for the number of studies
we the trials of amantadine for flu,
effects measured using odds ratio.
The idea of r squared is that we look at this
and it's a
measure of the extent to which these agree
with each other or disagree with each other.
And the idea is that what we mean?
There's not a lot of disagreement you can look at has
and I mean maybe these are slightly different, but
yeah, they essentially are consistent with each other.
And the idea is that I can compared that intuition
with your intuition from a different set of studies.
But I'm only I can only I've get everything I need for this picture.
And here these two really don't agree with each other.
So there's more inconsistency here,
even though these are in a completely different topic
and measured on a completely different measurement scale.
So I squared intuitively has got nothing to do with the measurement scale.
Yeah.
What's the problem in practice.
Well,
I couldn't access the first paper in so have to go to the second one.
Might be interesting.
Might be having just fine.
there we go.
It's a meta analysis in,
looking at the concern at treating this practice,
you go to the method section.
Heterogeneity across studies
was assessed using the I squared statistic.
And these values represent
low moderate and high heterogeneity.
Now to a statistician heterogeneity means
the variability in effects of studies.
And it means the variability in standardized
mean differences, all the variability in log odds ratios it.
And that is simply not measured by I square.
And then I use they say I squared a 55%.
so there was moderate heterogeneity.
this one's okay I squared zero.
That does reflect no.
But the problem with the first one,
the point here is that I squared does not measure the amount of heterogeneity
in the way that most people do
or should think about what that means,
either clinically or, statistically.
So if you go back to this formula, I'm just writing in words.
Now, what it measures is the amount of heterogeneity, which is variability
in percentage or in local expression resulting,
prevalence, whatever we're looking at,
over the sum of the Metropolis sampling error.
The sampling error is essentially a measure of how big the studies are
typically.
So a very big study gives a very small sampling error.
So one over the study size.
Now we can see this doesn't
measure heterogeneity because it's fairly common to big studies.
So typically this is a big number.
This is a very small number.
If you got lots of big studies this will tend towards one.
So we'll get a nice square towards 100 towards one.
But being 100% inconsistent.
Whereas if
if the studies are generally very small and this is a bigger number,
and as they get smaller and smaller, this goes to zero
for the same amount catchment.
So I hope it's very clear that R-squared doesn't directly
measure heterogeneity because it depends critically on
how big our studies are.
It's a relative measure, not an absolute measure.
a lot of people seem to understand this.
And it can lead to
synthesis interpretations.
Is there a solution?
Well,
I think so.
We've got a there is a statistic that we use in meta analysis
quite a lot that it perfectly measures the heterogeneity.
It measures the variability in effect sizes
across studies, something we generally call tool.
How you use the influence already.
And it's the standard deviation of effects across studies
through effects
not not not across the point estimates.
So it's actually quite difficult to estimate because it's conceptual.
I can't rely on something to calculate very directly recalculated
from data yesterday from data.
how often can be a bit difficult.
And a deviation in log odds ratios is a really challenging thing,
but we can express it right.
And I'll return to that later.
Okay.
Paper number three.
Damn Thessalonian into that paper.
Great paper.
landmark paper that introduced the idea of random effects.
Meta analysis.
What's a random effects meta analysis?
I just quickly review it.
let us look back at those studies courses with these.
With the second set of studies I showed you, Isaac, right there.
In fact, that is looking at, exercise as a gene for depression.
the idea of a random
effects match analysis is that
each study is estimating
its own unique true effect.
We can depict that with some dotted dashed lines here.
So we don't know where these are.
The idea is that there's one of these lines for each of the studies.
So say we don't know where they are,
but we assume that the
that they are distributed according to a specific distribution,
typically a normal distribution, so that most that have effects
in the middle of this and rarely though have studies outline
to the left or the right.
And then we can estimate where the middle of that distribution is.
And that's the timing that is portrayed,
is typically the result of a random text that matches.
So what's the problem with this?
Well, there are lots of problems with that analysis.
I'm just not too out.
So these are not what I'm talking about.
There are technical problems,
with the particular
method that estimating light introduced, which is a nice, simple one.
but there's a lot of those particular technical problems
have been overcome by more sophisticated methods.
There are really important practical methods.
as I told you, I said it's difficult to estimate the probability
and a lot of meta analysis and very few studies.
So we're going to get really bad estimates of the variability across studies.
And that impacts on how well we estimate the middle of the distribution.
So that's a real practical problem when you've got few studies.
But the the problem I really want to get at is a more conceptual,
which is the misunderstanding of the model.
going back
to most recent citation of the paper,
looking at some,
cough treatment for the infection and stroke and a meta umbrella review.
That's a new one on me.
I'm not quite sure what it is.
but they talk about using,
random effects meta analysis to obtain a pooled estimate of the effect size.
So I'm a bit fussy about the language here,
but I think betrays a misunderstanding of what the model is doing.
The effect size.
they say, interestingly, they say, well, follow a normal distribution.
but but this variance equal to the sum of the weights,
that is about the coefficient for
is actually nothing to do with the the random effects distribution.
So I that they did understand it, but they didn't or they didn't at all.
I know what I mean.
and so, so
they get these results and it was kept near the top result of 23
studies is in is a result of random effects meta analysis.
And they reported saying, well, it's 2.4.
Take the findings indicate the patients with severe Covid significant
more likely to vote stroke than those who are less with less severe forms.
I mean, that may be true, but it doesn't necessarily
follow from the random effects analysis they've done.
And you can see that the absolute value is quite large here.
that doesn't measure the amount of functionality,
but it does correlate with the amount of energy.
So that does tell me that there's probably a lot of probably reasonable about that.
You know, depending on how the studies are,
especially when this study is a small. So.
What's the problem here
that the problem is that random effects analysis are designed to learn
about distribution to the effects, not about single effects.
And when people right the effects with this,
then that is not aligned with the model that they used.
And it was very clear to me at the very beginning of detriment
in the LED discussion,
when I say that characterize the distribution of treatment effects,
but people don't read these papers closely, just citing.
So what's the solution to this one?
This one's more difficult.
honestly, one option is using a fixed effect,
model,
which may be very reasonable if the studies of questionable.
but I don't want to get into that debate if people are going to
use a random effects model,
then they need to recognize
that the estimate that comes out is an estimate of a mean or an average
and and not a single effect.
And ideally they should also portray
the rest of the distribution, the spread of effects clustering.
Because that's what they've assumed by saying I'm doing a random effects meta
analysis, people should know what you've found
from the assumptions you made.
so we can,
we can portray the full distribution
and my reporting tool, but that's, as I say, very difficult to interpret.
So one option is to, to present a prediction interval,
which is a way of expressing talk in a way that's more interpretable.
So here's the result.
This is an estimate of the mean.
A prediction for would extend
this to say, well this is where I think the mean is and the uncertainty.
But this is where I think the effects could be beyond the mean.
The spread
that some of the studies have effects down here and some of them up here.
There's an average somewhere around here.
Now these things are problematic for very small numbers of studied.
but when you got enough studies,
I think this got quite a useful way
of looking at the results.
so number four.
This is Matthias Eggers paper,
bias in meta analysis detected by a simple graphical test.
Okay. Something simple.
So it's become very popular.
What's it about? What?
It's really all about using funnel plots, some plots, plots of,
the size of effect in each study
and how big each study was.
Here is some from the past.
The original paper is the effect.
It measures just knots ratio.
And here is essentially how big the studies were measured.
Just precision. Yeah.
So the big studies with
higher precision are up and the smaller ones are down.
Now what should happen in an ideal world,
a big study should be close to the truth
because of the small statistical error and the small studies,
assuming they're there also
trying to learn about roughly the same thing is the big studies
I for a fixed effect model or a random effects
model should be distributed more widely
because of sampling error, but you should have the same mean.
So we would expect, as in these two examples,
the plot to appear roughly symmetrical
with a lot of scatter among the big studies,
the less scatter among the big studies,
but centered on the same underlying truth,
the basis under which we brought together the studies into one better thing
they're all trying to tell us about approximately the same thing
and exactly or an average.
But sometimes they don't.
And and this test is about identifying the situations in which the,
the symmetry doesn't hurt.
And with some nice examples, the, the I for trial runs from Oxford
50,000 patients randomized to magnesium or placebo.
strongly disagreed with all of the earlier
small trials when it landed back on the no.
But most of the others, indicated to just to the benefit of magnesium.
A very interesting study.
if you plot them, it looks asymmetrical.
So maybe there are
some studies missing down here.
And here's another example of the matrix.
So what's the problem?
Well, let's look at, the most recent citation.
It's looking at it's prognostic match analysis, looking at whether
and now three seven, eight predicts
mean some
cancers, cancer.
And they use
to assess publication bias back in aggression.
A similar test were applied.
and they got some small values
that indicated there was publication bias.
And they got some no values that suggest there is no publication. But.
I think it's just
plain I'm going to come up with now
I can just plain wrong.
I guess test is not a test of publication bias.
It's a test for asymmetry.
Now, the problem is that
the vast majority of people using the test equate
publication bias, and some of the testing,
and I can that can be quite dangerous, misleading things to do.
Yes, publication bias may introduce
asymmetry and have some problems.
It may be something to back
this picture that there were small studies here
that showed that magnesium was harmful
and they were suppressed under that for mission critical block.
But. There are other
very plausible reasons why some plot may be.
It may be.
Well, this is another example
of publication bias, but it may be just to do with quality
methodological quality of smaller studies being potentially worst.
In bigger studies you also get an asymmetric clinical.
It may be true heterogeneity.
I mean there is actually a statistical
not a problem but that is tested statistically flawed.
So any small flaw but you can get
asymmetry out of the test when it's not really that
chances are always an explanation, but these two,
highly plausible explanations.
Let me just show you, here's an example where the smallest studies
this is an artificial example, but small studies are all done with a,
you know, a bit of bias,
and it shifts
the results of those small studies to the left.
And when you put them together, you get something.
here's a plot with asymmetry.
maybe
you think there's some missing studies down here,
but then if you look at them in more detail,
there are three populations of trials here.
And in this set there's no symmetry in this set.
Looks fine.
This is actually when you put them all together it looks asymmetrical.
So that's an example of true heterogeneity.
Then we go back to this example.
What do they do.
Well the studies they put into this meta analysis
have looked at completely different cancers.
These are the results.
There are some that say the same predicts and others it doesn't seem to predict
the whole basis of assuming a percent of asymmetry.
For me, it doesn't really hold you.
It's I mean, it's
all it's all based on a theory that a lot of people
don't realize, based on the theory that all big studies are published,
but small studies with results in one direction
are selectively withheld.
Now, that is probably a good model
for randomized trials on the same topic, but there's no particular reason
for belief that theory holds in observational work,
where, you know, big studies may have really
crude measures of exposure,
so they may be the worst studies
and small studies may have really accurate ones,
and maybe they're giving ones with less past results.
And so this idea that the big ones are near the truth and the small ones bias
will suppress one side might not hold.
And I think we need to understand these things a lot better.
and not just naively assume that some of what I think should be psychology
studies through population bias.
It's a very common problem.
So what should we do if we worried about publication bias?
I say don't start with these tests.
Start by using the brain.
You know, what is the likelihood that studies would have suppressed results
or that studies would have been hidden altogether?
And you can get quite far by understanding the field and then some fields,
a lot of small studies are done by psychology students, for example.
and they're easy to suppress or just written into a thesis
and not written up in other areas.
randomized trials are registered.
We know about them.
if we can trace all of the registered trials
and we're still estimates, it's probably due to some other reason.
So we need to think about the situation we're in.
And, it's not a plug of work I was involved in.
The rope and Army tool is an attempt to put a framework
into thinking more sensibly about publication bias
and related biases that lead to missing evidence in next analysis,
and the funnel plots are mentioned.
But right at the end, after you've done all the other thinking.
So if we use
funnel plots, actually there is a I mean, if,
you know, I'm sure there's a lovely, modification around.
You, improvement we can make to funnel.
So that's you put controls on it controls
that describes statistical significance.
because for any
if we're going to plot the effect size against precision,
which is basically the nature of the standard error with effect size,
any combination of estimating standard error leads to.
So that's what I call the z statistic.
The meta a p value.
and so there's a p value
representing any point on this plot.
And then we draw contours for the specific p values that we're interested in.
So here's control. The first one is
point one.
So that's kind of getting towards the significance.
And then further up we get point naught five,
which I often use as a threshold for setting significance.
And then point one, which is one.
And what we see here is a lot of the studies we've got
were statistically significant.
And so and
and if this were due to publication bias, it would mean that
studies down here, emission
studies down here, non-significant studies.
So it's quite plausible that this is due to publication bias,
because it fits what we'd expect.
If the publication bias is the cause of this
small null studies emission.
Whereas in this example we've got asymmetry.
If you look at the controls you might think, oh that's the top.
There are studies missing down here.
But here the missing
studies are just the significant other.
and so,
I mean, it may be that people that found results down has pressed them, but
it's just a different set of assumptions about the mechanism for publication bias.
But it helps you
think about those and produce situations where it's a plausible explanation.
So the longer you need to think more carefully.
Right.
My last one I didn't have
I get time to do this, I think I do.
This is risk of bias.
Assessment tools.
this is a framework for assessing risk of bias
in a randomized trial included in a systematic review.
And it,
this is the original tool published in 2011.
Once through six items and 5 to 9 different phases
were updated to rock two.
and this is one we hoped that people would use now,
but the mis uses
I want to talk about relate to both of these.
Now here my my training
for most recent papers.
wasn't quite so successful.
The first one
for that patient to some of my colleagues in Bristol.
and it's not so it's not so bad.
But what what they what they're doing here is a descriptive review,
just describing interventions and outcomes. But,
that have been used in studies of,
American emergency death of me.
I have to care for that. But you can see that.
And and they used, promises another tool
and this tool to assess risk bias in the studies
which, which, which is fine.
Maybe it's interesting, but it's not what we designed
the tools for really, because they're not talking about results.
but I'm,
I'm not going to criticize this personal work.
The next one I came from actually illustrated the problem, but it was not
problematic. The problem.
it was,
they looked at,
some randomized trials, they assess risk bias,
and they took that,
towards the beginning of the results section.
They described their risk assessments here and refer to the figure
that shows that these studies, this the different domains of bias.
In fact, everything looks pretty good here.
A bit of uncertainty but a lot of low risk of bias.
they then go on to report the results of the meta analysis,
but make no mention of the risk assessment.
That's the problem I have here.
It doesn't really matter.
But the next one I found,
I think illustrates the problem that I see over and over again.
Here the results section starts
by presenting the match analysis results.
The results yesterday they do some, publication by stuff,
but they don't mention the quality of the trials going into the outcomes.
And then later on they have a little section
that that describes the risk of bias results.
And they show,
this profile
suggesting that there a problem in nearly every study,
if we believe their assessment.
But there is actually no linking
of the fact that there were problems with the results of the meta analysis,
and there is not a mention of risk of bias in the entire discussion.
So this is the problem that I if
it annoys me, it was to
value the doing.
What is risk of our system if you're not going to use it?
People don't routinely ignore risk assessments
and doing fancy reporting services and drawing conclusions
is the solution.
Well, I mean, this is one of the reasons why,
a modification in Rock two was should be really explicit.
This is risk assessment in a specific result,
not in a study.
And that means risk of bias assessment is
what two really need to be presented alongside those results.
And that was why we probably why we did this.
So that we would encourage people to present risk profiles
alongside the results.
And from the first part.
and we see it increasingly which is, which is great.
And Cochrane, has
shifted its guidance on where to report risk bias
from an overall think towards
and may affect interventions.
A comment here that great timing
of how because great is outcome specific.
And we think about risk assessments
in the specific results for the particular outcome.
Okay.
So I try to bring things to close.
Now what what are my reflections on all this.
Well I mean to summarize, I've looked at what I see
as some really, really common problems in match analysis.
I've built them on the left hand side of papers.
but that makes them highly, prevalent problems,
approachable solutions.
This is the fourth time I've given this talk.
Each time I go and look at the latest citations
the day before, and in almost every instance across the four,
I find that the most recent papers misuse or the sense
that the method now,
I think that means it's probably
the probably very commonly misused,
but it would be nice to have some empirical evidence.
And actually the project would be to look empirically
at a random sample and to see how much these problems.
Why did they happen?
And that's an interesting thing we could talk about
if we want to have a discussion.
I think it's largely people just copying
what previous people have done.
there could be an element of,
Dangerous thing to say in the context of a short course.
I'm not involved in teaching.
I'm sure it hasn't been the case this week.
but I think, I mean, this is Oxford, and I'm sure we got from the.
We can one can imagine that in other places, there are people
who don't understand things quite as well and perpetuate the sort of things
they read time and time again, and reports of systematic reviews.
Or it may just be that, you know, oh,
I mean, I but just inconsistency
that means that it must measure hedge matches and I just want it to measure.
That's you
make a combination of all these things.
Now, one last thing.
I mean, being highly cited is nice.
People introduce you with brand words,
but it doesn't mean you're having a positive influence.
Let's say,
I don't I never
use this little stuff that encourages
us so that thank you very much for the attention.