So hi everyone. I am Jamie Hartman Boice. I'm based at the Nuffield Department of Primary Care Health Sciences, just up the road in Jericho.
And I also work closely with the Cochrane Tobacco Addiction Group and today I'm going to
talk about why and when it all sides of the same question don't agree with each other,
focusing on electronic cigarettes as an example. So I'm funded by the National Institute for Health Research and the British Heart Foundation.
I don't have any complex to declare.
So what I'll cover today is I'll talk very briefly about Cochrane, give a bit of an introduction to the issue of e-cigarettes.
For some of you. You might know quite a lot of this, but for others you might not. And I think it's important context.
Before we go into our Cochrane Review, some other meta analyses that set out to answer the same questions we did in our Cochrane Review.
And then also looking into the wider literature for examples of where meta analysis might
not agree with one another and end up talking about implications and potential next steps.
So Cochrane is a global non-profit that primarily exists to do systematic reviews,
and we aim to do this in a very robust, transparent, sometimes excessively thorough manner.
So anyone who's read a Cochrane review knows that they are very long documents.
But the reason we do this is we have very specific guidelines that we follow and we do this because we want
to make sure that people making decisions about health care have the best available evidence to hand.
And that includes patients, carers, clinicians and policymakers.
And Cochrane is split into a number of different review groups. And those are subjects with this specific.
And the Cochrane Tobacco Addiction Group is based here in Oxford.
So now a little bit about e-cigarettes.
E-cigarettes I think first came into the public consciousness in the mainstream way probably in the past ten years or so,
but they've been around for a little bit longer than that.
They were developed around 20 years ago now by a pharmacist in China who developed them for the purpose of them being a smoking cessation device.
Within the field of smoking cessation, there are lots of evidence based things that can help people quit smoking.
But unfortunately, even the best of those have success rates that really leave room for improvement.
So in a best case scenario, let's say you're looking at doubling your quit rate.
So you might be looking at a population who wants to quit smoking, doubling them from 10% without treatment to 20% within with treatment.
And that's still leaving 80% of people who really want to quit smoking.
We know the vast majority of adults who smoke want to quit, who just can't, because it is such a notoriously hard thing to do.
And we also know that smoking is one of the worst things you can do for your health.
And therefore the best thing a smoker can do for their health is quit.
So e-cigarettes came into a market where we actually did need some better treatments to help people quit smoking and people were excited about them.
And there's a number of reasons they were excited.
So what they do is they heat a liquid into an aerosol for inhalation that usually comprises propylene glycol and glycerol,
and it can or cannot contain flavours, counter, cannot contain nicotine, etc.
And the key thing about e-cigarettes is that when they contain nicotine,
they're delivering that nicotine without combustion because the harmful thing about smoking,
the harmful thing about cigarettes is actually what happens when the product burns.
It's not the nicotine itself. It's not the other chemicals on their own. It's what happens when they burn.
And essentially what they were trying to do when they developed e-cigarettes was deliver a cigarette without the combustion.
So make a less harmful alternative. Another really important thing about e-cigarettes is that they are not all the same.
So it's really tempting when we talk about them to be like, Oh, they're dangerous or they're not dangerous.
They do this or they do that. There's a huge range of e-cigarettes.
So there are four generations, as they're commonly referred to, and that's what that picture is there.
The first one, which is the first ones that really came out, are cigar like.
So they look a lot like cigarettes. If you were 20 feet away, you'd probably think it was a cigarette.
And over time, they've evolved to devices that look pretty much nothing like cigarettes.
So there are vape pens which are second generation, those box models, which are those larger tank models.
And in the last couple of years, we've seen a lot more vape pods, which look quite a lot like USB sticks.
And probably the most popular brand of those are tool and take a seat.
So in any cigarette, the e-liquid or the juice is stored in a refillable or disposable cartridge or reservoir.
And that totally just depends on the type of e-cigarette, and it can vary dramatically in its nicotine content,
so it can vary from absolutely no nicotine to here if it was a regulated product,
not more than 20 milligrams per millilitre, that is specific to the EU.
So that's a regulation we have here on our nicotine content.
In the States, you might get concentrations up to, for example, 60 milligrams.
They're very, very much in their flavours. And there's also an opportunity, especially with some of the newer devices,
for people to put what they want in those devices and to tamper with them in some cases.
So trying to make a guess at what an e-cigarette contains if you're not buying it from a reputable retailer, is a risky thing to do.
And so here in the UK we're pretty well regulated and most people who use e-cigarettes
buy them from a shop and they can trust that it's going to conform to all of these EU.
Relations in the States. That's not necessarily the case.
And e-cigarette users are sometimes described as vapers e-cigarette use as vaping.
So if you hear me mention it, that's what I mean. And probably the other thing to know about e-cigarettes is that they cause a lot of controversy.
So I don't know how many people have seen them in the news in the past year.
I'm certainly very aware of it. And when I first started working in tobacco control research about eight years ago,
you would go to a conference and generally everyone was on the same page about everything.
We might have minor disagreements on analysis methods, but essentially we were all there with the same basic aim.
That has dramatically changed. So these conferences have become a lot more exciting, but also a lot more conflicting.
And people feel very, very strongly on that. Either pro e-cigarettes or anti e-cigarettes front.
And broadly, those divisions go by countries.
So in the U.K. here we are often criticised for being too favourable towards
e-cigarettes and we in the UK often criticised our US colleagues for just the opposite.
And what this means is that the public in particular get very, very conflicting messages.
And there is a concern not only about this affecting the public's understanding of e-cigarettes and the relative risks and benefits,
but also about possibly undermining public trust in scientists.
Because when you have big scientists on either side kind of slinging insults at each other and saying dramatically different things,
why does that put you in terms of trust in these experts?
So as I said, researchers can't agree just a quick like I spent 5 minutes on Twitter to come up with some of these reasons why researchers can agree.
And these are all people who are followed by thousands of people besides myself at the end.
They don't have that many followers. But I stand glance at the University of California.
San Francisco is a major name in tobacco control research, and he is majorly anti e-cigarettes.
And we're going to look at some of his research further on down the line.
And what he's arguing here is that e-cigarettes make it harder for smokers to quit.
So that is an interesting and very strong message there.
And he's also saying dual use, which means using both an e-cigarette and a cigarette at the same time is more dangerous than just smoking.
So, again, very bold claim. But on the other hand, we have Linda Bauld is a professor up in Edinburgh.
She's the Cancer Research UK is cancer prevention champion and this is her tweeting about a recent presentation which looked at the
effects of e-cigarettes in the cardiovascular system and found significant risk reduction when people switched completely to vaping.
So how do those things balance with that and have Martin Dockrell, who's the head of tobacco control for Public Health England,
and he tends to be quite pro vaping and calls out some of the research saying that there are vaping epidemics that have Robert West at UCL,
again, extremely senior professor who is getting extremely angry with the W.H.O.
This was last week. The W.H.O. came out with another report on e-cigarettes saying they're harmful to health and are not safe,
particularly risky when used by adolescents. Nicotine is highly addictive and young people's brains develop up to their mid twenties.
Exposure to nicotine can have long lasting damaging effects.
And Robert's response to this was to say this is propaganda that fuels distrust of experts
and provides a spawning ground for conspiracy theories such as anti-vaccine movement.
So these arguments are deep, people feel them strongly,
and these are all relatively reputable people and reputable organisations saying very different things.
Stern Glantz I think is right about at least one thing, which is that there's a war between scientists over the health risks of vaping.
And that is one thing that we definitely agree on because I've written on it too.
Ultimately, these conflicts boil down to a few key points and these key points drive policy,
and they're related to how we interpret evidence, but they're also related to what questions were asking of the evidence to start with.
So the first big question is, do they cause more kids to smoke?
And I think it's fair to say when e-cigarettes first came on the market within tobacco control,
that was probably the thing we were the most worried about because we knew that the tobacco industry was interested in e-cigarettes.
They were buying up these companies and we definitely didn't trust them after, you know,
decades and decades of horrible misconduct with are they going to use this as another route to advertise cigarettes, get people hooked, etc.?
There are lots of cohort studies which show that kids who use e-cigarettes are more likely to go on and smoke.
A lot of the time that is interpreted as e-cigarettes, leaving people to smoke and a gateway effect.
That story changes slightly when we look at the prevalence of smoking in young people, that continues to decline.
So there are some unanswered questions around that particular issue.
Another issue also around young people is do young people vape who wouldn't otherwise have smoked?
Because we know that e-cigarettes aren't completely safe, it's not a good idea to inhale anything other than fresh air.
But compared to smoking, they do appear a lot safer.
And there's a big debate here because actually, if these kids who are vaping would have otherwise ended up becoming addicted to cigarettes,
we'd probably prefer them to be addicted to vaping compared to being addicted to cigarettes.
But if they weren't going to be addicted to anything, then we don't want them to start vaping and become addicted.
And this is something that's causing.
A lot of concern, particularly in the US around this terminology, around a vaping epidemic and lots of teams starting to use e-cigarettes.
And then what I'm going to focus on for the rest of the talk is really the issue about adult smokers and do they help adult smokers to quit?
Public Health England has come up quite strongly on this. So every October they run a stop smoking campaign called Stop Tober,
and this year they proactively encourage people to use e-cigarettes to switch and to stop smoking.
And ultimately, of course, the main question here, too, is, are they safe?
And here again, we have massively conflicting information.
So the Centre for Disease Control in the US came out last year with quite a lot of
advertisements targeting young people with quite scary messages about damaging their brains.
Put aside that cancer research UK came out last year and kept on putting out infographics talking about
their relative safety of e-cigarettes compared to smoking and saying pretty positive things about nicotine.
So it's addictive, but it doesn't cause cancer. So that's a slightly different story you're seeing there.
Public Health England has an often cited figure that e-cigarettes are not risk free but are estimated to be about 95% safer than regular cigarettes.
And then we have the World Health Organisation last week answering the question are e-cigarettes more dangerous than regular cigarettes?
And saying this depends on a range of factors, including the amount of nicotine and other toxicants in the heated liquids.
But we know they pose clear health risks and are by no means safe.
And all of this has become more of an issue because in the last year there
have been a number of vaping related serious illnesses and deaths in the US.
And one of the reasons why the conversations here get a bit confusing is that in the
US it very much seems the case and the CDC has come out and said this is the case,
that what is the risk here is an additive called vitamin E acetate.
That is something that is not allowed in e-cigarettes in the EU because we
absolutely knew from the start that if you vape that you're going to get sick.
And it's used in e-cigarettes in the US, primarily ones that are sold on the black market,
off the street that contain cannabis because it looks very similar to THC oils.
So if you're buying something, you can basically cut it with vitamin E acetate and make more money that way.
So we have all of that going on and no one quite knows what to think.
And whenever me, with my own biases towards Cochrane systematic reviews see something like this,
I'm like, Well, let's just do a systematic review and then we'll figure out the answer.
It'll be unbiased, it will be transparent, everything will be great. So we did a Cochrane review.
We first published our Cochrane Review in 2014. We updated it in 2016, and that's the version I'm going to be focusing on in this talk.
And we're in the midst of updating again currently, so we're expecting a new version to be out later this year.
In terms of what we included typically in our group, when we're looking at a smoking cessation therapy,
we would only look at randomised controlled trials and here absolutely we preferred randomised controlled trials but we
knew that there just weren't that many studies out there and indeed there still aren't that many studies out there.
So we widened our inclusion criteria to also include uncontrolled intervention studies.
So these were studies where everyone was given some sort of e-cigarette intervention and also to
include observational studies of the type that basically survey a group of smokers at the beginning,
ask them whether or not they're using e-cigarettes.
Follow them up sometime later and see if there are differences in quit rates between those using e-cigarettes and those not using e-cigarettes.
Again, as standard in our group, we are only interested in smoking cessation at six months or longer because unfortunately a
lot of people who try to quit might do so successfully in the first week and then relapse,
and that's not going to confer any significant long term health benefit.
So we're really interested in lasting effects and we're interested in adverse effects after a week of use.
And that third group of studies, which I will talk about quite a bit in a little while, we're not included in this current update,
and that is because the nature of their design and serious risks of confounding just means
we felt that actually we're getting at least a few more randomised controlled trials,
a few more uncontrolled intervention studies. Let's leave this group of studies behind moving forward.
So in terms of our outcomes, we're interested in cessation at six months or longer.
We treat people who don't come to follow up as smoking in our analyses, and that's pretty standard across the field.
And that's because if you're having contact with a researcher who's encouraging you to quit smoking and you have successfully quit smoking,
it can be really lovely and quite an incentive to go back and tell them.
And if you haven't quit smoking and they've invested time and effort in you, it can be quite a disincentive to go and tell them.
And we use the strictest available definition of abstinence.
We also looked at a range of adverse events and various safety profiles.
In terms of our numerical analysis, we pooled data where appropriate and here again,
we just used our standard Cochrane methods that we use for any cessation intervention.
So this involves calculating risk ratios which we calculate as the number of people
quit in the intervention group over the number of people in the intervention.
And group. And then over that, the number of people quit the control group and the number of people in the control group.
And so risk ratio because it's a ratio.
If you end up with one, it means that you have the same proportion of people quitting in the intervention and control arms.
If you end up with a risk ratio less than one, it means that more people quit in the control group than in the intervention group.
And on the opposite side, if you have one greater than one, then you know that more people are quitting in the intervention group.
And if an intervention is successful, that's what we're hoping to see is a risk ratio greater than one.
And preferably if we want statistical significance, a risk ratio that doesn't confidence interval that doesn't encompass one.
So unfortunately back in 2016 there were only two randomised controlled trials that looked at our question and even
more unfortunately these are the same two randomised controlled trials that were included in our 2014 review.
So these are both two relatively small studies, one conducted in Italy which followed up over 12 months,
one conducted in New Zealand which followed up for six months.
And the comparison we're interested in here are the comparison between people given
e-cigarettes with nicotine and then those given e-cigarettes without nicotine.
And the reason that's an interesting comparison is it's pretty much the closest we could get to a placebo controlled trial of this device.
And there are more studies coming out now, which I think more interestingly are comparing e-cigarettes with nicotine to, for example,
other forms of nicotine replacement therapy, which might be a more interesting question, but that's where we were then.
And what we found is that neither of these studies on their own found a significant effect.
But when we pulled them, all of a sudden we did have a statistically and clinically significant effect in favour of the intervention.
So in favour of e-cigarettes with nicotine.
As you'll note, the lower confidence interval is quite close to one, so it's just narrowly achieved statistical significance as it were.
And as with all Cochrane reviews, what we do is we grade the certainty of our evidence.
So we think about how much do we trust that, how much do we trust this effect estimate?
And what we said was not very much. Essentially, our certainty in this is low.
We think no studies are very likely to change our effect estimate. And there are two issues here.
One was imprecision. So basically wide confidence intervals, and those are coming from very small numbers of events.
So even though we have about 600 people in these analyses, we only have seven people in the control arm who quit in 43 in the intervention arm.
So that's not very many. And another issue which is not as common is an issue of indirectness here.
And this comes because e-cigarettes are relatively new. The technology is changing rapidly.
It takes a very long time to get a randomised controlled trial off the ground.
You apply for funding, you apply for ethics, you run the trial and then you need to publish it.
And so the unfortunate reality in this field is that by the time most of these studies are published,
the devices they tested are no longer on the market. And that's the case here.
So both of the devices tested in these two studies by the time they published were
pulled from the market because they were simply terrible at delivering nicotine.
And bear in mind, the comparison we have is between a nicotine delivery device and a non nicotine delivering device.
And so how relevant is this to someone who might show up to their clinician and ask, Should I go buy an e-cigarette?
These studies aren't going to tell them quite as much about the e-cigarette they're going to buy now as what they would have bought back in 2014,
let's say, in terms of adverse events. Again, we had low certainty in the evidence because of a small number of studies,
but we didn't find anything particularly alarming in the studies that we looked at.
There were no serious adverse events related to e-cigarette use.
The non serious events didn't tend to vary between arms and the cohort studies.
All told, a similar picture of mouth and throat irritation that seemed to dissipate over time.
But major caveat. Our longest follow up here was two years.
And of course, in most of the studies, what we're looking at are nicotine containing e-cigarettes that are regulated.
So we are going to see a different picture from, for example,
tempered products that someone might be buying off the street now, somewhat distressingly.
So we published a review in 2014 with those two studies because we're Cochrane We update our review, so we published it again in 2016.
We included the same two studies and in that time 20 other systematic reviews were published which looked at this exact same question.
And I think you do have to then start to worry a little bit about whether or not that's a good use of time.
There are so many unanswered research questions is this is what is this what all the researchers need to be looking at?
I would argue possibly not all of them agreed that more evidence was needed, and I think that's a no brainer in this area.
That's absolutely the case, particularly around long term safety and long term efficacy.
And five of them conducted meta analyses.
And in the next slides, I'm going to focus on this meta analysis and how they may be similar or different to ours.
So my first one, and I'm not trying to pick on any of these because I understand why they did it.
I published a second review that had the exact same two studies that my first review did.
So I'm part of the problem too, arguably, but the first one was published in 2015.
It included the exact same two randomised controlled trials.
We did it use the exact same methods we did and it found the exact same thing and I think that is both trouble.
And reassurance, certainly for me as the author and the one who did that analysis,
it was wonderful that someone had reproduced it and found the exact same thing.
However, was it really a useful thing for them to have spent their time doing?
I'm not entirely sure, but we certainly found the same thing.
Then there were three studies which came out in 2016 which had some similarities and some differences, and it's worth focusing on these.
So just like us, they included the exact same two randomised controlled trials.
Again, quite reassuring. We're not missing anything here. A big difference for them is that whereas we looked at results of the longest follow up,
which in the case of that Copper Nettle study was 12 months and then followed with six months.
They all pooled results at six months follow up losses were treated as continuing smokers just like for us.
And the really key difference in their results was that they all found an effect size favour in the intervention,
but none of their effect sizes were statistically significant. And what this comes down to is that difference between six and 12 months.
And worryingly, what is driving that is one person.
So in the cap, another study, which is the one that followed up at 12 months, there was one person in the control group had quit at six months.
At 12 months, they'd relapse to smoking. And that difference in one person shifted the whole result to no longer being statistically significant.
So I think that makes us feel better about downgrading for imprecision because
clearly that is an issue and I think it also highlights some of the issues
around overly relying on statistically significant is something seriously meaningful if one person starting to smoke again switches the whole thing?
How much of our messaging should be focussed on that?
You know, there are some minor variations to in their individual effect estimates and
those are basically due to different models and random versus fixed effect,
etc. But essentially they're all using six month data and they're all finding basically the same thing.
Then the review will focus on the longest is one where the senior author is Dan Glantz,
who I talked about a little bit earlier and promised I'd come back to.
And this one is I'm focusing on because it found something dramatically different from ours and it came out the same year ours did.
So we ended up having to do quite a lot of looking into why is this coming out so different?
Where do we stand on this? And the main difference here and the reason that's driving their different results
is that they included a much wider range of studies and their meta analysis.
So rather than just restricting to the two controlled trials, which are the same two trials we had,
they included 15 cohort studies and three cross-sectional studies.
And they pooled all of these together and it is rare that you'll see lots of different study types combined
in one analysis and hopefully I'll be able to convince you of some of the issues with doing that.
I think sometimes it's legitimate, but they were heavily criticised for doing this and it is, I think, fair to say, quite unusual.
We reported results at longest follow up. They reported whatever was in the original paper and didn't have any cut off based on length.
So it could have been a month that could have been two years. They used whatever methods were used in the original paper for Lost to follow up,
which I think is a problem because I think particularly if you're working in an area like
this where you have lost the follow up that you absolutely know is not at all random.
You probably want to be using the same method to impute.
Otherwise, you're introducing spurious differences between studies and their effect estimate was 0.72 with confidence intervals that did not span one.
So they found that e-cigarettes made it harder to quit smoking.
And that is why stand keeps tweeting that and saying they're really stopping people from quitting.
We found that e-cigarettes did help people quit smoking. And what is driving that absolutely is their inclusion of this observational data.
And there's a number of reasons why we might expect the observational data in
this area to show something different from the randomised controlled trials.
The first one is that there's very good reasons to believe that the effectiveness of
e-cigarettes for a smoking cessation aid is very dependent on the amount of support provided.
So we know that even with something like nicotine replacement therapy,
if you give everyone nicotine replacement therapy and randomised one arm to receive extra behavioural support,
whether or not that's just talking to someone once, that will increase their chances of quitting.
And then the trials of e-cigarettes, absolutely they were interacting with people.
The other thing about e-cigarettes is that they are not necessarily the most intuitive to use.
My in-laws use e-cigarettes. They're both in their late sixties and they needed a lot of guidance on how to use that e-cigarette.
And if you're just buying one from a shop and you don't necessarily know what you're doing with it,
you may well not be getting great nicotine delivery, for example.
The second issue with these observational studies is that there are serious issues around the baseline definition of e-cigarette uses.
So just to remind you, these are studies which go out,
they get a group of smokers and they ask them at baseline if they use e-cigarettes and then they follow them up later.
And these studies, some of them are saying, okay, we define e-cigarettes as using an e-cigarette at least once a week or at least once a day.
Others are at any point in your entire life have you ever tried an e-cigarette?
So someone who tried one once because their friend had one in the pub.
And they took a path they decided it wasn't for them would also still be included in this group of e-cigarette users,
which is again introducing quite a lot of variability.
But the most to the two most important issues are just issues around design.
And one of those is of course confounding. We're looking at observational studies.
There are definitely some unexplored confounders here. And one of the reasons we know that these are going on is nicotine replacement therapy.
It's been around for years. It's a first line treatment for smoking cessation.
We have 113 randomised controlled trials with long term follow up that very clearly show
with high certainty evidence that nicotine replacement therapy helps people quit smoking.
But if you did the same study about nicotine replacement therapy and you surveyed smokers at baseline and said,
are you using nicotine replacement therapy or not? And then you followed them up.
Six months later, it would appear that nicotine replacement therapy was hindering their quit attempts, so fewer people using NRT would have quit.
And the main driver for this is the level of addictiveness.
So it's something that drives whether or not you use an R, T or e-cigarettes.
And it's something that also makes it less likely for you to quit. So if you're not very addicted, you're someone who has a cigarette.
Every once in a while you decide you're going to stop. You may well not try to switch to using it to use an e-cigarette.
You may well not use NRT. You may find it very easy to quit smoking.
If you are someone who has tried to quit smoking again and again and again, which most people are at this point,
then you probably are going to try and use some product to help you, and you're also going to be less likely to quit if you are heavily addicted.
So that is a very important confounder. And there are also some other confounders that we suspect are at play here.
And then the other issue, which is very specific to this individual type of study,
is that by their very definition, these studies are biased because if you imagine,
let's say let's say we imagine a study that was conducted in 2016,
and you imagine a group of people who in 2015 decided they were going to try to use an e-cigarette to quit smoking.
All of those who use the e-cigarette to successfully quit smoking would not be
included in our 2016 study because they would no longer be classed as smokers.
So anyone who used them and it worked for them were ignoring them.
The only people left in there are the people who are using an e-cigarette still smoking and haven't managed to switch completely to an e-cigarette.
So we have a real issue there in terms of essentially only retaining treatment failures or people who have only just started to try and switch over.
And so for all of those reasons,
that's why we of course prefer randomised controlled trials and why that's what we focus on or look at cessation outcomes.
Now it would be really nice to say, but this problem is pretty much just an issue because we don't have that many randomised controlled trials.
If we had more than imprecision wouldn't be an issue.
If we had more there would be much less valid reason for saying let's include observational studies.
Unfortunately, it doesn't necessarily seem to be the case and what they've tried to defend themselves with here is that they did
a sensitivity analysis to check that everything they did was fine and they did some kind of weird things here,
which is probably worth talking about.
It's probably also worth noting, for those of you who might be doing a systematic review, that what they didn't do was have a protocol.
So there's no way for me to look back at that protocol and check that these were predefined sensitivity analysis.
They seem weird to me because they did lots of sensitivity analysis, including this first one,
my list, which is real world versus clinical and what they mean by that,
it's observational versus randomised controlled trials and because they did lots of them, they controlled for multiple comparisons.
And what that's essentially doing is it's making their P values quite a lot higher.
And when you look at their table, which lists a very long list of sensitivity analysis,
I've just taken out the main one that's relevant to us, which is the study type.
We have a P value that's not at all statistically significant. It's 0.90.
And yet when you look at the study types, it's a bit weird.
So first of all, it seems kind of an underpowered sensitivity analysis if you're only taking out two studies from a meta analysis of 21.
And secondly, as we would expect, we have quite estimates going in very different directions and we also have confidence intervals that don't overlap.
So even though the P value here is not statistically significant, if I saw this as a sensitivity analysis,
I don't think I'd find it reassuring, which is how they interpreted it. They said This is fine.
It was totally appropriate to pull all the stuff. I would say that and think your confidence intervals don't overlap.
Your point estimates going in completely the opposite direction. Maybe there's something more going on here.
As I mentioned, and somewhat disappointingly,
this is not just a problem for e-cigarettes and it's not just a problem because there aren't enough randomised controlled trials.
So David Noonan, who's a colleague of ours based in the centre of evidence based medicine here in Oxford, and a colleague,
Claudia Hack, did a review article that just came out early this year which looks at this issue outside of e-cigarettes.
So it's looking at meta analyses which set out to answer the exact same clinical question in terms of their picos and why there might be different.
Is between them. And what they did was they went out and they identified 24 match pairs.
So 48 meta analysis where one of them was from a Cochrane review and one of them was from a non Cochrane review and they had the exact same picos,
the exact same inclusion criteria.
There was no reason you would think from reading them that they'd find something different, however, and only one pair were they the same.
So that's a bit of a problem. In fact, it's a really, really big problem when we want to trust better analysis.
And it turns out that everyone's saying they're doing the same thing and getting totally different results.
On average,
the non Cochrane Effect estimates were higher and in four of the pairs that was in the magnitude of at least two fold increase in the effect estimate,
which is a pretty dramatic change for 14 pairs they issued.
They looked at discrepancies in interpretation and the inclusion criteria.
So even though they had the same inclusion criteria, I think this kind of spells out all the grey areas that even if we have a PCO,
we can kind of work around or have differing opinions on.
And I think this is why it's particularly important to nail down your protocol and exactly what you mean by your PCO early on.
Another really big issue was the numerical data.
So even if they agreed on the same studies, the same studies were in there, often there was not the same data in the analysis.
So only two pairs agreed on the numerical data presented for the same studies.
And what they did is they took a sample of 50% of the discordant pairs of the studies that didn't match up.
So they looked at 45 of them out of 90, and in only 15 could they figure out why the data extracted was different?
So in a minority, could they make sense of that? I think that's extremely worrying.
Definitely points to a need for transparency, but also as a reader, what are you supposed to trust?
In that case? They hypothesise that there are a number of reasons this is happening.
Search strategies are of course one of them. Cochrane reviews tend to be quite extensive in their searching and one of the main things that
can get missed in other reviews if they don't do thorough searches is the grey literature.
So some of these Cochrane reviews were including unpublished studies that weren't showing up.
In other reviews, which makes sense, this issue around the interpretation of the eligibility criteria was a reason, probably just some simple errors.
We are humans and we are humans doing that analysis and we will make some errors.
And those could come up in screening or in data extraction and then also probably some genuine differences in the extracted data.
And a lot of the time what they speculate this might be is people going back to the authors.
So if one review team thinks I can't actually quite make sense of this, if you do a lot of systematic review and I regret to tell you,
there are many cases where you read the text and you read the table and the two things don't match up.
And either you can say, I'm going to take the text or I take the table,
or you can say I'm going to email these authors and figure out what's going on.
And if you've emailed the authors and if the authors have gotten back to you, that is wonderful.
That's the data that you use. But a lot of the time people aren't recording that.
So it's like these reviews are being published with this data and that doesn't match up to anything in the published version.
How do we know what's going on?
And of course, I would argue there's a strong risk of bias here that comes into play both around what data we choose to use,
around what studies we choose to include.
I think we automatically think of financial conflicts of interest first, and of course some of those might be relevant,
but there are also a lot of ideological conflicts of interest that come in here with e-cigarettes.
Unfortunately, I'd say in about 80% of the papers I see, if you showed me the study authors,
I could tell you what they found without reading the title or the abstract.
And that's because it's almost all observational data. You can fiddle with confounders in one direction or another.
There's no kind of standard guidelines for this, and therefore people end up finding things that fit their agenda.
And I love tobacco control. It's the area I've worked in for a long time now.
I'm super passionate about it.
One of the reasons I'm passionate about it is that our research drives policy, but I think that's another reason why bias creeps in.
Because if you have a strong policy agenda that's anti tobacco,
you might try and be more inclined to do research that is going to support that agenda.
And that's coming from a place that's not financially conflicted, that's coming from a moral place of saying, I've worked in this field for decades.
I've seen all the harm that's done. I want to try and stop that.
But it is an issue and it's one we have to be aware of.
And of course, another issue here is that a lot of people who work in this field also do the primary research,
which can be great because it means they know the area really well, they know the ins and outs of the studies.
But it can also be an issue if what they're finding are studies that contradict their findings.
So if they've come out with this big paper,
a lot of the e-cigarette papers are published in very big journals now because it's a topical issue and then they do the
systematic review and for other papers found something different and their meta analysis isn't agreeing with our study.
That's going to be a bit of an issue for them, too. And all of these things are places where, of course,
critical appraisal comes into play and something we want to be thinking about when we're looking at these reviews as readers.
So what can we do? I mean, if you have time, the first thing I do is investigate and I do this in that.
Two ways. One is actively search for other media analyses.
So I do this whenever we do a Cochrane review. I try and do a search for all of the other analysis out there.
And then because I have time and it's my job, I can actually spend time looking at them quite closely and trying to figure out the reasons.
And if I can't figure out the reason, it drives me crazy.
And I email the authors because I think, am I doing something wrong or are they doing something wrong?
What's happening here? But as readers, we can definitely do that. And as readers we can also critically appraise.
And I think if you have ten reviews and nine of them are saying vaguely the same thing, I would be inclined to trust those nine.
What do you do when you have two reviews and they're saying something that's diametrically opposed?
I think that's what we need to bring in. Our critical appraisal skills and more understanding of analysis.
And where things can go wrong is really important as authors and I won't say it enough,
I think it is incredibly important that we pre-register our protocols.
That's partly so we can be transparent, but it's also partly because if I'm doing a review, I will search for protocols.
I'll see if someone else is doing what I'm setting out to do.
And if they are, I will ask myself some serious questions about whether or not it's a good use of my time,
of the public's money to be doing the same review. I might get in touch with those authors and they might have published their protocols in journals.
Or often what they do is they publish them on a register called Prospero.
You can look at that. What I found from looking at that is often people say they're doing things and then you check in with them.
They're like, Oh, no, I didn't actually do that, which is fine. I view that as a green light. I'm going to go ahead and do it.
But if they say and this did happen to me once, yep, I'm doing exactly what you set out to do.
I was a bit gutted because I'd been like, This is a clear area where we need a systematic review.
This is great with part of my Dphil research. No one's done this before.
Did a quick photo search. Someone else is doing exactly the same thing. I changed my review question because I contacted them.
I knew they were going to do a good job. It wasn't the best thing for me to do.
So people should really be checking these and thinking about research, wastage and duplication.
Of course, another thing you can do, and I would say this because I'm part of Cochrane, is update your review.
So every Cochrane review I do has a section on the on the other reviews published where I'll
go through and I'll pick them apart and I'll see exactly what we might agree or disagree.
Transparency, I think, is absolutely key. And of course, as a review author, you walk a fine line because if you report absolutely everything you do,
your article is going to be longer than a journal will take.
But that's why we have supplementary appendices. That's why we have online repositories where we can be very transparent about exactly what we did.
And I think adhering to reporting guidelines really helps there.
And finally, as authors, I think one of the things we might do when we see a meta analysis that disagree with each other is communicate,
particularly if it's an area like this where we know people are actually looking
at these merit analyses and making health care decisions on the back of them.
And when it came to e-cigarettes, I was feeling pretty reticent about this particular issue because I thought, we don't know very much about them.
It could be that in ten years we find that they're causing some horrible thing that none of us could have guessed.
And Oh my God, out there saying, Yeah, okay. On the best available evidence, people should think about switching.
But I have ended up staking my claim on that one because I thought about it and I thought, actually,
once you've done a review, you probably are one of the people who knows the most about it.
I actually don't have any vested interest in whether or not e-cigarettes work or not.
I think it helps that I'm both American and British, so I kind of can sit firmly on the fence in that regard,
and I would want the best available evidence out there.
And I have family members who smoke and with them I've encouraged them to switch to e-cigarettes.
And one of them said at a family dinner a few years ago, he said, you know, I debated switching,
but you scientists can't agree, so I'm just not going to want to keep on smoking.
I thought, this is terrible. We have to say more. You know, this is awful.
If I'm feeling like I can tell the people I care about that they can switch, why can't I say it a little bit more loudly?
So on a Light-hearted note, I wanted to talk about how I said that more loudly and to give a little bit of background.
Back in 2013, when we first started our review of e-cigarettes, 7% of the British public thought that e-cigarettes were more or equally harmful.
That's this orange bar here. Since then, all of the evidence that came out has very, very, very much supported e-cigarettes being safer.
Right? So all of the studies we have of biomarkers of acute effects, of long term effects,
all of them suggested that e-cigarettes were not completely safe but were much safer than smoking.
And yet that orange bar is going in exactly the wrong direction.
So from 7% in 2013 to 26% in 2019,
of the British public who think e-cigarettes are more or equally harmful then literally the most dangerous legal product we have.
Anything else that had the safety profile of an e-cigarette would not be legal.
So they are uniquely deadly. They kill one in two people who use that product as they are intended in the States.
Guns obviously are legal too, but here we're a little bit more sensible.
So we just have the cigarette problem in terms of things that kill people that are easily available to them.
So I've done lots of things. I've talked to journalists, I've written blogs about probably the most ridiculous thing I did.
Was this? This is performed throughout the streets of Oxford, covered down deep in hills.
There is a video to but you don't need to see. Don't want to graze if it's safe.
Brian Not so you might as well smoke anyway.
What your mate needs is a Cochrane review. All my friends have been checked at least twice.
They find there's a lot that the experts agree on.
I did get to. It's not a very good video, to be honest.
So you're not missing too much. Now a cigarette burns rather horribly, wreaking all sorts of snakes.
Cyanide. Which is why it will.
This may look like personal preference so gonna say.
Enforcing sections of. That's what the experts.
And you leave this little bit of a concern. Checking evidence from the two killers, Ken.
And thanks to many. Three have been around 1700 thousand.
Participants in this training say that they are.
That's out. Thank you, guys. And I'm here for questions.