So you're all here for results. Anybody do resuscitation in the room and yet we've got A&E.
So this is all about resuscitation, so you should help us out. It was a very, very busy keep coming in.
Well, this is a lot of still my thought process is I've been involved in evidence based health care now, evidence based medicine for 25 years.
And it's really interesting. I'm starting to really get get the hang of it after 25 years.
And so lots of this is just my thinking.
If you ask me about research methods, I could probably spend the whole week here actually talking to you and we'd still be.
I'll be interested. But actually this talk is about thinking about what is it.
The crux of evidence is, is about we think about evidence, high quality evidence and what it means for practice.
And at the moment, I've got to switch this on. Is it recording? So be careful.
If I say anything stupid, please let me know. If you say anything stupid, it'll be on record.
Okay. Alright, so it's not so many of you on the IBM course or think about going back.
It was Archie Cochrane who came along and said, Look,
we need a summary of the evidence of the critical evidence that underpinned health care and this is in the 1970s.
So it's about 50 years ago, people were looking at health care and going, look,
there's so much fundamentally wrong in how we practice lots of it's opinion based and we have to get some
way of summarising the evidence to improve and understand what we need to do and we need randomised trials.
Okay. Second, as many of you know, the theme into late men today with particularly in the debate is England best research evidence.
It wasn't just evidence, it was the best research evidence underpins evidence based medicine and the practice of health care.
And when David arrived in 1994, when I was here as a medical student, there was lots of activity there.
There was the birth of the Cochrane Library in 1993, Centre for Statistics in Medicine,
that with about 94 and the Centre for CBM that's the first ever website is 1990 495.
So lots happening in that time.
But I wanted to draw attention to this one particular, which was wrote by Doug Altman, who is still professor of statistics here,
who wrote an editorial in the BMJ in 1994 and offered, If you want to get a paper cited a lot, it's all about the title, isn't it?
So the scandal of poor medical research. But particularly we need less research.
Better research and research done for the right reasons. The really nice sentences in it.
That's what got thousands of citations on this paper. I think if you read it, it's really interesting.
What he's saying at the time is with almost like developing research evidence
with a hobby and it was particularly a hobby for many clinicians and so it
was so poorly done that actually nobody could actually tell what was good
research and bad research and very little of it with translated into practice.
But what I want you to draw your attention to now is to jump forward. Okay, let's jump forward.
If we say need less research, better research and research done for the right reasons,
if you go into PubMed and type in using the number of randomised trials, you can go back to 1995 and trace the increase in trials.
Okay. So it's been about a three fold increase in the number of randomised controlled trials.
So you could say well actually that's better research, we've had much better research done in that period of time, so we've done the good thing.
But you can also then say, well, let's look at observational research and do the same thing.
You can time the thing and look at the terms and then you can see actually
there's been a nearly four fold growth of observational research in that time.
So we've gone from about 100,000 observational studies when Douglas said there's a huge problem to actually now there's 400,000.
And in fact, it's more than that because that's 214 I can go through each year they get indexed and then you can pull them through.
And if you look at the whole of evidence, if you look at all the research in PubMed is actually a huge phenomenon.
Research, it's a huge industry, it's a huge business.
Over a million articles published each year and it's growing a percentage that's almost exponential.
It's going this way now. So the first thing is to say it's not difficult to get published.
All right. Anybody think it's difficult to get published in the room unless there's somebody with someone with somebody?
First article is always the hardest. But basically, after about the first five or ten, it becomes ridiculously easy and a million per year.
So actually there's a huge growth in research. Okay.
But if you take them 31,000 trials. Okay. In the statisticians in the room.
Well, we'll see. This is a normal distribution. Can I look at the stats people and they say, happy with that?
And if you seem like out of the 31,000 trials, if we say let's just say on average when you before you start, they make no difference to practice.
Okay. So we sum up all the trials positive and negative, and on average, they make no difference.
So if you take your normal distribution and you say, well, let's take two and a half percent out,
you would expect to see about 792 positive trials in that distribution.
Okay, everybody happy with that? Following that, you would also expect to see 792 negative trials.
If you take that distribution. Okay. However, you would think with all the money that we spend,
we have biomedical research centres that get hundreds of millions of pounds that we will be a bit better than on an average, no benefit.
Wouldn't you think we were a bit positive? Hello there.
Come to the right talk. Okay, very good.
We'd like to start again. All right. This is a difficult.
But you have to follow. All right. So if you expect the old and if there are any basic scientists in the room.
Okay. One person at the back put their hand up there. Hello, sir. On average, the research you do makes no difference.
Is that fair to say? I don't have anybody in the room like to claim.
On average, my research is slightly better than average or slightly worth.
What do you think? Because if you think about it, it should look like that.
Shouldn't. They should look on average, we should be positive.
And if you do that, you would be then able to say, well, actually, this is how many impact on clinical practice.
And actually out of the 31,000, it's 30%.
So it's ten, 15,000. So you can say, well, actually, this line is about 70% chance at the outset.
On average, 70% to see that follow. Okay.
So that's interesting when you think about it all with research that should translate into practice.
Well, if you do follow that, if you're in trouble in Oxford,
you should always talk to somebody who's one of your neighbours and say, I've got this issue, I'd like to talk to you.
So I went to talk to Rafael PEREIRA about this paper, about this issue, and he said, Oh, we've published on this already.
So there's always somebody smarter than you in the room you can turn to in Oxford.
So this is about the proportion of proposed new treatments that are successful.
Okay. And how often did new experimental treatments evaluated in trials superior to what established treatments?
That's what we really want to know. The sum of all that research, what difference does it make?
Well, when you do that, this is what they did. They looked at cohort studies.
This is their systematic review, which is to try and collect all the studies that have looked at this one question.
Consecutive series of randomised trials registered after before study onset and compared new against established treatments in humans.
To look at the effect and what they found is four cohorts of randomised trials that provided data from 743 athletes involving 297,744 patients.
Happy with that? Okay. And what did they find? They found that basically new treatments are only slightly superior to old treatments.
Okay. And that's really and you can go and read the paper and you can go into it.
And what's interesting, this is like over the last 50, 40, 50 years, it looks like the ability to provide new results is pretty stable over time.
So when you look at that and the other thing, it's to say that when you look at it,
it's also a normal distribution and that if you actually translate it,
what they say in the discussion is slightly more than half will prove to be better and slightly less than half will prove to be worse.
So this line is a little bit this way.
Yeah. So out of our 31,000 plus trials, you can expect about a thousand are positive.
Okay. So first. But when you think about that, that means, wow, aren't we pretty bad at doing research because about 30,000 trials are of no benefit.
So immediately we get rid of a huge wave of research.
So there must be some fundamental problem right at the outset to get to the stages of randomised trials to say so many fail.
So that's the first issue. However, track back a bit.
Okay. It's still not a bad number, isn't it? 1000 to 1500 are positive each year and should impact on clinical practice.
That's quite a lot if you think about it.
And that means as clinicians, we should be in this sort of almost all the time looking up new evidence going,
Oh, I'm going to change my practice this week. Some new evidence come. But how many clinicians in the room?
Other clinicians in the room? Okay. Clinicians beyond sort of first year health officers will probably change their practice daily.
Let me ask you a question. When was the last time you changed your practice? Of different antibiotic treatments for drug resistant.
Okay. And that within the last in the last month.
Okay. So we've got one of the one files and, um, the ones that are popular with OC and Loughrea is OC a chronic stomach acid,
poly dental extractions with all our water. Okay.
And how long with that when? Twice. It's past three months.
So you've done it in the last year. You've changed practice once. That's it.
So again, for me, having said that once in the last year.
Okay, yeah. When you have nobody ever do the survey but great, we had some ongoing survey.
I've changed my practice and to put it on the register, I've done somewhat different based on a bit of evidence.
So when we look at it, it's actually very small number. So they must be somewhat fundamental going wrong.
And generally that might be on evidence from three or four years ago, not from there, but in fact, as it comes out.
So our ability to take up evidence is delayed already.
So when you ask clinicians, actually, we don't translate evidence into practice very often do it.
So that's an interesting concept. So we've already got rid of about 95% of all research that needs resuscitation already.
So let's just focus on the 5%, okay? Maybe 3 to 5% that actually could make a difference.
Okay. But this is the issue.
The reason that bit doesn't translate is because there are significant problems in the evidence that exist within IBN.
But even is then 1500 trials a year.
We can forget the 95 if you if that thousand to 5000 we want to know about and see if we can translate them into practice.
Okay. When you think about why do so little,
this was my first and often my first thought of thinking about this issue is it's often you can just pull out a paper that says,
well, it's external validity to internal validity, and there's some problem with clinical significance.
And for many years I won't have said that to you. And then I think, what does that actually mean?
What do we mean by external validity should apply to the trial populations we see in practice?
The trials internal validity should be robust and have valid methods and clinical significance should mean it makes a difference.
Sounds good. You have to learn it and you have to do it. And then if it is not quite right and interesting, while all this going on,
even the CMO is so worried about the translation of evidence that she's come out now and
said they must be able to report public trust in the safety and effectiveness of medicines.
The CMO, Sally DAVIES, has come out and said We need to follow iterative report of what to do about the
evidence and how we should inform public because they don't trust us anymore.
So while we are, clinicians are developing the evidence and creating a lot of the problems and she the patients
are starting to see through the issues and are doing much of the work for us and saying,
hey, there's a problem, there's a problem, we don't believe you when you come about vaccines.
We don't believe you when you come about medications. We don't believe you when you say you shouldn't take your antibiotics because of resistance.
I don't believe you still want me antibiotics. The patients don't trust us.
So they're getting this message then the media doesn't trust us. So this is a really interesting paper.
I would put this on your reading list. This is a paper from 2001 from de facto it was started was when you go through lots of days articles.
He's wrote an article about once every 18 months, which is really interesting to read because he was obviously thinking a lot about these issues,
why randomised controlled trials fail but don't need to.
And this is a bit of a really interesting the only formula and we can take the word clinician.
Anybody ever needs to know about trials, okay, this is the only form and then I'm going to show and make it a bit simpler in my own thinking.
So this is the formula. Our confidence, which means how confident are we in the effect is directly proportional to the signal.
Divided by the noise times, the square root, the sample size.
Okay. Now, stay with me.
I'm going to translate a few words for you. Confidence in the balance of benefits and harms of a treatment.
Yeah, because it's about the benefits and the harms signal with the effect size.
And the noise is the amount of ice. Okay. Happy with it.
And I'm come at the square of the sample size.
I'm going to come back to believe that because it's it's I understand it, but it would take me 5 minutes to explain here.
And if you take 5 minutes, it's of no value because going to make it even simpler, hopefully.
All right. Read the paper. You will understand it. All right.
So I want to do that. Everybody happy with that for now? Careful.
I might start asking you about it in a minute if you've memorised it.
Right now, when I realise a lot of the things we write in, I can see then a lot of things we're writing is in there.
So when we wrote this editorial with Ben Goldacre about how medicine is broken and how we can fix it.
What I realised is a lot of the time we're mentioning the biases.
That's what we do it. Yeah. And because it's, it's proportional to one over the bias, it's a bias that increases.
You can see you have to have a bigger effect size to overcome them both to see that it's really simple to think about.
So if you want to have an effect based on an observational study, you should have a rather large effect, though, shouldn't you?
Because we know observational studies are biased compared to randomised trials.
If you have defects in the randomised trial you should have larger effect sizes and it's a simple way of thinking what you're starting to look at.
Okay. So I know a lot about their thinking that actually there is these systemic problems, conflicts of interest,
entrenched problems in Ibn Amr in deep trouble because they are becoming so problematic.
What do we do about it? So it's trying to provide a system so people can think simply about what to do.
Okay, now here's how I rewrote it.
Okay. So patient benefit is proportional to the outcome.
Divided by the amount of both. And multiply by what we call the optimal information site.
Okay. Optimal information size means you've got to the point where you've got enough information to answer the question and you can stop.
Because what happens now is people say, we've got big data, don't it?
And our numbers are so big that actually it's going to overwhelm this.
They're not going to care about that. Patient benefit equals big data.
Big data equals patient benefit. But actually, in any scenario, you can calculate the optimal information size where you can say,
based on what we'd expect to see, this is how much data we like to see in our randomised trials, in our systematic reviews.
And at this point if we haven't developed any certainty, we can reject the treatment and so it's not worth carrying on.
Or sometimes you can stop and say, actually, this is beneficial. We don't need another trial.
We can stop. Now, that's an important concept that nobody's ever talked about and talked about very little.
I have a Ph.D. student about to start to publish on this because it's really important
how many treatments reach the threshold of optimal information where you can say,
we're happy to be certain about this treatment. Okay, so what you want to do with these three phenomenons is you want to optimise the outcomes.
You want to minimise the bias and you want to achieve the optimal information size.
That's it, really, for any treatment, that's what you want to achieve.
And if you achieve that and you say we've optimised the outcome, it's an important outcome.
There are minimised biases and we've achieved the optimal information size. We don't need guidelines.
We should get on with it. Okay. So a very good example of a treatment that might meet that is aspirin in MRI.
The outcomes important because it's death. It's quite a big outcome.
At one month, it's about 4% absolute reduction in mortality, three or four somewhere around there, 3%, 4%.
The bias is a minimise because we've got 200 different randomised trials that it's been replicated in
and then 200 trials have been put into an IPD collaboration to achieve the optimal information size.
You don't need a guideline. 99% of people on aspirin get aspirin at heart attack.
And clinicians will respond when the evidence meets this criteria.
The problem is often evidence doesn't meet this criteria. And we're in an uncertain world where the outcomes are meaningless.
The biases are quite high and we haven't quite got enough information.
And we're going to end up with a guideline. Okay. And people hate guidelines.
Right. Now, let's just take one of them. I can't give you more, but we could.
And I can tell you what we're going to do about this at the end.
So let's just think about outcomes. How. Optimising outcomes.
Because this made me think about this. How do you optimise outcomes? What do we mean? But actually there are major problems in outcomes.
When you think about it and it's really interesting and anybody think so, if I said to you,
I've done a new randomised trial and I'm about to publish my outcomes and here they are.
Can anybody think of one type of problem that might occur with the outcomes?
I think. Yeah. So actually that's a really one.
They're of no value to patients, aren't they? You look at anything that's not important to me.
Why do I care? That's a really important one. That's probably number one.
To measure it. You didn't measure it, right? Yep.
That's one of them. But you didn't make that up. You didn't measure it at all.
Okay. You didn't measure sugar, right? You didn't measure it at the right time.
For some people, it doesn't work. For others, and we don't know. We have a mess there.
You have a mess. Okay. I'm happy with the word mess. How significant?
Okay. So how big is the outcome? You don't quite understand how big it is.
I didn't link the patient to patient outcomes from the beginning.
He didn't link the patient benefit with outcomes that beginning. Yeah, that's fair.
So when it 40 down there and you think in the context of this I'm forcing you to start to think about now is what does he mean by important outcomes?
Valid outcomes are both problems in outcomes what they're actually they're just outcomes and not outcomes.
Yeah. Now think about it. We've got a world in millions of people involved in health care, in research, in epidemiology,
and nobody really thinks about this and they plan whole research budgets around it.
So this is what we've been thinking about. When you look at his recent ones, his one about problems of surrogate outcomes.
Okay. You understand what surrogates are. Surrogates are an outcome on the real disease of interest.
So on the way to cancer, we may look at disease progression.
And often people would like to assume that they actually do predict the outcome of interest, but often they don't and they're of limited value.
And here is this is based on Rita Freedberg saying that basically, first,
the FDA should promptly withdraw approval for cancer drugs that are proven to be to have no
clinical benefit because they're all suddenly approved on the basis of surrogate outcomes.
That's one problem. Here's another one. Here's what happened is there's a whole research agenda from a priority setting exercise.
How people people involved in and research. Number three in the Delphi of the priority exercise was here you go so consensus was
achieved among CTU among clinical trial unit directors across the whole of the UK.
Number one, with research into methods to use recruitment into trials, methods to minimise attrition, a choosing appropriate outcomes to measure.
So in 2006 did all the clinical trials.
People are telling you that we need research in how to achieve an appropriate measure.
We have we don't really know what we do. Huge issues and huge problem.
There's more of this stuff. Here's the surrogate victim on tour.
John YUTKIN Telling you in about diabetes, how things like HBA one, C home and fasting glucose sort of not what you're interested in.
What you really interested in is important vascular outcomes.
You get more of them composite endpoints. These are fantastic way of of of of not optimising outcomes because what you do is you
combine a number of different endpoints to try and get the optimal information size.
That's what you're trying to do when you do that.
But when you do that, you're often combining endpoints of different severity which are meaningless and then are of not value to pensions.
There's more when you keep going.
Did deviation from intention to treat analysis in randomised trials and treatment effective at the meta epidemiological study?
I love the word and it's like trials that deviated from the IPT showed larger intervention effects in trials that reported the standard approach.
So you have these biases that affect the outcome don't really make a difference and it keeps going.
The outcome selection and role of patient reported outcomes in cardiovascular treatment trials.
Down here, there's many trials in which patient reported outcomes would have been important of crucial or crucial for clinical decision making.
Did not report such outcomes. 122 of 174 trials.
70% of trials did not report.
The critical patient outcome shows you got a huge problem.
So we got all this research which is fundamentally distorted just because of the quality of the outcome, and you can keep going.
And then this one selective reporting bias of harming outcomes to maximise the benefit.
You don't report the beneficial outcomes and you forget to report the harms.
And so you got a problem of publication bias and a problem of reporting bias when it comes to outcomes.
Somebody said missed, you didn't collect them. We may have collected them, but then you decided not to report them.
Okay, huge problem. And you can do that in two ways.
You can say, I'm not publishing the paper, or you can say I'm publishing the paper and I'm take them out.
And do you know what? Nobody can do anything about that whatsoever.
So putting all together, I thought, wow, I didn't realise when I start thinking about their all these harm outcomes.
So this is something that we've just submitted to trials and it's come back with Ben Goldacre and Kemal Mouton.
Thinking about trial outcomes.
Surely there's a better way of putting all these together and because they're difficult to interpret and they do have a huge impact.
But what will happen happens is the articles are one over here, one over here and one over here,
and nobody's connecting the dots in terms of all the problems. This is this is for all the researchers.
This is how I go about it when I'm writing an article. It's this is a big Post-it note and these are small Post-it notes.
And I stick them on my wall and I write in Post-it notes and then I move them around.
And you can see this is what it starts out. And this is what you're doing to try to change the title of of an editorial or commentary.
So that's one of you involved in thinking about ideas. This is some of the ways.
And that that is that table fit.
Okay. Patients in co-design poorly chosen methods poorly collected equator concerned
thinking about them selective reported I sigma that it is obviously inappropriate.
So that translates into that further analysis and it is like magic. But what it shows you is there are four fundamental issue.
That's what we think at the design outcome, the badly chosen.
At the method studies that Bradley collected at the publication, they selectively reported and then interpreted.
We all do this a bit inappropriately or interpreted, so you can pick one area of this.
And so here's the problem here at the design. This is a particular aspect.
At the outset, you really are designing this in a way to maximise your chance of having a positive outcome.
Yeah. And that's where both with in lack of relevance to patient and decision make of it in the you are collecting them badly.
Well missing data, poorly specified outcomes is a really important issue.
You may think you're sometimes looking at an outcome like pneumonia. And when you look actually in the in the protocol, it's not actually pneumonia.
You wouldn't define it. That publication, selective reporting, publication bias reporting by underreporting of adverse outcomes and switched outcomes.
And then finally, relative measures, spin, multiplicity.
You can just publish as many outcomes as you want and at some point you'll have one positive one, won't you, by chance, alone?
Yeah, I'd be happy with that. If you have a if you're the slave to the p value, you're going to get a problem.
And then this lack of core out concepts makes a huge problem for us in evidence synthesis,
because often you're looking at seven trials with seven different outcomes and nobody can bring anything together than what you have that.
But what do you think? So we'll talk some more. He's still up for tomorrow.
All right, good. We can leave now if you want.
But if you've really everybody said at the back, somebody said, how can we?
So this is when I go to even commissioning groups, the MSI, I'd start for the minimal, clinically important difference.
At what point would you accept this difference to make it make a decision to treat somebody or not?
And it's really interesting if you, I suppose to hear some of that statistically significant take it or leave it,
what how much difference do you want to change? Practice. Before we stop him, before we look at evidence, what's the minimum?
And then with clinicians, we will do that. We will treat people.
I will treat people tonight when they may have a 50% chance of pneumonia or a 30% chance.
Knowing that actually if I leave them, the utilities, they may go on to develop pneumonia and die.
So actually I'm going to treat you at 30% jump. It's alright by me. Otherwise to get 200, then I've got standing to hospital.
They've got to have the x ray, got to have the gold standard cost of thousands.
I'm going to treat people. But when it comes to important differences in treatment, nobody has a clue.
Wherever have been and said right and left treatment you used when you decided the ones that we decided.
What was it in terms of the difference between treating somebody,
not treating that made a difference to you saying going to do this as opposed to know what
is it you as a group of doctors deciding together to say this benefit is so significant,
we are going to change? What we're looking for is a 1% reduction in mortality with this new treatment, 0.1%, 0.01%.
And whatever ago, nobody actually could pick trends up and go. Yeah, we thought about that.
We really did for it. And when people understand the cost of new treatments, you actually sometimes they're adding in cost for no benefit whatsoever.
So that's the IDs. But when you go and ask the same patients and say, well, this is what the doctors think.
They actually want really important differences, which are about 3 to 4 times larger than what the doctors will say is important to them.
So patients want and this is in a group of rheumatologic patients that say values are 3 to 4 times greater than the MCI ID values.
The patients want more than what we would accept.
So actually what we do do is often stop research too quickly without optimising the intervention and thinking, where do we want to get to?
Huge issue. And that should be really important in people developing new services, commissioning, thinking What's our target?
What do we want to achieve? How much difference do we want to make?
And I can tell you, I've never been with a clinical commissioning group who thinks like that yet ever.
The only two people have seen that is the W.H.O. So it's interesting.
They go to the lowest resource settings in the world.
It's almost you've got to get to the place where they've got nothing to then get somebody to set an NCD and they've got 2025,
oh, 2020 so they want a 20% reduction in cardiovascular disease by 2020.
Yeah. And they're the one. The second group I've noticed is Cancer Research UK is setting themselves a target by 2034.
They want freedom for people to be free of cancer, don't they will be cured from cancer and they've tied it to 24.
Quite smart when you think about it, but at least have set themselves a clinically important difference.
I think you could answer and I go and talk to people, right? If I go into this, anybody in the room tell me in any of their settings,
what's a clinically important difference that your organisation is working through right now?
Anybody? 20% improvement.
So I can actually see that it happened. Yeah.
So you might say, well, if your organisation talking to you or you were the practice going, this is what we're working to.
This is our, this is what would be important to us. So that said, no, you might think something don't so but then talk to you.
But no organisation tells me or can talk to me and say this is what's important to
do and what we've got is a problem in the distortion of what's important to us.
Right now. It seems to be whether you wait half an hour or 40 minutes, not whether you get a cure or treatment.
So we get distortions in what's important to us because. No, no, no organisation is that what's important to us?
Okay. So much of what we do in is an example is compare.
We work towards trying to fix some of the solution. And so this has been a project led by Ben Goldacre that's included medical students.
And what we tried to do is there's been this long term problem of outcome switching in randomised trials.
That was one of my problems in the outcomes.
You what you do is you see in the systematic review, 31% of trials have had discrepancies between the pre-specified in reported primary outcomes.
We had ten trials just basically say here's our primary outcome, but actually it's now different.
When we published the paper, we switched it and you can switch from the one with negative or inconclusive to one that's positive.
Well, that for me, things are ridiculous scandal, isn't it? That's what I feel about it.
And that's what we all feel about it. And 15%, about one in ten, one in 15 introduces some outcome that they didn't even pre-specified.
So one of the things we did with try and repeat this, but actually having a third feature with the see, could you correct science?
Can the journals correct themselves? And these are the outcome report.
So 67 trials checked, nine were perfect fringing 54 outcomes not reported infringed and 57 were silently added.
So you can basically pull out and put in. You can basically say this is a mess, 58%.
This was quite a big operation because that meant we had to check them all to students,
would check them independently, and then they get one of the senior people like myself to go,
Oh yeah, let's go over this, because we've got to do it within the window of a journal, which again seems absolutely ridiculous.
So some of them are two weeks, a maximum four weeks. If you send after that, they'll send you a letter.
In fact, you very much time the window. So you've got to get really organised to correct sign.
You can't come a month later and go, there's a problem. They'll just reject the letter apart from one journal.
Okay, so we wrote to these all them and here's some of the responses.
So the BMJ is interested. They didn't have many trials, but they did have some with small problems and some of them led to correction.
Okay. But I know that we wrote to no correction.
That seems to be up to the journal editor. Same problem.
Same issues when we correct. When we didn't. Different editors of The Lancet, they published.
Many of them took over 160 days for some of them to be published and they're all in the off with replies.
Got them to reply and no comment from the editors on the paper saying there's a correction problem here, just a letter.
Some hundred and 50 days later, JAMA rejected all the letters.
Not interested whatsoever. Too vague. And if you look to the there were only 250 words that we couldn't be that vague in
250 words because there's a word limit on them and repetition between letters.
So one of the things we did is to say, look, we can't make any mistakes, can we?
We would have a call letter that would say this paper insert name has four primary outcomes, three of which were.
So it was a standard letter so we couldn't make her. So that was what they were saying about repetition.
Annals of Internal Medicine went to war in some way, started writing editorials,
were pointing out all sorts of things in their journal that they could write.
So you see, they are the editors to the editor. And in response, they were getting as much as they wanted to respond.
So they wrote this whole diatribe thing for one trial.
So this is the editors one trial compare apparently considered the protocol published well after data collection ended.
Okay. However, they did not consider the protocol published two years before Matt Firth, an associate primary trial, was published.
So that's what they say. But what this protocol was published one year after the trial, it started.
Okay. You can see that the problem in research, we go to the registry, the rules.
Where you go to the registry, I take that as gold standard.
Unless there's a protocol published prior and we look at the protocol and say, when is it published?
And that's what we do. Okay.
Protocols were available for none of the 12. Yeah.
And that's the statement. They go, okay. And that's the reply we got when we email.
Outrageous when you think about what's going on in science, isn't it? So we have a reproducible statement.
This is available from Dr. Edmondson. If you email Dr. Everson, that's the response you get.
Try out if you're interested to see what happens.
New England Journal of Medicine. Fantastic.
Any introverted reader can compare the published article, the trial registration,
the protocol with the reported results to view discrepancies themselves.
It took us up to 4 hours for two people on the scene in person to do it.
The quickest we did it within one hour. If you want to go down that route, it'll take you somewhere between one and 4 hours per paper.
Okay, so that's where we are. That's an example of poorly reported outcomes of showing you that actually there are four fundamental features chosen,
collected, reported and interpreted about outcome. There are so many fundamental problems.
The next time you think about an outcome and you see a research outcome.
Just step back and think. I wonder how they were chosen one day, how they were collected, have they been reported and how am I interpreting it?
And particularly think about it in terms of a clinically important difference.
And part of that is what's led to all of this evidence log manifesto, which is got lots about certain outcomes in them,
and resuscitate in the deluge of poor quality evidence, which I'd if you want to leave a response.
Think about what we've said tonight. Any issues that would be great are probably quote you in some way.
But interestingly, however, everything I said is now up in the air because of the way the modern world is working in terms of fabrication,
research, fraud and trial. And this is the newly established Chinese FDA equivalent.
80% of data has been made up or fabricated in Chinese drug trials.
And that's because the overwhelming market forces are about people trying to make money.
And when we've got such issues with so much money is even whatever we do, that's still going to be the need for vigilance.
Thank you very much.