Well. Good morning, everybody. I'm Amanda.
And we're going to talk about making sense of results today.
I've discovered that the microphones only work for the cameras, so I'm going to shout because there's no generalised microphone.
So if you can't hear me at certain point, just sort of way even I'll speak up.
The other thing that I tend to do is I tend to speak a little bit fast when I get excited.
So if I start going too fast, especially for people who for whom English isn't your mother tongue, just look and I'll slow down, okay?
But you'll need to pointed out to me because I do get carried away.
This session is about making sense of results, and I've got a little cartoon for you to begin with, and this teacher's there.
And he says, Welcome to the basic astronomy. Before we start, are there any questions?
And he says yes. What makes astronomy different from astrology?
It's a lots and lots of maths. And they managed to lose the entire audience.
Actually, people are quite numerous phobic.
How many people here don't like numbers very much or are scared of numbers in papers?
Your your stronger group the most. I most. I sometimes bring calculators along and I sort of see people's white faces as I bought this.
This is a secession about statistics. It's not statistics, really.
It's not about numbers. It's not about maths, sport, common sense and concepts.
And I hope that that's going to come across today.
In this session, we're going to do the following. We're going to look at how measures of effect are reported.
We're going to look a bit about P values, then look at confidence intervals.
We're going to look at what we mean by statistical significance and test the statistical significance.
We're going to think about why we need systematic reviews and we're going to look at how to interpret global graphs.
Okay, so what's the blowback from. Oh, yeah, we're going to have fun.
And I hope so. And you're welcome to interrupt me at any point in time.
And I mean it. So here's a global crime, also known as a forest plot, also known as a Metro Analysis Graphed.
Now look at this block program and it is looking at if people have had a severe head injury,
how can we reduce their risk of death or long term morbidity?
Well, one idea is that if you call people right down, it will reduce the brain damage.
So this is a block diagram of the results of four trials of cooling people down and looking at how many die or severely incapacitated.
I want you to write down what you think the result of this systematic review showed.
This is taken from the Cochrane Library. I'm just going to give you 5 seconds to write that down.
What does this show? Okay.
Believe it or not, that was 5 seconds and quite a few of you didn't write anything down.
And I don't know whether that was because this was such a difficult question.
If I'm like God, which is jumping right in with the club gram and ask us to interpret one,
or whether it was just because it was so obvious you didn't have to write anything down.
Well, this session is for the people who thought, Oh, my goodness, what's she doing?
I do not know this. By the end of this session, when you see a graph like that, you'll be able to interpret it immediately.
That's that's the aim. That's how you know whether this session was worthwhile.
Okay. Let's go ahead.
So, as I say, you're welcome to ask questions.
There is no question that you can ask me. That's too stupid.
It's I'm serious. This is a safe environment. This is where you can learn the things that you need to know for the future.
And nobody's going to make fun of you because you don't know it.
This is. This is. You can ask the question. You're safe. We're going to look after each other.
Okay. But before we start,
we're just going to recap what are the important things you need to think about when you're using research evidence to inform your decisions?
Shout it out anyway. Is it randomised trial and is it relevant to your question to sorry.
Relative to reason? Is it valid? Do we always ask is it is it randomised?
Do we use any evidence? Not randomised. We have a want evidence.
That's not randomised. Depends on your question.
Depends on our question. If we want to know how women felt about a certain procedure, we go and ask them.
We do a qualitative study. So we don't necessarily want it randomised, but we do if it's an intervention study.
So for any study, whether it's a randomised controlled trial, whether it's a systematic review,
whether it's a qualitative study, economic evaluation, cohort study, whatever study we always want to know is it valid?
Somebody said only that if it is valid, what do the results tell us?
And then is it relevant to our decision? So the three things that you would ask of any study validity is can the results be trusted?
Results? Well, what are they? You know how people express them.
Could they be expressed in a different or more meaningful way? What do they actually mean?
And then relevance. Is this relevant to my particular patient or in the context where I have to make that particular decision?
So just because it's first thing in the morning, I'm just going to give you a little warming up exercise just to to stretch you a bit.
All on the validity for an intervention study. What sort of study design do we want for an intervention?
A randomised controlled trial and you have got 30 seconds to write down all the validity
questions or discuss with your neighbour what you do for a randomised controlled trial.
What makes a randomised controlled trial valid? You've only got 20 seconds left.
Okay. You've got 30 seconds. That validity questions.
You might not have had time to come up with them all, but perhaps between us we have.
Well, what questions are you going to ask randomised.
Well, was it randomised? Was it an adequate sample size which is what it matched.
Gold sample sort of. Okay. So were the groups comparable with baseline characteristics similar.
So in other words, the recruitment action people.
Okay. Yes. So and for that allocation to be okay, what do we want?
We want randomisation, but we want we want allocation, concealment.
Then we want blinding. Anything else we want the to maintain.
Sorry. Limiting. Yes. Where they kept the same way.
So were both groups treated equally? Anything else we'd like to see?
Adequate numbers. Adequate numbers? So that was sample size calculation.
And you want it analysed properly, so in intention to treat analysis.
So here you are going from the top. It's all about getting those groups to say was it randomised?
Was the allocation concealed? Did it work? Did you have similar baseline characteristics?
Was it blinded, treated the groups the same minimal losses to follow up and whether the same in both arms?
And did they do an intention to treat analysis? Nobody came up with that in 30 seconds.
Well, at least as a group we did.
The reason I've gone doing about results and I've gone back to validity is because results only have meaning if the study is valid.
If you're appraising, you're reading a trial and it's not been done validly, you know it.
You can't interpret the results.
So I'm going to give you a warning that everything I say now about what results mean, it's assuming that the study's been done in a valid way.
Okay. Most critical appraisal. That way you throw out the paper, it'll be long before you get to look at the results.
It's because it will just not be a robust paper. So everything I say from now onwards assumes the study was done validly.
So let's we've got a valid study. Most useful studies, cohort studies, case control studies.
Most useful quantitative studies. Compare two alternatives, at least alternatives.
You know, did the person have this treatment or did they have the control they exposed to this risk factor or were they not exposed to it?
Did they get a disease compared to the people who didn't get the disease?
So how can the results of such comparisons be expressed?
Glass, then put them in graphs. And what might you just show in the graph?
Relative risk, you might share the relative risk. So you could have a graph where you just share the proportion who got a certain
outcome in one arm and the proportion another in a sort of a bar chart.
Or you could actually do some kind of calculation to show the relative risk.
Some of these words, like relative risk, might not even be familiar to you, but the concepts are actually common sense.
So what we're going to do is we're going to do a little exercise. We're going to look at a randomised controlled trial.
This is one I plucked out of the air. It's just a nice easy numbers.
And we're looking at whether orthopaedic mattresses are better than normal mattresses for preventing backache.
Okay. And we've randomised 100 people to receive a firm orthopaedic mattress and 100 to receive a normal mattress,
a medium mattress and after three months we measure the backache in the groups.
80% got rid of that backache in the orthopaedic mattress group and 20 in the medium mattress group.
Right. I'm going to give you just one minute to summarise this result with your neighbour.
So talk about all the ways you can summarise that result.
She that? Right.
You know. Again.
Yes. Okay.
My friend Darci is going to help me. Are you Assam? Yes, I am.
Do you want to tell us how you summarise those results?
Well, I think for the person that takes the relative risk is 80% and for the early 1980s the minimum tax is 20%.
So the difference is 60% can see that for seven days.
Four out of five will benefit for that.
One out of five would benefit and will get better.
That means there is the risk is going to be four times for medium of matches.
Then further matters extra.
I make it too complicated. Perfect. I rarely get the full answer on the very first go.
Excellent. Okay. So exactly what?
As I'm says, 80 out of 100 get better in the four mattress group.
So we say 80% get better. Strangely enough, although it's getting better, that's what's known as a risk.
What's the risk of getting better in the mattress group is 80%.
It's a funny, funny word for English speakers to use the word risk like that, but that's what we do.
What's the risk of getting better? 80%. 20 out of a hundred.
What's the risk of getting better in the medium mattress group?
20%. So once you know that we're calling the probability of getting better risk,
the relative risk becomes how likely is you to get better in one group compared to the other, you know, the risk relative of one group to another.
So the relative risk of getting better is four.
You're four times as likely to get better with a firm mattress.
So when people talk about relative risk, they're talking about dividing one risk by another 80 divided by 20.
That's what relative risk means. Adam also did something else.
He said if 80% get better in the orthopaedic mattress group and only 20% get better in the regular mattress group, an extra 60% are getting better.
So instead of dividing one by another, he took 80 and he took 20 away from it.
And that's called the risk difference. An extra 60% of people get better with a mattress.
That's the risk difference. The only thing he missed was the number needed to treat.
This is an interesting concept. For every 1.7 people with back pain, given a firm instead of meeting one case, one case of back pain is improved.
That is the number needed to treat. I've got the definition on the next page.
The number needed to treat is the number of people who would have to be given an intervention
in this case of a mattress in order for a one extra case of back pain to be given.
So in this case, we give two people a mattress for a mattress instead of a medium.
Mattress will get a cure, a case of back pain. That's the number.
How did we get to that number? Needed to treat. Let's just go back.
Well, it's very easy. If a hundred people are treated with a mattress, an extra 60 get better.
Divide 100 by 60 and you get 1.7.
So it's how many to get one extra person better.
So 100%, 100 people. Extra 60% get better under divide by 61.7.
Excuse me. Actually, I will do passionate questions.
And we understand the last bit, which is about the 1.7.
Okay, I understand.
Do you understand the concept number needed to treat? Okay.
So, for example, if I give somebody an anaesthetic and they become unconscious and it happens every time I give somebody in on a static,
how many people would I have to give an anaesthetic to to make them unconscious?
Happens absolutely every time. I just have to give it to one person.
And they become unconscious, wouldn't they? So in that that would be the number needed to treat number of people.
I need to give an anaesthetic to. To make them unconscious is one.
Just give it to one person. They become unconscious. In this case, not everybody given a firm mattress got better.
Only 80 got better out of the hundred, so the number needed to treat can't be one and 20 would have got better anyway.
20 because we got in the regular mattress group, 20 got better anyway.
So we look at the risk difference. How many extra people got better?
Will 60%. 60% extra people got better.
Do you get at the risk different? You took them away from Rachel. Okay.
So if 60% of people extra get better.
For every hundred people we treat, how many extra people get better?
We treat 100 people. How many extra people will get better?
60. So the question is, how many people do we have to treat or one extra person to get better?
100 divided by 60. Yeah.
That's the number needed to trade. Now, if you're mathematical and you like formula, you can think of that as one over the risk difference.
But you don't have to think over the absolute risk difference. But formula sometimes confused people more than help.
Isn't it easier to see for only five patients?
For those who get better as compared to what you can.
And that is a good way of saying it.
It makes more sense to a lot of patients because if you told patients their risk is 0.3, they think what they mean.
It was a 30% risk. I can't understand that.
What you mean it's either going to happen or it's not. But if you tell them, well, for every three patients like you, one will get better.
They can understand that.
So it is people for if you're talking to patients, it's much better to keep things in natural frequency so they understand it a lot better.
Okay. But there are already two basic ways to summarise risk.
You either divide them or you take them away.
So you either divide things. You get a relative risk, you take them away.
And you get a risk difference. And. There are other things, ways of expressing risk like odds ratio.
Do you think that's a dividing or taking away way? There's a clue.
There's a clue in the title ratio actually means of dividing.
So it's dividing relative risk by dividing hazard ratios are dividing or taking away risk is dividing risk.
So any two things I want you know that life becomes very easy because you don't have to know how to cause working to be able to drive it.
You don't have to know the statistics to be able to interpret them.
All you need to know is whether someone is getting you are dividing or taking away risk.
So this diagram the slope of global Graham has as its basic structure.
This inverted t was the line down the middle on that inverted t.
When you look at those diagrams, what's that line down the centre is neutral.
No effect one. No, no, I'm I'm saying somebody saying it's no effect.
Somebody saying it's one. Any other options and the other there is no difference between the two.
You're too clever, Adam. He's just too clever, isn't he? It's actually not necessarily no effect.
And it's it's no difference between the two things you're comparing.
Obviously, if you're comparing placebo or or do nothing, then it is no effect.
But it's it's the line of no difference between the two things that you're comparing.
Somebody said it was one.
Is it one who thinks it's one, a number along the bottom who thinks it's one one person thinks it's one, two people think it's three, four, five, six.
We're getting more. Who thinks it's something else? Okay.
We've got quite a few votes for one and a lot of people not. But it depends.
It often is one, so I'm for it to be one.
It's got to be a divorce. It's got to be a dividing risk. So if you got 12 and you divide it by 12, what do you get?
Well, so if you're getting the same in one arm and the same in the other arm, you'll only get one.
If you get 43.9 and you divide it by 43.9.
What you get. So that what happens on that line?
It depends on whether you're looking at the relative risk or risk ratio, an odds ratio, hazard ratio, or whether you're looking at a risk difference.
Regardless of that to the left is always less be less than one.
If it's a ratio under the right, it's always more so.
If it's a dividing, it's a ratio, then the number there is one.
However, what, 43.9 -43.90.
So if it's a if it's if you're showing a risk difference and it's the same in both arms, you're going to get aa0 away.
So if you take away, you get a zero there. But it's still less on this side or more on that.
So it doesn't really matter. Okay.
I'm now going to do a randomised controlled trial.
I'm interested in backache and I looked on the web for cures and I actually came across this genuine herbal remedy for backache.
Okay, Potters. So I thought, well, I wonder if it works.
I'm being an evidence based person that I am. I couldn't find any trials on Medline for for Potter, so I decided to conduct my own trial.
I didn't get funding from the pharmaceutical industry or Potter's, so it's quite a small trial, but it was superb.
There was no bias whatsoever. I had the randomisation done centrally by a friend is absolutely superb trial.
If it's all the criteria for being biased.
Three five people ended up being randomised to get plus with Potter's and five people got absolutely identical and
indistinguishable placebo and four out of five people got that with Potter's and two out of five got better with placebo.
First of all I wanted to do before we even go on,
I want you to tell me what the relative risk of getting better with Potter's was and what the risk differences.
Before we even even start thinking about these results.
I said, Why don't you give the microphone to somebody else who's brave enough to have a go at answering this?
No, not brave enough. Pass it backwards. Now it's going to the cameras.
It's going to the cameras. Somebody who's brave enough to answer this and wants to appear on film.
No. Nathan, budding film stars up there.
That's a bit scary. It's possible.
Actor Paul Glacier. He must be brave enough to appear on camera.
Paul, what did you make the relative risk of getting better to be a we're not switched on.
Most of you are twice as likely to get hot as did everyone get that?
Twice as likely to get better with potters than with. Yeah.
Yeah. And you didn't dare say that on come.
You're so insecure. What was the difference?
What motivated you to exit your people out of five of their policies to two out of five, which is 40%?
Okay, so two out of five, 40% got better at a party.
So with a 40% extra get better, how many if we treated 100 people, how many extra people would get better?
1440. So for every hundred people we treat, 40 extra will get better.
So what's the number needed to treat? Two point something.
Yeah, 2.0. Somebody's good at maths over that.
Okay. So this is you can put it in a two by two table like that if you want.
And here we have for this, this is how on these global grams you get this, here's the study, this is my little study I did.
The next line will give you the intervention. The next line gives you the control and you get a little fraction like that.
Now that little fraction, it's telling you the first bit, the numerator is how many people got better.
The denominator, the bit underneath is how many people were in that arm.
So here's our study. Four out of five got better in Potter's and two out of five got better in placebo.
And there we are. It's Paul gave us the relative risk so that we know the number at the bottom is one and there's our blob above two.
That's our point estimate. Unfortunately, I don't think my little later works, but there you are.
You're there's your blob on your blood program. So that's how a blob of ground which is giving that the point.
Best guess of how good it is. It's twice. Twice as likely to get better with Potter's.
Yeah. So I immediately went out and invested in the potters industry and I now want to sell you all potters.
Are you going to buy this? Do you believe the Potters Q is back?
This is an unbiased study. We need to look at the chemical industry and the baseline characteristics.
Did my randomisation work? The application was just one test, only to be clear, and participants didn't have the characteristics.
In addition to finding the most of them, were they?
They had low chronic back pain of no particular cause.
And and we did seem to get a good randomisation only.
Yes. The numbers aren't significant. There must be something we you have a very small population to really come to the
conclusion that the number that significant difference makes complete sense.
So you're telling me you don't believe it? Not because the study is biased, but because it could have happened by chance.
Is that what you're telling me? Yes. Okay.
Every time you see a result and somebody comes up, the pharmaceutical rep comes again, look at this, look at this marvellous thing.
You stop and you ask yourself, Could this be due to bias?
That's the first thing I always ask myself. That study was damned nuts bursting.
Is the study valid? Is there something in the way they conducted the study that would give them the results?
The first thing I ask myself is, is it valid? Is the bias in there?
The next thing I want to know is, could this have just happened by chance?
And only if I can't find bias a chance do I start thinking, Oh, maybe there's something here.
Okay. So we don't believe it. It could have happened by chance.
So we're. We're very sceptical. Mm hmm. So.
I decide people aren't going to you know, I've got all these shares in Potter, so I've got to prove that it works.
So I do another superb study, totally bias free, and this time I randomise 2000 people.
Okay, I got exactly the same results. Twice as many people are getting better in the potter's arm as are getting better in the placebo arm.
Now, if this was unbiased, let's say I hadn't bought shares in the company, would you believe this result?
Yes. How many people would still be sceptical?
Right. Most people believe this. Let's hear from the sceptics.
Why aren't you going to believe this result? You need to.
Right. So you're saying you might believe this result, but you can't tell just from these numbers.
What about other people? I would like some confidence intervals.
Okay. So you're saying the same as in. Yeah, that's looking good.
I need to know how much uncertainty surrounding that.
Okay. The think is that. How do you assess getting better.
Ah, well, it's a it is a subjective outcome.
It is a subjective outcome. But they were blinded. Everyone was blinded in the assessment.
Even the assessor that it was patient reported.
But the the two tablets were indistinguishable. And we asked them to guess which they were on and they couldn't.
So we do think this is a valid trial. Okay.
Some people are going to believe this because their intuition is that the uncertainty surrounding that estimate is very low.
Some people want a little bit more information just going on, intuition alone.
We're going to we're going to back and have a bit of fun here just to wake you up.
I'm going to ask you, how many people you would want, would you guess you'd need in each arm before you're going to believe the result?
Okay. And we're going to do this by the secret ballot.
I want you all to tear off a piece of paper. A blank piece of paper.
And I want you to write on it how many people you would guess intuitively you would need in each arm before you begin to believe the results?
Before you got those results? Yeah, before you believe it.
And you do a trial. Yeah. And you're going to 80% to getting bedroom with Potter's 40% getting better with placebo.
In other words, you've got a relative risk of getting better of two. How many people would you guess?
You're not to be right, because the secret ballot. No one's ever going to know what you wrote.
Write your number on a piece of paper. Fold it in half and half again, and then swap it with someone.
Don't let go of your piece of paper till you've got one back from them.
Otherwise somebody will end up with two and somebody will end up or more.
Keep swapping it until you don't know whose piece of paper you've got.
You know, any any number you want, the number you sort of think, well, about 30, about 3000, whatever do you feel in your belly?
What do you think?
Yeah. Okay, just. Just. If you have followed the instructions correctly.
Everyone should have one piece of paper. No more in front of them.
Open the piece of paper in front of you. And this is why it's a secret ballot.
You're going to report on the piece of paper in front of, you know, what you thought.
So it is silly. It's someone else's fault. Okay.
Now, how many people have a number? Between nought and 20 on the piece of paper in front of you?
One. Okay. What are we going to do here? If I can get the mouse to work is we're going to write these down pointer options.
Felt pen. We have one for that. How many have got 21 to 40.
To. 41 to 60.
61 200. One, two, three, four, five, six, seven, eight, nine, ten, 11, 12, 13, 14, 15, 16, 17, 18, 18.
101 two, 200. Three.
And over 200 the overwhelming majority.
So one, two, three, four, five, six. And they're not alarmed that it will get to 689 in their own right very, very roughly.
There's some confidence intervals on here, about 33. I don't know if that's right, but something somewhere along that line.
Okay. So most of you want it over 200. Who's right?
How do you decide who's right? How do we decide who's right?
What we need to do is to have some way of of knowing of quantifying the uncertainty surrounding our results.
Everybody all of you knew that if only five people in each job, you could easy just by chance,
four get better in one arm and to get better in the other, all of you knew that intuitively.
However, what you don't know intuitively is how many people you need in each of the at least five people who,
even with a thousand in each arm, didn't intuitively feel that they could know that they wanted confidence intervals and so.
So how can we know how much fuzziness there is? Here's a highly fuzzy lots and lots of uncertainty.
You can hardly even see that there are trees and here's a little less.
So how can we talk about how much uncertainty, how fuzzy is our best estimate that you're twice as likely to get better?
How can we put a number on that? Who knows how we quantify that?
So we can we could do a power calculation and that would tell us how many we would need in each arm.
And then when we look at our results, it would tell us how much uncertainty in our results,
how much do how do we express, how much uncertainty there is in our results.
So confidence intervals. One way to you be that a p value is another way.
So we're going to look at P values first. So one way of expressing this is a P value.
And we're just going to look at this. You all said to me, hang on.
Five in each on that could be due to chance. So the question is how often would you get a result like that by chance?
Hmm. With nothing going on. If you have that little thought, we epidemiologists give it a name.
Does anyone know what name we give to that thought? Hang on.
If nothing's going on. No.
It's the null hypothesis. Let's assume that nothing's going on.
Is the null hypothesis. You hypothesise that there's nothing there.
Okay. So there's nothing going on. How often would we get a result like that?
By chance. And what do we give the answer in? We give the answer, how often?
What do we express at answering the probability, the p value?
So let's just have a look at P values. P values are actually quite easy.
They're very, very easy. First of all, you've got to know what P stands for.
And our friend also told us just then, what is P stand for probability and what values can probability take?
You look so wonderful. Sarah, don't ask.
I'm just too good. He is just too good. Here we are.
So we have 0 to 1.
Okay, I'm going to ask somebody else. What does zero mean? If you've got a probability of zero, what does it mean to me?
It's impossible. So if I have a bag of red sweets, there's nothing but red sweets in it.
I put my hand in and I pull out a sweet. What's the probability that it's blue?
What's the probability? The sweet I pull out is red. Fantastic.
So that's all you really need to know. Zero is impossible and one is absolutely certain and everything else is somewhere between.
So if we've got a p value of 0.5, what does that mean?
You know the coupon you.
In the Middle East. It's in the middle. It's 5050.
So if I toss a coin, what's the probability it will come up?
Heads Yeah. P 0.05.
So if I tell you, look, I'm, I've got this coin here and I can always make it come up on heads.
And he said, Oh yeah. I said I can always make it come up on heads when I toss it.
Look, it's not biased. There's a tail and I toss it.
And it's heads and you'll say, yes, that will happen by chance, even if you aren't able to control it.
So the null hypothesis is basically Amanda can't really control how many times the coin spins.
How often would you get a result like that by chance for it to come up heads when she said it was going to happen.
0.0. 0.5 of the time. Half the time.
I'll prove you. I'm right. People have made a fortune out of that.
They predict the sex of children. They say, Oh, I can predict the sex of your child with my my little court's thing.
Money back guarantee. You know, they get half the time they get to keep the money.
Okay. So 0.05 means that it's 50/50 halfway between.
Absolutely impossible. Not truly certain what 0.1 mean.
Pardon, 10%. So it's one in ten of the times it's there.
What does 0.05 mean? I try to use 5% of the time or one in 20.
That's a very familiar number of P equals 0.05, isn't it.
You've heard that one a lot. Why have we heard that one a lot?
Because. Because that's what we tend to use when people say it's statistically significant is when P is less than 0.05.
So statistical significance is nothing special.
It's nothing magical. When you get a result, what's statistically significant all it means up P less than 0.05.
All it means that if nothing is going on the null hypothesis,
you wouldn't get a result like the one you've seen or bigger more than one in 20 of the time.
It doesn't mean that there's anything real that.
So we call things statistically significant when the result is unlikely to have occurred more often than one in 24 time.
So there's nothing nothing special. And we tend to use P less than 0.05.
This is what I like, this little cartoon.
The reason I like this cartoon is because actually you do not have to know how a statistician has done the test for statistical significance.
All you need to know is how to interpret a p value.
So if they go to a test for statistical significance, p equals 0.01.
What does it mean? Well, not what it means.
It's statistically significant. What does it actually mean? On the first chance that the is correct.
Yeah. So if there was nothing going on.
One in 101% of the time you would get a result like this by chance, 99 times you wouldn't get a result this extreme by chance.
That's all it means. Okay.
So let's just have a look at this. This is the one we looked at before with partisan placebo.
We say we as a community start believing things when P is less than 0.05.
So if we wanted to do a power calculation to find our sample size, we would have been happy with 15 in each arm.
If you were getting as big an effect as a number needed to treat it to with 80% getting better in the orthopaedic mattress good.
You would only needed 15 in each arm and we'd start believing it.
So a p equals 0.05 isn't a very strong test.
If we go back to the slide before, which I believe is this one,
only one person was right and most of you will weigh down wanting way much more evidence than we accept to change your practice and treat people.
Scary, isn't it? Well, I find it quite scary. I don't know.
So that's interesting if you put a p value on it. Any questions?
What did you. You don't need to know that.
I'll tell you something. I hope I know why. I don't know it either.
I put it into computer program. That clever of people that may have done.
And it will tell me what the p value is if it you know, I've got little to buy to type on and I can share it with you later.
An Excel spreadsheet. And it just tells me at the end, give me p values,
gives me confidence and you don't need to know how the engine works to be able to make sensible use of the results.
And that's the point. But it's after you've got the results, you've got to see values.
Before I start a study, how do I know how many I need to enrol in the test again?
It's a nice little bits of software.
If you go into Google and put sample size calculation software, it will give you it will give you people like Rafael who are clever.
Where's Rafael who know you know, who think and swimming Greek letters and yeah, they do it.
And then ordinary people like you or me, all we have to do know to be able to make sense of results.
You don't have to be statisticians. Statistics is all about concept if you understand.
But the null hypothesis got A's in my exams putting 80i include zeros meant what the null hypothesis is.
All it means is that you're playing a mental game. So. Well, let's assume nothing's going on.
Nobody told me that I'm. I could still get a in the exam. It's weird.
Anyway, let's go back to where we were, which I think ought to be that what slide it is.
Okay, so you can see that with a thousand in each arm,
we could easily believe that result provided it was an unbiased study because with only a hundred it would only happen one in 10,000 of the time.
Okay. So why pay less than 0.05? Your gut feeling didn't want you to go for 0.05.
So why do we go for that? Okay. Just good to wake you up again.
Let's get you all active. They say if people sit for more than 10 minutes, they need to do something.
So it must be about 10 minutes since we did the secret ballot.
So get the coin out and toss it six times in a row and count the number of heads that came up.
Yeah. Oh, yes. I shall do it to you.
Oh, no. Oh, no, no, no. I've got it. So that one will count as a tail?
Yeah. Yes.
Yeah. Okay.
I got two heads. What did you get? I must be a one in five eagle.
You got five heads. Okay.
How many people got one head when they did it?
One, two. So we got another two people who got one head.
How many people got three heads? So one, two, three, four, five, six, seven, eight, nine, ten, 11, 12, 13, 14, 15, 16, 17, 18.
How many people. Oops I've got 18 on here. Three, four, five, six, seven, eight, nine, ten, 11.
Well it goes right up the top and further. How many people got?
Four. 19.
So that goes off the top. Very similar to the before to the three.
How many people got six? My goodness.
Two. How many people got five? One, two, three, four.
And plus the what we got here before. So let's here we got another four on top of the one we got before.
We got two with six. How many?
Got one. How many got none.
Okay. So the probability of getting six heads in a row in Atlanta is twice is less than 0.016 if they didn't have an unbiased coin.
But it happened. We have enough people throwing the coins for it to happen.
Why do we accept point nought five?
It's just a convention that I think is probably rather a two lower barrier myself,
but I do have a double headed coin and I play I play with groups like this.
I've stopped doing it cause I always drop them on the floor, but they start asking to see the coin.
After it's come up, it gets into row four or five times.
I went about people's 0.05. They stop trusting me.
And by the time it's six, everybody in the room thinks I pulled the double headed coin.
Nobody believes that I can actually control it that much.
Okay, so we're going to apply we're going to start just that.
We've learned a p value is saying if nothing's going on, how often would you get a result like this for a chance?
Now, I've been teaching people about statistics for quite a few years,
and I used to evaluate people's subjective understanding of certain terms before and after
workshops to see how effective I was being or how effective participants thought I was being.
And I'd give them words like odds ratio or risk ratio, and they would score it.
So five would be, I understand the term and could I explain it to somebody else?
I understand the term but couldn't define it. Three I have a vague idea what it knows what it means.
Don't ask me any more questions too. I've heard of it, but I've never understood it.
I'm the clue. And one was I've never heard of the term.
Okay, so five was better and with odds ratio.
This is before the workshop over here and this is after the workshops over there.
Before the workshop, 70% of people didn't really know what it means.
Certainly 60% hadn't a clue what it meant, even if they'd heard a bit before.
40% haven't even heard of it before. After the workshop, somehow, 2% still hadn't never heard of the trials before,
but a staggering 80% thought they understood it and a staggering 40% thought they could actually define it and explain it to somebody else.
Right. So these lines coming down like this is a good sign.
So my question to you is, is this a statistically significant result?
So I'm asking you, has this happened by chance or is this a real result of the workshop?
Now I'm going to get you to vote. And by now we're friends enough for you to be able to stick your hands in the air.
Okay. Who thinks this is statistically significant?
Hands up. Who thinks it's not statistically significant.
Thank you. Call to two of you and.
And who's sitting on the fence? I really.
Mike well done. Well done, my man. Okay. This was a highly statistically significant result.
P less one in 10,000 or less on.
When people voted. You should have voted.
Don't know, because I didn't tell you how many people that was, Donald.
Did I? And it could have been like my Potter trial.
We can infer that.
But Paul inferred that we should cleverer kids so the don't knows and the people who voted yes were right and call you were unique in being wrong.
It was statistically significant. Okay.
However, people are nice. People are nice. And you know, it doesn't mean I was teaching very well.
It just means that they were pretending it was self-assessed understanding.
So I'm not nice. So I decided to trick them.
And in the evaluation forms I also put a term I never, ever covered mog to see if they were just being nice.
They'd say they'd heard of it and were better at it, and lo and behold, the lines come down.
They know this term more at the end of the workshop. Okay.
A lot less. A lot less. But do you think this is a statistically significant result?
We get to vote again. Who thinks? Yes, this is a statistically significant result.
Hands off, same class. It's the same same same class.
Hands up. A statistically significant to hands off for not statistically significant hands off sitting on the fence, most of you.
You can't sit on the fence this time because it's the same class.
You yourself got the don't know it's been deleted here. You cannot sit on the fence because this is the same class you're only allowed yes or no.
It's highly statistically significant. This was not a chance occurrence.
This was a real effect. There's a couple of thousand people.
I thought, this is an utterly true effect. I think the clinical significance, that level of their understanding was negligible.
But this was a trip. So whether it was a social compliance effect, people being nice,
but it was a not a chance effect or whether it was because I said, Did you see that really weird word in that book?
What did that mean? And they actually do know it at the end of the workshop, but it's not a chance effect.
And this is very important because authors and editors love P values like that.
So they love P values like that. But they can belong to things as meaningless as this.
Hmm. Bit of a problem.
Any real difference between things, however tiny it is, will become statistically significant if you take a large enough sample size.
And I just thought. Lots and lots of lots and lots of people. So a very small p value doesn't mean you got a marvellous treatment necessarily.
All it means is the effect that's being reported is unlikely to have occurred by chance.
If there was nothing going on, there's something going on. Whether it's bias.
There's something going on. Hmm.
It's a very important message. The fact that something statistically significant or even highly statistically
significant does not mean it's clinically relevant can be due to bias like this lady.
She weighs herself like this, her weight style. It's there's a bias in the measurement system, I think a little bit.
Or it can be just you've got to have a huge sample size. So for example, let's say you've got children who are not going to be very tall.
They're obviously got small stature. Parents are small, but they want to be taller.
Right. And you give them a penny to put in each shoe. You are definitely going to make them taller, a trivial amount.
You're going to make them significantly, statistically, significantly taller, but you're not going to make them clinically,
significantly taller is going to alter their lives because they're walking around with a penny in each shape.
So you've got to distinguish clinical significance from statistical significance.
So the P value tells us something, but it doesn't actually allow us to see enough.
It would be nice to see a little bit further than the p value allows us to see, wouldn't it?
Something about you know, it just tells us that it's not a chance occurrence.
It doesn't tell us whether it's an important effect. So is there a better way of expressing uncertainty due to chance that gives us more information?
Come back. Confidence interval. Yes, excellent.
The confidence interval. So we now going to do an introduction to confidence intervals and learning from Rafael yesterday.
I've decided that I need to give you sweeties.
So I have at home a great big barrel of sweets.
And it's that kind of chocolate éclairs. And they've got green wrappers or they've got gold wrappers on them.
Okay. And I was told there were 50 of you here.
So this morning, I put my hand in the bag and I drew out 50 sweets without looking.
The question is how many of the sweets were green? I don't know.
I don't know how many sweets are green in this fight? If someone had to guess, you had to guess.
Let's say there was olives going to give £100. The person you guess is right.
How many green sweets would you get in the bag? 40.
30. Mike's guessing 25. Right.
Let's see if we can get this to right.
So Mike's getting 25 here. Somebody guessed 30.
Any other guesses? Somebody is guessing that there are none.
Yup. Oops. Some of these guessing this number will be true.
Pardon? 32, 33, 23, 23.
Okay. What's the least number there could be? We've had zero. What's the most?
5050. Okay. So actually, I can't really see this very well, but the truth could lie anywhere along this line from 0 to 50.
Couldn't take a truth. Could be anywhere between nought and 50.
If we were all coming in, you know, one, two, three, we could get £100 a hole by getting it out afterwards.
You got to quit it because the truth.
We know the truth. Well, 100% confident, but the truth lies somewhere between nought and 50.
So if you want as one 100% confidence interval where we know the truth lies.
Okay, Azzam, you have been very, very good today.
So I'm going to let you have a sweet thanks. But before you eat it, you have to show us what colour it is.
It's a gold one. Okay. So the question is, how many green suites did I put in the bag?
What, what? What what are a 100% confidence intervals?
Now, this for 39. Okay, I can't put 55.
Kind of like because we know at least one is gold in gold.
So now 100% confidence intervals lie somewhere between 49 and nought.
I want five of you to take a suite and then I'll take a suite each and then pass the bag on.
Okay. So five, you take the last week. Okay, now let's have a look what we've got here.
You want to hold them up? Okay, we've got three, four goals.
One green. So now we're schedule 100% confidence intervals.
Lie. One, two, one, two, 45 right there.
53 to this box. We better start playing this game quickly.
Keep passing the bag. You can take out more than can just keep passing the bank, taking on anyone who's got a suite so far.
Then the next round, raise your hands. Okay, so we've this time we've got four green and two gold.
So we've got two gold. So for green.
So we know another four of these if I'm doing this right to gold.
Okay. Well, next, a lot of people raise your hands and we have two green and four gold.
Okay. Next, two people. Raise your hands. We got three for gold and one green.
Yeah. I hope I'm doing this accurately.
Next, a lot of people hold you. Hold up your hands. Oh, can't see because of the light.
One green and three gold. Pass it back down to the front for the people who didn't get any.
If you take them off because I want to compromise on £3, so I pass it down that way any more.
People have got it. Haven't held up the hands yet? No.
Okay. A lot to hold up your hands. Bright one, green three.
Gold. I usually do this in workshops where there's 20 people.
But I didn't think you'd be happy if you didn't get a suite, so I had to sit next to people, hold up your hands.
So we've got three green, three gold.
Like a lot of people. Two of each.
Keep going. Next slot, hold up.
And so we got two more. Excellent.
Next slot. Next slot.
Empty. It's empty. There were, in fact, 18 free sweets in that bag.
Okay, so there were 18. Nobody guessed 18.
You're off the hook. Okay, so there were 18.
What do you notice about these lines? You'd think that getting shorter and shorter.
So we're 100% confident. We're getting more and more confident.
The uncertainty is getting less and less and less.
Why is the uncertainty getting less and less and less? Because we've got information.
Every time we take a suite out of the bag, it gives us information.
And the more suites we take out of the bag, the more information there is, the less uncertainty there is.
Okay. So the width of that is a measure of how much uncertainty.
And as we get more and more certainty till we know the truth, which was 18.
What else do we know to spot those lines? Because 100% confidence intervals.
The truth. All of them contain the truth. We knew they would.
We were 100% certain the truth lay between them. So if you draw a line down from 18, it intersects every line.
Every line, contain the truth. Confidence intervals are just like this.
They're measures of how much uncertainty there is. Okay.
But because there's always the future, things might happen again and we cannot be 100% sure.
We've only got a sample. This time we were able to take every suite out of the body.
We were able to look at the entire universe of sweet stuff. But actually, in the real world, we can only ever take a sample.
So what? We settle for a confidence intervals where we're certain that the truth lies what we usually say.
Well, we usually go for 95% certain the truth lies between those arms.
So 5% of the time will allow ourselves to be wrong, which is the same as people's 0.05.
If you notice, 5% of the time we'll be wrong.
So if we were to put the uncertainty around our Potters by our Potters Test when we only had five in each arm,
the uncertainty went all the way from 0.63 and it could have gone right up to six and a bit times better.
So here's another thing about these graphs.
You get your best estimate as the blob, and then you get your uncertainty in terms of confidence intervals.
And this graphical information is usually shown at the slide.
So it tells you where the blob is too, and that tells you the bottom end and the top end of your confidence intervals.
And it tells you whether you're being 95% certain the truth lies 80% certain where the truth lies or 100% certain.
Well, you never have hundred percent, but 99%.
So we're getting to be able to interpret these lines and lights out or an arrow.
That's simply because I didn't draw the graph. It means it goes on to six.
I shouldn't really have done it, but I could have done the graph a bit better.
But I was in a hurry. Here, here.
This one actually fits on the graph, which goes up to five, so we don't have to have the arrow on it.
So this is a bigger sample size.
This is when we had ten in each arm and notice because we've got more information, because we got more sweets out of the bag, as it were.
We have a bigger sample. Our uncertainty is much shorter.
So this time our uncertainty only goes from point eight, 824.54.
You can think of the bottom end and the top end of the confidence intervals as a best case, worst case scenario.
We think potters is twice as good as Potter's.
Yes, Potter's is twice as good as placebo, but actually it could be doing harm.
A 0.88. Or it could be as much as four and a half times as good.
So that tells us more than a peep. And it tells us, you know, what kind of effect we're talking about.
So on this one, what's happening on this one?
You remember we said if we got 15 in each arm, we now knew we now had a statistically significant result.
Why was it statistically significant?
Post-punk values, just that 105 because we wouldn't have got a result like that by chance, more than one in 20 of the time.
And if you look at this, these confidence intervals, there's a little tiny little gap there.
You can read it off here. It's bigger. It's not crossing this line of no difference between the two treatments.
So now, even in the worst case scenario, we think what is is probably better than placebo.
So if a 95% confidence interval exactly sits on that line, what's your p value?
If you're 0.05 and if it doesn't cross that line of no treatment difference, you know,
your P-value is what you know it's less than 0.05 because that's what you're 95% certain.
The truth is 95% of the time it's bigger than the no the null effect.
Okay. So here. Here, we've got the confidence intervals. So this is this is how your blogger gram looks.
It will give you all these studies down the side. It will usually, so to say carpenter and 2007 glaze you 1992 and that sort of thing all down the
side give you the names of the studies will tell you how many were in a study,
how many got the outcome of interest in the treatment arm, how many in the placebo arm or control arm?
And then you get your totals. Okay. What's this little diamond here?
We've been having blobs with with with uncertainty surrounding the best estimate.
That may turn out that we may do. Okay.
Meta analysis. What's that? Something special.
Yeah. Okay. Analysis is a nice, posh term.
It sounds really good. Doesn't that matter? Unless it's every size board.
But all it means is putting all the results we've got together, getting a summary for all the results.
So if we had all these these we have these five trials, we had the ball up, we said we were all like little parts of the same trial, this big trial.
So we add up all these there were 150 in total.
If you add up, all those all come to 60, add up all these 250 that come 220 and then this this bit here,
the pointy bits here, top and bottom are your best estimate.
So that's should two and it's equivalent to your blobs and these are your confidence intervals, right?
So that's how that diamonds interpreted. But it's when you add all the studies together and meta analysis.
When we do that, it has to be all studies. They have to be the same.
I mean, the time of the study, the size of the studies does not have to be the same.
In fact, we have very different sized studies here. And in fact, that's where meta analysis is useful, because if you've got lots of little studies,
none of which are statistically significant, all of which have the uncertainty, including a harmful effect, you put them all together.
Well, it will become statistically significant, maybe, you know, you don't get five studies all on the same time by chance.
But but there's another question. Do all the studies have to be the same?
They have to be the same about the same thing. There's no point having a study that's looking at warfarin and time to anticoagulation and heparin.
I don't know if the medics warfarin takes a long time to work.
Heparin works immediately. You can't put them together in the same.
Plotz So for example, if you say how long does a study is looking at how long bones take to repair and you have children and old people?
Well, maybe you can't really add that up together. Maybe it's different in children from what it is in old people.
So, yes. And if you look at the validity questions, when you get these checklists for systematic review,
one of the questions saying in those things, it says if the results of the study were combined, was it reasonable to do so?
And it's just saying with a study similar enough, looking at a similar population,
a similar enough intervention, similar enough outcomes to be able to be put together sensibly.
Now I said time, I mean is it randomised control.
It has to be the same type of same type of study.
Well not necessarily. But you could argue it's a judgement call.
Not necessarily. But if you have randomised controlled trials you'd be unlikely to want to add in a lesser quality study.
You'd be more likely to believe the randomised controlled trials.
But if they're both, you know, moving in the same direction and they're both giving similar estimates, then that's quite powerful evidence.
Any questions about confidence intervals or the blowback from.
Okay. It's just the size of the squid.
How do you decide the size of this? Excuse me?
The size of the square. Is tells you how much information is in a particular trial and is, if you like.
It's exactly a correlate with the arm. So the narrow the arms, the bigger the square.
They they're both telling the same information. And the reason the blob gets bigger is people saying, hang on.
The least informative studies are the ones with the biggest uncertainty surrounding
of these great big long arms that's giving the wrong visual impression.
They're the ones that you're noticing. And the really important studies with the little narrow arms, you hardly spot them.
We've got to correct this, but otherwise people are going to get a misleading impression.
They corrected it by making the blobs bigger. So the blob is how much information is in there?
I think I better stop now. I've gone on 10 minutes more than I should have.
Yeah. No. Now, you can now all interpret the blowback from I believe the next thing was a blowback from.
I think I had a blowback to go back to the one that we had right at the beginning.
I think the slide three now you can see that you can start interpreting these things very, very quickly.
You now know this. Are there any significant studies? They're on their own.
Mariam might have been. You can't really tell from this and might be borderline significant missing information now when you bung them all together.
Yeah. They are statistically safe when you do. Sorry.
Bung them all together is the epidemiological technical term for meta analysis.
Okay. But when you do the match analysis, you notice that the long arms of Diamond Dog crossing the treatment pool.
I'm going to go on for 2 minutes because I can't resist. Just going to say, does this mean the treatment works or it doesn't work?
Does hypothermia work or not work? Yes.
Right? Yeah, right. So before it was saying favours policy but favours plotters over here, wasn't it?
And now it's not telling you so. Does it always favour the treatment on one side or the other?
60 Minutes. Okay.
If you remember, this side is always less on this side is always more.
Okay. On to work out which side favours it.
You've got to think of the outcome. If it's something you don't want, like mortality and incapacity, you want less of it.
If it's something you do what like proportion giving up smoking and you want it on the other side.
So which side it sits on depends on whether the outcome is something you want or you don't want.
It's something you don't want. When you want less of it, it's something you do want.
Lives saved, number of people cured, and it's on the other side.
And that's true whether it's a risk difference or a relative risk. So this is easy, isn't it?
And we have done any equations. I've done any calculations.
Which is lucky because I don't know how to do them. Okay, well, that's it.
I'm going to have fun. Sorry. Sorry. I bit into your small group time.