1
00:00:00,520 --> 00:00:04,210
Well. Good morning, everybody. I'm Amanda.

2
00:00:04,690 --> 00:00:08,410
And we're going to talk about making sense of results today.

3
00:00:09,880 --> 00:00:17,350
I've discovered that the microphones only work for the cameras, so I'm going to shout because there's no generalised microphone.

4
00:00:17,360 --> 00:00:22,810
So if you can't hear me at certain point, just sort of way even I'll speak up.

5
00:00:23,350 --> 00:00:27,910
The other thing that I tend to do is I tend to speak a little bit fast when I get excited.

6
00:00:28,270 --> 00:00:36,550
So if I start going too fast, especially for people who for whom English isn't your mother tongue, just look and I'll slow down, okay?

7
00:00:37,840 --> 00:00:40,990
But you'll need to pointed out to me because I do get carried away.

8
00:00:43,060 --> 00:00:52,600
This session is about making sense of results, and I've got a little cartoon for you to begin with, and this teacher's there.

9
00:00:52,600 --> 00:00:56,860
And he says, Welcome to the basic astronomy. Before we start, are there any questions?

10
00:00:57,190 --> 00:01:00,370
And he says yes. What makes astronomy different from astrology?

11
00:01:00,980 --> 00:01:09,050
It's a lots and lots of maths. And they managed to lose the entire audience.

12
00:01:10,040 --> 00:01:13,660
Actually, people are quite numerous phobic.

13
00:01:13,670 --> 00:01:20,090
How many people here don't like numbers very much or are scared of numbers in papers?

14
00:01:23,420 --> 00:01:34,610
Your your stronger group the most. I most. I sometimes bring calculators along and I sort of see people's white faces as I bought this.

15
00:01:34,790 --> 00:01:39,649
This is a secession about statistics. It's not statistics, really.

16
00:01:39,650 --> 00:01:44,660
It's not about numbers. It's not about maths, sport, common sense and concepts.

17
00:01:45,080 --> 00:01:48,440
And I hope that that's going to come across today.

18
00:01:49,370 --> 00:01:53,210
In this session, we're going to do the following. We're going to look at how measures of effect are reported.

19
00:01:53,840 --> 00:01:58,340
We're going to look a bit about P values, then look at confidence intervals.

20
00:01:58,790 --> 00:02:04,400
We're going to look at what we mean by statistical significance and test the statistical significance.

21
00:02:04,940 --> 00:02:11,540
We're going to think about why we need systematic reviews and we're going to look at how to interpret global graphs.

22
00:02:12,620 --> 00:02:17,210
Okay, so what's the blowback from. Oh, yeah, we're going to have fun.

23
00:02:17,360 --> 00:02:22,220
And I hope so. And you're welcome to interrupt me at any point in time.

24
00:02:22,490 --> 00:02:30,350
And I mean it. So here's a global crime, also known as a forest plot, also known as a Metro Analysis Graphed.

25
00:02:30,770 --> 00:02:37,610
Now look at this block program and it is looking at if people have had a severe head injury,

26
00:02:37,610 --> 00:02:40,910
how can we reduce their risk of death or long term morbidity?

27
00:02:41,270 --> 00:02:46,700
Well, one idea is that if you call people right down, it will reduce the brain damage.

28
00:02:47,300 --> 00:02:56,960
So this is a block diagram of the results of four trials of cooling people down and looking at how many die or severely incapacitated.

29
00:02:57,560 --> 00:03:04,190
I want you to write down what you think the result of this systematic review showed.

30
00:03:04,340 --> 00:03:11,150
This is taken from the Cochrane Library. I'm just going to give you 5 seconds to write that down.

31
00:03:12,230 --> 00:03:23,790
What does this show? Okay.

32
00:03:23,790 --> 00:03:28,410
Believe it or not, that was 5 seconds and quite a few of you didn't write anything down.

33
00:03:28,680 --> 00:03:32,250
And I don't know whether that was because this was such a difficult question.

34
00:03:32,490 --> 00:03:37,710
If I'm like God, which is jumping right in with the club gram and ask us to interpret one,

35
00:03:38,520 --> 00:03:41,910
or whether it was just because it was so obvious you didn't have to write anything down.

36
00:03:42,570 --> 00:03:47,160
Well, this session is for the people who thought, Oh, my goodness, what's she doing?

37
00:03:47,160 --> 00:03:56,390
I do not know this. By the end of this session, when you see a graph like that, you'll be able to interpret it immediately.

38
00:03:56,400 --> 00:04:00,299
That's that's the aim. That's how you know whether this session was worthwhile.

39
00:04:00,300 --> 00:04:04,750
Okay. Let's go ahead.

40
00:04:05,290 --> 00:04:08,499
So, as I say, you're welcome to ask questions.

41
00:04:08,500 --> 00:04:12,160
There is no question that you can ask me. That's too stupid.

42
00:04:12,850 --> 00:04:18,040
It's I'm serious. This is a safe environment. This is where you can learn the things that you need to know for the future.

43
00:04:18,400 --> 00:04:23,560
And nobody's going to make fun of you because you don't know it.

44
00:04:23,590 --> 00:04:27,970
This is. This is. You can ask the question. You're safe. We're going to look after each other.

45
00:04:28,570 --> 00:04:30,370
Okay. But before we start,

46
00:04:30,370 --> 00:04:36,640
we're just going to recap what are the important things you need to think about when you're using research evidence to inform your decisions?

47
00:04:40,600 --> 00:04:51,090
Shout it out anyway. Is it randomised trial and is it relevant to your question to sorry.

48
00:04:51,340 --> 00:04:57,130
Relative to reason? Is it valid? Do we always ask is it is it randomised?

49
00:04:58,360 --> 00:05:02,860
Do we use any evidence? Not randomised. We have a want evidence.

50
00:05:02,860 --> 00:05:07,480
That's not randomised. Depends on your question.

51
00:05:08,020 --> 00:05:13,809
Depends on our question. If we want to know how women felt about a certain procedure, we go and ask them.

52
00:05:13,810 --> 00:05:23,410
We do a qualitative study. So we don't necessarily want it randomised, but we do if it's an intervention study.

53
00:05:23,740 --> 00:05:29,200
So for any study, whether it's a randomised controlled trial, whether it's a systematic review,

54
00:05:29,200 --> 00:05:36,490
whether it's a qualitative study, economic evaluation, cohort study, whatever study we always want to know is it valid?

55
00:05:36,790 --> 00:05:40,840
Somebody said only that if it is valid, what do the results tell us?

56
00:05:41,680 --> 00:05:49,930
And then is it relevant to our decision? So the three things that you would ask of any study validity is can the results be trusted?

57
00:05:50,560 --> 00:05:55,060
Results? Well, what are they? You know how people express them.

58
00:05:55,060 --> 00:05:59,140
Could they be expressed in a different or more meaningful way? What do they actually mean?

59
00:05:59,650 --> 00:06:06,550
And then relevance. Is this relevant to my particular patient or in the context where I have to make that particular decision?

60
00:06:07,480 --> 00:06:15,790
So just because it's first thing in the morning, I'm just going to give you a little warming up exercise just to to stretch you a bit.

61
00:06:16,150 --> 00:06:21,160
All on the validity for an intervention study. What sort of study design do we want for an intervention?

62
00:06:23,720 --> 00:06:29,900
A randomised controlled trial and you have got 30 seconds to write down all the validity

63
00:06:29,900 --> 00:06:34,010
questions or discuss with your neighbour what you do for a randomised controlled trial.

64
00:06:34,010 --> 00:06:38,360
What makes a randomised controlled trial valid? You've only got 20 seconds left.

65
00:06:58,950 --> 00:07:03,630
Okay. You've got 30 seconds. That validity questions.

66
00:07:03,870 --> 00:07:07,590
You might not have had time to come up with them all, but perhaps between us we have.

67
00:07:08,190 --> 00:07:12,330
Well, what questions are you going to ask randomised.

68
00:07:13,200 --> 00:07:19,919
Well, was it randomised? Was it an adequate sample size which is what it matched.

69
00:07:19,920 --> 00:07:26,220
Gold sample sort of. Okay. So were the groups comparable with baseline characteristics similar.

70
00:07:27,060 --> 00:07:30,650
So in other words, the recruitment action people.

71
00:07:31,140 --> 00:07:35,310
Okay. Yes. So and for that allocation to be okay, what do we want?

72
00:07:36,810 --> 00:07:42,000
We want randomisation, but we want we want allocation, concealment.

73
00:07:42,420 --> 00:07:48,940
Then we want blinding. Anything else we want the to maintain.

74
00:07:49,620 --> 00:07:52,700
Sorry. Limiting. Yes. Where they kept the same way.

75
00:07:52,710 --> 00:07:56,610
So were both groups treated equally? Anything else we'd like to see?

76
00:07:57,120 --> 00:08:01,560
Adequate numbers. Adequate numbers? So that was sample size calculation.

77
00:08:01,860 --> 00:08:05,370
And you want it analysed properly, so in intention to treat analysis.

78
00:08:05,380 --> 00:08:09,690
So here you are going from the top. It's all about getting those groups to say was it randomised?

79
00:08:09,690 --> 00:08:14,069
Was the allocation concealed? Did it work? Did you have similar baseline characteristics?

80
00:08:14,070 --> 00:08:20,400
Was it blinded, treated the groups the same minimal losses to follow up and whether the same in both arms?

81
00:08:20,700 --> 00:08:25,540
And did they do an intention to treat analysis? Nobody came up with that in 30 seconds.

82
00:08:25,830 --> 00:08:27,330
Well, at least as a group we did.

83
00:08:30,060 --> 00:08:39,270
The reason I've gone doing about results and I've gone back to validity is because results only have meaning if the study is valid.

84
00:08:39,570 --> 00:08:46,590
If you're appraising, you're reading a trial and it's not been done validly, you know it.

85
00:08:46,590 --> 00:08:48,510
You can't interpret the results.

86
00:08:49,500 --> 00:08:58,050
So I'm going to give you a warning that everything I say now about what results mean, it's assuming that the study's been done in a valid way.

87
00:08:59,210 --> 00:09:05,060
Okay. Most critical appraisal. That way you throw out the paper, it'll be long before you get to look at the results.

88
00:09:05,350 --> 00:09:13,870
It's because it will just not be a robust paper. So everything I say from now onwards assumes the study was done validly.

89
00:09:13,880 --> 00:09:20,150
So let's we've got a valid study. Most useful studies, cohort studies, case control studies.

90
00:09:20,390 --> 00:09:25,020
Most useful quantitative studies. Compare two alternatives, at least alternatives.

91
00:09:25,400 --> 00:09:34,220
You know, did the person have this treatment or did they have the control they exposed to this risk factor or were they not exposed to it?

92
00:09:35,660 --> 00:09:39,380
Did they get a disease compared to the people who didn't get the disease?

93
00:09:39,980 --> 00:09:43,190
So how can the results of such comparisons be expressed?

94
00:09:45,620 --> 00:09:50,120
Glass, then put them in graphs. And what might you just show in the graph?

95
00:09:54,770 --> 00:10:00,829
Relative risk, you might share the relative risk. So you could have a graph where you just share the proportion who got a certain

96
00:10:00,830 --> 00:10:04,970
outcome in one arm and the proportion another in a sort of a bar chart.

97
00:10:05,360 --> 00:10:09,050
Or you could actually do some kind of calculation to show the relative risk.

98
00:10:12,370 --> 00:10:18,010
Some of these words, like relative risk, might not even be familiar to you, but the concepts are actually common sense.

99
00:10:18,370 --> 00:10:23,979
So what we're going to do is we're going to do a little exercise. We're going to look at a randomised controlled trial.

100
00:10:23,980 --> 00:10:27,940
This is one I plucked out of the air. It's just a nice easy numbers.

101
00:10:28,390 --> 00:10:34,180
And we're looking at whether orthopaedic mattresses are better than normal mattresses for preventing backache.

102
00:10:34,570 --> 00:10:41,709
Okay. And we've randomised 100 people to receive a firm orthopaedic mattress and 100 to receive a normal mattress,

103
00:10:41,710 --> 00:10:46,540
a medium mattress and after three months we measure the backache in the groups.

104
00:10:46,540 --> 00:10:52,690
80% got rid of that backache in the orthopaedic mattress group and 20 in the medium mattress group.

105
00:10:53,080 --> 00:10:58,090
Right. I'm going to give you just one minute to summarise this result with your neighbour.

106
00:10:58,090 --> 00:11:01,090
So talk about all the ways you can summarise that result.

107
00:11:08,050 --> 00:11:11,350
She that? Right.

108
00:11:12,080 --> 00:11:16,980
You know. Again.

109
00:11:32,650 --> 00:11:46,610
Yes. Okay.

110
00:11:48,370 --> 00:11:51,610
My friend Darci is going to help me. Are you Assam? Yes, I am.

111
00:11:52,810 --> 00:11:56,200
Do you want to tell us how you summarise those results?

112
00:11:56,240 --> 00:12:07,780
Well, I think for the person that takes the relative risk is 80% and for the early 1980s the minimum tax is 20%.

113
00:12:08,230 --> 00:12:13,420
So the difference is 60% can see that for seven days.

114
00:12:13,810 --> 00:12:18,700
Four out of five will benefit for that.

115
00:12:22,630 --> 00:12:26,830
One out of five would benefit and will get better.

116
00:12:27,340 --> 00:12:33,610
That means there is the risk is going to be four times for medium of matches.

117
00:12:34,420 --> 00:12:38,079
Then further matters extra.

118
00:12:38,080 --> 00:12:43,630
I make it too complicated. Perfect. I rarely get the full answer on the very first go.

119
00:12:43,660 --> 00:12:47,739
Excellent. Okay. So exactly what?

120
00:12:47,740 --> 00:12:52,629
As I'm says, 80 out of 100 get better in the four mattress group.

121
00:12:52,630 --> 00:12:59,530
So we say 80% get better. Strangely enough, although it's getting better, that's what's known as a risk.

122
00:12:59,530 --> 00:13:04,450
What's the risk of getting better in the mattress group is 80%.

123
00:13:04,780 --> 00:13:08,740
It's a funny, funny word for English speakers to use the word risk like that, but that's what we do.

124
00:13:08,920 --> 00:13:13,030
What's the risk of getting better? 80%. 20 out of a hundred.

125
00:13:13,030 --> 00:13:16,210
What's the risk of getting better in the medium mattress group?

126
00:13:18,430 --> 00:13:24,579
20%. So once you know that we're calling the probability of getting better risk,

127
00:13:24,580 --> 00:13:32,920
the relative risk becomes how likely is you to get better in one group compared to the other, you know, the risk relative of one group to another.

128
00:13:33,550 --> 00:13:37,420
So the relative risk of getting better is four.

129
00:13:37,420 --> 00:13:41,380
You're four times as likely to get better with a firm mattress.

130
00:13:41,390 --> 00:13:48,820
So when people talk about relative risk, they're talking about dividing one risk by another 80 divided by 20.

131
00:13:49,450 --> 00:13:54,550
That's what relative risk means. Adam also did something else.

132
00:13:54,850 --> 00:14:04,930
He said if 80% get better in the orthopaedic mattress group and only 20% get better in the regular mattress group, an extra 60% are getting better.

133
00:14:05,440 --> 00:14:10,329
So instead of dividing one by another, he took 80 and he took 20 away from it.

134
00:14:10,330 --> 00:14:17,320
And that's called the risk difference. An extra 60% of people get better with a mattress.

135
00:14:17,410 --> 00:14:24,380
That's the risk difference. The only thing he missed was the number needed to treat.

136
00:14:24,410 --> 00:14:36,229
This is an interesting concept. For every 1.7 people with back pain, given a firm instead of meeting one case, one case of back pain is improved.

137
00:14:36,230 --> 00:14:39,680
That is the number needed to treat. I've got the definition on the next page.

138
00:14:40,130 --> 00:14:44,300
The number needed to treat is the number of people who would have to be given an intervention

139
00:14:44,300 --> 00:14:49,340
in this case of a mattress in order for a one extra case of back pain to be given.

140
00:14:49,350 --> 00:14:54,110
So in this case, we give two people a mattress for a mattress instead of a medium.

141
00:14:54,110 --> 00:14:57,780
Mattress will get a cure, a case of back pain. That's the number.

142
00:14:58,370 --> 00:15:03,590
How did we get to that number? Needed to treat. Let's just go back.

143
00:15:03,890 --> 00:15:11,720
Well, it's very easy. If a hundred people are treated with a mattress, an extra 60 get better.

144
00:15:12,200 --> 00:15:15,650
Divide 100 by 60 and you get 1.7.

145
00:15:15,860 --> 00:15:19,100
So it's how many to get one extra person better.

146
00:15:21,960 --> 00:15:29,820
So 100%, 100 people. Extra 60% get better under divide by 61.7.

147
00:15:30,330 --> 00:15:34,770
Excuse me. Actually, I will do passionate questions.

148
00:15:34,770 --> 00:15:38,160
And we understand the last bit, which is about the 1.7.

149
00:15:38,430 --> 00:15:41,570
Okay, I understand.

150
00:15:42,120 --> 00:15:46,560
Do you understand the concept number needed to treat? Okay.

151
00:15:46,770 --> 00:15:58,640
So, for example, if I give somebody an anaesthetic and they become unconscious and it happens every time I give somebody in on a static,

152
00:15:58,650 --> 00:16:02,850
how many people would I have to give an anaesthetic to to make them unconscious?

153
00:16:04,430 --> 00:16:11,370
Happens absolutely every time. I just have to give it to one person.

154
00:16:11,370 --> 00:16:18,659
And they become unconscious, wouldn't they? So in that that would be the number needed to treat number of people.

155
00:16:18,660 --> 00:16:22,260
I need to give an anaesthetic to. To make them unconscious is one.

156
00:16:22,680 --> 00:16:29,760
Just give it to one person. They become unconscious. In this case, not everybody given a firm mattress got better.

157
00:16:31,110 --> 00:16:39,509
Only 80 got better out of the hundred, so the number needed to treat can't be one and 20 would have got better anyway.

158
00:16:39,510 --> 00:16:44,010
20 because we got in the regular mattress group, 20 got better anyway.

159
00:16:44,400 --> 00:16:47,910
So we look at the risk difference. How many extra people got better?

160
00:16:48,480 --> 00:16:52,950
Will 60%. 60% extra people got better.

161
00:16:53,730 --> 00:16:56,850
Do you get at the risk different? You took them away from Rachel. Okay.

162
00:16:57,030 --> 00:17:00,390
So if 60% of people extra get better.

163
00:17:00,870 --> 00:17:05,040
For every hundred people we treat, how many extra people get better?

164
00:17:08,840 --> 00:17:12,350
We treat 100 people. How many extra people will get better?

165
00:17:14,120 --> 00:17:20,900
60. So the question is, how many people do we have to treat or one extra person to get better?

166
00:17:22,350 --> 00:17:25,530
100 divided by 60. Yeah.

167
00:17:26,250 --> 00:17:35,790
That's the number needed to trade. Now, if you're mathematical and you like formula, you can think of that as one over the risk difference.

168
00:17:36,000 --> 00:17:42,570
But you don't have to think over the absolute risk difference. But formula sometimes confused people more than help.

169
00:17:43,600 --> 00:17:49,500
Isn't it easier to see for only five patients?

170
00:17:50,340 --> 00:17:54,140
For those who get better as compared to what you can.

171
00:17:54,150 --> 00:17:55,890
And that is a good way of saying it.

172
00:17:56,370 --> 00:18:03,600
It makes more sense to a lot of patients because if you told patients their risk is 0.3, they think what they mean.

173
00:18:04,000 --> 00:18:07,010
It was a 30% risk. I can't understand that.

174
00:18:07,020 --> 00:18:15,720
What you mean it's either going to happen or it's not. But if you tell them, well, for every three patients like you, one will get better.

175
00:18:15,900 --> 00:18:17,100
They can understand that.

176
00:18:17,610 --> 00:18:26,130
So it is people for if you're talking to patients, it's much better to keep things in natural frequency so they understand it a lot better.

177
00:18:29,210 --> 00:18:33,450
Okay. But there are already two basic ways to summarise risk.

178
00:18:33,470 --> 00:18:37,240
You either divide them or you take them away.

179
00:18:37,250 --> 00:18:40,940
So you either divide things. You get a relative risk, you take them away.

180
00:18:40,940 --> 00:18:49,950
And you get a risk difference. And. There are other things, ways of expressing risk like odds ratio.

181
00:18:50,910 --> 00:18:56,010
Do you think that's a dividing or taking away way? There's a clue.

182
00:18:56,010 --> 00:18:59,040
There's a clue in the title ratio actually means of dividing.

183
00:18:59,280 --> 00:19:09,060
So it's dividing relative risk by dividing hazard ratios are dividing or taking away risk is dividing risk.

184
00:19:09,660 --> 00:19:18,629
So any two things I want you know that life becomes very easy because you don't have to know how to cause working to be able to drive it.

185
00:19:18,630 --> 00:19:21,600
You don't have to know the statistics to be able to interpret them.

186
00:19:21,960 --> 00:19:26,010
All you need to know is whether someone is getting you are dividing or taking away risk.

187
00:19:26,430 --> 00:19:32,459
So this diagram the slope of global Graham has as its basic structure.

188
00:19:32,460 --> 00:19:36,720
This inverted t was the line down the middle on that inverted t.

189
00:19:36,720 --> 00:19:41,910
When you look at those diagrams, what's that line down the centre is neutral.

190
00:19:41,970 --> 00:19:48,360
No effect one. No, no, I'm I'm saying somebody saying it's no effect.

191
00:19:48,360 --> 00:19:57,750
Somebody saying it's one. Any other options and the other there is no difference between the two.

192
00:19:58,140 --> 00:20:03,660
You're too clever, Adam. He's just too clever, isn't he? It's actually not necessarily no effect.

193
00:20:04,500 --> 00:20:09,180
And it's it's no difference between the two things you're comparing.

194
00:20:09,480 --> 00:20:14,280
Obviously, if you're comparing placebo or or do nothing, then it is no effect.

195
00:20:15,120 --> 00:20:20,610
But it's it's the line of no difference between the two things that you're comparing.

196
00:20:21,210 --> 00:20:22,650
Somebody said it was one.

197
00:20:23,310 --> 00:20:33,959
Is it one who thinks it's one, a number along the bottom who thinks it's one one person thinks it's one, two people think it's three, four, five, six.

198
00:20:33,960 --> 00:20:42,760
We're getting more. Who thinks it's something else? Okay.

199
00:20:42,760 --> 00:20:47,800
We've got quite a few votes for one and a lot of people not. But it depends.

200
00:20:47,830 --> 00:20:51,730
It often is one, so I'm for it to be one.

201
00:20:52,720 --> 00:20:59,950
It's got to be a divorce. It's got to be a dividing risk. So if you got 12 and you divide it by 12, what do you get?

202
00:21:00,350 --> 00:21:05,259
Well, so if you're getting the same in one arm and the same in the other arm, you'll only get one.

203
00:21:05,260 --> 00:21:08,770
If you get 43.9 and you divide it by 43.9.

204
00:21:08,770 --> 00:21:16,080
What you get. So that what happens on that line?

205
00:21:16,110 --> 00:21:22,980
It depends on whether you're looking at the relative risk or risk ratio, an odds ratio, hazard ratio, or whether you're looking at a risk difference.

206
00:21:25,090 --> 00:21:30,549
Regardless of that to the left is always less be less than one.

207
00:21:30,550 --> 00:21:34,150
If it's a ratio under the right, it's always more so.

208
00:21:37,050 --> 00:21:41,010
If it's a dividing, it's a ratio, then the number there is one.

209
00:21:42,270 --> 00:21:49,190
However, what, 43.9 -43.90.

210
00:21:49,620 --> 00:21:56,730
So if it's a if it's if you're showing a risk difference and it's the same in both arms, you're going to get aa0 away.

211
00:21:56,760 --> 00:22:04,120
So if you take away, you get a zero there. But it's still less on this side or more on that.

212
00:22:04,750 --> 00:22:09,079
So it doesn't really matter. Okay.

213
00:22:09,080 --> 00:22:11,560
I'm now going to do a randomised controlled trial.

214
00:22:11,570 --> 00:22:19,280
I'm interested in backache and I looked on the web for cures and I actually came across this genuine herbal remedy for backache.

215
00:22:19,280 --> 00:22:22,700
Okay, Potters. So I thought, well, I wonder if it works.

216
00:22:23,630 --> 00:22:32,300
I'm being an evidence based person that I am. I couldn't find any trials on Medline for for Potter, so I decided to conduct my own trial.

217
00:22:34,370 --> 00:22:40,790
I didn't get funding from the pharmaceutical industry or Potter's, so it's quite a small trial, but it was superb.

218
00:22:41,210 --> 00:22:48,920
There was no bias whatsoever. I had the randomisation done centrally by a friend is absolutely superb trial.

219
00:22:48,920 --> 00:22:50,870
If it's all the criteria for being biased.

220
00:22:50,870 --> 00:22:58,579
Three five people ended up being randomised to get plus with Potter's and five people got absolutely identical and

221
00:22:58,580 --> 00:23:06,620
indistinguishable placebo and four out of five people got that with Potter's and two out of five got better with placebo.

222
00:23:07,190 --> 00:23:09,590
First of all I wanted to do before we even go on,

223
00:23:09,890 --> 00:23:15,260
I want you to tell me what the relative risk of getting better with Potter's was and what the risk differences.

224
00:23:21,470 --> 00:23:24,380
Before we even even start thinking about these results.

225
00:23:37,200 --> 00:23:42,000
I said, Why don't you give the microphone to somebody else who's brave enough to have a go at answering this?

226
00:23:43,080 --> 00:23:48,360
No, not brave enough. Pass it backwards. Now it's going to the cameras.

227
00:23:48,390 --> 00:23:52,860
It's going to the cameras. Somebody who's brave enough to answer this and wants to appear on film.

228
00:23:54,060 --> 00:23:57,120
No. Nathan, budding film stars up there.

229
00:23:57,770 --> 00:24:02,489
That's a bit scary. It's possible.

230
00:24:02,490 --> 00:24:05,550
Actor Paul Glacier. He must be brave enough to appear on camera.

231
00:24:14,700 --> 00:24:20,790
Paul, what did you make the relative risk of getting better to be a we're not switched on.

232
00:24:22,910 --> 00:24:29,379
Most of you are twice as likely to get hot as did everyone get that?

233
00:24:29,380 --> 00:24:32,780
Twice as likely to get better with potters than with. Yeah.

234
00:24:32,850 --> 00:24:36,159
Yeah. And you didn't dare say that on come.

235
00:24:36,160 --> 00:24:40,720
You're so insecure. What was the difference?

236
00:24:42,730 --> 00:24:49,180
What motivated you to exit your people out of five of their policies to two out of five, which is 40%?

237
00:24:49,720 --> 00:24:53,230
Okay, so two out of five, 40% got better at a party.

238
00:24:53,230 --> 00:24:59,200
So with a 40% extra get better, how many if we treated 100 people, how many extra people would get better?

239
00:25:02,610 --> 00:25:07,860
1440. So for every hundred people we treat, 40 extra will get better.

240
00:25:08,550 --> 00:25:13,190
So what's the number needed to treat? Two point something.

241
00:25:13,200 --> 00:25:17,050
Yeah, 2.0. Somebody's good at maths over that.

242
00:25:17,760 --> 00:25:22,170
Okay. So this is you can put it in a two by two table like that if you want.

243
00:25:22,590 --> 00:25:30,210
And here we have for this, this is how on these global grams you get this, here's the study, this is my little study I did.

244
00:25:30,690 --> 00:25:37,860
The next line will give you the intervention. The next line gives you the control and you get a little fraction like that.

245
00:25:37,890 --> 00:25:42,330
Now that little fraction, it's telling you the first bit, the numerator is how many people got better.

246
00:25:42,630 --> 00:25:46,110
The denominator, the bit underneath is how many people were in that arm.

247
00:25:46,680 --> 00:25:53,280
So here's our study. Four out of five got better in Potter's and two out of five got better in placebo.

248
00:25:53,790 --> 00:26:05,240
And there we are. It's Paul gave us the relative risk so that we know the number at the bottom is one and there's our blob above two.

249
00:26:05,250 --> 00:26:10,740
That's our point estimate. Unfortunately, I don't think my little later works, but there you are.

250
00:26:11,100 --> 00:26:17,490
You're there's your blob on your blood program. So that's how a blob of ground which is giving that the point.

251
00:26:17,700 --> 00:26:22,980
Best guess of how good it is. It's twice. Twice as likely to get better with Potter's.

252
00:26:24,720 --> 00:26:33,680
Yeah. So I immediately went out and invested in the potters industry and I now want to sell you all potters.

253
00:26:33,690 --> 00:26:38,580
Are you going to buy this? Do you believe the Potters Q is back?

254
00:26:39,450 --> 00:26:47,159
This is an unbiased study. We need to look at the chemical industry and the baseline characteristics.

255
00:26:47,160 --> 00:26:58,770
Did my randomisation work? The application was just one test, only to be clear, and participants didn't have the characteristics.

256
00:26:58,770 --> 00:27:02,160
In addition to finding the most of them, were they?

257
00:27:03,600 --> 00:27:06,870
They had low chronic back pain of no particular cause.

258
00:27:06,870 --> 00:27:10,949
And and we did seem to get a good randomisation only.

259
00:27:10,950 --> 00:27:19,229
Yes. The numbers aren't significant. There must be something we you have a very small population to really come to the

260
00:27:19,230 --> 00:27:23,580
conclusion that the number that significant difference makes complete sense.

261
00:27:25,320 --> 00:27:31,800
So you're telling me you don't believe it? Not because the study is biased, but because it could have happened by chance.

262
00:27:31,860 --> 00:27:36,730
Is that what you're telling me? Yes. Okay.

263
00:27:40,180 --> 00:27:46,990
Every time you see a result and somebody comes up, the pharmaceutical rep comes again, look at this, look at this marvellous thing.

264
00:27:48,010 --> 00:27:51,890
You stop and you ask yourself, Could this be due to bias?

265
00:27:51,910 --> 00:27:56,200
That's the first thing I always ask myself. That study was damned nuts bursting.

266
00:27:56,650 --> 00:28:02,110
Is the study valid? Is there something in the way they conducted the study that would give them the results?

267
00:28:02,300 --> 00:28:06,190
The first thing I ask myself is, is it valid? Is the bias in there?

268
00:28:06,610 --> 00:28:10,810
The next thing I want to know is, could this have just happened by chance?

269
00:28:12,880 --> 00:28:19,840
And only if I can't find bias a chance do I start thinking, Oh, maybe there's something here.

270
00:28:20,910 --> 00:28:24,840
Okay. So we don't believe it. It could have happened by chance.

271
00:28:24,840 --> 00:28:29,610
So we're. We're very sceptical. Mm hmm. So.

272
00:28:32,330 --> 00:28:37,190
I decide people aren't going to you know, I've got all these shares in Potter, so I've got to prove that it works.

273
00:28:37,190 --> 00:28:46,340
So I do another superb study, totally bias free, and this time I randomise 2000 people.

274
00:28:47,270 --> 00:28:56,000
Okay, I got exactly the same results. Twice as many people are getting better in the potter's arm as are getting better in the placebo arm.

275
00:28:56,810 --> 00:29:03,770
Now, if this was unbiased, let's say I hadn't bought shares in the company, would you believe this result?

276
00:29:07,200 --> 00:29:10,500
Yes. How many people would still be sceptical?

277
00:29:13,730 --> 00:29:17,180
Right. Most people believe this. Let's hear from the sceptics.

278
00:29:17,180 --> 00:29:24,670
Why aren't you going to believe this result? You need to.

279
00:29:32,220 --> 00:29:37,800
Right. So you're saying you might believe this result, but you can't tell just from these numbers.

280
00:29:38,340 --> 00:29:43,170
What about other people? I would like some confidence intervals.

281
00:29:43,890 --> 00:29:47,460
Okay. So you're saying the same as in. Yeah, that's looking good.

282
00:29:47,730 --> 00:29:50,790
I need to know how much uncertainty surrounding that.

283
00:29:51,720 --> 00:29:56,250
Okay. The think is that. How do you assess getting better.

284
00:29:57,470 --> 00:30:00,490
Ah, well, it's a it is a subjective outcome.

285
00:30:00,510 --> 00:30:05,490
It is a subjective outcome. But they were blinded. Everyone was blinded in the assessment.

286
00:30:05,940 --> 00:30:08,969
Even the assessor that it was patient reported.

287
00:30:08,970 --> 00:30:14,490
But the the two tablets were indistinguishable. And we asked them to guess which they were on and they couldn't.

288
00:30:18,150 --> 00:30:21,870
So we do think this is a valid trial. Okay.

289
00:30:22,710 --> 00:30:28,080
Some people are going to believe this because their intuition is that the uncertainty surrounding that estimate is very low.

290
00:30:28,080 --> 00:30:32,970
Some people want a little bit more information just going on, intuition alone.

291
00:30:33,600 --> 00:30:37,590
We're going to we're going to back and have a bit of fun here just to wake you up.

292
00:30:38,100 --> 00:30:45,640
I'm going to ask you, how many people you would want, would you guess you'd need in each arm before you're going to believe the result?

293
00:30:45,660 --> 00:30:50,070
Okay. And we're going to do this by the secret ballot.

294
00:30:50,460 --> 00:30:54,210
I want you all to tear off a piece of paper. A blank piece of paper.

295
00:30:59,570 --> 00:31:09,560
And I want you to write on it how many people you would guess intuitively you would need in each arm before you begin to believe the results?

296
00:31:09,710 --> 00:31:12,920
Before you got those results? Yeah, before you believe it.

297
00:31:12,920 --> 00:31:21,320
And you do a trial. Yeah. And you're going to 80% to getting bedroom with Potter's 40% getting better with placebo.

298
00:31:21,500 --> 00:31:27,200
In other words, you've got a relative risk of getting better of two. How many people would you guess?

299
00:31:27,200 --> 00:31:30,410
You're not to be right, because the secret ballot. No one's ever going to know what you wrote.

300
00:31:31,520 --> 00:31:35,570
Write your number on a piece of paper. Fold it in half and half again, and then swap it with someone.

301
00:31:36,290 --> 00:31:39,889
Don't let go of your piece of paper till you've got one back from them.

302
00:31:39,890 --> 00:31:43,870
Otherwise somebody will end up with two and somebody will end up or more.

303
00:31:44,210 --> 00:31:47,510
Keep swapping it until you don't know whose piece of paper you've got.

304
00:31:50,120 --> 00:31:58,880
You know, any any number you want, the number you sort of think, well, about 30, about 3000, whatever do you feel in your belly?

305
00:32:07,240 --> 00:32:15,290
What do you think?

306
00:32:15,780 --> 00:32:22,220
Yeah. Okay, just. Just. If you have followed the instructions correctly.

307
00:32:22,640 --> 00:32:25,880
Everyone should have one piece of paper. No more in front of them.

308
00:32:27,110 --> 00:32:31,680
Open the piece of paper in front of you. And this is why it's a secret ballot.

309
00:32:31,700 --> 00:32:35,450
You're going to report on the piece of paper in front of, you know, what you thought.

310
00:32:35,870 --> 00:32:39,020
So it is silly. It's someone else's fault. Okay.

311
00:32:41,090 --> 00:32:47,000
Now, how many people have a number? Between nought and 20 on the piece of paper in front of you?

312
00:32:47,930 --> 00:32:57,110
One. Okay. What are we going to do here? If I can get the mouse to work is we're going to write these down pointer options.

313
00:32:57,110 --> 00:33:02,120
Felt pen. We have one for that. How many have got 21 to 40.

314
00:33:04,930 --> 00:33:10,860
To. 41 to 60.

315
00:33:14,010 --> 00:33:23,850
61 200. One, two, three, four, five, six, seven, eight, nine, ten, 11, 12, 13, 14, 15, 16, 17, 18, 18.

316
00:33:25,360 --> 00:33:30,580
101 two, 200. Three.

317
00:33:30,820 --> 00:33:34,420
And over 200 the overwhelming majority.

318
00:33:34,420 --> 00:33:42,490
So one, two, three, four, five, six. And they're not alarmed that it will get to 689 in their own right very, very roughly.

319
00:33:43,360 --> 00:33:48,520
There's some confidence intervals on here, about 33. I don't know if that's right, but something somewhere along that line.

320
00:33:48,820 --> 00:33:54,230
Okay. So most of you want it over 200. Who's right?

321
00:33:57,510 --> 00:34:01,980
How do you decide who's right? How do we decide who's right?

322
00:34:06,580 --> 00:34:13,930
What we need to do is to have some way of of knowing of quantifying the uncertainty surrounding our results.

323
00:34:14,110 --> 00:34:20,649
Everybody all of you knew that if only five people in each job, you could easy just by chance,

324
00:34:20,650 --> 00:34:24,970
four get better in one arm and to get better in the other, all of you knew that intuitively.

325
00:34:25,690 --> 00:34:32,530
However, what you don't know intuitively is how many people you need in each of the at least five people who,

326
00:34:32,530 --> 00:34:40,390
even with a thousand in each arm, didn't intuitively feel that they could know that they wanted confidence intervals and so.

327
00:34:40,960 --> 00:34:47,050
So how can we know how much fuzziness there is? Here's a highly fuzzy lots and lots of uncertainty.

328
00:34:47,050 --> 00:34:50,820
You can hardly even see that there are trees and here's a little less.

329
00:34:50,830 --> 00:34:58,180
So how can we talk about how much uncertainty, how fuzzy is our best estimate that you're twice as likely to get better?

330
00:34:59,080 --> 00:35:03,850
How can we put a number on that? Who knows how we quantify that?

331
00:35:05,700 --> 00:35:13,940
So we can we could do a power calculation and that would tell us how many we would need in each arm.

332
00:35:14,450 --> 00:35:18,829
And then when we look at our results, it would tell us how much uncertainty in our results,

333
00:35:18,830 --> 00:35:22,280
how much do how do we express, how much uncertainty there is in our results.

334
00:35:23,090 --> 00:35:32,060
So confidence intervals. One way to you be that a p value is another way.

335
00:35:32,330 --> 00:35:38,060
So we're going to look at P values first. So one way of expressing this is a P value.

336
00:35:39,320 --> 00:35:45,140
And we're just going to look at this. You all said to me, hang on.

337
00:35:45,260 --> 00:35:55,290
Five in each on that could be due to chance. So the question is how often would you get a result like that by chance?

338
00:35:57,530 --> 00:36:04,780
Hmm. With nothing going on. If you have that little thought, we epidemiologists give it a name.

339
00:36:04,800 --> 00:36:08,610
Does anyone know what name we give to that thought? Hang on.

340
00:36:08,610 --> 00:36:11,690
If nothing's going on. No.

341
00:36:12,310 --> 00:36:16,000
It's the null hypothesis. Let's assume that nothing's going on.

342
00:36:18,060 --> 00:36:21,240
Is the null hypothesis. You hypothesise that there's nothing there.

343
00:36:21,720 --> 00:36:26,549
Okay. So there's nothing going on. How often would we get a result like that?

344
00:36:26,550 --> 00:36:31,980
By chance. And what do we give the answer in? We give the answer, how often?

345
00:36:32,190 --> 00:36:37,530
What do we express at answering the probability, the p value?

346
00:36:37,920 --> 00:36:43,320
So let's just have a look at P values. P values are actually quite easy.

347
00:36:43,620 --> 00:36:48,610
They're very, very easy. First of all, you've got to know what P stands for.

348
00:36:48,630 --> 00:36:57,420
And our friend also told us just then, what is P stand for probability and what values can probability take?

349
00:36:59,980 --> 00:37:03,700
You look so wonderful. Sarah, don't ask.

350
00:37:03,700 --> 00:37:06,850
I'm just too good. He is just too good. Here we are.

351
00:37:07,420 --> 00:37:11,740
So we have 0 to 1.

352
00:37:12,190 --> 00:37:21,230
Okay, I'm going to ask somebody else. What does zero mean? If you've got a probability of zero, what does it mean to me?

353
00:37:22,040 --> 00:37:27,050
It's impossible. So if I have a bag of red sweets, there's nothing but red sweets in it.

354
00:37:27,350 --> 00:37:32,420
I put my hand in and I pull out a sweet. What's the probability that it's blue?

355
00:37:35,210 --> 00:37:40,310
What's the probability? The sweet I pull out is red. Fantastic.

356
00:37:40,580 --> 00:37:50,360
So that's all you really need to know. Zero is impossible and one is absolutely certain and everything else is somewhere between.

357
00:37:50,600 --> 00:37:53,960
So if we've got a p value of 0.5, what does that mean?

358
00:37:55,720 --> 00:37:59,410
You know the coupon you.

359
00:38:00,940 --> 00:38:04,000
In the Middle East. It's in the middle. It's 5050.

360
00:38:04,010 --> 00:38:09,010
So if I toss a coin, what's the probability it will come up?

361
00:38:09,010 --> 00:38:12,850
Heads Yeah. P 0.05.

362
00:38:13,780 --> 00:38:20,590
So if I tell you, look, I'm, I've got this coin here and I can always make it come up on heads.

363
00:38:21,430 --> 00:38:24,549
And he said, Oh yeah. I said I can always make it come up on heads when I toss it.

364
00:38:24,550 --> 00:38:27,790
Look, it's not biased. There's a tail and I toss it.

365
00:38:30,090 --> 00:38:36,750
And it's heads and you'll say, yes, that will happen by chance, even if you aren't able to control it.

366
00:38:37,050 --> 00:38:43,170
So the null hypothesis is basically Amanda can't really control how many times the coin spins.

367
00:38:43,920 --> 00:38:49,620
How often would you get a result like that by chance for it to come up heads when she said it was going to happen.

368
00:38:49,920 --> 00:38:53,800
0.0. 0.5 of the time. Half the time.

369
00:38:54,210 --> 00:38:57,210
I'll prove you. I'm right. People have made a fortune out of that.

370
00:38:57,540 --> 00:39:03,659
They predict the sex of children. They say, Oh, I can predict the sex of your child with my my little court's thing.

371
00:39:03,660 --> 00:39:08,310
Money back guarantee. You know, they get half the time they get to keep the money.

372
00:39:10,070 --> 00:39:15,600
Okay. So 0.05 means that it's 50/50 halfway between.

373
00:39:15,780 --> 00:39:19,860
Absolutely impossible. Not truly certain what 0.1 mean.

374
00:39:22,430 --> 00:39:26,930
Pardon, 10%. So it's one in ten of the times it's there.

375
00:39:27,890 --> 00:39:34,430
What does 0.05 mean? I try to use 5% of the time or one in 20.

376
00:39:34,610 --> 00:39:37,670
That's a very familiar number of P equals 0.05, isn't it.

377
00:39:38,210 --> 00:39:42,230
You've heard that one a lot. Why have we heard that one a lot?

378
00:39:44,820 --> 00:39:52,920
Because. Because that's what we tend to use when people say it's statistically significant is when P is less than 0.05.

379
00:39:58,660 --> 00:40:02,799
So statistical significance is nothing special.

380
00:40:02,800 --> 00:40:10,510
It's nothing magical. When you get a result, what's statistically significant all it means up P less than 0.05.

381
00:40:10,600 --> 00:40:14,590
All it means that if nothing is going on the null hypothesis,

382
00:40:14,890 --> 00:40:20,469
you wouldn't get a result like the one you've seen or bigger more than one in 20 of the time.

383
00:40:20,470 --> 00:40:22,090
It doesn't mean that there's anything real that.

384
00:40:24,010 --> 00:40:30,850
So we call things statistically significant when the result is unlikely to have occurred more often than one in 24 time.

385
00:40:32,060 --> 00:40:38,060
So there's nothing nothing special. And we tend to use P less than 0.05.

386
00:40:44,380 --> 00:40:46,000
This is what I like, this little cartoon.

387
00:40:48,940 --> 00:41:01,810
The reason I like this cartoon is because actually you do not have to know how a statistician has done the test for statistical significance.

388
00:41:02,020 --> 00:41:05,350
All you need to know is how to interpret a p value.

389
00:41:06,190 --> 00:41:12,040
So if they go to a test for statistical significance, p equals 0.01.

390
00:41:12,280 --> 00:41:19,089
What does it mean? Well, not what it means.

391
00:41:19,090 --> 00:41:24,310
It's statistically significant. What does it actually mean? On the first chance that the is correct.

392
00:41:24,730 --> 00:41:27,340
Yeah. So if there was nothing going on.

393
00:41:29,580 --> 00:41:38,550
One in 101% of the time you would get a result like this by chance, 99 times you wouldn't get a result this extreme by chance.

394
00:41:38,700 --> 00:41:44,620
That's all it means. Okay.

395
00:41:45,160 --> 00:41:49,030
So let's just have a look at this. This is the one we looked at before with partisan placebo.

396
00:41:49,240 --> 00:41:57,400
We say we as a community start believing things when P is less than 0.05.

397
00:41:57,940 --> 00:42:06,729
So if we wanted to do a power calculation to find our sample size, we would have been happy with 15 in each arm.

398
00:42:06,730 --> 00:42:13,930
If you were getting as big an effect as a number needed to treat it to with 80% getting better in the orthopaedic mattress good.

399
00:42:14,200 --> 00:42:20,230
You would only needed 15 in each arm and we'd start believing it.

400
00:42:21,190 --> 00:42:24,610
So a p equals 0.05 isn't a very strong test.

401
00:42:24,610 --> 00:42:28,540
If we go back to the slide before, which I believe is this one,

402
00:42:30,970 --> 00:42:41,650
only one person was right and most of you will weigh down wanting way much more evidence than we accept to change your practice and treat people.

403
00:42:42,520 --> 00:42:47,230
Scary, isn't it? Well, I find it quite scary. I don't know.

404
00:42:48,220 --> 00:42:53,140
So that's interesting if you put a p value on it. Any questions?

405
00:42:53,500 --> 00:42:57,630
What did you. You don't need to know that.

406
00:43:00,560 --> 00:43:04,340
I'll tell you something. I hope I know why. I don't know it either.

407
00:43:04,340 --> 00:43:08,000
I put it into computer program. That clever of people that may have done.

408
00:43:08,390 --> 00:43:14,510
And it will tell me what the p value is if it you know, I've got little to buy to type on and I can share it with you later.

409
00:43:14,510 --> 00:43:17,810
An Excel spreadsheet. And it just tells me at the end, give me p values,

410
00:43:17,820 --> 00:43:25,280
gives me confidence and you don't need to know how the engine works to be able to make sensible use of the results.

411
00:43:25,610 --> 00:43:29,780
And that's the point. But it's after you've got the results, you've got to see values.

412
00:43:29,780 --> 00:43:33,380
Before I start a study, how do I know how many I need to enrol in the test again?

413
00:43:33,500 --> 00:43:34,819
It's a nice little bits of software.

414
00:43:34,820 --> 00:43:41,780
If you go into Google and put sample size calculation software, it will give you it will give you people like Rafael who are clever.

415
00:43:41,900 --> 00:43:49,700
Where's Rafael who know you know, who think and swimming Greek letters and yeah, they do it.

416
00:43:49,700 --> 00:43:55,760
And then ordinary people like you or me, all we have to do know to be able to make sense of results.

417
00:43:56,000 --> 00:44:00,590
You don't have to be statisticians. Statistics is all about concept if you understand.

418
00:44:01,130 --> 00:44:08,750
But the null hypothesis got A's in my exams putting 80i include zeros meant what the null hypothesis is.

419
00:44:09,380 --> 00:44:13,730
All it means is that you're playing a mental game. So. Well, let's assume nothing's going on.

420
00:44:14,360 --> 00:44:18,410
Nobody told me that I'm. I could still get a in the exam. It's weird.

421
00:44:19,490 --> 00:44:24,050
Anyway, let's go back to where we were, which I think ought to be that what slide it is.

422
00:44:24,440 --> 00:44:28,370
Okay, so you can see that with a thousand in each arm,

423
00:44:29,540 --> 00:44:37,940
we could easily believe that result provided it was an unbiased study because with only a hundred it would only happen one in 10,000 of the time.

424
00:44:39,510 --> 00:44:45,089
Okay. So why pay less than 0.05? Your gut feeling didn't want you to go for 0.05.

425
00:44:45,090 --> 00:44:49,260
So why do we go for that? Okay. Just good to wake you up again.

426
00:44:49,470 --> 00:44:53,340
Let's get you all active. They say if people sit for more than 10 minutes, they need to do something.

427
00:44:53,760 --> 00:44:57,569
So it must be about 10 minutes since we did the secret ballot.

428
00:44:57,570 --> 00:45:04,620
So get the coin out and toss it six times in a row and count the number of heads that came up.

429
00:45:05,850 --> 00:45:09,180
Yeah. Oh, yes. I shall do it to you.

430
00:45:09,330 --> 00:45:14,460
Oh, no. Oh, no, no, no. I've got it. So that one will count as a tail?

431
00:45:16,420 --> 00:45:58,880
Yeah. Yes.

432
00:46:01,770 --> 00:46:06,190
Yeah. Okay.

433
00:46:06,250 --> 00:46:11,960
I got two heads. What did you get? I must be a one in five eagle.

434
00:46:12,120 --> 00:46:17,010
You got five heads. Okay.

435
00:46:18,510 --> 00:46:21,540
How many people got one head when they did it?

436
00:46:23,760 --> 00:46:27,270
One, two. So we got another two people who got one head.

437
00:46:28,560 --> 00:46:39,660
How many people got three heads? So one, two, three, four, five, six, seven, eight, nine, ten, 11, 12, 13, 14, 15, 16, 17, 18.

438
00:46:40,080 --> 00:46:45,650
How many people. Oops I've got 18 on here. Three, four, five, six, seven, eight, nine, ten, 11.

439
00:46:45,660 --> 00:46:49,110
Well it goes right up the top and further. How many people got?

440
00:46:49,110 --> 00:46:57,049
Four. 19.

441
00:46:57,050 --> 00:47:00,530
So that goes off the top. Very similar to the before to the three.

442
00:47:00,740 --> 00:47:05,390
How many people got six? My goodness.

443
00:47:05,800 --> 00:47:11,320
Two. How many people got five? One, two, three, four.

444
00:47:11,330 --> 00:47:17,030
And plus the what we got here before. So let's here we got another four on top of the one we got before.

445
00:47:17,030 --> 00:47:20,359
We got two with six. How many?

446
00:47:20,360 --> 00:47:23,480
Got one. How many got none.

447
00:47:25,660 --> 00:47:35,260
Okay. So the probability of getting six heads in a row in Atlanta is twice is less than 0.016 if they didn't have an unbiased coin.

448
00:47:35,500 --> 00:47:40,960
But it happened. We have enough people throwing the coins for it to happen.

449
00:47:42,900 --> 00:47:46,410
Why do we accept point nought five?

450
00:47:46,830 --> 00:47:52,560
It's just a convention that I think is probably rather a two lower barrier myself,

451
00:47:52,860 --> 00:47:57,190
but I do have a double headed coin and I play I play with groups like this.

452
00:47:57,210 --> 00:48:02,520
I've stopped doing it cause I always drop them on the floor, but they start asking to see the coin.

453
00:48:02,670 --> 00:48:06,479
After it's come up, it gets into row four or five times.

454
00:48:06,480 --> 00:48:09,750
I went about people's 0.05. They stop trusting me.

455
00:48:10,110 --> 00:48:13,770
And by the time it's six, everybody in the room thinks I pulled the double headed coin.

456
00:48:14,010 --> 00:48:17,160
Nobody believes that I can actually control it that much.

457
00:48:18,060 --> 00:48:21,550
Okay, so we're going to apply we're going to start just that.

458
00:48:21,570 --> 00:48:28,080
We've learned a p value is saying if nothing's going on, how often would you get a result like this for a chance?

459
00:48:28,380 --> 00:48:32,280
Now, I've been teaching people about statistics for quite a few years,

460
00:48:32,280 --> 00:48:38,669
and I used to evaluate people's subjective understanding of certain terms before and after

461
00:48:38,670 --> 00:48:44,340
workshops to see how effective I was being or how effective participants thought I was being.

462
00:48:44,670 --> 00:48:48,900
And I'd give them words like odds ratio or risk ratio, and they would score it.

463
00:48:48,910 --> 00:48:54,020
So five would be, I understand the term and could I explain it to somebody else?

464
00:48:54,030 --> 00:48:59,910
I understand the term but couldn't define it. Three I have a vague idea what it knows what it means.

465
00:49:00,300 --> 00:49:04,470
Don't ask me any more questions too. I've heard of it, but I've never understood it.

466
00:49:04,470 --> 00:49:07,950
I'm the clue. And one was I've never heard of the term.

467
00:49:08,190 --> 00:49:13,380
Okay, so five was better and with odds ratio.

468
00:49:13,830 --> 00:49:18,450
This is before the workshop over here and this is after the workshops over there.

469
00:49:18,720 --> 00:49:24,540
Before the workshop, 70% of people didn't really know what it means.

470
00:49:24,930 --> 00:49:28,950
Certainly 60% hadn't a clue what it meant, even if they'd heard a bit before.

471
00:49:29,460 --> 00:49:38,550
40% haven't even heard of it before. After the workshop, somehow, 2% still hadn't never heard of the trials before,

472
00:49:40,080 --> 00:49:50,520
but a staggering 80% thought they understood it and a staggering 40% thought they could actually define it and explain it to somebody else.

473
00:49:50,520 --> 00:49:54,180
Right. So these lines coming down like this is a good sign.

474
00:49:55,080 --> 00:50:00,090
So my question to you is, is this a statistically significant result?

475
00:50:01,680 --> 00:50:07,020
So I'm asking you, has this happened by chance or is this a real result of the workshop?

476
00:50:07,830 --> 00:50:12,569
Now I'm going to get you to vote. And by now we're friends enough for you to be able to stick your hands in the air.

477
00:50:12,570 --> 00:50:15,780
Okay. Who thinks this is statistically significant?

478
00:50:16,260 --> 00:50:21,070
Hands up. Who thinks it's not statistically significant.

479
00:50:21,670 --> 00:50:24,700
Thank you. Call to two of you and.

480
00:50:24,910 --> 00:50:29,350
And who's sitting on the fence? I really.

481
00:50:29,380 --> 00:50:36,280
Mike well done. Well done, my man. Okay. This was a highly statistically significant result.

482
00:50:36,370 --> 00:50:39,770
P less one in 10,000 or less on.

483
00:50:42,920 --> 00:50:47,400
When people voted. You should have voted.

484
00:50:47,420 --> 00:50:52,640
Don't know, because I didn't tell you how many people that was, Donald.

485
00:50:53,000 --> 00:50:56,270
Did I? And it could have been like my Potter trial.

486
00:50:57,020 --> 00:50:57,800
We can infer that.

487
00:50:58,860 --> 00:51:07,790
But Paul inferred that we should cleverer kids so the don't knows and the people who voted yes were right and call you were unique in being wrong.

488
00:51:07,880 --> 00:51:13,700
It was statistically significant. Okay.

489
00:51:14,240 --> 00:51:19,549
However, people are nice. People are nice. And you know, it doesn't mean I was teaching very well.

490
00:51:19,550 --> 00:51:23,510
It just means that they were pretending it was self-assessed understanding.

491
00:51:23,900 --> 00:51:28,190
So I'm not nice. So I decided to trick them.

492
00:51:28,400 --> 00:51:35,180
And in the evaluation forms I also put a term I never, ever covered mog to see if they were just being nice.

493
00:51:35,180 --> 00:51:40,490
They'd say they'd heard of it and were better at it, and lo and behold, the lines come down.

494
00:51:40,940 --> 00:51:46,360
They know this term more at the end of the workshop. Okay.

495
00:51:46,440 --> 00:51:51,470
A lot less. A lot less. But do you think this is a statistically significant result?

496
00:51:52,010 --> 00:51:56,270
We get to vote again. Who thinks? Yes, this is a statistically significant result.

497
00:51:56,270 --> 00:52:00,139
Hands off, same class. It's the same same same class.

498
00:52:00,140 --> 00:52:09,740
Hands up. A statistically significant to hands off for not statistically significant hands off sitting on the fence, most of you.

499
00:52:09,980 --> 00:52:14,030
You can't sit on the fence this time because it's the same class.

500
00:52:14,030 --> 00:52:21,650
You yourself got the don't know it's been deleted here. You cannot sit on the fence because this is the same class you're only allowed yes or no.

501
00:52:24,450 --> 00:52:29,040
It's highly statistically significant. This was not a chance occurrence.

502
00:52:29,820 --> 00:52:33,540
This was a real effect. There's a couple of thousand people.

503
00:52:33,540 --> 00:52:41,939
I thought, this is an utterly true effect. I think the clinical significance, that level of their understanding was negligible.

504
00:52:41,940 --> 00:52:47,040
But this was a trip. So whether it was a social compliance effect, people being nice,

505
00:52:47,850 --> 00:52:53,270
but it was a not a chance effect or whether it was because I said, Did you see that really weird word in that book?

506
00:52:53,330 --> 00:52:59,700
What did that mean? And they actually do know it at the end of the workshop, but it's not a chance effect.

507
00:53:02,050 --> 00:53:08,080
And this is very important because authors and editors love P values like that.

508
00:53:08,740 --> 00:53:15,160
So they love P values like that. But they can belong to things as meaningless as this.

509
00:53:17,950 --> 00:53:20,710
Hmm. Bit of a problem.

510
00:53:23,350 --> 00:53:31,630
Any real difference between things, however tiny it is, will become statistically significant if you take a large enough sample size.

511
00:53:31,900 --> 00:53:43,800
And I just thought. Lots and lots of lots and lots of people. So a very small p value doesn't mean you got a marvellous treatment necessarily.

512
00:53:44,400 --> 00:53:50,340
All it means is the effect that's being reported is unlikely to have occurred by chance.

513
00:53:51,060 --> 00:53:54,960
If there was nothing going on, there's something going on. Whether it's bias.

514
00:53:56,730 --> 00:54:00,460
There's something going on. Hmm.

515
00:54:04,420 --> 00:54:10,840
It's a very important message. The fact that something statistically significant or even highly statistically

516
00:54:10,840 --> 00:54:16,600
significant does not mean it's clinically relevant can be due to bias like this lady.

517
00:54:16,930 --> 00:54:23,170
She weighs herself like this, her weight style. It's there's a bias in the measurement system, I think a little bit.

518
00:54:25,570 --> 00:54:34,180
Or it can be just you've got to have a huge sample size. So for example, let's say you've got children who are not going to be very tall.

519
00:54:34,180 --> 00:54:38,590
They're obviously got small stature. Parents are small, but they want to be taller.

520
00:54:39,190 --> 00:54:46,450
Right. And you give them a penny to put in each shoe. You are definitely going to make them taller, a trivial amount.

521
00:54:46,540 --> 00:54:53,229
You're going to make them significantly, statistically, significantly taller, but you're not going to make them clinically,

522
00:54:53,230 --> 00:54:57,020
significantly taller is going to alter their lives because they're walking around with a penny in each shape.

523
00:54:58,180 --> 00:55:02,560
So you've got to distinguish clinical significance from statistical significance.

524
00:55:02,980 --> 00:55:07,240
So the P value tells us something, but it doesn't actually allow us to see enough.

525
00:55:07,270 --> 00:55:12,880
It would be nice to see a little bit further than the p value allows us to see, wouldn't it?

526
00:55:13,390 --> 00:55:16,830
Something about you know, it just tells us that it's not a chance occurrence.

527
00:55:16,840 --> 00:55:27,280
It doesn't tell us whether it's an important effect. So is there a better way of expressing uncertainty due to chance that gives us more information?

528
00:55:29,430 --> 00:55:35,700
Come back. Confidence interval. Yes, excellent.

529
00:55:35,700 --> 00:55:46,800
The confidence interval. So we now going to do an introduction to confidence intervals and learning from Rafael yesterday.

530
00:55:47,340 --> 00:55:50,670
I've decided that I need to give you sweeties.

531
00:55:53,190 --> 00:55:57,840
So I have at home a great big barrel of sweets.

532
00:55:58,170 --> 00:56:04,680
And it's that kind of chocolate éclairs. And they've got green wrappers or they've got gold wrappers on them.

533
00:56:05,670 --> 00:56:08,850
Okay. And I was told there were 50 of you here.

534
00:56:09,060 --> 00:56:15,000
So this morning, I put my hand in the bag and I drew out 50 sweets without looking.

535
00:56:17,220 --> 00:56:23,200
The question is how many of the sweets were green? I don't know.

536
00:56:24,220 --> 00:56:28,870
I don't know how many sweets are green in this fight? If someone had to guess, you had to guess.

537
00:56:28,960 --> 00:56:32,450
Let's say there was olives going to give £100. The person you guess is right.

538
00:56:32,470 --> 00:56:36,790
How many green sweets would you get in the bag? 40.

539
00:56:38,530 --> 00:56:42,910
30. Mike's guessing 25. Right.

540
00:56:42,940 --> 00:56:46,690
Let's see if we can get this to right.

541
00:56:46,690 --> 00:56:50,500
So Mike's getting 25 here. Somebody guessed 30.

542
00:56:51,400 --> 00:56:55,180
Any other guesses? Somebody is guessing that there are none.

543
00:56:55,870 --> 00:57:01,180
Yup. Oops. Some of these guessing this number will be true.

544
00:57:02,090 --> 00:57:09,110
Pardon? 32, 33, 23, 23.

545
00:57:09,770 --> 00:57:13,760
Okay. What's the least number there could be? We've had zero. What's the most?

546
00:57:14,360 --> 00:57:26,630
5050. Okay. So actually, I can't really see this very well, but the truth could lie anywhere along this line from 0 to 50.

547
00:57:26,780 --> 00:57:29,720
Couldn't take a truth. Could be anywhere between nought and 50.

548
00:57:30,320 --> 00:57:37,330
If we were all coming in, you know, one, two, three, we could get £100 a hole by getting it out afterwards.

549
00:57:37,340 --> 00:57:40,430
You got to quit it because the truth.

550
00:57:40,470 --> 00:57:47,180
We know the truth. Well, 100% confident, but the truth lies somewhere between nought and 50.

551
00:57:48,590 --> 00:57:53,210
So if you want as one 100% confidence interval where we know the truth lies.

552
00:57:54,020 --> 00:57:57,200
Okay, Azzam, you have been very, very good today.

553
00:57:57,770 --> 00:58:04,490
So I'm going to let you have a sweet thanks. But before you eat it, you have to show us what colour it is.

554
00:58:06,770 --> 00:58:12,860
It's a gold one. Okay. So the question is, how many green suites did I put in the bag?

555
00:58:13,910 --> 00:58:17,450
What, what? What what are a 100% confidence intervals?

556
00:58:17,450 --> 00:58:23,320
Now, this for 39. Okay, I can't put 55.

557
00:58:23,330 --> 00:58:28,310
Kind of like because we know at least one is gold in gold.

558
00:58:28,610 --> 00:58:35,330
So now 100% confidence intervals lie somewhere between 49 and nought.

559
00:58:39,110 --> 00:58:45,020
I want five of you to take a suite and then I'll take a suite each and then pass the bag on.

560
00:58:53,830 --> 00:58:58,030
Okay. So five, you take the last week. Okay, now let's have a look what we've got here.

561
00:58:58,060 --> 00:59:01,690
You want to hold them up? Okay, we've got three, four goals.

562
00:59:01,690 --> 00:59:05,349
One green. So now we're schedule 100% confidence intervals.

563
00:59:05,350 --> 00:59:11,940
Lie. One, two, one, two, 45 right there.

564
00:59:11,950 --> 00:59:14,950
53 to this box. We better start playing this game quickly.

565
00:59:14,950 --> 00:59:32,740
Keep passing the bag. You can take out more than can just keep passing the bank, taking on anyone who's got a suite so far.

566
00:59:33,010 --> 00:59:40,870
Then the next round, raise your hands. Okay, so we've this time we've got four green and two gold.

567
00:59:41,830 --> 00:59:47,940
So we've got two gold. So for green.

568
00:59:47,940 --> 00:59:52,080
So we know another four of these if I'm doing this right to gold.

569
00:59:53,100 --> 01:00:02,910
Okay. Well, next, a lot of people raise your hands and we have two green and four gold.

570
01:00:12,020 --> 01:00:18,530
Okay. Next, two people. Raise your hands. We got three for gold and one green.

571
01:00:22,720 --> 01:00:25,780
Yeah. I hope I'm doing this accurately.

572
01:00:25,780 --> 01:00:32,580
Next, a lot of people hold you. Hold up your hands. Oh, can't see because of the light.

573
01:00:32,590 --> 01:00:37,830
One green and three gold. Pass it back down to the front for the people who didn't get any.

574
01:00:37,880 --> 01:00:48,220
If you take them off because I want to compromise on £3, so I pass it down that way any more.

575
01:00:48,220 --> 01:00:51,430
People have got it. Haven't held up the hands yet? No.

576
01:00:59,570 --> 01:01:03,559
Okay. A lot to hold up your hands. Bright one, green three.

577
01:01:03,560 --> 01:01:11,020
Gold. I usually do this in workshops where there's 20 people.

578
01:01:11,230 --> 01:01:20,770
But I didn't think you'd be happy if you didn't get a suite, so I had to sit next to people, hold up your hands.

579
01:01:21,220 --> 01:01:25,630
So we've got three green, three gold.

580
01:01:29,140 --> 01:01:35,500
Like a lot of people. Two of each.

581
01:01:39,940 --> 01:01:47,829
Keep going. Next slot, hold up.

582
01:01:47,830 --> 01:01:56,010
And so we got two more. Excellent.

583
01:02:01,720 --> 01:02:07,810
Next slot. Next slot.

584
01:02:09,010 --> 01:02:15,550
Empty. It's empty. There were, in fact, 18 free sweets in that bag.

585
01:02:16,090 --> 01:02:19,090
Okay, so there were 18. Nobody guessed 18.

586
01:02:19,090 --> 01:02:23,440
You're off the hook. Okay, so there were 18.

587
01:02:23,830 --> 01:02:33,230
What do you notice about these lines? You'd think that getting shorter and shorter.

588
01:02:33,560 --> 01:02:38,460
So we're 100% confident. We're getting more and more confident.

589
01:02:38,480 --> 01:02:41,990
The uncertainty is getting less and less and less.

590
01:02:42,560 --> 01:02:49,300
Why is the uncertainty getting less and less and less? Because we've got information.

591
01:02:49,510 --> 01:02:52,630
Every time we take a suite out of the bag, it gives us information.

592
01:02:53,410 --> 01:02:59,230
And the more suites we take out of the bag, the more information there is, the less uncertainty there is.

593
01:03:00,400 --> 01:03:04,750
Okay. So the width of that is a measure of how much uncertainty.

594
01:03:04,990 --> 01:03:08,470
And as we get more and more certainty till we know the truth, which was 18.

595
01:03:09,790 --> 01:03:14,680
What else do we know to spot those lines? Because 100% confidence intervals.

596
01:03:14,890 --> 01:03:18,520
The truth. All of them contain the truth. We knew they would.

597
01:03:18,730 --> 01:03:26,010
We were 100% certain the truth lay between them. So if you draw a line down from 18, it intersects every line.

598
01:03:26,020 --> 01:03:33,620
Every line, contain the truth. Confidence intervals are just like this.

599
01:03:34,220 --> 01:03:38,030
They're measures of how much uncertainty there is. Okay.

600
01:03:38,630 --> 01:03:44,180
But because there's always the future, things might happen again and we cannot be 100% sure.

601
01:03:44,210 --> 01:03:47,900
We've only got a sample. This time we were able to take every suite out of the body.

602
01:03:47,930 --> 01:03:53,510
We were able to look at the entire universe of sweet stuff. But actually, in the real world, we can only ever take a sample.

603
01:03:54,140 --> 01:03:59,720
So what? We settle for a confidence intervals where we're certain that the truth lies what we usually say.

604
01:04:04,150 --> 01:04:08,500
Well, we usually go for 95% certain the truth lies between those arms.

605
01:04:09,010 --> 01:04:14,649
So 5% of the time will allow ourselves to be wrong, which is the same as people's 0.05.

606
01:04:14,650 --> 01:04:18,310
If you notice, 5% of the time we'll be wrong.

607
01:04:20,710 --> 01:04:29,890
So if we were to put the uncertainty around our Potters by our Potters Test when we only had five in each arm,

608
01:04:30,220 --> 01:04:38,980
the uncertainty went all the way from 0.63 and it could have gone right up to six and a bit times better.

609
01:04:39,460 --> 01:04:41,560
So here's another thing about these graphs.

610
01:04:42,520 --> 01:04:48,190
You get your best estimate as the blob, and then you get your uncertainty in terms of confidence intervals.

611
01:04:49,810 --> 01:04:53,140
And this graphical information is usually shown at the slide.

612
01:04:53,440 --> 01:04:59,740
So it tells you where the blob is too, and that tells you the bottom end and the top end of your confidence intervals.

613
01:05:00,310 --> 01:05:07,270
And it tells you whether you're being 95% certain the truth lies 80% certain where the truth lies or 100% certain.

614
01:05:07,450 --> 01:05:11,260
Well, you never have hundred percent, but 99%.

615
01:05:11,590 --> 01:05:19,240
So we're getting to be able to interpret these lines and lights out or an arrow.

616
01:05:19,480 --> 01:05:22,720
That's simply because I didn't draw the graph. It means it goes on to six.

617
01:05:22,990 --> 01:05:26,649
I shouldn't really have done it, but I could have done the graph a bit better.

618
01:05:26,650 --> 01:05:29,980
But I was in a hurry. Here, here.

619
01:05:30,040 --> 01:05:34,690
This one actually fits on the graph, which goes up to five, so we don't have to have the arrow on it.

620
01:05:35,050 --> 01:05:36,640
So this is a bigger sample size.

621
01:05:36,640 --> 01:05:43,660
This is when we had ten in each arm and notice because we've got more information, because we got more sweets out of the bag, as it were.

622
01:05:43,660 --> 01:05:48,880
We have a bigger sample. Our uncertainty is much shorter.

623
01:05:49,870 --> 01:05:54,550
So this time our uncertainty only goes from point eight, 824.54.

624
01:05:55,150 --> 01:06:03,400
You can think of the bottom end and the top end of the confidence intervals as a best case, worst case scenario.

625
01:06:03,610 --> 01:06:07,749
We think potters is twice as good as Potter's.

626
01:06:07,750 --> 01:06:12,940
Yes, Potter's is twice as good as placebo, but actually it could be doing harm.

627
01:06:13,420 --> 01:06:17,680
A 0.88. Or it could be as much as four and a half times as good.

628
01:06:18,600 --> 01:06:24,700
So that tells us more than a peep. And it tells us, you know, what kind of effect we're talking about.

629
01:06:25,810 --> 01:06:28,720
So on this one, what's happening on this one?

630
01:06:31,510 --> 01:06:39,490
You remember we said if we got 15 in each arm, we now knew we now had a statistically significant result.

631
01:06:39,520 --> 01:06:41,200
Why was it statistically significant?

632
01:06:43,030 --> 01:06:52,810
Post-punk values, just that 105 because we wouldn't have got a result like that by chance, more than one in 20 of the time.

633
01:06:52,960 --> 01:06:57,770
And if you look at this, these confidence intervals, there's a little tiny little gap there.

634
01:06:57,790 --> 01:07:05,740
You can read it off here. It's bigger. It's not crossing this line of no difference between the two treatments.

635
01:07:06,190 --> 01:07:12,400
So now, even in the worst case scenario, we think what is is probably better than placebo.

636
01:07:13,840 --> 01:07:21,280
So if a 95% confidence interval exactly sits on that line, what's your p value?

637
01:07:25,460 --> 01:07:33,230
If you're 0.05 and if it doesn't cross that line of no treatment difference, you know,

638
01:07:33,230 --> 01:07:40,309
your P-value is what you know it's less than 0.05 because that's what you're 95% certain.

639
01:07:40,310 --> 01:07:46,610
The truth is 95% of the time it's bigger than the no the null effect.

640
01:07:47,620 --> 01:07:55,720
Okay. So here. Here, we've got the confidence intervals. So this is this is how your blogger gram looks.

641
01:07:55,730 --> 01:08:04,219
It will give you all these studies down the side. It will usually, so to say carpenter and 2007 glaze you 1992 and that sort of thing all down the

642
01:08:04,220 --> 01:08:09,140
side give you the names of the studies will tell you how many were in a study,

643
01:08:09,680 --> 01:08:16,970
how many got the outcome of interest in the treatment arm, how many in the placebo arm or control arm?

644
01:08:17,240 --> 01:08:22,890
And then you get your totals. Okay. What's this little diamond here?

645
01:08:23,970 --> 01:08:28,980
We've been having blobs with with with uncertainty surrounding the best estimate.

646
01:08:29,430 --> 01:08:32,640
That may turn out that we may do. Okay.

647
01:08:33,150 --> 01:08:39,710
Meta analysis. What's that? Something special.

648
01:08:42,830 --> 01:08:50,389
Yeah. Okay. Analysis is a nice, posh term.

649
01:08:50,390 --> 01:08:53,330
It sounds really good. Doesn't that matter? Unless it's every size board.

650
01:08:53,630 --> 01:09:00,770
But all it means is putting all the results we've got together, getting a summary for all the results.

651
01:09:01,100 --> 01:09:11,149
So if we had all these these we have these five trials, we had the ball up, we said we were all like little parts of the same trial, this big trial.

652
01:09:11,150 --> 01:09:14,540
So we add up all these there were 150 in total.

653
01:09:15,050 --> 01:09:22,550
If you add up, all those all come to 60, add up all these 250 that come 220 and then this this bit here,

654
01:09:22,580 --> 01:09:26,810
the pointy bits here, top and bottom are your best estimate.

655
01:09:27,020 --> 01:09:35,930
So that's should two and it's equivalent to your blobs and these are your confidence intervals, right?

656
01:09:36,740 --> 01:09:42,649
So that's how that diamonds interpreted. But it's when you add all the studies together and meta analysis.

657
01:09:42,650 --> 01:09:47,210
When we do that, it has to be all studies. They have to be the same.

658
01:09:47,330 --> 01:09:52,040
I mean, the time of the study, the size of the studies does not have to be the same.

659
01:09:52,220 --> 01:09:59,390
In fact, we have very different sized studies here. And in fact, that's where meta analysis is useful, because if you've got lots of little studies,

660
01:09:59,780 --> 01:10:08,510
none of which are statistically significant, all of which have the uncertainty, including a harmful effect, you put them all together.

661
01:10:08,570 --> 01:10:14,540
Well, it will become statistically significant, maybe, you know, you don't get five studies all on the same time by chance.

662
01:10:14,810 --> 01:10:17,990
But but there's another question. Do all the studies have to be the same?

663
01:10:20,570 --> 01:10:29,899
They have to be the same about the same thing. There's no point having a study that's looking at warfarin and time to anticoagulation and heparin.

664
01:10:29,900 --> 01:10:33,500
I don't know if the medics warfarin takes a long time to work.

665
01:10:33,500 --> 01:10:36,320
Heparin works immediately. You can't put them together in the same.

666
01:10:37,470 --> 01:10:46,200
Plotz So for example, if you say how long does a study is looking at how long bones take to repair and you have children and old people?

667
01:10:46,470 --> 01:10:51,930
Well, maybe you can't really add that up together. Maybe it's different in children from what it is in old people.

668
01:10:52,380 --> 01:10:58,050
So, yes. And if you look at the validity questions, when you get these checklists for systematic review,

669
01:10:58,050 --> 01:11:05,700
one of the questions saying in those things, it says if the results of the study were combined, was it reasonable to do so?

670
01:11:06,120 --> 01:11:10,199
And it's just saying with a study similar enough, looking at a similar population,

671
01:11:10,200 --> 01:11:15,480
a similar enough intervention, similar enough outcomes to be able to be put together sensibly.

672
01:11:15,660 --> 01:11:19,220
Now I said time, I mean is it randomised control.

673
01:11:19,230 --> 01:11:22,530
It has to be the same type of same type of study.

674
01:11:22,830 --> 01:11:29,400
Well not necessarily. But you could argue it's a judgement call.

675
01:11:29,430 --> 01:11:37,470
Not necessarily. But if you have randomised controlled trials you'd be unlikely to want to add in a lesser quality study.

676
01:11:37,710 --> 01:11:41,580
You'd be more likely to believe the randomised controlled trials.

677
01:11:42,700 --> 01:11:50,890
But if they're both, you know, moving in the same direction and they're both giving similar estimates, then that's quite powerful evidence.

678
01:11:54,720 --> 01:11:58,290
Any questions about confidence intervals or the blowback from.

679
01:12:02,610 --> 01:12:06,990
Okay. It's just the size of the squid.

680
01:12:08,730 --> 01:12:15,180
How do you decide the size of this? Excuse me?

681
01:12:15,750 --> 01:12:26,139
The size of the square. Is tells you how much information is in a particular trial and is, if you like.

682
01:12:26,140 --> 01:12:31,120
It's exactly a correlate with the arm. So the narrow the arms, the bigger the square.

683
01:12:31,540 --> 01:12:36,730
They they're both telling the same information. And the reason the blob gets bigger is people saying, hang on.

684
01:12:37,120 --> 01:12:41,560
The least informative studies are the ones with the biggest uncertainty surrounding

685
01:12:41,650 --> 01:12:44,770
of these great big long arms that's giving the wrong visual impression.

686
01:12:44,770 --> 01:12:49,750
They're the ones that you're noticing. And the really important studies with the little narrow arms, you hardly spot them.

687
01:12:50,020 --> 01:12:53,730
We've got to correct this, but otherwise people are going to get a misleading impression.

688
01:12:53,750 --> 01:12:58,570
They corrected it by making the blobs bigger. So the blob is how much information is in there?

689
01:13:00,370 --> 01:13:06,490
I think I better stop now. I've gone on 10 minutes more than I should have.

690
01:13:10,050 --> 01:13:17,590
Yeah. No. Now, you can now all interpret the blowback from I believe the next thing was a blowback from.

691
01:13:17,800 --> 01:13:21,360
I think I had a blowback to go back to the one that we had right at the beginning.

692
01:13:21,480 --> 01:13:28,690
I think the slide three now you can see that you can start interpreting these things very, very quickly.

693
01:13:30,610 --> 01:13:34,270
You now know this. Are there any significant studies? They're on their own.

694
01:13:39,530 --> 01:13:47,660
Mariam might have been. You can't really tell from this and might be borderline significant missing information now when you bung them all together.

695
01:13:51,490 --> 01:13:54,490
Yeah. They are statistically safe when you do. Sorry.

696
01:13:54,490 --> 01:14:00,190
Bung them all together is the epidemiological technical term for meta analysis.

697
01:14:00,340 --> 01:14:09,380
Okay. But when you do the match analysis, you notice that the long arms of Diamond Dog crossing the treatment pool.

698
01:14:09,400 --> 01:14:16,090
I'm going to go on for 2 minutes because I can't resist. Just going to say, does this mean the treatment works or it doesn't work?

699
01:14:16,540 --> 01:14:19,800
Does hypothermia work or not work? Yes.

700
01:14:22,510 --> 01:14:28,840
Right? Yeah, right. So before it was saying favours policy but favours plotters over here, wasn't it?

701
01:14:29,140 --> 01:14:35,050
And now it's not telling you so. Does it always favour the treatment on one side or the other?

702
01:14:36,400 --> 01:14:39,430
60 Minutes. Okay.

703
01:14:39,790 --> 01:14:44,740
If you remember, this side is always less on this side is always more.

704
01:14:45,200 --> 01:14:48,549
Okay. On to work out which side favours it.

705
01:14:48,550 --> 01:14:54,550
You've got to think of the outcome. If it's something you don't want, like mortality and incapacity, you want less of it.

706
01:14:55,020 --> 01:15:00,069
If it's something you do what like proportion giving up smoking and you want it on the other side.

707
01:15:00,070 --> 01:15:06,580
So which side it sits on depends on whether the outcome is something you want or you don't want.

708
01:15:06,940 --> 01:15:10,120
It's something you don't want. When you want less of it, it's something you do want.

709
01:15:10,720 --> 01:15:14,440
Lives saved, number of people cured, and it's on the other side.

710
01:15:14,440 --> 01:15:19,750
And that's true whether it's a risk difference or a relative risk. So this is easy, isn't it?

711
01:15:20,380 --> 01:15:24,460
And we have done any equations. I've done any calculations.

712
01:15:26,350 --> 01:15:32,350
Which is lucky because I don't know how to do them. Okay, well, that's it.

713
01:15:32,650 --> 01:15:35,440
I'm going to have fun. Sorry. Sorry. I bit into your small group time.