1 00:00:00,866 --> 00:00:02,133 And thank you so much. 2 00:00:02,133 --> 00:00:03,666 I'm delighted to be here. 3 00:00:03,666 --> 00:00:07,366 The opportunity to address, to talk of participants and people online 4 00:00:07,832 --> 00:00:11,665 to talk about really some of my bugbears when I read papers 5 00:00:11,665 --> 00:00:14,665 about systematic reviews and meta analysis. 6 00:00:16,498 --> 00:00:18,898 of course, these things have become critical 7 00:00:18,898 --> 00:00:23,031 for decision making and chiefly in health through nice and simple 8 00:00:23,031 --> 00:00:26,498 organizations around the world, but across various disciplines. 9 00:00:27,364 --> 00:00:30,664 and so they're really one of the most influential types of research, 10 00:00:30,964 --> 00:00:34,197 which means that articles about systematic reviews 11 00:00:34,197 --> 00:00:37,597 and that analysis, are really highly cited. 12 00:00:38,197 --> 00:00:42,630 I should just permit permeate so often appear at the top of pyramid. 13 00:00:43,263 --> 00:00:46,263 This is one of my favorite pyramids. 14 00:00:48,163 --> 00:00:50,996 so the high five just met. 15 00:00:50,996 --> 00:00:54,096 You just mentioned that I've got a lot of citations, and I 16 00:00:55,196 --> 00:00:55,996 quite often get 17 00:00:55,996 --> 00:00:58,996 well, so sometimes get emails like this. 18 00:00:58,996 --> 00:01:01,595 Congratulations. You're very well cited. 19 00:01:01,595 --> 00:01:05,528 This and I, it's nice, but I don't take it very seriously. 20 00:01:05,528 --> 00:01:08,528 Although I did picture on this spot from silent 21 00:01:08,695 --> 00:01:12,695 that, is a really interesting one because it allows you to rank, 22 00:01:14,195 --> 00:01:16,294 174 million publications 23 00:01:16,294 --> 00:01:19,927 by number of citations across everything and all time. 24 00:01:20,527 --> 00:01:23,394 And if you look at that and the most cited papers of all time 25 00:01:23,394 --> 00:01:28,727 back from 1951, it's got 229,000 about proteins. 26 00:01:28,994 --> 00:01:32,993 But if you look through the list, you find various papers about systematic 27 00:01:32,993 --> 00:01:36,193 reviews and meta Nazis, and these are the ones 28 00:01:36,193 --> 00:01:39,193 that rank high in that list. 29 00:01:39,393 --> 00:01:42,393 these are the number of citations, 30 00:01:43,226 --> 00:01:43,926 and obesity, 31 00:01:43,926 --> 00:01:47,026 I mean, quite a lot in the top 120, 32 00:01:48,092 --> 00:01:50,425 for a relatively young field. 33 00:01:50,425 --> 00:01:52,759 So I thought, I want to look through this list. 34 00:01:52,759 --> 00:01:55,992 I think, oh, gosh, people keep citing that in ways that I don't like. 35 00:01:56,658 --> 00:01:59,658 So I thought I would try to talk around the idea, 36 00:02:00,225 --> 00:02:03,225 what I'm going to do if these are the papers, slightly reordered, 37 00:02:03,758 --> 00:02:07,191 and there are two paths which are essentially about the same thing. 38 00:02:07,924 --> 00:02:12,157 So I'm going to go through these five next time. 39 00:02:13,824 --> 00:02:17,690 and talk about why I think 40 00:02:17,690 --> 00:02:21,957 they are very often misused or misunderstood. 41 00:02:22,457 --> 00:02:24,423 Try and offer some solutions. 42 00:02:24,423 --> 00:02:26,290 And, and 43 00:02:26,290 --> 00:02:28,823 that's it then. Drawing to a close. 44 00:02:28,823 --> 00:02:32,089 so one of the topics first is misuse of reporting guidelines. 45 00:02:32,456 --> 00:02:35,356 The second is misuse of this I squared statistic. 46 00:02:35,356 --> 00:02:39,089 And the third is misuse of random effects meta analysis. 47 00:02:39,622 --> 00:02:43,189 The fourth is misuse of tests for funnel plot asymmetry. 48 00:02:43,755 --> 00:02:46,188 And the fifth is misuse of risk of bias. 49 00:02:46,188 --> 00:02:48,722 Assessment tools. 50 00:02:48,722 --> 00:02:50,555 Before I start, 51 00:02:50,555 --> 00:02:53,555 I'm sure I know you all know this, but just, 52 00:02:54,288 --> 00:02:57,054 for completeness, I just want to have one slide 53 00:02:57,054 --> 00:02:58,654 on each of what is a systematic treatment. 54 00:02:58,654 --> 00:03:01,654 What is a that analysis, at least in my head. 55 00:03:02,587 --> 00:03:05,154 systematic reviews, I mean, are often thought of as reviews 56 00:03:05,154 --> 00:03:06,854 or the method section. 57 00:03:06,854 --> 00:03:10,053 when they first came about, it's my impression 58 00:03:10,053 --> 00:03:14,186 was that they really thought of it as studies of studies and meta analysis. 59 00:03:14,586 --> 00:03:18,053 combined results of studies increasingly, 60 00:03:18,053 --> 00:03:21,386 that the term systematic review and the idea is applied 61 00:03:21,586 --> 00:03:24,886 to what I would consider reviews of literature rather than user studies. 62 00:03:25,952 --> 00:03:29,219 Now that a mixture of the two and some pieces work are actually 63 00:03:29,219 --> 00:03:33,252 a mixture of review, discussion, reviews of a papers, 64 00:03:34,785 --> 00:03:37,185 we can talk about that more if you don't know quite what I mean by that. 65 00:03:37,185 --> 00:03:38,951 But whichever approach one takes, 66 00:03:38,951 --> 00:03:41,951 I think they're characterized by following a protocol 67 00:03:42,185 --> 00:03:45,484 I'm using methods that are systematic, transparent, and rigorous, 68 00:03:46,384 --> 00:03:49,584 really trying to minimize biases. 69 00:03:49,617 --> 00:03:53,584 I systematic errors and random errors by bringing together, 70 00:03:54,017 --> 00:03:56,617 all of the information that is under the tackled. 71 00:03:56,617 --> 00:03:59,617 The question that we're trying to answer 72 00:04:00,117 --> 00:04:01,083 meta analysis. 73 00:04:01,083 --> 00:04:02,083 As I already mentioned, 74 00:04:02,083 --> 00:04:05,083 the statistical combination of results from multiple studies. 75 00:04:05,750 --> 00:04:09,216 This is a probably my favorite definition of a meta analysis for my coffee. 76 00:04:09,583 --> 00:04:10,583 What I like about it, 77 00:04:10,583 --> 00:04:14,382 I like a lot of them, is that it explicitly acknowledges the judgment 78 00:04:14,916 --> 00:04:19,415 that studies should be combined, which is a non statistical issue. 79 00:04:19,415 --> 00:04:21,115 And that's a really important part of the meta. 80 00:04:22,882 --> 00:04:23,815 It's such a pleasure to 81 00:04:23,815 --> 00:04:26,815 be in Oxford because it really is the home of 82 00:04:27,182 --> 00:04:30,181 so many pivotal ideas in this area too. 83 00:04:30,615 --> 00:04:34,314 you know, which Peter and his colleagues and in Thomas and colleagues 84 00:04:34,748 --> 00:04:37,748 copycat versions of it's a, 85 00:04:37,781 --> 00:04:40,114 very special place in this area. 86 00:04:40,114 --> 00:04:41,347 So let me get started. 87 00:04:41,347 --> 00:04:44,347 Paper number one, the Prisma statement, 88 00:04:45,447 --> 00:04:48,447 originally published in 2009. 89 00:04:48,713 --> 00:04:51,713 updated 2020. 90 00:04:51,913 --> 00:04:53,347 What's it all about? 91 00:04:53,347 --> 00:04:54,480 I'm sure you will know. 92 00:04:54,480 --> 00:04:57,446 It's a checklist and a flowchart 93 00:04:57,446 --> 00:05:00,446 for reporting, systematic review and all of that analysis. 94 00:05:00,679 --> 00:05:03,679 I was lucky enough to be at the Prisma meeting 95 00:05:03,846 --> 00:05:07,146 for this one back in June 2005. 96 00:05:09,212 --> 00:05:12,212 so it's something that's quite close to my heart, 97 00:05:12,379 --> 00:05:13,412 what it looks like. 98 00:05:13,412 --> 00:05:15,378 It's a checklist. 99 00:05:15,378 --> 00:05:18,178 A number of things it suggests you should do. 100 00:05:18,178 --> 00:05:20,645 And there's a checkbox to say whether you've done them, 101 00:05:20,645 --> 00:05:21,978 where you've done them in the report. 102 00:05:24,644 --> 00:05:26,778 And the flowchart is a 103 00:05:26,778 --> 00:05:29,777 suggested way of illustrating the numbers. 104 00:05:30,744 --> 00:05:33,744 of all the articles or studies that you've started with 105 00:05:34,244 --> 00:05:36,777 in your research and how they were whittled down 106 00:05:36,777 --> 00:05:40,043 to the number of included studies on the completed reports. 107 00:05:42,943 --> 00:05:44,376 What's the problem there? 108 00:05:44,376 --> 00:05:48,909 Well, how can I introduce these who have decided to do this? 109 00:05:48,909 --> 00:05:52,043 I thought, well, let's pick a pseudo random paper and see 110 00:05:52,043 --> 00:05:55,042 whether they illustrate the problem that I think is right. 111 00:05:55,142 --> 00:05:58,142 So I just looked at the most recent citation of Prisma, 112 00:05:59,475 --> 00:06:02,475 in scope, because if you do that, grab them by date. 113 00:06:02,775 --> 00:06:05,975 And this one was not in English, so I didn't look at that one. 114 00:06:06,608 --> 00:06:07,775 the next one was in English. 115 00:06:07,775 --> 00:06:11,075 It's around birds, urinary continents, in pregnancy. 116 00:06:11,841 --> 00:06:15,974 And if you can jump to the beginning of the method section, you read this text. 117 00:06:15,974 --> 00:06:20,407 The review is conduct it based on a guidelines from Prisma. 118 00:06:24,507 --> 00:06:24,874 Can you 119 00:06:24,874 --> 00:06:27,974 do a review according to Prisma will look at Prisma. 120 00:06:28,807 --> 00:06:30,840 let's just look at one item in more detail. 121 00:06:30,840 --> 00:06:34,606 Specify the methods used to decide whether study 122 00:06:34,606 --> 00:06:38,273 might actually complete two criteria or how many reviews, and so on. 123 00:06:38,439 --> 00:06:39,473 Well, okay. 124 00:06:39,473 --> 00:06:43,606 One reviewer looked at the keywords and drew a decision based on that. 125 00:06:44,206 --> 00:06:47,206 Now, clearly that's really ridiculous thing to do. 126 00:06:47,806 --> 00:06:50,172 but you get full marks in the Prisma 127 00:06:50,172 --> 00:06:53,172 and by telling people that's what you get, 128 00:06:53,438 --> 00:06:56,705 it's really clear that this should not be used. 129 00:06:57,305 --> 00:07:00,305 It is not intended as a guideline for doing systematic reviews. 130 00:07:00,305 --> 00:07:03,371 Despite almost 3.5 to you saying that's 131 00:07:03,371 --> 00:07:06,371 what you did. 132 00:07:06,804 --> 00:07:10,104 I have to say, the prisma elaboration has got 133 00:07:10,104 --> 00:07:13,537 lots of great, ideas and, and, 134 00:07:14,370 --> 00:07:16,937 information about how good reviews are done, 135 00:07:16,937 --> 00:07:21,737 but I think it's, it's still stopped short of being a guide for doing reviews. 136 00:07:22,237 --> 00:07:25,770 And the Prisma paper is very clear about what that is. 137 00:07:27,636 --> 00:07:28,369 What's the solution? 138 00:07:28,369 --> 00:07:30,069 Why? This would be easy. 139 00:07:30,069 --> 00:07:31,336 Don't use Prisma. 140 00:07:31,336 --> 00:07:34,769 Use one of the many documents, books, papers, 141 00:07:36,202 --> 00:07:38,835 standards that actually guide you in doing 142 00:07:38,835 --> 00:07:41,835 a systematic review. 143 00:07:43,802 --> 00:07:44,402 Okay. 144 00:07:44,402 --> 00:07:46,402 Number two, 145 00:07:46,402 --> 00:07:49,401 I squared the 146 00:07:49,535 --> 00:07:51,635 what is I squared. 147 00:07:51,635 --> 00:07:54,501 Well, it's a it's a simple statistic 148 00:07:54,501 --> 00:07:57,834 perhaps why it's an unpopular it. 149 00:07:59,901 --> 00:08:02,467 It measures inconsistency. 150 00:08:02,467 --> 00:08:06,534 That means the extent to which the results of the studies are consistent. 151 00:08:06,967 --> 00:08:09,033 Now when I say the results, I mean the estimates 152 00:08:09,033 --> 00:08:12,033 and the confidence interval that the results, 153 00:08:12,367 --> 00:08:16,233 and you can think of it as an, a measure of the extent to which they line up 154 00:08:16,233 --> 00:08:20,033 for the fact that they don't line up because of the converse consistency. 155 00:08:21,666 --> 00:08:24,666 I should be thought of as inconsistency. 156 00:08:25,299 --> 00:08:28,465 It so happens that it can also be interpreted in a different way. 157 00:08:28,865 --> 00:08:30,965 Perhaps statisticians. Right. 158 00:08:30,965 --> 00:08:35,432 As the proportion of variability in in the point estimates 159 00:08:36,298 --> 00:08:39,398 that is due to heterogeneity rather than chance. 160 00:08:40,265 --> 00:08:43,631 What's heterogeneity heterogeneity variation 161 00:08:43,931 --> 00:08:46,764 in the true effects across the studies. 162 00:08:48,997 --> 00:08:49,997 I've got some math here. 163 00:08:49,997 --> 00:08:52,464 That's not it's not important. 164 00:08:52,464 --> 00:08:53,664 This is a representation. 165 00:08:53,664 --> 00:08:57,830 The top one is a representation of the the second understanding where torsemide 166 00:08:57,830 --> 00:09:01,897 represents the amount of heterogeneity, which I'll talk about more as we go along. 167 00:09:02,130 --> 00:09:06,796 And the bottom is, you can break out the total variation 168 00:09:06,996 --> 00:09:09,863 into components due to heterogeneity, components 169 00:09:09,863 --> 00:09:12,862 due to sampling error. 170 00:09:13,862 --> 00:09:16,862 And I'll come back to that equation. 171 00:09:17,229 --> 00:09:19,129 but I do want to talk about what a bit more about 172 00:09:19,129 --> 00:09:22,962 what it means conceptually and what what we had in mind when we developed this, 173 00:09:23,028 --> 00:09:26,028 what we were trying to do, we think we achieved. 174 00:09:26,162 --> 00:09:28,295 Here's a match analysis for the number of studies 175 00:09:28,295 --> 00:09:31,295 we the trials of amantadine for flu, 176 00:09:32,494 --> 00:09:35,494 effects measured using odds ratio. 177 00:09:35,794 --> 00:09:40,327 The idea of r squared is that we look at this 178 00:09:41,427 --> 00:09:41,894 and it's a 179 00:09:41,894 --> 00:09:44,894 measure of the extent to which these agree 180 00:09:44,894 --> 00:09:47,893 with each other or disagree with each other. 181 00:09:48,960 --> 00:09:51,960 And the idea is that what we mean? 182 00:09:52,026 --> 00:09:54,826 There's not a lot of disagreement you can look at has 183 00:09:54,826 --> 00:09:57,826 and I mean maybe these are slightly different, but 184 00:09:57,859 --> 00:10:00,859 yeah, they essentially are consistent with each other. 185 00:10:01,293 --> 00:10:04,292 And the idea is that I can compared that intuition 186 00:10:04,426 --> 00:10:07,059 with your intuition from a different set of studies. 187 00:10:07,059 --> 00:10:11,459 But I'm only I can only I've get everything I need for this picture. 188 00:10:12,492 --> 00:10:14,925 And here these two really don't agree with each other. 189 00:10:14,925 --> 00:10:17,491 So there's more inconsistency here, 190 00:10:17,491 --> 00:10:19,858 even though these are in a completely different topic 191 00:10:19,858 --> 00:10:22,858 and measured on a completely different measurement scale. 192 00:10:23,291 --> 00:10:27,291 So I squared intuitively has got nothing to do with the measurement scale. 193 00:10:29,557 --> 00:10:30,391 Yeah. 194 00:10:30,391 --> 00:10:32,157 What's the problem in practice. 195 00:10:32,157 --> 00:10:34,357 Well, 196 00:10:34,357 --> 00:10:37,757 I couldn't access the first paper in so have to go to the second one. 197 00:10:37,857 --> 00:10:38,590 Might be interesting. 198 00:10:38,590 --> 00:10:40,890 Might be having just fine. 199 00:10:40,890 --> 00:10:41,423 there we go. 200 00:10:41,423 --> 00:10:44,356 It's a meta analysis in, 201 00:10:44,356 --> 00:10:47,723 looking at the concern at treating this practice, 202 00:10:48,689 --> 00:10:51,689 you go to the method section. 203 00:10:52,322 --> 00:10:54,722 Heterogeneity across studies 204 00:10:54,722 --> 00:10:57,722 was assessed using the I squared statistic. 205 00:10:58,022 --> 00:11:00,622 And these values represent 206 00:11:00,622 --> 00:11:03,622 low moderate and high heterogeneity. 207 00:11:04,522 --> 00:11:07,488 Now to a statistician heterogeneity means 208 00:11:07,488 --> 00:11:10,121 the variability in effects of studies. 209 00:11:10,121 --> 00:11:12,754 And it means the variability in standardized 210 00:11:12,754 --> 00:11:16,154 mean differences, all the variability in log odds ratios it. 211 00:11:16,654 --> 00:11:19,687 And that is simply not measured by I square. 212 00:11:20,221 --> 00:11:24,187 And then I use they say I squared a 55%. 213 00:11:25,454 --> 00:11:28,187 so there was moderate heterogeneity. 214 00:11:28,187 --> 00:11:30,053 this one's okay I squared zero. 215 00:11:30,053 --> 00:11:31,120 That does reflect no. 216 00:11:32,586 --> 00:11:35,186 But the problem with the first one, 217 00:11:35,186 --> 00:11:39,586 the point here is that I squared does not measure the amount of heterogeneity 218 00:11:39,786 --> 00:11:42,819 in the way that most people do 219 00:11:42,819 --> 00:11:45,819 or should think about what that means, 220 00:11:45,986 --> 00:11:48,985 either clinically or, statistically. 221 00:11:50,085 --> 00:11:52,385 So if you go back to this formula, I'm just writing in words. 222 00:11:52,385 --> 00:11:56,585 Now, what it measures is the amount of heterogeneity, which is variability 223 00:11:56,585 --> 00:11:59,585 in percentage or in local expression resulting, 224 00:12:01,185 --> 00:12:04,018 prevalence, whatever we're looking at, 225 00:12:04,018 --> 00:12:07,017 over the sum of the Metropolis sampling error. 226 00:12:07,884 --> 00:12:11,350 The sampling error is essentially a measure of how big the studies are 227 00:12:12,517 --> 00:12:13,684 typically. 228 00:12:13,684 --> 00:12:16,283 So a very big study gives a very small sampling error. 229 00:12:16,283 --> 00:12:19,283 So one over the study size. 230 00:12:19,283 --> 00:12:20,483 Now we can see this doesn't 231 00:12:20,483 --> 00:12:23,916 measure heterogeneity because it's fairly common to big studies. 232 00:12:24,316 --> 00:12:25,850 So typically this is a big number. 233 00:12:25,850 --> 00:12:27,383 This is a very small number. 234 00:12:27,383 --> 00:12:31,316 If you got lots of big studies this will tend towards one. 235 00:12:31,316 --> 00:12:34,149 So we'll get a nice square towards 100 towards one. 236 00:12:34,149 --> 00:12:36,082 But being 100% inconsistent. 237 00:12:37,715 --> 00:12:38,282 Whereas if 238 00:12:38,282 --> 00:12:41,848 if the studies are generally very small and this is a bigger number, 239 00:12:42,215 --> 00:12:45,215 and as they get smaller and smaller, this goes to zero 240 00:12:46,748 --> 00:12:48,548 for the same amount catchment. 241 00:12:48,548 --> 00:12:53,081 So I hope it's very clear that R-squared doesn't directly 242 00:12:53,081 --> 00:12:56,147 measure heterogeneity because it depends critically on 243 00:12:56,147 --> 00:12:59,147 how big our studies are. 244 00:13:00,614 --> 00:13:03,614 It's a relative measure, not an absolute measure. 245 00:13:03,947 --> 00:13:06,947 a lot of people seem to understand this. 246 00:13:09,847 --> 00:13:10,913 And it can lead to 247 00:13:10,913 --> 00:13:13,913 synthesis interpretations. 248 00:13:14,346 --> 00:13:15,613 Is there a solution? 249 00:13:15,613 --> 00:13:17,479 Well, 250 00:13:17,479 --> 00:13:18,479 I think so. 251 00:13:18,479 --> 00:13:22,379 We've got a there is a statistic that we use in meta analysis 252 00:13:22,546 --> 00:13:26,079 quite a lot that it perfectly measures the heterogeneity. 253 00:13:26,512 --> 00:13:29,445 It measures the variability in effect sizes 254 00:13:29,445 --> 00:13:32,445 across studies, something we generally call tool. 255 00:13:32,778 --> 00:13:35,612 How you use the influence already. 256 00:13:35,612 --> 00:13:38,278 And it's the standard deviation of effects across studies 257 00:13:39,278 --> 00:13:40,878 through effects 258 00:13:40,878 --> 00:13:43,744 not not not across the point estimates. 259 00:13:43,744 --> 00:13:46,977 So it's actually quite difficult to estimate because it's conceptual. 260 00:13:46,977 --> 00:13:50,611 I can't rely on something to calculate very directly recalculated 261 00:13:50,611 --> 00:13:53,610 from data yesterday from data. 262 00:13:54,577 --> 00:13:56,743 how often can be a bit difficult. 263 00:13:56,743 --> 00:14:00,210 And a deviation in log odds ratios is a really challenging thing, 264 00:14:00,677 --> 00:14:03,076 but we can express it right. 265 00:14:03,076 --> 00:14:06,076 And I'll return to that later. 266 00:14:07,676 --> 00:14:08,143 Okay. 267 00:14:08,143 --> 00:14:11,142 Paper number three. 268 00:14:11,376 --> 00:14:14,376 Damn Thessalonian into that paper. 269 00:14:14,609 --> 00:14:16,642 Great paper. 270 00:14:16,642 --> 00:14:19,942 landmark paper that introduced the idea of random effects. 271 00:14:20,109 --> 00:14:23,108 Meta analysis. 272 00:14:23,342 --> 00:14:26,341 What's a random effects meta analysis? 273 00:14:27,208 --> 00:14:29,241 I just quickly review it. 274 00:14:29,241 --> 00:14:31,441 let us look back at those studies courses with these. 275 00:14:31,441 --> 00:14:34,241 With the second set of studies I showed you, Isaac, right there. 276 00:14:34,241 --> 00:14:38,574 In fact, that is looking at, exercise as a gene for depression. 277 00:14:39,841 --> 00:14:41,107 the idea of a random 278 00:14:41,107 --> 00:14:44,107 effects match analysis is that 279 00:14:45,107 --> 00:14:47,840 each study is estimating 280 00:14:47,840 --> 00:14:50,706 its own unique true effect. 281 00:14:50,706 --> 00:14:54,273 We can depict that with some dotted dashed lines here. 282 00:14:54,540 --> 00:14:55,806 So we don't know where these are. 283 00:14:55,806 --> 00:14:59,073 The idea is that there's one of these lines for each of the studies. 284 00:15:00,939 --> 00:15:02,972 So say we don't know where they are, 285 00:15:02,972 --> 00:15:05,605 but we assume that the 286 00:15:05,605 --> 00:15:10,205 that they are distributed according to a specific distribution, 287 00:15:10,738 --> 00:15:13,505 typically a normal distribution, so that most that have effects 288 00:15:13,505 --> 00:15:16,938 in the middle of this and rarely though have studies outline 289 00:15:17,438 --> 00:15:18,905 to the left or the right. 290 00:15:18,905 --> 00:15:22,238 And then we can estimate where the middle of that distribution is. 291 00:15:22,838 --> 00:15:25,404 And that's the timing that is portrayed, 292 00:15:25,404 --> 00:15:28,537 is typically the result of a random text that matches. 293 00:15:30,071 --> 00:15:32,070 So what's the problem with this? 294 00:15:32,070 --> 00:15:35,070 Well, there are lots of problems with that analysis. 295 00:15:35,237 --> 00:15:37,403 I'm just not too out. 296 00:15:37,403 --> 00:15:39,303 So these are not what I'm talking about. 297 00:15:39,303 --> 00:15:42,303 There are technical problems, 298 00:15:42,770 --> 00:15:43,803 with the particular 299 00:15:43,803 --> 00:15:46,836 method that estimating light introduced, which is a nice, simple one. 300 00:15:47,436 --> 00:15:49,869 but there's a lot of those particular technical problems 301 00:15:49,869 --> 00:15:52,869 have been overcome by more sophisticated methods. 302 00:15:54,069 --> 00:15:57,069 There are really important practical methods. 303 00:15:58,069 --> 00:16:01,702 as I told you, I said it's difficult to estimate the probability 304 00:16:02,435 --> 00:16:05,401 and a lot of meta analysis and very few studies. 305 00:16:05,401 --> 00:16:08,701 So we're going to get really bad estimates of the variability across studies. 306 00:16:08,935 --> 00:16:13,468 And that impacts on how well we estimate the middle of the distribution. 307 00:16:14,201 --> 00:16:17,201 So that's a real practical problem when you've got few studies. 308 00:16:17,601 --> 00:16:21,700 But the the problem I really want to get at is a more conceptual, 309 00:16:23,167 --> 00:16:26,167 which is the misunderstanding of the model. 310 00:16:28,533 --> 00:16:28,900 going back 311 00:16:28,900 --> 00:16:31,900 to most recent citation of the paper, 312 00:16:33,000 --> 00:16:35,166 looking at some, 313 00:16:35,166 --> 00:16:40,666 cough treatment for the infection and stroke and a meta umbrella review. 314 00:16:40,666 --> 00:16:42,832 That's a new one on me. 315 00:16:42,832 --> 00:16:45,699 I'm not quite sure what it is. 316 00:16:45,699 --> 00:16:48,432 but they talk about using, 317 00:16:48,432 --> 00:16:52,698 random effects meta analysis to obtain a pooled estimate of the effect size. 318 00:16:53,165 --> 00:16:55,365 So I'm a bit fussy about the language here, 319 00:16:55,365 --> 00:16:58,365 but I think betrays a misunderstanding of what the model is doing. 320 00:16:58,431 --> 00:17:00,431 The effect size. 321 00:17:00,431 --> 00:17:03,431 they say, interestingly, they say, well, follow a normal distribution. 322 00:17:03,964 --> 00:17:07,431 but but this variance equal to the sum of the weights, 323 00:17:07,797 --> 00:17:09,197 that is about the coefficient for 324 00:17:09,197 --> 00:17:12,564 is actually nothing to do with the the random effects distribution. 325 00:17:13,230 --> 00:17:17,497 So I that they did understand it, but they didn't or they didn't at all. 326 00:17:19,730 --> 00:17:22,730 I know what I mean. 327 00:17:23,130 --> 00:17:24,096 and so, so 328 00:17:24,096 --> 00:17:27,496 they get these results and it was kept near the top result of 23 329 00:17:27,496 --> 00:17:30,762 studies is in is a result of random effects meta analysis. 330 00:17:31,796 --> 00:17:34,762 And they reported saying, well, it's 2.4. 331 00:17:34,762 --> 00:17:38,229 Take the findings indicate the patients with severe Covid significant 332 00:17:38,229 --> 00:17:42,428 more likely to vote stroke than those who are less with less severe forms. 333 00:17:42,562 --> 00:17:46,228 I mean, that may be true, but it doesn't necessarily 334 00:17:46,228 --> 00:17:49,961 follow from the random effects analysis they've done. 335 00:17:50,228 --> 00:17:53,228 And you can see that the absolute value is quite large here. 336 00:17:53,794 --> 00:17:56,627 that doesn't measure the amount of functionality, 337 00:17:56,627 --> 00:17:59,794 but it does correlate with the amount of energy. 338 00:18:00,060 --> 00:18:03,860 So that does tell me that there's probably a lot of probably reasonable about that. 339 00:18:03,860 --> 00:18:05,693 You know, depending on how the studies are, 340 00:18:08,227 --> 00:18:12,126 especially when this study is a small. So. 341 00:18:14,826 --> 00:18:15,626 What's the problem here 342 00:18:15,626 --> 00:18:20,526 that the problem is that random effects analysis are designed to learn 343 00:18:20,526 --> 00:18:24,725 about distribution to the effects, not about single effects. 344 00:18:24,725 --> 00:18:27,725 And when people right the effects with this, 345 00:18:28,492 --> 00:18:32,825 then that is not aligned with the model that they used. 346 00:18:33,992 --> 00:18:37,058 And it was very clear to me at the very beginning of detriment 347 00:18:37,058 --> 00:18:38,091 in the LED discussion, 348 00:18:38,091 --> 00:18:41,091 when I say that characterize the distribution of treatment effects, 349 00:18:41,991 --> 00:18:44,991 but people don't read these papers closely, just citing. 350 00:18:46,791 --> 00:18:49,191 So what's the solution to this one? 351 00:18:49,191 --> 00:18:51,690 This one's more difficult. 352 00:18:51,690 --> 00:18:54,690 honestly, one option is using a fixed effect, 353 00:18:54,757 --> 00:18:56,223 model, 354 00:18:56,223 --> 00:18:59,223 which may be very reasonable if the studies of questionable. 355 00:19:00,423 --> 00:19:03,090 but I don't want to get into that debate if people are going to 356 00:19:03,090 --> 00:19:04,356 use a random effects model, 357 00:19:05,523 --> 00:19:07,256 then they need to recognize 358 00:19:07,256 --> 00:19:10,289 that the estimate that comes out is an estimate of a mean or an average 359 00:19:11,289 --> 00:19:13,522 and and not a single effect. 360 00:19:13,522 --> 00:19:16,355 And ideally they should also portray 361 00:19:16,355 --> 00:19:19,555 the rest of the distribution, the spread of effects clustering. 362 00:19:19,788 --> 00:19:23,622 Because that's what they've assumed by saying I'm doing a random effects meta 363 00:19:23,688 --> 00:19:27,155 analysis, people should know what you've found 364 00:19:27,721 --> 00:19:30,721 from the assumptions you made. 365 00:19:31,288 --> 00:19:33,888 so we can, 366 00:19:33,888 --> 00:19:36,887 we can portray the full distribution 367 00:19:37,087 --> 00:19:40,287 and my reporting tool, but that's, as I say, very difficult to interpret. 368 00:19:40,720 --> 00:19:44,453 So one option is to, to present a prediction interval, 369 00:19:44,553 --> 00:19:48,787 which is a way of expressing talk in a way that's more interpretable. 370 00:19:49,586 --> 00:19:50,420 So here's the result. 371 00:19:50,420 --> 00:19:53,020 This is an estimate of the mean. 372 00:19:53,020 --> 00:19:55,786 A prediction for would extend 373 00:19:55,786 --> 00:19:59,852 this to say, well this is where I think the mean is and the uncertainty. 374 00:20:00,086 --> 00:20:03,552 But this is where I think the effects could be beyond the mean. 375 00:20:04,119 --> 00:20:04,685 The spread 376 00:20:06,152 --> 00:20:09,152 that some of the studies have effects down here and some of them up here. 377 00:20:09,452 --> 00:20:12,385 There's an average somewhere around here. 378 00:20:12,385 --> 00:20:15,385 Now these things are problematic for very small numbers of studied. 379 00:20:17,151 --> 00:20:19,318 but when you got enough studies, 380 00:20:19,318 --> 00:20:22,318 I think this got quite a useful way 381 00:20:22,584 --> 00:20:25,584 of looking at the results. 382 00:20:28,351 --> 00:20:31,350 so number four. 383 00:20:34,050 --> 00:20:37,050 This is Matthias Eggers paper, 384 00:20:38,217 --> 00:20:41,750 bias in meta analysis detected by a simple graphical test. 385 00:20:41,750 --> 00:20:43,550 Okay. Something simple. 386 00:20:43,550 --> 00:20:46,549 So it's become very popular. 387 00:20:47,283 --> 00:20:48,149 What's it about? What? 388 00:20:48,149 --> 00:20:53,316 It's really all about using funnel plots, some plots, plots of, 389 00:20:53,949 --> 00:20:56,382 the size of effect in each study 390 00:20:56,382 --> 00:20:59,382 and how big each study was. 391 00:21:00,115 --> 00:21:01,815 Here is some from the past. 392 00:21:01,815 --> 00:21:04,815 The original paper is the effect. 393 00:21:05,281 --> 00:21:06,915 It measures just knots ratio. 394 00:21:06,915 --> 00:21:10,181 And here is essentially how big the studies were measured. 395 00:21:10,181 --> 00:21:12,348 Just precision. Yeah. 396 00:21:12,348 --> 00:21:14,547 So the big studies with 397 00:21:14,547 --> 00:21:18,381 higher precision are up and the smaller ones are down. 398 00:21:19,814 --> 00:21:23,180 Now what should happen in an ideal world, 399 00:21:25,147 --> 00:21:27,613 a big study should be close to the truth 400 00:21:27,613 --> 00:21:31,413 because of the small statistical error and the small studies, 401 00:21:32,046 --> 00:21:35,046 assuming they're there also 402 00:21:35,179 --> 00:21:38,946 trying to learn about roughly the same thing is the big studies 403 00:21:39,512 --> 00:21:41,946 I for a fixed effect model or a random effects 404 00:21:41,946 --> 00:21:45,845 model should be distributed more widely 405 00:21:45,845 --> 00:21:48,845 because of sampling error, but you should have the same mean. 406 00:21:49,678 --> 00:21:52,678 So we would expect, as in these two examples, 407 00:21:53,711 --> 00:21:56,711 the plot to appear roughly symmetrical 408 00:21:57,545 --> 00:21:59,911 with a lot of scatter among the big studies, 409 00:21:59,911 --> 00:22:02,911 the less scatter among the big studies, 410 00:22:03,111 --> 00:22:06,111 but centered on the same underlying truth, 411 00:22:06,311 --> 00:22:10,910 the basis under which we brought together the studies into one better thing 412 00:22:10,910 --> 00:22:13,777 they're all trying to tell us about approximately the same thing 413 00:22:13,777 --> 00:22:15,143 and exactly or an average. 414 00:22:18,010 --> 00:22:20,676 But sometimes they don't. 415 00:22:20,676 --> 00:22:23,943 And and this test is about identifying the situations in which the, 416 00:22:24,209 --> 00:22:26,309 the symmetry doesn't hurt. 417 00:22:26,309 --> 00:22:31,209 And with some nice examples, the, the I for trial runs from Oxford 418 00:22:31,209 --> 00:22:36,375 50,000 patients randomized to magnesium or placebo. 419 00:22:37,409 --> 00:22:40,275 strongly disagreed with all of the earlier 420 00:22:40,275 --> 00:22:43,275 small trials when it landed back on the no. 421 00:22:43,408 --> 00:22:48,041 But most of the others, indicated to just to the benefit of magnesium. 422 00:22:48,741 --> 00:22:51,741 A very interesting study. 423 00:22:52,141 --> 00:22:55,141 if you plot them, it looks asymmetrical. 424 00:22:57,607 --> 00:22:58,440 So maybe there are 425 00:22:58,440 --> 00:23:01,440 some studies missing down here. 426 00:23:01,840 --> 00:23:04,840 And here's another example of the matrix. 427 00:23:05,807 --> 00:23:06,907 So what's the problem? 428 00:23:06,907 --> 00:23:10,640 Well, let's look at, the most recent citation. 429 00:23:11,373 --> 00:23:14,839 It's looking at it's prognostic match analysis, looking at whether 430 00:23:15,339 --> 00:23:18,339 and now three seven, eight predicts 431 00:23:19,706 --> 00:23:21,239 mean some 432 00:23:21,239 --> 00:23:22,772 cancers, cancer. 433 00:23:24,005 --> 00:23:26,539 And they use 434 00:23:26,539 --> 00:23:29,505 to assess publication bias back in aggression. 435 00:23:29,505 --> 00:23:32,138 A similar test were applied. 436 00:23:32,138 --> 00:23:35,171 and they got some small values 437 00:23:35,171 --> 00:23:38,171 that indicated there was publication bias. 438 00:23:38,538 --> 00:23:42,304 And they got some no values that suggest there is no publication. But. 439 00:23:45,404 --> 00:23:47,504 I think it's just 440 00:23:47,504 --> 00:23:49,337 plain I'm going to come up with now 441 00:23:49,337 --> 00:23:52,337 I can just plain wrong. 442 00:23:52,637 --> 00:23:55,603 I guess test is not a test of publication bias. 443 00:23:55,603 --> 00:23:58,470 It's a test for asymmetry. 444 00:23:58,470 --> 00:24:01,370 Now, the problem is that 445 00:24:01,370 --> 00:24:05,003 the vast majority of people using the test equate 446 00:24:05,469 --> 00:24:08,469 publication bias, and some of the testing, 447 00:24:09,369 --> 00:24:13,102 and I can that can be quite dangerous, misleading things to do. 448 00:24:13,302 --> 00:24:16,235 Yes, publication bias may introduce 449 00:24:16,235 --> 00:24:19,235 asymmetry and have some problems. 450 00:24:19,302 --> 00:24:22,202 It may be something to back 451 00:24:22,202 --> 00:24:25,201 this picture that there were small studies here 452 00:24:25,535 --> 00:24:28,534 that showed that magnesium was harmful 453 00:24:29,234 --> 00:24:32,567 and they were suppressed under that for mission critical block. 454 00:24:34,467 --> 00:24:39,967 But. There are other 455 00:24:40,267 --> 00:24:44,833 very plausible reasons why some plot may be. 456 00:24:46,667 --> 00:24:47,500 It may be. 457 00:24:47,500 --> 00:24:48,566 Well, this is another example 458 00:24:48,566 --> 00:24:52,100 of publication bias, but it may be just to do with quality 459 00:24:52,799 --> 00:24:56,933 methodological quality of smaller studies being potentially worst. 460 00:24:56,933 --> 00:25:00,266 In bigger studies you also get an asymmetric clinical. 461 00:25:00,499 --> 00:25:03,299 It may be true heterogeneity. 462 00:25:03,299 --> 00:25:05,565 I mean there is actually a statistical 463 00:25:05,565 --> 00:25:08,765 not a problem but that is tested statistically flawed. 464 00:25:09,098 --> 00:25:12,098 So any small flaw but you can get 465 00:25:12,398 --> 00:25:15,265 asymmetry out of the test when it's not really that 466 00:25:15,265 --> 00:25:18,264 chances are always an explanation, but these two, 467 00:25:18,831 --> 00:25:20,598 highly plausible explanations. 468 00:25:20,598 --> 00:25:25,331 Let me just show you, here's an example where the smallest studies 469 00:25:25,331 --> 00:25:28,330 this is an artificial example, but small studies are all done with a, 470 00:25:28,864 --> 00:25:30,064 you know, a bit of bias, 471 00:25:31,197 --> 00:25:32,930 and it shifts 472 00:25:32,930 --> 00:25:35,930 the results of those small studies to the left. 473 00:25:36,130 --> 00:25:39,130 And when you put them together, you get something. 474 00:25:40,696 --> 00:25:43,596 here's a plot with asymmetry. 475 00:25:43,596 --> 00:25:45,096 maybe 476 00:25:45,096 --> 00:25:48,229 you think there's some missing studies down here, 477 00:25:48,996 --> 00:25:50,729 but then if you look at them in more detail, 478 00:25:50,729 --> 00:25:53,495 there are three populations of trials here. 479 00:25:53,495 --> 00:25:57,362 And in this set there's no symmetry in this set. 480 00:25:57,362 --> 00:25:58,362 Looks fine. 481 00:25:58,362 --> 00:26:03,595 This is actually when you put them all together it looks asymmetrical. 482 00:26:03,595 --> 00:26:06,595 So that's an example of true heterogeneity. 483 00:26:06,628 --> 00:26:07,894 Then we go back to this example. 484 00:26:07,894 --> 00:26:10,894 What do they do. 485 00:26:11,461 --> 00:26:14,461 Well the studies they put into this meta analysis 486 00:26:15,427 --> 00:26:18,427 have looked at completely different cancers. 487 00:26:18,760 --> 00:26:19,860 These are the results. 488 00:26:19,860 --> 00:26:23,060 There are some that say the same predicts and others it doesn't seem to predict 489 00:26:23,660 --> 00:26:28,360 the whole basis of assuming a percent of asymmetry. 490 00:26:28,360 --> 00:26:29,926 For me, it doesn't really hold you. 491 00:26:32,126 --> 00:26:32,826 It's I mean, it's 492 00:26:32,826 --> 00:26:35,826 all it's all based on a theory that a lot of people 493 00:26:36,593 --> 00:26:41,226 don't realize, based on the theory that all big studies are published, 494 00:26:41,992 --> 00:26:44,825 but small studies with results in one direction 495 00:26:44,825 --> 00:26:47,825 are selectively withheld. 496 00:26:47,825 --> 00:26:50,225 Now, that is probably a good model 497 00:26:50,225 --> 00:26:55,158 for randomized trials on the same topic, but there's no particular reason 498 00:26:55,158 --> 00:26:58,591 for belief that theory holds in observational work, 499 00:26:59,591 --> 00:27:02,191 where, you know, big studies may have really 500 00:27:02,191 --> 00:27:05,191 crude measures of exposure, 501 00:27:05,491 --> 00:27:08,357 so they may be the worst studies 502 00:27:08,357 --> 00:27:11,390 and small studies may have really accurate ones, 503 00:27:11,390 --> 00:27:14,390 and maybe they're giving ones with less past results. 504 00:27:14,457 --> 00:27:17,556 And so this idea that the big ones are near the truth and the small ones bias 505 00:27:18,123 --> 00:27:21,156 will suppress one side might not hold. 506 00:27:21,390 --> 00:27:24,389 And I think we need to understand these things a lot better. 507 00:27:24,656 --> 00:27:28,456 and not just naively assume that some of what I think should be psychology 508 00:27:28,456 --> 00:27:29,856 studies through population bias. 509 00:27:30,989 --> 00:27:33,989 It's a very common problem. 510 00:27:34,122 --> 00:27:37,655 So what should we do if we worried about publication bias? 511 00:27:38,855 --> 00:27:41,855 I say don't start with these tests. 512 00:27:41,955 --> 00:27:44,955 Start by using the brain. 513 00:27:45,155 --> 00:27:50,488 You know, what is the likelihood that studies would have suppressed results 514 00:27:51,288 --> 00:27:54,221 or that studies would have been hidden altogether? 515 00:27:54,221 --> 00:27:58,387 And you can get quite far by understanding the field and then some fields, 516 00:27:59,220 --> 00:28:03,420 a lot of small studies are done by psychology students, for example. 517 00:28:04,320 --> 00:28:07,186 and they're easy to suppress or just written into a thesis 518 00:28:07,186 --> 00:28:10,186 and not written up in other areas. 519 00:28:10,520 --> 00:28:12,453 randomized trials are registered. 520 00:28:12,453 --> 00:28:14,286 We know about them. 521 00:28:14,286 --> 00:28:17,052 if we can trace all of the registered trials 522 00:28:17,052 --> 00:28:20,952 and we're still estimates, it's probably due to some other reason. 523 00:28:22,719 --> 00:28:26,285 So we need to think about the situation we're in. 524 00:28:27,185 --> 00:28:30,518 And, it's not a plug of work I was involved in. 525 00:28:30,918 --> 00:28:34,518 The rope and Army tool is an attempt to put a framework 526 00:28:34,518 --> 00:28:38,051 into thinking more sensibly about publication bias 527 00:28:38,051 --> 00:28:42,251 and related biases that lead to missing evidence in next analysis, 528 00:28:43,684 --> 00:28:45,351 and the funnel plots are mentioned. 529 00:28:45,351 --> 00:28:48,450 But right at the end, after you've done all the other thinking. 530 00:28:52,750 --> 00:28:53,950 So if we use 531 00:28:53,950 --> 00:28:57,116 funnel plots, actually there is a I mean, if, 532 00:28:57,150 --> 00:29:00,716 you know, I'm sure there's a lovely, modification around. 533 00:29:02,916 --> 00:29:06,949 You, improvement we can make to funnel. 534 00:29:07,216 --> 00:29:10,716 So that's you put controls on it controls 535 00:29:10,716 --> 00:29:13,715 that describes statistical significance. 536 00:29:15,082 --> 00:29:16,582 because for any 537 00:29:16,582 --> 00:29:20,582 if we're going to plot the effect size against precision, 538 00:29:20,815 --> 00:29:24,215 which is basically the nature of the standard error with effect size, 539 00:29:24,781 --> 00:29:29,148 any combination of estimating standard error leads to. 540 00:29:29,681 --> 00:29:31,447 So that's what I call the z statistic. 541 00:29:31,447 --> 00:29:34,447 The meta a p value. 542 00:29:35,481 --> 00:29:37,547 and so there's a p value 543 00:29:37,547 --> 00:29:40,547 representing any point on this plot. 544 00:29:40,647 --> 00:29:45,480 And then we draw contours for the specific p values that we're interested in. 545 00:29:45,480 --> 00:29:48,246 So here's control. The first one is 546 00:29:49,646 --> 00:29:50,780 point one. 547 00:29:50,780 --> 00:29:53,646 So that's kind of getting towards the significance. 548 00:29:53,646 --> 00:29:56,079 And then further up we get point naught five, 549 00:29:56,079 --> 00:29:59,079 which I often use as a threshold for setting significance. 550 00:29:59,212 --> 00:30:01,245 And then point one, which is one. 551 00:30:01,245 --> 00:30:04,245 And what we see here is a lot of the studies we've got 552 00:30:04,245 --> 00:30:07,245 were statistically significant. 553 00:30:07,678 --> 00:30:09,045 And so and 554 00:30:09,045 --> 00:30:12,345 and if this were due to publication bias, it would mean that 555 00:30:13,778 --> 00:30:15,844 studies down here, emission 556 00:30:15,844 --> 00:30:18,844 studies down here, non-significant studies. 557 00:30:18,844 --> 00:30:21,844 So it's quite plausible that this is due to publication bias, 558 00:30:22,977 --> 00:30:25,244 because it fits what we'd expect. 559 00:30:25,244 --> 00:30:27,844 If the publication bias is the cause of this 560 00:30:27,844 --> 00:30:30,843 small null studies emission. 561 00:30:31,177 --> 00:30:34,343 Whereas in this example we've got asymmetry. 562 00:30:34,343 --> 00:30:37,410 If you look at the controls you might think, oh that's the top. 563 00:30:37,410 --> 00:30:38,810 There are studies missing down here. 564 00:30:39,810 --> 00:30:40,776 But here the missing 565 00:30:40,776 --> 00:30:43,776 studies are just the significant other. 566 00:30:44,176 --> 00:30:45,143 and so, 567 00:30:45,143 --> 00:30:49,042 I mean, it may be that people that found results down has pressed them, but 568 00:30:49,042 --> 00:30:53,575 it's just a different set of assumptions about the mechanism for publication bias. 569 00:30:53,809 --> 00:30:54,442 But it helps you 570 00:30:54,442 --> 00:30:58,375 think about those and produce situations where it's a plausible explanation. 571 00:30:58,675 --> 00:31:01,675 So the longer you need to think more carefully. 572 00:31:05,841 --> 00:31:08,208 Right. 573 00:31:08,208 --> 00:31:09,641 My last one I didn't have 574 00:31:09,641 --> 00:31:12,641 I get time to do this, I think I do. 575 00:31:13,474 --> 00:31:16,140 This is risk of bias. 576 00:31:16,140 --> 00:31:19,140 Assessment tools. 577 00:31:20,840 --> 00:31:24,240 this is a framework for assessing risk of bias 578 00:31:24,407 --> 00:31:27,640 in a randomized trial included in a systematic review. 579 00:31:28,340 --> 00:31:30,539 And it, 580 00:31:30,539 --> 00:31:33,306 this is the original tool published in 2011. 581 00:31:33,306 --> 00:31:37,206 Once through six items and 5 to 9 different phases 582 00:31:38,239 --> 00:31:39,939 were updated to rock two. 583 00:31:41,339 --> 00:31:44,339 and this is one we hoped that people would use now, 584 00:31:44,405 --> 00:31:48,805 but the mis uses 585 00:31:49,172 --> 00:31:52,171 I want to talk about relate to both of these. 586 00:31:54,038 --> 00:31:55,838 Now here my my training 587 00:31:55,838 --> 00:31:58,838 for most recent papers. 588 00:31:59,004 --> 00:32:01,237 wasn't quite so successful. 589 00:32:01,237 --> 00:32:03,537 The first one 590 00:32:03,537 --> 00:32:06,537 for that patient to some of my colleagues in Bristol. 591 00:32:07,704 --> 00:32:09,937 and it's not so it's not so bad. 592 00:32:09,937 --> 00:32:14,803 But what what they what they're doing here is a descriptive review, 593 00:32:15,703 --> 00:32:19,570 just describing interventions and outcomes. But, 594 00:32:20,669 --> 00:32:23,669 that have been used in studies of, 595 00:32:24,269 --> 00:32:26,469 American emergency death of me. 596 00:32:26,469 --> 00:32:28,969 I have to care for that. But you can see that. 597 00:32:28,969 --> 00:32:32,335 And and they used, promises another tool 598 00:32:32,369 --> 00:32:35,935 and this tool to assess risk bias in the studies 599 00:32:37,168 --> 00:32:38,668 which, which, which is fine. 600 00:32:38,668 --> 00:32:41,068 Maybe it's interesting, but it's not what we designed 601 00:32:41,068 --> 00:32:44,901 the tools for really, because they're not talking about results. 602 00:32:46,668 --> 00:32:48,334 but I'm, 603 00:32:48,334 --> 00:32:51,334 I'm not going to criticize this personal work. 604 00:32:51,867 --> 00:32:56,100 The next one I came from actually illustrated the problem, but it was not 605 00:32:57,200 --> 00:32:59,700 problematic. The problem. 606 00:32:59,700 --> 00:33:01,933 it was, 607 00:33:01,933 --> 00:33:04,900 they looked at, 608 00:33:04,900 --> 00:33:07,566 some randomized trials, they assess risk bias, 609 00:33:07,566 --> 00:33:10,466 and they took that, 610 00:33:10,466 --> 00:33:12,299 towards the beginning of the results section. 611 00:33:12,299 --> 00:33:17,166 They described their risk assessments here and refer to the figure 612 00:33:17,399 --> 00:33:21,032 that shows that these studies, this the different domains of bias. 613 00:33:23,932 --> 00:33:26,232 In fact, everything looks pretty good here. 614 00:33:26,232 --> 00:33:29,465 A bit of uncertainty but a lot of low risk of bias. 615 00:33:30,965 --> 00:33:34,065 they then go on to report the results of the meta analysis, 616 00:33:34,665 --> 00:33:37,664 but make no mention of the risk assessment. 617 00:33:38,564 --> 00:33:40,431 That's the problem I have here. 618 00:33:40,431 --> 00:33:41,931 It doesn't really matter. 619 00:33:41,931 --> 00:33:44,664 But the next one I found, 620 00:33:44,664 --> 00:33:47,864 I think illustrates the problem that I see over and over again. 621 00:33:50,197 --> 00:33:52,630 Here the results section starts 622 00:33:52,630 --> 00:33:55,630 by presenting the match analysis results. 623 00:33:56,096 --> 00:33:59,796 The results yesterday they do some, publication by stuff, 624 00:34:00,063 --> 00:34:04,296 but they don't mention the quality of the trials going into the outcomes. 625 00:34:05,829 --> 00:34:08,862 And then later on they have a little section 626 00:34:08,862 --> 00:34:11,929 that that describes the risk of bias results. 627 00:34:11,929 --> 00:34:14,362 And they show, 628 00:34:14,362 --> 00:34:15,762 this profile 629 00:34:15,762 --> 00:34:18,762 suggesting that there a problem in nearly every study, 630 00:34:19,562 --> 00:34:21,895 if we believe their assessment. 631 00:34:21,895 --> 00:34:24,428 But there is actually no linking 632 00:34:24,428 --> 00:34:27,728 of the fact that there were problems with the results of the meta analysis, 633 00:34:27,994 --> 00:34:31,161 and there is not a mention of risk of bias in the entire discussion. 634 00:34:32,561 --> 00:34:36,527 So this is the problem that I if 635 00:34:37,827 --> 00:34:40,827 it annoys me, it was to 636 00:34:41,027 --> 00:34:41,827 value the doing. 637 00:34:41,827 --> 00:34:44,693 What is risk of our system if you're not going to use it? 638 00:34:44,693 --> 00:34:48,060 People don't routinely ignore risk assessments 639 00:34:48,060 --> 00:34:51,226 and doing fancy reporting services and drawing conclusions 640 00:34:53,159 --> 00:34:54,159 is the solution. 641 00:34:54,159 --> 00:34:57,059 Well, I mean, this is one of the reasons why, 642 00:34:57,059 --> 00:35:00,625 a modification in Rock two was should be really explicit. 643 00:35:00,625 --> 00:35:03,825 This is risk assessment in a specific result, 644 00:35:05,425 --> 00:35:08,325 not in a study. 645 00:35:08,325 --> 00:35:10,058 And that means risk of bias assessment is 646 00:35:10,058 --> 00:35:14,358 what two really need to be presented alongside those results. 647 00:35:14,724 --> 00:35:17,124 And that was why we probably why we did this. 648 00:35:17,124 --> 00:35:20,124 So that we would encourage people to present risk profiles 649 00:35:20,591 --> 00:35:21,891 alongside the results. 650 00:35:21,891 --> 00:35:24,890 And from the first part. 651 00:35:26,057 --> 00:35:28,757 and we see it increasingly which is, which is great. 652 00:35:28,757 --> 00:35:31,757 And Cochrane, has 653 00:35:31,790 --> 00:35:34,790 shifted its guidance on where to report risk bias 654 00:35:35,190 --> 00:35:38,090 from an overall think towards 655 00:35:38,090 --> 00:35:41,089 and may affect interventions. 656 00:35:42,923 --> 00:35:44,656 A comment here that great timing 657 00:35:44,656 --> 00:35:47,656 of how because great is outcome specific. 658 00:35:48,422 --> 00:35:50,889 And we think about risk assessments 659 00:35:50,889 --> 00:35:54,522 in the specific results for the particular outcome. 660 00:35:57,455 --> 00:35:58,688 Okay. 661 00:35:58,688 --> 00:36:01,421 So I try to bring things to close. 662 00:36:01,421 --> 00:36:04,254 Now what what are my reflections on all this. 663 00:36:04,254 --> 00:36:07,988 Well I mean to summarize, I've looked at what I see 664 00:36:07,988 --> 00:36:11,387 as some really, really common problems in match analysis. 665 00:36:11,887 --> 00:36:14,887 I've built them on the left hand side of papers. 666 00:36:15,220 --> 00:36:18,254 but that makes them highly, prevalent problems, 667 00:36:19,353 --> 00:36:21,720 approachable solutions. 668 00:36:21,720 --> 00:36:23,320 This is the fourth time I've given this talk. 669 00:36:23,320 --> 00:36:26,520 Each time I go and look at the latest citations 670 00:36:26,520 --> 00:36:30,919 the day before, and in almost every instance across the four, 671 00:36:31,486 --> 00:36:34,386 I find that the most recent papers misuse or the sense 672 00:36:34,386 --> 00:36:37,386 that the method now, 673 00:36:37,852 --> 00:36:39,685 I think that means it's probably 674 00:36:39,685 --> 00:36:42,685 the probably very commonly misused, 675 00:36:42,852 --> 00:36:44,952 but it would be nice to have some empirical evidence. 676 00:36:44,952 --> 00:36:48,085 And actually the project would be to look empirically 677 00:36:48,085 --> 00:36:51,485 at a random sample and to see how much these problems. 678 00:36:54,051 --> 00:36:55,684 Why did they happen? 679 00:36:55,684 --> 00:36:58,451 And that's an interesting thing we could talk about 680 00:36:58,451 --> 00:37:00,584 if we want to have a discussion. 681 00:37:00,584 --> 00:37:03,484 I think it's largely people just copying 682 00:37:03,484 --> 00:37:06,484 what previous people have done. 683 00:37:07,517 --> 00:37:10,517 there could be an element of, 684 00:37:12,817 --> 00:37:14,716 Dangerous thing to say in the context of a short course. 685 00:37:14,716 --> 00:37:16,883 I'm not involved in teaching. 686 00:37:16,883 --> 00:37:19,883 I'm sure it hasn't been the case this week. 687 00:37:20,916 --> 00:37:23,916 but I think, I mean, this is Oxford, and I'm sure we got from the. 688 00:37:24,749 --> 00:37:27,582 We can one can imagine that in other places, there are people 689 00:37:27,582 --> 00:37:32,149 who don't understand things quite as well and perpetuate the sort of things 690 00:37:32,149 --> 00:37:35,748 they read time and time again, and reports of systematic reviews. 691 00:37:36,848 --> 00:37:39,848 Or it may just be that, you know, oh, 692 00:37:42,081 --> 00:37:44,181 I mean, I but just inconsistency 693 00:37:44,181 --> 00:37:48,381 that means that it must measure hedge matches and I just want it to measure. 694 00:37:48,414 --> 00:37:49,647 That's you 695 00:37:51,281 --> 00:37:53,381 make a combination of all these things. 696 00:37:53,381 --> 00:37:54,547 Now, one last thing. 697 00:37:54,547 --> 00:37:56,714 I mean, being highly cited is nice. 698 00:37:56,714 --> 00:37:59,713 People introduce you with brand words, 699 00:37:59,747 --> 00:38:02,547 but it doesn't mean you're having a positive influence. 700 00:38:02,547 --> 00:38:04,680 Let's say, 701 00:38:04,680 --> 00:38:07,646 I don't I never 702 00:38:07,646 --> 00:38:10,646 use this little stuff that encourages 703 00:38:11,946 --> 00:38:13,846 us so that thank you very much for the attention.