1 00:00:00,120 --> 00:00:03,480 Over to you, Gordon. Thank you. Okay, great. Thanks very much. 2 00:00:04,410 --> 00:00:10,950 So, people, it sounds like in this audience know me by what I've written about evidence based medicine. 3 00:00:12,090 --> 00:00:20,399 What you may not know about me is that the process started in when I took over 4 00:00:20,400 --> 00:00:26,070 as director of the Internal Medicine Residency Programme at McMaster in 1990. 5 00:00:26,310 --> 00:00:28,840 And that's where my story of today will start. 6 00:00:29,940 --> 00:00:40,260 And I still do clinical practise as a hospital based general internist, and I've just come off two weeks on clinical service, so I'm still at it. 7 00:00:41,100 --> 00:00:47,250 So the topic is how should we teach evidence based medicine in the 21st century? 8 00:00:47,970 --> 00:00:51,840 Conflict of interest. And you're going to hear about grade. 9 00:00:52,140 --> 00:00:55,320 And I'm co-chair of the Great Working Group. 10 00:00:56,460 --> 00:01:03,840 There may be something in this that indirectly refers to up to date. 11 00:01:04,590 --> 00:01:13,200 Not directly, I don't think. But at any rate, I'm done a lot of consulting work with up to date the for any of you who want 12 00:01:13,200 --> 00:01:20,280 to take a slower review of what I'm going to cover everything in this talk. 13 00:01:21,420 --> 00:01:27,989 I think what works is in the article about understanding results, evidence, summaries, 14 00:01:27,990 --> 00:01:32,730 applicability, not critical appraisal or the core skills of the medical curriculum. 15 00:01:33,000 --> 00:01:42,600 That's the point. I'm going to try and make curry chicken and I published a paper making that case in BMJ IBN in 2020. 16 00:01:43,620 --> 00:01:54,239 So in a relatively short period of time, I'm going to talk about critical appraisal before evidence based medicine, evidence based medicine. 17 00:01:54,240 --> 00:02:00,810 At the start, it's every illusion and the implications for evidence based medicine education. 18 00:02:01,260 --> 00:02:08,340 And I'm going to identify what I think are three challenges for evidence based medicine and education. 19 00:02:08,850 --> 00:02:12,750 One, physicians must understand the results. 20 00:02:13,020 --> 00:02:18,720 I'm convinced that that is the case. Conditions must understand grade. 21 00:02:19,740 --> 00:02:24,180 I have confessed my conflict of interest as far as that is concerned. 22 00:02:25,230 --> 00:02:34,380 But in a more general sense, perhaps they must understand the concept of quality and certainty of evidence and be able to apply it. 23 00:02:34,830 --> 00:02:41,190 And thirdly, the underappreciated aspect of evidence based medicine, 24 00:02:41,460 --> 00:02:49,440 even though it's more than 20 years since we've been highlighting the DBM, is all about patient values and preferences. 25 00:02:49,800 --> 00:02:55,860 Some folks still don't get it and challenge three completely consistent. 26 00:02:55,860 --> 00:03:04,050 In fact, at the heart of the beam is that we must teach physicians to understand the importance of patient values and preferences. 27 00:03:05,010 --> 00:03:07,229 So this is Dave Sackett. 28 00:03:07,230 --> 00:03:20,820 He was my mentor and in 1981 he published a very influential series that looked at critical appraisal of the medical literature, 29 00:03:21,090 --> 00:03:28,590 which started with an article about how to read clinical journals, why to read them, and how to start reading the critically. 30 00:03:29,490 --> 00:03:34,290 And this is what he started out a little later. 31 00:03:34,300 --> 00:03:38,220 I'm not sure he would have well, I'm sure he wouldn't have agreed necessarily to all of this. 32 00:03:38,580 --> 00:03:42,030 But he said, look at the title. Interesting or useful? That's probably a good one. 33 00:03:42,420 --> 00:03:46,230 Review the authors. Good track record. That's probably not such a good one. 34 00:03:46,650 --> 00:03:50,240 Read the summary. Evaluate with these results. Be useful. 35 00:03:50,250 --> 00:03:55,680 Pretty good. Considered the site valid with these results apply to your practise? 36 00:03:56,100 --> 00:04:00,240 Maybe. But then what was the focus after that? 37 00:04:00,840 --> 00:04:04,139 Well, if it was a diagnostic test, look, 38 00:04:04,140 --> 00:04:11,010 whether there was an independent blind comparison to learn about the prognosis was there in a sectional cohort, 39 00:04:11,580 --> 00:04:21,150 aetiology or causation, which we've sort of abandoned as just the specific character characterisation, what are the basic methods? 40 00:04:21,420 --> 00:04:25,350 And to distinguish useful from useless or even harmful therapy? 41 00:04:25,710 --> 00:04:31,050 Was the assignment of patients really randomised risk of bias criteria? 42 00:04:31,440 --> 00:04:39,780 So the focus was risk of bias criteria and then if the initial was met, read the patients and methods section. 43 00:04:40,320 --> 00:04:43,710 Nothing here about the results. 44 00:04:44,010 --> 00:04:48,390 The focus is on what we think of as risk of bias. 45 00:04:49,800 --> 00:04:59,850 So that was critical appraisal. And then in 1990, when I took over the residency programme in Internal Medicine, 46 00:05:00,300 --> 00:05:07,760 I had a vision of what I wanted to teach people and I thought, I need to call this something. 47 00:05:07,770 --> 00:05:14,700 Come to McMaster to learn a particular approach to medicine, which is evidence based medicine. 48 00:05:15,000 --> 00:05:25,230 And this is the first article that appeared about with using the term evidence based medicine in the medical literature in 1991. 49 00:05:25,560 --> 00:05:31,380 And just to tell you, that's what I looked like in 1991 when that was published. 50 00:05:31,680 --> 00:05:34,560 Sadly, very different from how I look now. 51 00:05:35,370 --> 00:05:44,280 So I started this internal medicine residency programme and it was essentially a laboratory for IBM education. 52 00:05:44,850 --> 00:05:51,410 And my initial goal was every graduate would be able to read the methods and results. 53 00:05:51,420 --> 00:05:59,670 I was trained by Dave Sackett and I was very familiar with the initial reader's guides to the medical literature. 54 00:06:00,000 --> 00:06:06,150 I was aware of the importance of what we then called validity, which we now call risk of bias. 55 00:06:06,870 --> 00:06:15,750 However, at the end of seven years running this programme, I had observed some distressing findings. 56 00:06:16,680 --> 00:06:22,320 Very few of the residents, despite my best efforts who finished my programme, 57 00:06:22,470 --> 00:06:33,030 were able to appraise the methods and concluded that the advice in the CMAG series that they've led, 58 00:06:33,330 --> 00:06:40,620 while a brilliant innovation at the time and crucial to the development of what eventually 59 00:06:40,620 --> 00:06:47,670 became evidence based medicine in its focus on validity or risk of bias was misguided. 60 00:06:48,550 --> 00:06:52,090 And so we wanted to get the word out. 61 00:06:52,600 --> 00:07:01,719 So I finished the resume, my residency programme directorship in 1997 and in 2007, sorry. 62 00:07:01,720 --> 00:07:12,250 In 2000 we published this article in the BMJ, which made the case that critical appraisal was not for everybody. 63 00:07:12,490 --> 00:07:21,790 In fact, critical appraisal was for very few, but it was not realistic to ask physicians to find and appraise the results. 64 00:07:22,510 --> 00:07:33,460 Learning even skills, if by what even skills one means critical appraisal and risk of bias is not the best way to achieve evidence based practise. 65 00:07:34,160 --> 00:07:40,370 You will have the training. Those who do, including me, often won't have the time. 66 00:07:40,690 --> 00:07:50,559 I still high, as you see, will do often what other clinicians do, which is not to look carefully at the literature myself, 67 00:07:50,560 --> 00:07:55,990 or even critically appraise the literature summaries at the time. 68 00:07:56,650 --> 00:07:58,629 In 2000, when we wrote this, 69 00:07:58,630 --> 00:08:08,950 what was our alternative to learning Hebrew skills and being able to critical appraise it was pre appraised systematic reviews. 70 00:08:09,520 --> 00:08:18,190 Well, maybe. But 22 years later, we actually make another suggestion about what the alternative is. 71 00:08:19,060 --> 00:08:30,010 So the implications for education is that the goal is not critical appraisal of the primary studies, 72 00:08:30,250 --> 00:08:34,480 nor is critical appraisal criteria for systematic reviews. 73 00:08:34,750 --> 00:08:37,900 The structured clinical question. The comprehensive search. And so. 74 00:08:39,320 --> 00:08:44,100 That's not the business of the clinicians either. We still teach it. 75 00:08:44,120 --> 00:08:51,110 I still teach it, but I don't teach it with the expectation that anybody's going to remember any of those details. 76 00:08:51,620 --> 00:08:57,170 And if you try it, I would predict that after a little while, 77 00:08:57,380 --> 00:09:06,440 when you ask your your clinicians who've been through one of your courses about what they remember about concealment of application, 78 00:09:06,440 --> 00:09:09,800 blinding and loss, to follow up, it will be very little. 79 00:09:10,280 --> 00:09:14,450 So I don't teach that so that they can do it themselves. 80 00:09:14,480 --> 00:09:24,260 They're not going to that I can do it only so that they have an appreciation that there is a process that is legitimate, 81 00:09:25,160 --> 00:09:29,930 that is logical, that is trustworthy. 82 00:09:30,020 --> 00:09:36,980 There is a process that allows the identification of high, moderate, low and very low quality evidence. 83 00:09:37,520 --> 00:09:47,690 And so at the end, when somebody tells them this is high, moderate, low or very low quality evidence, they say, well, I can't do that myself. 84 00:09:47,900 --> 00:09:59,210 But I remember that there is a structured and and trustworthy process of making these characterisations of the quality of the evidence. 85 00:09:59,810 --> 00:10:03,080 But they still have to understand the results. 86 00:10:03,680 --> 00:10:09,950 They have to understand the results because our interventions come with benefits and with burdens and harms. 87 00:10:10,490 --> 00:10:18,230 And in particular, some of us believe that it should be the patient's values and preferences that bear on the issue. 88 00:10:18,770 --> 00:10:22,790 And to do shared decision making, you need to understand. 89 00:10:24,090 --> 00:10:28,110 So do clinicians currently understand the results? 90 00:10:28,380 --> 00:10:43,620 We did a study in which we took a survey survey in eight countries of primary and internal medicine physicians in training who 91 00:10:43,620 --> 00:10:52,559 came to their rounds and we presented them with six ways of presenting continuous variables which standardised mean difference. 92 00:10:52,560 --> 00:10:58,410 Mean difference ratio means minimally important different units, relative and absolute effects. 93 00:10:58,920 --> 00:11:06,120 So we start out continuous variables which can be appropriately modified to prevent present relative and absolute effects. 94 00:11:06,810 --> 00:11:17,160 So they were attending teaching Secessions, Canada, Chile, Norway, Spain, Lebanon, US, Switzerland and Finland because we had a captive audience. 95 00:11:17,430 --> 00:11:20,520 87% completed the survey. 96 00:11:21,760 --> 00:11:26,890 So they were randomly assigned one of four versions of the questionnaire. 97 00:11:27,490 --> 00:11:34,690 They either had small or large effects, and the six measures were either presented in order 1 to 6 or 6 to 1. 98 00:11:34,930 --> 00:11:42,250 And then we tested their understanding of whether the effect was trivial, small, moderate or large. 99 00:11:43,180 --> 00:11:48,850 Here was their understanding. Not surprisingly, we predicted this in advance. 100 00:11:49,270 --> 00:11:53,140 Risk difference was the one they understood best. Relative risk. 101 00:11:53,170 --> 00:12:01,670 Next. Next ratio of mean standardised mean difference that minimally important difference and the natural units of the instrument were less. 102 00:12:02,360 --> 00:12:12,640 So that's what we anticipated. What we didn't anticipate is that even with risk difference, the one they did best, it was awful. 103 00:12:13,000 --> 00:12:20,770 Only 40% of them got what we thought was the right answer in terms of the effect trivial, small, moderate or large. 104 00:12:21,400 --> 00:12:28,540 This was the stri found this quite distressing and this hastened my I don't I was 105 00:12:28,540 --> 00:12:33,820 already obviously since 2000 moving toward a focus on understanding results. 106 00:12:34,270 --> 00:12:39,580 This shifted me further. This shifted me to say understanding results are what's important. 107 00:12:39,790 --> 00:12:47,080 I should be spending even less time on the critical appraisal and more time helping clinicians learn to understand the results. 108 00:12:47,950 --> 00:12:59,150 So. But. My experience today, we said it in 2000, is that most even teachers have not got the message. 109 00:12:59,540 --> 00:13:04,880 They still think that clinicians are going to go out and do critical appraisal. 110 00:13:05,390 --> 00:13:13,520 And our first challenge, I think my I perceive our first challenge is to get Hebrew teachers to accept reality. 111 00:13:14,030 --> 00:13:17,570 Most of the nations will seldom or ever do critical appraisal. 112 00:13:18,160 --> 00:13:23,990 I had. They will still be making decisions and hopefully doing that in. 113 00:13:25,110 --> 00:13:28,139 The process of shared decision making with their patients. 114 00:13:28,140 --> 00:13:34,530 They'll be counselling patients. And our studies suggest they can interpret the results to do that well. 115 00:13:35,280 --> 00:13:39,600 So we need to get even teachers to understand the implications. 116 00:13:40,110 --> 00:13:46,440 Markedly De-Emphasise critical appraisal markedly emphasise understanding the results. 117 00:13:47,220 --> 00:13:52,210 When will we be successful? When we repeat that survey I told you about? 118 00:13:52,620 --> 00:14:01,170 And at least with respect to relative risk and risk difference, maybe some of the others clinicians can actually understand the results. 119 00:14:01,440 --> 00:14:11,040 That will be progress. So next thing I will tell you or remind you if you already know a little bit about grade. 120 00:14:11,460 --> 00:14:23,790 So in 2004, we published a paper in the BMJ that presented this process of looking at quality of evidence and strength of recommendations. 121 00:14:24,390 --> 00:14:29,010 And the missions now in my neck of the woods, actually worldwide, 122 00:14:29,010 --> 00:14:36,540 I go worldwide and I say at least in academic centres who uses up to date and everybody extraordinary 123 00:14:36,540 --> 00:14:42,570 number of people around the world are using up to date update has over 10,000 graded recommendations. 124 00:14:43,080 --> 00:14:47,340 They present the quality of the evidence and the strength of recommendations. 125 00:14:47,820 --> 00:14:57,210 A lot of people who use Uptodate don't understand, but up to date says at one or two C they don't understand what it means. 126 00:14:57,540 --> 00:15:06,300 And they need to they need to understand that ABC is high, moderate and low quality evidence, and one and two is strong and weak recommendations. 127 00:15:07,020 --> 00:15:11,909 So the next 2004, 128 00:15:11,910 --> 00:15:23,040 we published the first presentation of grade in the BMJ and then we published the six part series about grade for clinicians if they see great use, 129 00:15:23,310 --> 00:15:28,210 how to understand it. And we've been pretty successful. 130 00:15:28,510 --> 00:15:38,379 So over 110 organisations have adopted grade include World Health Organisation, prominent American organisations, American Thoracic Society, 131 00:15:38,380 --> 00:15:47,950 American College of Physicians, British NHS, Cochrane Collaboration and both the two leading electronic both up to date and that. 132 00:15:48,870 --> 00:15:55,529 So what are grades, criteria, confidence or certainty or quality of evidence? 133 00:15:55,530 --> 00:15:59,040 They're all synonyms. Can be high, moderate, low and very low. 134 00:15:59,250 --> 00:16:05,579 Randomised trials start as high, but they may be rated down by risk of bias, inconsistency. 135 00:16:05,580 --> 00:16:09,030 Indirectness Imprecision and publication bias. 136 00:16:09,780 --> 00:16:16,800 Observational studies start as low. They may go to very low because of those problems, or typically because of large effects. 137 00:16:17,250 --> 00:16:24,809 Things like insulin for diabetic ketoacidosis or hip replacement or dialysis do 138 00:16:24,810 --> 00:16:29,670 not need randomised trials to show to provide high quality evidence of benefit. 139 00:16:30,630 --> 00:16:38,130 So challenge number two, way back in 2000, we said clinicians look at systematic reviews. 140 00:16:38,550 --> 00:16:41,970 We don't tell clinicians to look at systematic reviews anymore. 141 00:16:42,180 --> 00:16:50,070 We tell them to look at trustworthy guidelines and we don't need to tell them that is what they in fact are looking at. 142 00:16:50,790 --> 00:16:56,700 Good guidelines present the quality of the evidence. Most good ones will be using grade. 143 00:16:57,180 --> 00:17:04,200 So our success with Challenge two will be met when clinicians understand the quality or certainty of evidence synonyms, 144 00:17:04,560 --> 00:17:07,320 and they understand the basics of the great approach. 145 00:17:08,220 --> 00:17:15,120 Now, in terms of the third challenge, I'm going to tell you a story of a study done by PJ Devereux, 146 00:17:15,120 --> 00:17:22,710 who is now leading the world in terms of perioperative medicine, who way back did a study looking at values and preferences. 147 00:17:23,160 --> 00:17:26,520 He took patients with atrial fibrillation at high risk of stroke. 148 00:17:26,820 --> 00:17:33,000 Anticoagulation lessens the risk of stroke and the risk at the cost of more GI bleeds. 149 00:17:33,330 --> 00:17:42,600 He presented that without treatment, 100 patients will suffer 12 strokes, six major and six minor anticoagulation decreased strokes to four. 150 00:17:42,900 --> 00:17:49,950 And asked how many bleeds would you accept in 100 patients and still be willing to administer or take anticoagulants? 151 00:17:50,760 --> 00:18:00,660 And if we were doing this interactively, I would ask you, how many bleeds would you accept to prevent eight strokes in 100 patients? 152 00:18:01,170 --> 00:18:05,910 Five or fewer? 6 to 10, 11 to 15, six or 20 or more than 20. 153 00:18:06,420 --> 00:18:08,129 Think for just a few seconds. 154 00:18:08,130 --> 00:18:15,510 If you are finishing or even if you're not a clinician, how many bleeds would you be willing to accept to prevent eight strokes? 155 00:18:16,260 --> 00:18:26,100 Here's what happens when Dr. Devereux asks physicians and patients with respect to the patient physician's flat distribution, 156 00:18:26,520 --> 00:18:31,920 they went over right from. Very few under five. 157 00:18:32,130 --> 00:18:35,430 Right up to a large number and across the whole range. 158 00:18:35,820 --> 00:18:41,670 Physicians, values and preferences in terms of stroke or bleeding aversion or very variable. 159 00:18:42,300 --> 00:18:51,540 The patients, the majority of patients overwhelmingly chose that they would be ready to accept 22 bleeds, two thirds of them to prevent eight strokes. 160 00:18:51,900 --> 00:19:00,780 The patients were much more stroke averse and much less leaning averse than were clinicians, although a few of the patients were at the other end. 161 00:19:00,780 --> 00:19:09,540 More like the clinicians. More bleeding averse messages is if you use the clinician's values and preferences, you're going to get it wrong, 162 00:19:09,540 --> 00:19:15,960 at least in this case, if you're much better to use the values and preferences of the majority of patients. 163 00:19:16,410 --> 00:19:22,050 But if you really want to get it right, you have to find out about individual patient values and preferences. 164 00:19:22,710 --> 00:19:32,550 So great then says. And a strong recommendation would be when all or almost half of fully informed patients would make that same choice. 165 00:19:33,120 --> 00:19:36,359 Interaction with the patient strong recommendation. 166 00:19:36,360 --> 00:19:41,850 You can just inform the patient's week you need to do shared decision making. 167 00:19:42,180 --> 00:19:49,650 Sure decision making can be aided by decision aid, and you're only going to use quality of care criteria for strong recommendations. 168 00:19:50,310 --> 00:19:55,440 So challenge number three Achieve shared decision making when appropriate. 169 00:19:55,890 --> 00:20:00,090 Clinicians and guideline developers need to understand two situations. 170 00:20:00,390 --> 00:20:05,070 Just do it. The strong recommendations, weak recommendations, 171 00:20:05,070 --> 00:20:13,139 or conditional that are value in preference sensitive success will be achieved when clinicians do shared decision making. 172 00:20:13,140 --> 00:20:17,640 When appropriate, they understand the great, strong and weak recommendation. 173 00:20:17,940 --> 00:20:28,020 And when every guideline tells you what their underlying values and preferences are that drive the direction and strength of their recommendation. 174 00:20:28,860 --> 00:20:33,030 Thus, three challenges to teaching evidence based medicine. 175 00:20:33,510 --> 00:20:43,560 First, get clinicians to understand the results, how to achieve it, teach guidelines, grades and results, not critical appraisal. 176 00:20:44,950 --> 00:20:50,560 We will be successful when clinicians understand the guidelines that they are using. 177 00:20:51,860 --> 00:20:58,250 Clinicians all over the world rely on guidelines in ways totally different than they did 20 years ago. 178 00:20:58,820 --> 00:21:05,630 But to understand guidelines, they need to understand the concepts of quality of evidence and strength of recommendations. 179 00:21:06,140 --> 00:21:12,440 We will be successful when clinical and guideline developers understand values and preferences, 180 00:21:12,740 --> 00:21:18,860 and the clinicians who use those guidelines will understand it and applied in shared decision making. 181 00:21:19,460 --> 00:21:22,280 So thanks very much for listening.