1 00:00:00,060 --> 00:00:06,840 Title of the talk is how do you conduct a really synthesis when there is no evidence and I guess, you know, disclosures first? 2 00:00:06,840 --> 00:00:11,130 This project that I'm going to talk about was funded by an I child. 3 00:00:11,130 --> 00:00:16,920 And I'll explain what problems are in a minute. But of course, the usual disclosures apply. 4 00:00:16,920 --> 00:00:24,330 And now, you know, this is not their opinion. It's mine. I also want to acknowledge the brilliant team we had as a big team. 5 00:00:24,330 --> 00:00:30,190 So lots of different people from different backgrounds, so very grateful for them. 6 00:00:30,190 --> 00:00:36,150 And that was that was great. So you have an eye to the inspiration. 7 00:00:36,150 --> 00:00:40,920 I guess for this talk comes from the feedback we got from it. 8 00:00:40,920 --> 00:00:49,230 And I child when I put in my grant to do a really synthesis of feedback of patient reported outcome measures. 9 00:00:49,230 --> 00:00:56,500 And I will not assume that you will know what those they are, but I will explain them in due course as the talk goes on. 10 00:00:56,500 --> 00:00:59,490 But first, I just wanted to give you some insight. 11 00:00:59,490 --> 00:01:05,670 I don't know if any of you you've applied for an outside of funding, you know, you get the email and you, Oh, goodness me, what's it going to say? 12 00:01:05,670 --> 00:01:13,710 And you have to calm yourself down to to read the feedback and, you know, remind yourself that it's really not that bad and you can't deal with it. 13 00:01:13,710 --> 00:01:20,970 But one of the questions that they really interested me, I had that you can see here is is, you know, 14 00:01:20,970 --> 00:01:25,800 the feedback that they wanted to give us was whether researchers will gain sufficient examples of 15 00:01:25,800 --> 00:01:31,530 where problems have and have not led to improvements for an unbiased formulation of the model. 16 00:01:31,530 --> 00:01:39,630 And I think ultimately what this reviewer was saying is how will you do this really synthesis when there isn't any or much evidence? 17 00:01:39,630 --> 00:01:42,870 Because at the time we submitted the grant, well, 18 00:01:42,870 --> 00:01:52,920 patient reported outcome measures were often used in trials as indicators of the outcome of an intervention or treatment. 19 00:01:52,920 --> 00:01:58,500 There wasn't a lot of evidence if they if they use performance data, so is at the very beginning. 20 00:01:58,500 --> 00:02:07,560 So when this project got funded, it was 2014. So I think it was probably about 2012 2013 when we got this feedback and the National 21 00:02:07,560 --> 00:02:12,600 English National Problems programme had only been running really since 2010. 22 00:02:12,600 --> 00:02:17,910 So this was the use of patient reported outcome measures as indicators of the quality of hospitals. 23 00:02:17,910 --> 00:02:23,520 So there was a feeling from the reviewer to say, Well, this is a very new intervention and you know, 24 00:02:23,520 --> 00:02:28,260 nobody's there's no evidence out there about its effectiveness or its impacts. 25 00:02:28,260 --> 00:02:36,930 How are you going to do this review? And, you know, I chuckled to myself and thought, I know how to answer that question. 26 00:02:36,930 --> 00:02:42,900 And I suppose the summary really of this talk and what I'm going to sort of elaborate in more detail as we go 27 00:02:42,900 --> 00:02:50,160 along is that I think the review was coming very much coming from a traditional systematic review perspective. 28 00:02:50,160 --> 00:02:56,340 So when you're doing a traditional systematic review, a Cochrane review, however you want to call them, 29 00:02:56,340 --> 00:03:03,960 and I have to say also that I think realist reviews are systematic too, but that's another of the side discussion. 30 00:03:03,960 --> 00:03:11,580 But I'm talking here, I suppose, about sort of Cochrane style reviews. And in those kind of reviews, the unit of analysis is the intervention itself. 31 00:03:11,580 --> 00:03:13,710 It's the programme, it's the intervention. 32 00:03:13,710 --> 00:03:22,440 And so evidence synthesis really does rely on there being evidence trials that have evaluated that specific intervention. 33 00:03:22,440 --> 00:03:26,190 So it really is a problem if there aren't many trials available. 34 00:03:26,190 --> 00:03:31,620 If there isn't any cohort studies or quasi experimental studies of the of the intervention, 35 00:03:31,620 --> 00:03:35,700 you really do have a problem if there's not any any evidence. 36 00:03:35,700 --> 00:03:44,440 And that's why, you know, an actual wisely require people to do some scoping work to look at the volume of evidence before they submit a grant. 37 00:03:44,440 --> 00:03:48,250 Now really sort of a different, well, 38 00:03:48,250 --> 00:03:54,270 one of the fundamental differences between realist reviews and what I call traditional systematic reviews 39 00:03:54,270 --> 00:03:59,220 is the unit of analysis and the unit of analysis in Middle East reviews is the programme theory. 40 00:03:59,220 --> 00:04:04,500 Now I'm sure you'll have heard Jeff talking about this at some point over the next two days, 41 00:04:04,500 --> 00:04:10,950 and I am going to elaborate a little bit more about what I mean by programme theory, but just go with me here. 42 00:04:10,950 --> 00:04:17,130 So evidence synthesis relies on the evaluation of interventions sharing the same programme theory. 43 00:04:17,130 --> 00:04:22,290 So is the programme theory that you're testing that you're evaluating, not the intervention itself. 44 00:04:22,290 --> 00:04:29,400 Now this is quite a big thing to get your head around. I'm not expecting you to automatically go, Oh, right, yeah, I get it. 45 00:04:29,400 --> 00:04:34,110 But I do hope that during the talk, it will be made clearer. 46 00:04:34,110 --> 00:04:40,740 And I think one of the things to say here is that different interventions can share the same programme theory, 47 00:04:40,740 --> 00:04:47,100 but at the same time, the same intervention can have a number of different programme theories underlying it. 48 00:04:47,100 --> 00:04:52,350 This sounds confusing, but it's also one of the great things about realist methods is that they can be flexible. 49 00:04:52,350 --> 00:04:59,680 So just because there isn't any evidence about promise specifically, I think you'll find that the programme theories and. 50 00:04:59,680 --> 00:05:04,420 The lying problems and their feedback and they use as a an indicator of hospital 51 00:05:04,420 --> 00:05:09,940 quality is not that new is certainly not as new as as problems are themselves. 52 00:05:09,940 --> 00:05:13,870 There are lots of other interventions that share that same programme theory. 53 00:05:13,870 --> 00:05:18,100 So, you know, this is a really important and fundamental difference, I think, 54 00:05:18,100 --> 00:05:24,820 which is sort of the basis of this talk about the difference between a realist review and a traditional systematic review. 55 00:05:24,820 --> 00:05:31,390 I would say, however, that there are many similarities, and one of those includes the requirement to read lots of papers. 56 00:05:31,390 --> 00:05:36,880 So that doesn't change. There are some similarities there. 57 00:05:36,880 --> 00:05:40,960 But just before I dive into sort of talk a little bit more about the really synthesis, 58 00:05:40,960 --> 00:05:46,970 I'm I'm not making the assumption that you will know what problems are. And for many people, when they hear the word problems, 59 00:05:46,970 --> 00:05:52,930 they think of something that year 11 and year 13 students do at the end of their school year to celebrate, 60 00:05:52,930 --> 00:06:00,940 as many of them are about to do at the moment. But but my my version of problems is it short for patient reported outcome measures? 61 00:06:00,940 --> 00:06:05,560 And these are questionnaires that measure patients perceptions of the impact of 62 00:06:05,560 --> 00:06:10,930 their condition and its treatment on their health and their the structured. 63 00:06:10,930 --> 00:06:13,990 They ask patients to rate the health on various different dimensions. 64 00:06:13,990 --> 00:06:23,620 So the example you can see here is from the Oxford Hip School, which was developed in Oxford, and they can be generic. 65 00:06:23,620 --> 00:06:30,550 That is, they apply to anybody. So there are problems like the SF 36, which some of you may have heard of, 66 00:06:30,550 --> 00:06:37,690 which are suitable for people with a variety of conditions and also people that don't have any health problems as well. 67 00:06:37,690 --> 00:06:42,590 Well, they can be specifically focussed on looking at the impact of a specific condition. 68 00:06:42,590 --> 00:06:52,240 So in the case of the Oxford Hip School, this is a condition specific from looking at the impact of arthritis on on people's hips. 69 00:06:52,240 --> 00:07:00,370 And it's usually used to evaluate hip operations, which is part of the English Promise programme, which I'll explain in a minute. 70 00:07:00,370 --> 00:07:09,010 But the big idea about prongs is that they sort of capture and quantify the patient's perspective on their health and their condition. 71 00:07:09,010 --> 00:07:15,460 And, you know, when they were first developed, they were largely used as as I said before outcome measures in in trials of different treatments. 72 00:07:15,460 --> 00:07:21,850 But more recently, there's been a use of them in the health service in two different ways. 73 00:07:21,850 --> 00:07:28,300 So the main way, well, not the main way, but the focus of this talk is they use what I call an aggregate level. 74 00:07:28,300 --> 00:07:31,840 So this is where data patients are asked. 75 00:07:31,840 --> 00:07:36,070 So, for example, in the English National Problems programme here, 76 00:07:36,070 --> 00:07:42,130 anybody that's undergoing either a knee replacement or hip replacement is asked to complete a 77 00:07:42,130 --> 00:07:48,880 baseline patient approach that could measure the condition specific one four hips and one for knees. 78 00:07:48,880 --> 00:07:53,710 Both developed in Oxford so that the data is collected before the procedure. 79 00:07:53,710 --> 00:08:00,040 And then I think it's either six months or three months, depending on the condition they complete it again. 80 00:08:00,040 --> 00:08:07,780 Those data are just off the case mix and the change that people experience from before the hip 81 00:08:07,780 --> 00:08:13,450 replacement or the knee replacement after the knee replacement is aggregated at a hospital level. 82 00:08:13,450 --> 00:08:24,250 So each hospital has a case mix, adjusted average change in the hip or knee outcome score for patients, 83 00:08:24,250 --> 00:08:28,570 and this is used as an indicator of the amounts of health gain. 84 00:08:28,570 --> 00:08:36,100 I guess that patients have gained from going through that procedure and is used as an indicator of the quality of care in that hospital. 85 00:08:36,100 --> 00:08:39,160 They're also used to an individual level of synthesis. 86 00:08:39,160 --> 00:08:45,840 Looked at this as well, but I'm not going to really talk about this for now, but just to provide a bit more context, 87 00:08:45,840 --> 00:08:50,620 an individual level patients perhaps might complete a problem before they go visit the clinician. 88 00:08:50,620 --> 00:08:56,590 This is feedback back to the clinician during the visit. The idea is this this enables patients to raise issues with clinicians, 89 00:08:56,590 --> 00:09:03,010 or it helps clinicians understand what's going on for the patient, and there's a number of international examples. 90 00:09:03,010 --> 00:09:10,450 But as I mentioned for for this tool, what we were interested in was understanding how and why problems, 91 00:09:10,450 --> 00:09:14,980 data and aggregate level might improve patient care through what mechanisms. 92 00:09:14,980 --> 00:09:23,080 How will context effect that? What are they? What are the outcomes? And this is the data offered that that providers are presented with. 93 00:09:23,080 --> 00:09:27,610 So each of those little dots on what's known as a funnel plot. 94 00:09:27,610 --> 00:09:33,070 It's the main post-operative Oxford hip score against the provides a sample size of the number of 95 00:09:33,070 --> 00:09:38,860 procedures where there's an expectation that perhaps centres that do more procedures might be. 96 00:09:38,860 --> 00:09:44,440 It might be slightly better at it, but there's 95 percent and 99 percent confidence intervals. 97 00:09:44,440 --> 00:09:52,360 And if your hospital is one of the dots that's outside of these confidence intervals, you're labelled as an outlier. 98 00:09:52,360 --> 00:09:58,030 So really, what you want to be is somewhere I think you can see my cursory somewhere up here. 99 00:09:58,030 --> 00:10:02,350 These are positive out. So they're doing almost like better than is expected. 100 00:10:02,350 --> 00:10:10,900 These ones are what's known as negative outliers and labelled as as hospitals they're not doing as well as as they're expected. 101 00:10:10,900 --> 00:10:16,870 So you know what a provider supposed to do with this information is one of the things that we were really interested. 102 00:10:16,870 --> 00:10:21,340 How is this supposed to incentivise create? 103 00:10:21,340 --> 00:10:26,620 How do responders providers respond to this? You know, what are the mechanisms through this might work? 104 00:10:26,620 --> 00:10:29,380 And you know how? How will context of this? 105 00:10:29,380 --> 00:10:34,660 How does that, you know, perhaps the features of the intervention or the context into which it's implemented? 106 00:10:34,660 --> 00:10:39,300 These are all very, really good questions. As you look at it from from Jeff. 107 00:10:39,300 --> 00:10:42,690 So it's also interesting to note that's an interesting, 108 00:10:42,690 --> 00:10:50,130 contextual feature about problems was that they were incentivised as part of local sequence payments for four particular years. 109 00:10:50,130 --> 00:10:58,620 They're not, they're not. So now, so you can see this is quite a complex intervention requiring a lot of, I suppose, interpretation. 110 00:10:58,620 --> 00:11:02,280 And you know, one of the things we were really interested in is, well, you know, 111 00:11:02,280 --> 00:11:07,230 when this was set up, what were the ideas and assumptions underlying how this was supposed to work? 112 00:11:07,230 --> 00:11:11,790 And in fact, this was partly why Nick Black, we were very privileged to have him on our team. 113 00:11:11,790 --> 00:11:14,680 He was instrumental in setting up the National Problems programme. 114 00:11:14,680 --> 00:11:21,630 And one of the reasons he joined the project was because he was really interested in thinking about, you know, what are the underlying? 115 00:11:21,630 --> 00:11:25,560 Why did we set this up in the first place, which was which is very interesting. 116 00:11:25,560 --> 00:11:32,140 So now I'm going to talk a little bit more about the process that we did this and also illustrator methodological point, 117 00:11:32,140 --> 00:11:42,540 as I do so about the importance of programme theory and how that's used as a unit of analysis in this context. 118 00:11:42,540 --> 00:11:47,520 But just as a quick reminder, and I'm not sure where on your realist journey you are, 119 00:11:47,520 --> 00:11:54,180 whether Geoff's talked you through the process of doing a realist synthesis, but I just thought I'd provide a really quick slide here. 120 00:11:54,180 --> 00:12:00,180 I'm not going to go into detail as to every single one of these processes, 121 00:12:00,180 --> 00:12:10,620 but I'm just going to give you an overview and then talk to a couple of these issues to illustrate the idea about programme theory and how 122 00:12:10,620 --> 00:12:18,360 you can use that as a unit of analysis and use evidence from interventions that share the same programme theory to to do your analysis. 123 00:12:18,360 --> 00:12:26,070 So the basic idea is that the first sort of process of doing a really synthesis is you need to get an understanding of the programme theory, 124 00:12:26,070 --> 00:12:30,750 the ideas and assumptions. And I'm going to talk to you a little bit more about programme theory in a minute. 125 00:12:30,750 --> 00:12:34,260 As Jeff mentioned, you can't often look at everything in a real synthesis. 126 00:12:34,260 --> 00:12:39,600 You have to be focussed. You have to decide which theories are we going to test? 127 00:12:39,600 --> 00:12:43,500 Are we going to jettison some? Are we going to focus on particular ones? 128 00:12:43,500 --> 00:12:48,420 Again, that's that's an important decision and offered a difficult one. 129 00:12:48,420 --> 00:12:54,480 And then there's a second, sorry second phase of searching, and it's different to the first phase. 130 00:12:54,480 --> 00:12:56,790 The first phase is much more interested in programme theory. 131 00:12:56,790 --> 00:13:01,740 So you look at different sorts of evidence, and I'll talk a bit more about that in a minute. 132 00:13:01,740 --> 00:13:09,300 So the second phase, you're looking for empirical evidence in order to test your programme theories, the quality appraisal. 133 00:13:09,300 --> 00:13:16,050 And again, this is different to a standard, you know, systematic review or Cochrane review. 134 00:13:16,050 --> 00:13:23,220 So you're not selecting studies or appraising quality against a hierarchy of evidence where randomised controlled trials sit at the top. 135 00:13:23,220 --> 00:13:30,090 It's more about the study's relevance to testing your theory, so a whole range of different study designs are used. 136 00:13:30,090 --> 00:13:37,740 You know, quantitative studies really good for looking at outcome patterns and perhaps the relationships that outcomes might have with context. 137 00:13:37,740 --> 00:13:42,030 Qualitative studies are really useful for looking at mechanisms and perhaps 138 00:13:42,030 --> 00:13:47,100 some links between particular contexts giving rise to particular mechanisms. 139 00:13:47,100 --> 00:13:53,340 So one of the strengths I think of early synthesis is that you use different study designs and 140 00:13:53,340 --> 00:13:58,390 their their importance in their their inclusion is based on their theory testing potential, 141 00:13:58,390 --> 00:14:03,510 not about whether they're an asset or not. Then there's a process of data extraction. 142 00:14:03,510 --> 00:14:05,190 I'm not going to go into detail about this. 143 00:14:05,190 --> 00:14:12,750 Very happy to share our data extraction with you, but I'm going to talk more sort of an abstract level, I suppose. 144 00:14:12,750 --> 00:14:17,220 But again, you know, this is not necessarily to a strict standard matrix. 145 00:14:17,220 --> 00:14:22,470 You know, I've done Cochrane reviews and used a a an accent spreadsheet. 146 00:14:22,470 --> 00:14:33,210 When I do various reviews, I tend to use word documents and do much more of a narrative commentary and then, you know, synthesise it in words tables. 147 00:14:33,210 --> 00:14:36,030 But anyway. And then there's a process of this, 148 00:14:36,030 --> 00:14:43,650 which is where you're bringing your programme theories into conversation with various different processes of empirical evidence. 149 00:14:43,650 --> 00:14:48,330 And then there's this dissemination production of abstract, middle range theory. 150 00:14:48,330 --> 00:14:53,760 And that's what I'm going to talk a little bit about much more about this middle aged programme theory. 151 00:14:53,760 --> 00:15:01,830 So that's just a quick overview. So I want to go back to this idea of Programme three, which I've been sort of banding about with, you know, 152 00:15:01,830 --> 00:15:07,440 with gay abandon, without thinking about and explaining what it is, which is which must be frustrating. 153 00:15:07,440 --> 00:15:15,660 So. And this was one of the first questions I remember when I first met Ray White and a long time ago when I was putting in a night fellowship, 154 00:15:15,660 --> 00:15:21,060 actually a personal one, which I didn't get, but everything worked out anyway. 155 00:15:21,060 --> 00:15:24,180 And I thought, Well, what is the programme say? What do you mean? 156 00:15:24,180 --> 00:15:28,560 You know, at the time I was saying, Well, is it just sort of psychological or sociological theory? 157 00:15:28,560 --> 00:15:34,020 And he was like, Well, it can be, but actually, it's something more simple than that. 158 00:15:34,020 --> 00:15:39,440 It's you know what? What are the ideas and assumptions underlying how this intervention will work? 159 00:15:39,440 --> 00:15:47,150 What are what's the what? What was going on in the policymakers heads when they developed this intervention to to think 160 00:15:47,150 --> 00:15:52,770 about how it was going to work and really think about how things work in a particular way? 161 00:15:52,770 --> 00:16:00,530 So you will have heard Jeff, talk about this idea of mechanisms. So how are the various recipients or the stakeholders involved in the intervention 162 00:16:00,530 --> 00:16:06,590 intended to respond to the resources that are offered by the intervention and how, 163 00:16:06,590 --> 00:16:14,130 you know, how will context shape that? And I think what we find often is that interventions are often developed in almost like a context free way. 164 00:16:14,130 --> 00:16:17,900 That's an assumption often that they're going to work the same way everywhere. 165 00:16:17,900 --> 00:16:21,440 You know, we do this in pool by magic. Something happens. 166 00:16:21,440 --> 00:16:29,630 And I think sometimes also those those ideas and assumptions are deemed to be so obvious that they're not spelled out. 167 00:16:29,630 --> 00:16:34,070 But I think one of the things that we have to do when we're thinking as a real estate is really interrogate that. 168 00:16:34,070 --> 00:16:39,530 And I think this was what attracted Nick Black to the project. He was saying, I'm not sure we've done that yet with problems. 169 00:16:39,530 --> 00:16:45,740 So, so that was interesting. And I think when you're first developing your programme theory ideas, as I'll illustrate, 170 00:16:45,740 --> 00:16:52,760 when I talk about how we did this in the project, they often stay very close at the beginning to the particular programme. 171 00:16:52,760 --> 00:16:57,470 So how is this programme supposed to work? How are people supposed to respond? 172 00:16:57,470 --> 00:17:02,420 But once you've done that, you'll then find actually those ideas are not new. 173 00:17:02,420 --> 00:17:07,970 They're very similar to something else I've heard of or another intervention that's got the same ideas. 174 00:17:07,970 --> 00:17:14,630 And I guess this is the other picture that I really like that that that we did as part of the Ramos's project. 175 00:17:14,630 --> 00:17:20,060 One of the great joys of that project was working with a cartoonist who also was an evaluator. 176 00:17:20,060 --> 00:17:24,170 I mean, what are the chances of finding somebody with that skill set? 177 00:17:24,170 --> 00:17:29,120 But happily, they exist, and they did these wonderful cartoons for as they call Chris Lissy. 178 00:17:29,120 --> 00:17:37,010 And he was absolutely brilliant, and he developed these great cartoons and these this is what I think like about this one is that they show that, 179 00:17:37,010 --> 00:17:43,490 you know, often there are some very general and abstract and very familiar ideas underlying all the things we do. 180 00:17:43,490 --> 00:17:48,260 And, you know, some people would argue actually all the interventions that anybody ever does just, 181 00:17:48,260 --> 00:17:56,210 you know, can be narrowed down to whether they're a carrot, a stick or a sermon, you know, so you know, carrots are things like, you know, 182 00:17:56,210 --> 00:18:02,390 reward stickers for children at school to financial incentives for us to do particular tests. 183 00:18:02,390 --> 00:18:05,480 You know, they all share this sort of idea of an incentive. 184 00:18:05,480 --> 00:18:15,200 Similarly, you know, stakes are behaviour points that children get at school or sanctions that people get for not following the rules. 185 00:18:15,200 --> 00:18:22,040 Similarly, all sorts of teaching is providing people with information training equivalent to being a sermon. 186 00:18:22,040 --> 00:18:28,340 And Paulson calls these sort of I you know these this idea that many interventions 187 00:18:28,340 --> 00:18:33,230 share the same underlying ideas and solutions as reusable conceptual platforms, 188 00:18:33,230 --> 00:18:39,260 and we really kind of try to embrace this idea as we as we did as we did the review. 189 00:18:39,260 --> 00:18:44,510 So when we were beginning this review, we we really, as I mentioned before, stuck quite closely to the idea of, 190 00:18:44,510 --> 00:18:49,940 you know, what are the underlying ideas about how the aggregate user from stage is going to improve care? 191 00:18:49,940 --> 00:18:55,550 And we drew on various different policy documents, editorials, think pieces, letters. 192 00:18:55,550 --> 00:18:58,520 So this was a study, a study. 193 00:18:58,520 --> 00:19:06,590 It was more like an extended review of the literature that was undertaken by Nancy Devlin and John Appleby at the King's Fund. 194 00:19:06,590 --> 00:19:14,720 There's also guidance that the NHS issued about how Proms data should be collected and how providers should respond to it, 195 00:19:14,720 --> 00:19:20,660 the steps they should take if they're an outlier. Again, methodological guidance as well. 196 00:19:20,660 --> 00:19:29,180 You know how how are these outlier allies identified, but also kind of editorials and analysis by particularly this one by Nick Black, 197 00:19:29,180 --> 00:19:35,030 where various commentators set out, you know, this is how I think this intervention is going to work. 198 00:19:35,030 --> 00:19:38,360 And what's really interesting are the kind of sometimes editors set up. 199 00:19:38,360 --> 00:19:44,000 These debate articles don't know whether one person's arguing one side and another person's arguing another. 200 00:19:44,000 --> 00:19:51,980 So these are really rich opportunities for getting at those underlying ideas about, well, how is it that problems are supposed to work? 201 00:19:51,980 --> 00:20:00,260 And I know Jeff, when you were doing the smoking cars carrying children's review with Ray, 202 00:20:00,260 --> 00:20:07,370 and they were really nice paper about how the Today programme on Radio four is a really interesting sort of programme series as well, 203 00:20:07,370 --> 00:20:11,000 because there's that debate about how this programme is supposed to work. 204 00:20:11,000 --> 00:20:17,090 We also did a stakeholder workshop where we presented some of these ideas to stakeholders and said, What do you think? 205 00:20:17,090 --> 00:20:23,570 You know, is there anything we've missed? What does this really mean? I also did a few interviews with various stakeholders as well. 206 00:20:23,570 --> 00:20:29,030 So we really tried to sort of get get a feel for what's what are the underlying ideas. 207 00:20:29,030 --> 00:20:36,230 And this revealed, I guess, we narrowed it down after a long iterative process into three sort of main idea. 208 00:20:36,230 --> 00:20:42,780 So the first one? And, you know, if you think that this was, you know, the Proms programme did emerge from the dance, 209 00:20:42,780 --> 00:20:46,770 I report he was he was very interested in patient choice. 210 00:20:46,770 --> 00:20:54,750 So one of the ideas behind the aggregate use of proms was that they were useful to evaluate the relative clinical quality providers 211 00:20:54,750 --> 00:21:02,460 of elective procedures because hip and knee replacements were elective and it can be used by patients in exercising choice. 212 00:21:02,460 --> 00:21:07,350 In other words, the idea was by presenting these data on proms, 213 00:21:07,350 --> 00:21:13,080 on the outcomes and the different providers and where they stood in relation to each other. 214 00:21:13,080 --> 00:21:20,730 This could be used by GP's and patients in choosing where to have their hip operation and where to have their knee operation. 215 00:21:20,730 --> 00:21:24,870 So it was, you know, it was sort of in the context of the the policy at the time, 216 00:21:24,870 --> 00:21:30,330 which was very big on expanding and supporting patient choice in the NHS. 217 00:21:30,330 --> 00:21:33,840 Another key. So we sort of called this choice theory. 218 00:21:33,840 --> 00:21:40,360 Another key idea was this idea that, you know, I could get from this data can be used to empower a commission, 219 00:21:40,360 --> 00:21:43,470 as you can see how old this is because they talk about pets who are now, 220 00:21:43,470 --> 00:21:48,960 of course, CCGs and now, you know, integrated care systems and that they can use the data, 221 00:21:48,960 --> 00:21:52,680 establish the quality of services which they are contracting providers for. 222 00:21:52,680 --> 00:21:58,620 In other words, they can hold providers accountable for the quality of the services that they are providing. 223 00:21:58,620 --> 00:22:05,190 So commissioners can use these data to say, actually, you know, you said you were going to provide this service. 224 00:22:05,190 --> 00:22:07,980 This is the quality. You know this, we're not happy with this. 225 00:22:07,980 --> 00:22:14,520 So they can presumably impose sanctions or reward it, for example, with sequin payments. 226 00:22:14,520 --> 00:22:22,500 And then another idea was that promise will be useful for assessing the relative clinical quality of providers of elective procedures for clinicians, 227 00:22:22,500 --> 00:22:25,680 managers and commissioners to benchmark their own performance. 228 00:22:25,680 --> 00:22:33,300 And I remember asking one of the persons in the Department of Health about this, and I said, Well, what do you think, clinicians? 229 00:22:33,300 --> 00:22:36,540 You know, how do you think surgeons or clinicians should respond to these days? 230 00:22:36,540 --> 00:22:42,900 And they said, Well, what I hope they'll do is they'll go, Oh, the hospital down the road is doing a really good job. 231 00:22:42,900 --> 00:22:45,900 How can we learn from them? Can I go and talk to them? 232 00:22:45,900 --> 00:22:53,790 So the eye is very much the idea of this intrinsic motivation that providers might have to improve their own practise to learn from others. 233 00:22:53,790 --> 00:23:01,050 So you can see these are sort of three different ideas underlying these problems, and this is where we sort of started our synthesis from. 234 00:23:01,050 --> 00:23:07,830 And then we we sat down the three of us. I'm not very good at drawing, so I did stick people at me. 235 00:23:07,830 --> 00:23:17,400 Sonya Duncan, who is the research fellow on the project and was great to work with, and André, who sort of an adviser with us, you know, what is this? 236 00:23:17,400 --> 00:23:25,140 What is this an example of this sound? Quite familiar ideas that many other interventions also share these ideas. 237 00:23:25,140 --> 00:23:28,530 So we tried to think about what? What were these similarities? 238 00:23:28,530 --> 00:23:35,910 So, you know, one of the specific things was patients will choose proms will use proms to choose a provider, 239 00:23:35,910 --> 00:23:39,450 which is very reminiscent of, you know, patient choice theory. 240 00:23:39,450 --> 00:23:46,530 This idea, you know, many other interventions were underpinned by this idea of promoting patient choice. 241 00:23:46,530 --> 00:23:51,030 And then there were other also quality data initiatives that were also underpinned by this idea, 242 00:23:51,030 --> 00:23:55,080 such as mortality report cause in the U.S. patient experience data, 243 00:23:55,080 --> 00:24:01,020 as well various different forms of hospital performance data and also the My NHS website, 244 00:24:01,020 --> 00:24:05,970 which was also set up with the intent of providing patients with with these data. 245 00:24:05,970 --> 00:24:11,970 Similarly, this idea of proms will enable commissioners to hold providers to account again. 246 00:24:11,970 --> 00:24:15,990 There were similar. The other interventions that share this idea. 247 00:24:15,990 --> 00:24:24,300 So this idea of accountability, public disclosure. You know, we can see in secrecy inspections, quality accounts that hospitals have to produce. 248 00:24:24,300 --> 00:24:27,090 And again, my NHS website. 249 00:24:27,090 --> 00:24:33,840 And then this idea of this intrinsic motivation that clinicians want to improve their practise and learn from each other and 250 00:24:33,840 --> 00:24:40,290 they will take action if they perceive that they are better or worse than their colleagues again shared ideas with benchmarking, 251 00:24:40,290 --> 00:24:41,250 audit and feedback. 252 00:24:41,250 --> 00:24:52,770 Ideas and similar interventions also share these ideas, such as clinical audits as the NHS Benchmarking Club so far from there being, 253 00:24:52,770 --> 00:24:59,670 you know, problems being this sort of singular, unique intervention that had a particular set of programme theories. 254 00:24:59,670 --> 00:25:07,590 When we looked at that, we actually saw that actually there's lots of other interventions using quality data that share these similar ideas. 255 00:25:07,590 --> 00:25:13,590 And so we could draw on the evidence in our review to test our theories. 256 00:25:13,590 --> 00:25:18,030 And also the other thing that we we explored was, OK, well, what are the intended? 257 00:25:18,030 --> 00:25:24,360 And I'm sure Jeff has mentioned this. That context shapes the ways in which interventions are implemented, 258 00:25:24,360 --> 00:25:29,160 how people respond, and sometimes interventions have a set of unintended consequences. 259 00:25:29,160 --> 00:25:34,140 But they also might have a set of unintended consequences, and we could draw on this literature, 260 00:25:34,140 --> 00:25:38,180 this knowledge to to help us think about, well, what might we look? 261 00:25:38,180 --> 00:25:42,350 For when we're synthesising this evidence, what might we expect to find what's going on? 262 00:25:42,350 --> 00:25:45,710 So in terms of the intended consequence of the patient choice idea, 263 00:25:45,710 --> 00:25:51,740 the idea was that patients would use these data to choose high performing providers and consequently the 264 00:25:51,740 --> 00:25:56,600 lower performing providers might respond by improving their care in order to attract more patients. 265 00:25:56,600 --> 00:26:01,250 That's how it was intended to work. But actually, one of the you know, it can always go wrong. 266 00:26:01,250 --> 00:26:06,170 One of the unintended consequences might be that patients are not even aware of these data, 267 00:26:06,170 --> 00:26:10,910 or that clinicians might refuse to treat sicker patients because they don't want those patients 268 00:26:10,910 --> 00:26:16,820 to bring down their average score because they don't improve as much in terms of accountability, 269 00:26:16,820 --> 00:26:20,300 stakeholders might impose sanctions or offer rewards. 270 00:26:20,300 --> 00:26:28,670 And this in turn was expected to to, you know, stimulate providers to either maintain the good, good care that they provided or improve it. 271 00:26:28,670 --> 00:26:40,040 However, you know, many commentators talked about concerns that actually, if you focus, you know, if you if you focus on particular indicators, 272 00:26:40,040 --> 00:26:49,100 this might actually result in what's known as tunnel vision, and that is exclusive focus on what is measured to the exclusion of other areas of care. 273 00:26:49,100 --> 00:26:53,570 All providers might gain the data in terms of benchmarking providers. 274 00:26:53,570 --> 00:27:00,260 You know, the idea was compiled. This will compare themselves to their peers and take steps to improve the buffalo average. 275 00:27:00,260 --> 00:27:07,580 But actually one of the understand the consequences might be that providers don't even understand the data, or they might dismiss or ignore it. 276 00:27:07,580 --> 00:27:12,410 So there were all these different possible outcomes, and I think that's another thing that that, you know, 277 00:27:12,410 --> 00:27:17,600 working with Ray was always keen to emphasise is that interventions don't just produce one outcome. 278 00:27:17,600 --> 00:27:24,560 They produce a whole range of different possible outcomes. Some of these are intended, some of them are unintended, 279 00:27:24,560 --> 00:27:31,130 and contexts often has a big role in shaping whether the intervention produces the intended or the unintended ones. 280 00:27:31,130 --> 00:27:36,200 So we also tried to think, Well, what are the kind of, you know, from the literature? What are they also? 281 00:27:36,200 --> 00:27:42,170 What are the important contextual features that might play a role? And this suggested that whether there are any, you know, 282 00:27:42,170 --> 00:27:50,540 existing concurrent incentives or sanctions associated with the the intervention itself and the credibility of the data, 283 00:27:50,540 --> 00:27:54,890 whether whether providers take the data seriously, which raised the question of, well, 284 00:27:54,890 --> 00:28:02,240 what features might support to constrain this and also the action ability of the data, how how easy is it that providers to act on it? 285 00:28:02,240 --> 00:28:07,640 Let me see. So these are some of the examples. As you can see, I'm not going to go into these in great detail, 286 00:28:07,640 --> 00:28:16,730 but they give you a flavour of we drew on evidence from many different kinds of interventions that shared the same underlying programme theories, 287 00:28:16,730 --> 00:28:24,290 from clinical audits to the cardiac surgery reports in the U.S. audits a clinical audit patient experience, 288 00:28:24,290 --> 00:28:28,490 plus some patients, some older indicators that were used in the UK. 289 00:28:28,490 --> 00:28:32,690 And for those of you that like PRISMA diagrams, here's a present at what what. 290 00:28:32,690 --> 00:28:41,060 The one thing I want you to take from this is how messy is compared to, say, a PRISMA diagram of a traditional systematic review. 291 00:28:41,060 --> 00:28:44,840 You can see we did lots of different searches. There was lots of ins and outs. 292 00:28:44,840 --> 00:28:51,500 There was large numbers of screening involved and basically it was a very messy and complex process. 293 00:28:51,500 --> 00:28:56,930 So it's not as simple that you do one search. You have a set of criteria, you select them. 294 00:28:56,930 --> 00:28:58,730 It was much more iterative than that. 295 00:28:58,730 --> 00:29:06,200 And I guess if you take one thing from this rather messy and difficult to read slide is that it was a messy process. 296 00:29:06,200 --> 00:29:08,030 It also took a little bit about context. 297 00:29:08,030 --> 00:29:14,300 I've mentioned this a little bit as well, and this was a really important sort of thing that helped us to think about it. 298 00:29:14,300 --> 00:29:20,450 And this is the idea that interventions might work differently in different contexts that you implement them in different contexts, 299 00:29:20,450 --> 00:29:24,680 you end up with different outcomes. And I suppose the context, you know, 300 00:29:24,680 --> 00:29:31,550 I and I recently wrote a paper on context where we really thought about that inspired by the work that we did on the remix project. 301 00:29:31,550 --> 00:29:37,130 And I guess part of context, might you know for this project, we we interpret it quite broadly and thought about, 302 00:29:37,130 --> 00:29:43,260 well, also what about the intervention features in terms of how you know, the intervention was implemented? 303 00:29:43,260 --> 00:29:47,600 Is that part of the intervention resources or is it part of context? It might be both. 304 00:29:47,600 --> 00:29:52,100 And but it's this idea that context shapes the way that the intervention works, 305 00:29:52,100 --> 00:29:57,950 but also this idea that context doesn't necessarily come in neat little packages. 306 00:29:57,950 --> 00:30:05,270 You know, when, when, when programmes are implemented, they're implemented into messy into messy contexts, 307 00:30:05,270 --> 00:30:11,960 into messy hospitals, where there's lots of other stuff going on that having to compete with all sorts of different things. 308 00:30:11,960 --> 00:30:19,970 And it was this idea that, you know, as you get more contextual features involved the outcomes that you might find a much more complex. 309 00:30:19,970 --> 00:30:27,140 And so we tried to get our heads around what the characteristics of the different interventions that we 310 00:30:27,140 --> 00:30:33,050 were looking at in terms of context and looked at the different outcome patterns that we found there. 311 00:30:33,050 --> 00:30:37,970 So to go back to all of the different evidence, all of these different interventions. 312 00:30:37,970 --> 00:30:43,040 She had different characteristics that we then tried to map and think about how 313 00:30:43,040 --> 00:30:46,850 they related to the kind of contextual features that we might be important. 314 00:30:46,850 --> 00:30:55,940 And then we we sort of grouped the studies relating to each particular intervention to find out, well, how do people respond what you know, 315 00:30:55,940 --> 00:31:06,170 we looked at qualitative studies about surgeons opinions or surgeon's ideas and feelings about the about the intervention, 316 00:31:06,170 --> 00:31:11,240 but also some of the quantitative studies looking at the outcome data to say, Well, did things actually change? 317 00:31:11,240 --> 00:31:15,380 Did outcomes improve? So just to give you an idea about how we did this, 318 00:31:15,380 --> 00:31:26,360 we set up some propositions in relation to our programme theories to try and understand how different contextual features might make a difference. 319 00:31:26,360 --> 00:31:31,880 And you can see that we drew on. As I mentioned before, what we were testing was the programme theory, 320 00:31:31,880 --> 00:31:38,210 so we drew on evidence from a whole range of these different interventions that shared the same underlying programme theories. 321 00:31:38,210 --> 00:31:44,960 But we looked at the different contextual features. So a key important one was whether the data are public and reported. 322 00:31:44,960 --> 00:31:49,130 Do does the public have access to them or not? You know, 323 00:31:49,130 --> 00:31:55,580 it was reminiscent of kind of this naming and shaming idea in terms of whether this idea of 324 00:31:55,580 --> 00:32:01,280 people were was supposed to respond because they've been outed as a badge poor provider. 325 00:32:01,280 --> 00:32:05,600 And also whether they had any sanctions or incentives attached to them. 326 00:32:05,600 --> 00:32:14,690 And we looked at a number of different interventions that the crack crack indicators were something that was implemented in Scotland a long time ago. 327 00:32:14,690 --> 00:32:19,280 They were not publicly reported. There were no sanctions attached to them. 328 00:32:19,280 --> 00:32:24,620 When we looked at the data and when we looked at the studies, what we found was that clinicians just ignored the data. 329 00:32:24,620 --> 00:32:28,430 There was absolutely no change to patient care. They had no impact whatsoever. 330 00:32:28,430 --> 00:32:35,330 Similarly, you know, there was a couple of studies looking at these hypothetical indicators based on national survey frameworks, 331 00:32:35,330 --> 00:32:39,810 similar outcome percent, they were not public reported there were no sanctions. 332 00:32:39,810 --> 00:32:46,730 And what happened was that the data was ignored. There was absolutely no changes to patient care, so it made no difference to patient care. 333 00:32:46,730 --> 00:32:50,600 We looked at star ratings. These were publicly reported. 334 00:32:50,600 --> 00:33:00,410 These were something that was, I think, around in the late 90s, early 2000s associated with the National Service Framework. 335 00:33:00,410 --> 00:33:04,560 And we looked at studies that was carried out by Russell Manian and colleagues. 336 00:33:04,560 --> 00:33:08,390 And what happened here was that the providers just, you know, 337 00:33:08,390 --> 00:33:13,610 if they were a poor outlier or a poor provider, they dismissed the credibility of the data. 338 00:33:13,610 --> 00:33:18,350 They said, We don't believe that we're not, you know, these data do not summarise. 339 00:33:18,350 --> 00:33:25,370 They do not represent our hospital. And there was some evidence that what happened is that the providers focussed on, you know, 340 00:33:25,370 --> 00:33:36,050 changing care in relation to particular indicators or changing care so that they could tick the boxes rather than actually improving the care. 341 00:33:36,050 --> 00:33:42,200 Similarly, we looked at the quality and outcomes framework and found a similar pattern in terms of, 342 00:33:42,200 --> 00:33:44,570 you know, the public reports they were there were sanctions, 343 00:33:44,570 --> 00:33:51,800 but again, they they tended to lead to tunnel vision such that the data, the indicators that were measured improved. 344 00:33:51,800 --> 00:33:54,770 But but the other ones didn't. In general, the care didn't. 345 00:33:54,770 --> 00:34:04,970 So this begins to give some idea about, you know, some of the patterns in terms of some of the contextual features and some of the findings. 346 00:34:04,970 --> 00:34:13,490 And then we added, we added another layer of context. So if you think back to the the the slide I had with the contextual features, you know, 347 00:34:13,490 --> 00:34:17,660 in addition to whether the data were public reported and sanctions were received, 348 00:34:17,660 --> 00:34:19,850 we were also interested in, well, you know, 349 00:34:19,850 --> 00:34:26,000 one of the things that seemed to be in was whether providers trusted the credibility, whether they found the data credible. 350 00:34:26,000 --> 00:34:31,850 This begs the question of what influences credibility in some of our reading suggested that one of the things that 351 00:34:31,850 --> 00:34:39,050 might influence credibility was whether clinicians were involved in setting up the measurement system or not, 352 00:34:39,050 --> 00:34:42,740 and also whether the data was the quality of the data. 353 00:34:42,740 --> 00:34:49,310 So an indicator of that quality we took to be whether the data were collected from patient notes, you know, 354 00:34:49,310 --> 00:34:54,920 in terms of they accurately reflected patient activity or whether they would gain from billing data. 355 00:34:54,920 --> 00:34:57,320 And again, we looked at a number of different initiatives, 356 00:34:57,320 --> 00:35:07,400 most at some of these that the top reports were a mortality report cards in cause yak surgery that were implemented in California. 357 00:35:07,400 --> 00:35:12,430 A similar scheme was implemented in New York, but it had some important differences. 358 00:35:12,430 --> 00:35:18,080 As you can see, they were both publicly reported and they had sanctions about them. 359 00:35:18,080 --> 00:35:21,080 But in contrast to the California one, 360 00:35:21,080 --> 00:35:29,360 the New York one was had a lot of clinician involvement and also it was based on patient data, not on billing data. 361 00:35:29,360 --> 00:35:34,070 And we found a slightly different outcome in response to those the reports. 362 00:35:34,070 --> 00:35:37,670 The the there were fewer quality improvement initiatives. 363 00:35:37,670 --> 00:35:44,510 The shouted on those and clinicians tended to dismiss the credibility of the data, whereas in contrast, the New York one, 364 00:35:44,510 --> 00:35:50,120 there was much greater acceptance of the data, but still not that many quality improvement initiatives. 365 00:35:50,120 --> 00:36:00,380 The gaokao is like the American version of our quality commission, and they had a number of incentives that they wanted to implement again. 366 00:36:00,380 --> 00:36:03,980 But they were not clinician led. They were based on billing data and again, 367 00:36:03,980 --> 00:36:11,600 a similar outcome to two of the forms of a patient report data found previously where you just had incentives, 368 00:36:11,600 --> 00:36:19,880 a public report, it tended to leave to just focus on the indicators and not thinking about the more global things about patient care. 369 00:36:19,880 --> 00:36:25,760 And again, similarly with the GP patient experience data which was implemented in England. 370 00:36:25,760 --> 00:36:30,440 What we found was that providers tended to dismiss the credibility of the data. 371 00:36:30,440 --> 00:36:39,500 They didn't believe the credibility. However, we also found a small number of studies that were exploring quality improvement, 372 00:36:39,500 --> 00:36:43,070 sort of quality collaboratives that were set up by clinicians themselves. 373 00:36:43,070 --> 00:36:47,060 They had a lot of clinician involvement from the clinicians. 374 00:36:47,060 --> 00:36:52,100 And what we found there was that that data was accepted by the clinicians and 375 00:36:52,100 --> 00:36:55,730 there was some quality improvement initiatives initiated on the back of that. 376 00:36:55,730 --> 00:37:03,350 So how do we make sense of this? Looking across all those outcomes patterns and looking at those contexts, what we what we found was that, 377 00:37:03,350 --> 00:37:08,330 you know, externally mandated data was perceived as being driven by political motives. 378 00:37:08,330 --> 00:37:14,300 The data was questioned because it was perceived as measuring what's important to clinicians and regulator. 379 00:37:14,300 --> 00:37:24,350 Not sorry politicians and regulators, not the clinicians. So the clinicians didn't trust it, and they didn't tend to respond to it. 380 00:37:24,350 --> 00:37:27,980 The public reporting of these data, especially if there were sanctions, 381 00:37:27,980 --> 00:37:31,760 you know it could lead to tunnel vision or at worst gaming to protect providers, 382 00:37:31,760 --> 00:37:38,180 represent reputation in the eyes of regulators or other responsible bodies. 383 00:37:38,180 --> 00:37:46,580 And I think what we found from looking at the studies was that externally mandated didn't always enable providers to identify the causes of poor care, 384 00:37:46,580 --> 00:37:50,600 so it might signal that they were outliers. But it didn't always tell the why. 385 00:37:50,600 --> 00:37:54,260 So it was very difficult for them to act on it. 386 00:37:54,260 --> 00:38:03,990 In contrast, internally collected data that was either clinician led or professionally led was much more accepted by clinicians. 387 00:38:03,990 --> 00:38:10,880 They they perceived it as a as being based on a desire to improve patient care rather than just ticking boxes. 388 00:38:10,880 --> 00:38:19,340 And also because they had a say in specifying the indicators it was perceived it was felt to be measuring what was important to clinicians. 389 00:38:19,340 --> 00:38:28,610 So it has much more credibility. And because they had a stake in specifying the indicators, they found it was much harder to dismiss or ignore. 390 00:38:28,610 --> 00:38:30,860 So there's much more pressure to act on it. 391 00:38:30,860 --> 00:38:40,700 And they were motivated to respond to protect their reputation in the eyes of their peers, which was a very an interesting motivator. 392 00:38:40,700 --> 00:38:47,270 And they also found that actually this internally collected data was really useful in telling them what was going on. 393 00:38:47,270 --> 00:38:52,880 So obviously, externally mandated indicators wouldn't tell them what was going on, 394 00:38:52,880 --> 00:39:01,550 whereas this internal data that was often generated by clinicians was able to much better pinpoint the causes of poor care. 395 00:39:01,550 --> 00:39:07,550 However, it still depended on the resources and capacity, collect the data and then to act on it. 396 00:39:07,550 --> 00:39:15,920 So it wasn't the end of the story. But what this enabled us to do was identify some sort of x CMO patterns, I guess, 397 00:39:15,920 --> 00:39:20,960 which explains these patterns that we observed in terms of contextual features about 398 00:39:20,960 --> 00:39:30,590 whether the use of quality data did motivate or lead to clinicians improving care or not. 399 00:39:30,590 --> 00:39:34,910 And so, well, how does this relate to problems we can then sort of in a way, 400 00:39:34,910 --> 00:39:47,450 go back down the ladder of abstraction and think about patient reported data in terms of these features that we found and, you know, problems. 401 00:39:47,450 --> 00:39:54,830 Yes, they are in theory that publicly reported. Although you often have to look quite hard to find data, they're not, you know, 402 00:39:54,830 --> 00:39:58,910 they are available on a database, but it's whether patients know how to find them. 403 00:39:58,910 --> 00:40:02,870 Yes, there are sometimes incentives attached to them. 404 00:40:02,870 --> 00:40:09,440 There wasn't a great deal of certainly not in the English National Promes programme, although the National College of Surgeons were involved in it. 405 00:40:09,440 --> 00:40:14,780 Local clinicians? Not necessarily. And the data are placed on subjective report. 406 00:40:14,780 --> 00:40:22,000 And when we looked at the evidence that the few studies, as you recall, there was only about four studies at the time. 407 00:40:22,000 --> 00:40:29,270 And you know, we found that actually evidence that providers tended to dismiss the quality of the data, 408 00:40:29,270 --> 00:40:32,720 rather not necessarily because it was based on billing data, 409 00:40:32,720 --> 00:40:37,310 but this time because it was based on patients reporting and they were a little bit sceptical. 410 00:40:37,310 --> 00:40:44,510 A. Outs her patients reports and their own opinions about the improvements that patients might experience in surgery. 411 00:40:44,510 --> 00:40:49,340 And there is very little evidence that certainly in the early days that this had resulted, 412 00:40:49,340 --> 00:40:55,730 that the promise programme has actually resulted in an improvement in any change in any patient outcomes. 413 00:40:55,730 --> 00:41:03,380 So to go back and this is my sort of last slide really to answer the question of how can you do a really synthesis when there isn't any evidence? 414 00:41:03,380 --> 00:41:09,260 The answer is that in a real synthesis, the unit of analysis is the programme theory, not the intervention itself. 415 00:41:09,260 --> 00:41:14,360 So you can test your programme face against evidence of interventions that share the same programme theory. 416 00:41:14,360 --> 00:41:22,850 And as you can see in this example, we tested our theories in relation to evidence from the range of interventions, 417 00:41:22,850 --> 00:41:31,190 clinical audits, patient experience, data mortality, report cards that shared the same underlying programme theories. 418 00:41:31,190 --> 00:41:40,500 And we came up with a number of CMO configurations to explain the patterns that we observed in the data and quantitative studies. 419 00:41:40,500 --> 00:41:42,740 Really useful for looking at these contexts. 420 00:41:42,740 --> 00:41:49,310 Outcome patterns, but also qualitative studies are useful for thinking about the mechanisms and how people respond. 421 00:41:49,310 --> 00:41:50,870 So putting all of that together, 422 00:41:50,870 --> 00:41:58,970 it's like doing a big jigsaw puzzle because you don't often find a perfect test of your CMO configuration in any particular study. 423 00:41:58,970 --> 00:42:05,330 But one of the things that we did and was assembling the data in the way that we did, you know, 424 00:42:05,330 --> 00:42:12,470 in terms of those tables where we had contextual features in the findings that enabled us to think 425 00:42:12,470 --> 00:42:18,920 about and compare and contrast and come up with some explanations to explain those theories. 426 00:42:18,920 --> 00:42:20,810 And if you want to read any more information, 427 00:42:20,810 --> 00:42:27,860 we've we've published a few studies and there's also a Nature Journals library report, but that's very long. 428 00:42:27,860 --> 00:42:32,930 So you have to have had a good cup of tea and a chocolate biscuit if you want to read it. 429 00:42:32,930 --> 00:42:34,241 But thank you. That's the end of my talk.