1 00:00:00,060 --> 00:00:05,640 Welcome, everybody, to this evening's talk of mixed methods research in the real world. 2 00:00:05,940 --> 00:00:09,540 I'm delighted to welcome a former colleague and friend, 3 00:00:09,540 --> 00:00:15,750 Dr. Cathy Pollard from University of the West of England in Bristol, who going to to to do this talk. 4 00:00:16,470 --> 00:00:23,640 She's going to be sort of talking for about 40 to 40 to 45 minutes and then we'll have a little bit of time at the end. 5 00:00:23,640 --> 00:00:27,750 The question so if you can say your questions up until the end, that would be great. 6 00:00:28,920 --> 00:00:34,440 Cathy Okay. Well, well, thank you very much for coming along here this evening. 7 00:00:34,440 --> 00:00:37,890 And Margaret, thank you for inviting me to come here. I'm delighted to be here. 8 00:00:38,560 --> 00:00:45,360 So as you will see, my title, I think says quite a lot about what I think about mixed methods research. 9 00:00:46,680 --> 00:00:53,940 So for those of you who are wondering about it, I've been working in research, 10 00:00:54,870 --> 00:01:00,780 mixed methods and both quantitative and qualitative now for about 20 years. 11 00:01:01,200 --> 00:01:08,339 And I found that the assumptions often, particularly on the part of students often, is that research is this nice, 12 00:01:08,340 --> 00:01:16,770 neat, tidy process, because of course we produce nice, neat, tidy looking findings or results. 13 00:01:18,420 --> 00:01:22,740 And of course, that that may be true for lab based work. 14 00:01:23,310 --> 00:01:26,580 It's certainly not true for social research. 15 00:01:26,970 --> 00:01:31,320 And all my work has been in the field of health and social care. 16 00:01:31,410 --> 00:01:38,490 So most health and social care with lots of health and social care research actually sort of qualifies as social research. 17 00:01:39,330 --> 00:01:42,630 And of course, I mean, if one just stops to think about it for a second, 18 00:01:43,050 --> 00:01:50,340 the reason one of the two main reasons I think for why research, social research is so messy. 19 00:01:51,300 --> 00:01:59,970 And the one is just the vagaries of human nature, where we're always you know, it's people who are actually conducting research. 20 00:02:00,240 --> 00:02:07,860 It's people who are participating in research. And we like to think of ourselves as rational beings. 21 00:02:07,860 --> 00:02:10,960 But frankly, I don't think that's the case at all. 22 00:02:10,980 --> 00:02:17,879 We certainly have some rational faculties, but a lot of our behaviour is driven by impulses that are very, 23 00:02:17,880 --> 00:02:26,430 very far from being rational and that situation doesn't suddenly get held in abeyance because we are involved in a research project from either end. 24 00:02:26,820 --> 00:02:35,850 If anything, once the sort of pressures, those sort of impulses can can be highlighted or emphasised. 25 00:02:36,690 --> 00:02:46,440 So there's that aspect of it. The other aspect is actually our external environment, because of course, we're going out into the world to do research. 26 00:02:47,220 --> 00:02:52,200 We are actually having to travel to go to places and I'll just say leaves on the line. 27 00:02:52,480 --> 00:02:58,260 You know, you just never know when something is going to happen that's going to literally derail the process. 28 00:02:58,980 --> 00:03:07,980 So we're always, always having to deal with the possibility of error when we're doing social research because it's a very complex process. 29 00:03:09,750 --> 00:03:23,310 When you start thinking about doing mixed methods as part of social research, you actually then increase the possibilities of messiness exponentially. 30 00:03:25,260 --> 00:03:31,830 So I suppose the question then becomes if that is the problem is why do mixed methods research? 31 00:03:35,100 --> 00:03:40,940 Well, as I said, I'd been working in the in the field now for for quite a long time. 32 00:03:40,940 --> 00:03:50,250 And I've been working in health and social care generally by the clinically or in as a researcher for about 25 years. 33 00:03:50,940 --> 00:03:55,469 And I think the more I stay in it, the more I feel very strongly that if we are producing, 34 00:03:55,470 --> 00:04:02,920 if we're doing research, we actually need to produce results that are useful and can be practically implemented. 35 00:04:03,840 --> 00:04:10,530 And as far as I'm concerned, doing mixed methods research really aids us in that process. 36 00:04:10,530 --> 00:04:20,430 And so I think I sort of reveal myself very firmly as a pragmatist by actually sort of laying out my stall in this way. 37 00:04:21,870 --> 00:04:29,220 So there, you know, there's quite a lot of there's been a lot of theory written about mixed methods research. 38 00:04:29,670 --> 00:04:35,970 The different researchers have got different ideas and attitudes towards mixed methods research. 39 00:04:36,340 --> 00:04:42,240 You know, there is the idea that that's quantitative research and qualitative research are actually dichotomous. 40 00:04:42,780 --> 00:04:52,919 I think that's a pretty outdated stance. If you have a lot of people now writing about mixed methods, research, 41 00:04:52,920 --> 00:04:58,290 being on a continuum where you have qualitative research at one end and quantitative research at the other, 42 00:04:59,100 --> 00:05:05,910 and a lot of researchers talk about using mixed methods in terms of depth and breadth 43 00:05:06,510 --> 00:05:12,570 enough for me that that's that sort of chimes with my own feelings about it. 44 00:05:13,050 --> 00:05:22,379 And in fact, when I think about mixed methods research, I tend to sort of, if you like, visualise it almost as a cross in that you've got a broad. 45 00:05:22,380 --> 00:05:28,800 The quantitative research and quantitative approaches give you a nice broad picture of what's going on. 46 00:05:29,640 --> 00:05:32,760 So you can find out what's happening if you use. 47 00:05:32,760 --> 00:05:36,629 And the qualitative research for me is like the vertical arm where you're actually 48 00:05:36,630 --> 00:05:42,210 really being able to get a go into depth to find out why something's happening, 49 00:05:42,420 --> 00:05:48,090 to understand it. So if we're going to actually do come in this couple of seats up at the front, 50 00:05:49,470 --> 00:05:58,740 so if we are going to really understand and produce work that's actually useful in health and social care, 51 00:05:59,460 --> 00:06:04,410 I think it's somehow that intersection of the quantitative and qualitative, 52 00:06:04,410 --> 00:06:13,830 which really gives us a it's our best chance of getting a comprehensive overview about a topic and really understanding it. 53 00:06:15,570 --> 00:06:23,310 We still have to deal with the problem of messiness and particularly we need to think about how can we produce robust results. 54 00:06:23,400 --> 00:06:31,379 We want to be able to trust our findings because otherwise it's much more to go home and not bother to even get started. 55 00:06:31,380 --> 00:06:37,020 If you think you're going to produce findings that are not reliable, not trustworthy, not credible. 56 00:06:38,880 --> 00:06:42,660 So I'm going to we'll come back to that a little bit later. 57 00:06:42,930 --> 00:06:53,100 But what I do want to do is at this point is just talk through some of the processes involved in conducting this sort of research. 58 00:06:53,460 --> 00:07:02,640 And to do that, I'm going to draw on my own experiences of two fairly large mixed methods, projects or programs that I've actually been involved in. 59 00:07:07,980 --> 00:07:16,530 Right. So the first one I want to talk about is an evaluation of the interprofessional curriculum. 60 00:07:16,530 --> 00:07:21,840 And as you can see from the slide, it went from it ran from 2001 to 2008. 61 00:07:22,800 --> 00:07:27,600 And this was actually my my first post. My first research post was on this project. 62 00:07:27,600 --> 00:07:30,960 I was appointed as the research associate on the project. 63 00:07:31,440 --> 00:07:35,010 And there was it was funded by what was then. 64 00:07:35,820 --> 00:07:43,979 There was a point some of you will remember some of you weren't that the the the way that health services were organised. 65 00:07:43,980 --> 00:07:49,740 They were these workforce development confederations funded by the Department of Health in each region. 66 00:07:50,040 --> 00:08:00,630 And so this project was funded by the Avon and Work even Workforce and Avon Workforce Development Corporation, all Confederation. 67 00:08:01,830 --> 00:08:12,780 So that was the first one I want to talk about. The second one is one that we've just finished, as you can see, started in 2014, carried on to 2017. 68 00:08:13,320 --> 00:08:21,150 The data collection analysis stopped a year ago and it's been tweaking and amending the final report for the last year. 69 00:08:21,990 --> 00:08:27,300 And this was, again, an investigation of quality measurement in English community nursing, 70 00:08:27,660 --> 00:08:33,600 and that's quite a mouthful, so we shouldn't get too quick in. So that's how we refer to it is Quicken the Quicken Study, 71 00:08:33,930 --> 00:08:43,800 and that was funded by the National Institute of Health Research and that was a multi site, multi organisation study. 72 00:08:44,340 --> 00:08:47,399 So I'm going to talk about both of those. So firstly, 73 00:08:47,400 --> 00:08:52,200 what I want to do is just tell you a little bit more about each of them and then start 74 00:08:52,200 --> 00:08:57,720 thinking about some of the the issues that arose when we were actually doing these projects. 75 00:09:01,230 --> 00:09:03,470 So the U.S. interprofessional curriculum. 76 00:09:03,480 --> 00:09:14,430 Now, at the time that this curriculum was introduced, it was the idea of giving undergraduates the opportunity for interprofessional education, 77 00:09:14,430 --> 00:09:18,150 for learning together, for working together, which was brand new. 78 00:09:18,870 --> 00:09:23,010 It hadn't really been done very much at all. It was it was starting to come in. 79 00:09:23,310 --> 00:09:28,140 So a degree was one of the first universities to introduce this curriculum, 80 00:09:28,530 --> 00:09:33,900 right throughout the whole professional program for undergraduate students. 81 00:09:34,230 --> 00:09:36,000 But also, more importantly, 82 00:09:36,240 --> 00:09:44,819 it was there was a module in each year and those modules were compulsory and they were assessed so students could choose whether or not to do them. 83 00:09:44,820 --> 00:09:51,150 They had to do them and they couldn't progress or graduate unless they'd actually passed those modules. 84 00:09:52,410 --> 00:09:59,580 And as you said, at that point, we didn't have medical students actually in the university because they're educated at the University of Bristol. 85 00:10:00,210 --> 00:10:04,470 But as in a slightly later iteration of the curriculum, 86 00:10:04,590 --> 00:10:11,160 we had medical students coming over from Bristol to join us in some of our interprofessional work and interprofessional curricula. 87 00:10:11,370 --> 00:10:15,870 But the evaluation at the time didn't include medical students, 88 00:10:15,870 --> 00:10:24,480 but it had a lot of different of the health professionals, all on separate programs, educational programs. 89 00:10:27,930 --> 00:10:37,290 And in their wisdom, the faculty decided it would be a good idea to evaluate this new, new curriculum. 90 00:10:37,710 --> 00:10:39,690 So that got it sort of set up quite well. 91 00:10:39,960 --> 00:10:47,850 And these were the things we were interested really interesting in what the student attitudes towards learning and working into professionally, 92 00:10:47,850 --> 00:10:57,540 collaboratively. We were also looking at not just the attitudes, but what how did they find the experience of learning into professionally, 93 00:10:57,540 --> 00:11:00,180 both in academic settings and in placement settings. 94 00:11:00,510 --> 00:11:05,980 And finally, we were interested in what was happened to the staff because again, this was very new for staff. 95 00:11:06,000 --> 00:11:17,760 Staff were not used to having to work and facilitate students who actually were from a different professional discipline from themselves. 96 00:11:19,140 --> 00:11:31,350 So the whole thing was pretty brand new. So what we decided or what the design for the study was, was a really realist evaluation. 97 00:11:31,740 --> 00:11:39,300 So there was very much a focus on linking contexts, mechanisms and outcomes. 98 00:11:40,050 --> 00:11:46,860 And one of the key questions that one asks in realist evaluation is what works for whom and why? 99 00:11:47,040 --> 00:11:50,459 Because, I mean, it's fairly obvious that, you know, 100 00:11:50,460 --> 00:11:55,170 you can have the same situation and bring people into it and they'll get different things from it. 101 00:11:55,350 --> 00:12:05,350 So we were really trying to tease out, again, the complexities of the processes that students were actually being exposed to. 102 00:12:06,900 --> 00:12:16,380 So we designed a mixed methods research program, and it was concurrent, independent equal status program, and it incorporated five distinct studies. 103 00:12:21,420 --> 00:12:29,969 So it's mostly quantitative, although this study had some qualitative aspects to it, in fact was the serve. 104 00:12:29,970 --> 00:12:35,580 Our survey was a mixed methods, but it was it was very heavy on the qualitative and quantitative side. 105 00:12:36,450 --> 00:12:39,899 We wanted to find out about student attitudes and staff attitudes. 106 00:12:39,900 --> 00:12:50,700 So you can see over the course of the of the of the program, we can we collected, you know, to just about 2700 questionnaires. 107 00:12:51,390 --> 00:12:55,770 And please also do remember that this was way before online surveys. 108 00:12:55,890 --> 00:13:02,100 We had to do everything with paper copies. So which had its own interesting times. 109 00:13:03,630 --> 00:13:14,220 Of those 2700 rough, just over two, two and a half thousand of them were actually student surveys. 110 00:13:14,940 --> 00:13:19,509 I think we had about we only had about 60 or 70 staff questionnaires. 111 00:13:19,510 --> 00:13:27,990 So most of those were student. And what we did was we we had to do we had for administration. 112 00:13:28,000 --> 00:13:35,610 So as they came in the doors point in their second year when they qualified and a year out once they were in practice, 113 00:13:36,000 --> 00:13:39,750 practice for a year, we also did it. 114 00:13:40,110 --> 00:13:48,929 We had three qualitative studies looking at sorry for qualitative studies, looking at non participant observations. 115 00:13:48,930 --> 00:13:57,270 So we observed these interprofessional learning groups and we all went out and observed what was happening in the students on placement. 116 00:13:58,140 --> 00:14:00,810 We did interviews, loads of interviews with students, 117 00:14:01,050 --> 00:14:11,340 some interviews with staff in both placement areas and the academic staff and we interviewed did focus groups with students. 118 00:14:11,970 --> 00:14:16,140 The third year of the interprofessional module was delivered online. 119 00:14:16,680 --> 00:14:23,730 So we analysed discussion boards from the groups and then we also analysed some of the module evaluation forms. 120 00:14:24,030 --> 00:14:29,819 So as I'm sure you can appreciate, we had a huge amount of data both to collect and to deal with. 121 00:14:29,820 --> 00:14:33,270 And I'll talk about some of the issues around all that in a minute. 122 00:14:33,270 --> 00:14:40,150 Once I've just told you a bit about the other project. So the quick and steady. 123 00:14:45,190 --> 00:14:53,020 So as it says there, the aim was to determine, just find out how is the quality of care actually measured in community nursing services. 124 00:14:54,220 --> 00:14:58,940 And the reason why this was of interest and why got funded is a couple of things are happening. 125 00:14:59,380 --> 00:15:04,480 You may I don't know how many of you have got any knowledge of the community nursing services, 126 00:15:04,930 --> 00:15:12,129 but the community services generally are being used more and more and more that the landscape has changed dramatically 127 00:15:12,130 --> 00:15:17,620 over the last ten years or so and with a drive to get people out of hospital and not have people in hospitals. 128 00:15:18,880 --> 00:15:27,250 The community nurses are having to provide extremely skilled care to people who actually 129 00:15:27,250 --> 00:15:33,670 have very complex medical needs and who would previously have been cared for in hospital, 130 00:15:33,670 --> 00:15:38,650 not at home. So that landscape has changed completely in terms of community nursing care. 131 00:15:39,370 --> 00:15:51,310 And while it's actually quite easy in a ward or residential setting to see what's happening in terms of quality of care, 132 00:15:51,670 --> 00:15:57,040 because things are happening and other people are watching in the domiciliary setting, 133 00:15:57,040 --> 00:16:02,950 the only people who know what the quality of care is like is the person receiving it and the person providing it. 134 00:16:04,450 --> 00:16:14,800 So obviously there is a need to make sure that there are robust measures for measuring the quality of community uptake of community nursing. 135 00:16:15,160 --> 00:16:23,470 Now in again, I'll just digress slightly, but in health care generally in the UK and elsewhere, 136 00:16:24,550 --> 00:16:31,750 a very common way of measuring quality of the health care being delivered is by using quality indicators. 137 00:16:31,930 --> 00:16:39,700 That is simply just a measure of care quality used to judge how good the services and the sort of indicators. 138 00:16:40,380 --> 00:16:46,959 An example of an indicator is something like, you know, staff should be trained, should be updated regularly about medicine, 139 00:16:46,960 --> 00:16:54,400 administration or every older person should be assessed for the risk of a fall risk of falling. 140 00:16:54,610 --> 00:16:58,450 So you can see the very, very specific statements. 141 00:16:58,450 --> 00:17:05,140 Usually these quality indicators, which means that organisations providing care have rafts of them. 142 00:17:05,740 --> 00:17:15,700 Some of them are nationally mandated, but a lot of the of the indicators that actually are news are negotiated between clinical commissioning groups, 143 00:17:15,700 --> 00:17:21,250 CCGs and the organisations providing the nursing services. 144 00:17:23,080 --> 00:17:31,240 And obviously this applies to all the services that this is commissioned from, from any, any health care providers, acute or community based. 145 00:17:32,980 --> 00:17:39,850 Anyway, for these reasons, what we were interested in was seeing is really finding out about the selection, 146 00:17:40,030 --> 00:17:43,870 how acute care quality indicators chosen and why, 147 00:17:44,710 --> 00:17:51,880 how are they applied and why, what happens then and how useful they are both for, 148 00:17:52,360 --> 00:17:57,670 you know, commissioners, managers, frontline nurses and patients and carers. 149 00:17:57,880 --> 00:18:02,980 So how useful are these indicators? So we were looking at the ones that are currently in use for community nursing. 150 00:18:04,600 --> 00:18:11,350 So it was an exploratory design. Again, it was a mixed method research project, sequential, interactive. 151 00:18:11,560 --> 00:18:19,200 But this one really had a qualitative bias as opposed to the the curriculum, which was more an equal status program. 152 00:18:21,460 --> 00:18:29,680 So we did a national survey to find out what quality indicators were being applied to community nursing. 153 00:18:29,680 --> 00:18:30,970 At 2014, 154 00:18:31,150 --> 00:18:43,000 we contacted every every CCG in the country and sort of 211 at the time to find out what indicators they were using so that we could actually map. 155 00:18:46,090 --> 00:18:48,400 What was actually happening across the country. 156 00:18:48,790 --> 00:18:56,110 And then also one of the other reasons for doing the survey was to provide us with a pool from which we could select. 157 00:18:58,030 --> 00:19:01,330 Case sites because we wanted to do in-depth case sites as well. 158 00:19:01,540 --> 00:19:05,050 And we had five case sites in England. 159 00:19:05,350 --> 00:19:13,930 And each case site was a clinical commissioning group and a and the provider, the community nursing services. 160 00:19:14,020 --> 00:19:20,230 So they would sort of dyads. So that actually meant that we were dealing with ten different organisations in our case sites. 161 00:19:21,520 --> 00:19:26,320 We did individual interviews or group interviews and with Right. 162 00:19:26,350 --> 00:19:27,880 All the stakeholders involved. 163 00:19:27,890 --> 00:19:37,990 So again, that was NHS England representatives, commissioners, provider managers, nursing team leaders, frontline nurses, patients, carers. 164 00:19:38,230 --> 00:19:42,850 We managed to get a sort of pretty good representation across those groups. 165 00:19:43,690 --> 00:19:48,249 We also sat in on 25 organisational meetings, some of them which were huge, 166 00:19:48,250 --> 00:19:52,840 as you can see by the the number there of the people who actually attended. 167 00:19:53,380 --> 00:19:58,330 And so we were interested in any, any meetings where quality was on the agenda. 168 00:20:00,580 --> 00:20:05,590 One of our we had a couple of registered nurses working on the project, including the PR. 169 00:20:05,590 --> 00:20:14,620 I was a registered nurse and she actually went out and did and shadowed community nurses to see what was actually happening out on the ground. 170 00:20:15,370 --> 00:20:20,470 And then also we were able to scrutinise some of the documentary data that was made available to us. 171 00:20:22,060 --> 00:20:29,740 So again, both these projects had a huge amount of, of, of data in them. 172 00:20:30,310 --> 00:20:39,640 The what I wouldn't say about the curriculum, the curriculum evaluation is that we mixed methods sort of all the way along, 173 00:20:39,760 --> 00:20:45,250 both in terms of data collection, data analysis and also in the interpretation. 174 00:20:46,000 --> 00:20:55,960 Whereas with the the quickened study, it was very much in the initial data collection and analysis phases that we that we mixed the methods. 175 00:20:59,690 --> 00:21:08,170 So Tony, in recruitment, I think that the the challenges of doing these sort of large studies where you're mixing methods is, 176 00:21:08,240 --> 00:21:14,060 is you have to take into account that actually you're going to run into problems, 177 00:21:14,240 --> 00:21:20,390 otherwise you are really going to run into problems if you haven't sort of tried to anticipate them a little beforehand. 178 00:21:24,860 --> 00:21:33,140 So with the IPA curriculum evaluation, given that it was an evaluation, it was incredibly important to have baseline data. 179 00:21:34,390 --> 00:21:41,990 Right. You can't evaluate the effect of something if you don't know what's actually the status quo before you actually start implementing it. 180 00:21:42,770 --> 00:21:47,629 So the way the and this is compounded, 181 00:21:47,630 --> 00:21:55,910 our difficulties around this were compounded by the fact that the students were on ten different timetables because they were ten different programs, 182 00:21:55,910 --> 00:22:01,370 they were ten different timetables. And as I said, this is my first job as a research associate. 183 00:22:01,370 --> 00:22:09,380 I think I started in the beginning of the September in the job and the we started data collection in the last week of September. 184 00:22:09,710 --> 00:22:21,770 And in my naivete, I just assumed that somewhere in the faculty there would be some sort of central information about all the students. 185 00:22:23,390 --> 00:22:36,230 Not so because this had been because of the the the history of how, you know, that areas like health care, education have come into higher education. 186 00:22:39,170 --> 00:22:51,950 You had absorbed courses from about ten different colleges, which meant that they all came in and they all had their way of doing things. 187 00:22:52,020 --> 00:22:56,090 They all had their separate admin staff and their separate admin processes. 188 00:22:56,240 --> 00:23:04,520 And although it was about ten years after that amalgamation, it hadn't actually then really moved much further in terms of centralising it. 189 00:23:04,970 --> 00:23:10,970 So the first thing I had to do was to try and go around and find out where these students were going to be, 190 00:23:11,840 --> 00:23:15,200 because we were absolutely bound by the student timetable. 191 00:23:15,200 --> 00:23:21,439 There was no slippage. We could not slip because if those students were in, they were end. 192 00:23:21,440 --> 00:23:27,140 And if they were up, they were out and there was nothing that could be changed to accommodate the fact that we were doing research. 193 00:23:28,430 --> 00:23:34,459 And again, this was made a little bit more difficult because the only time that actually suited all the 194 00:23:34,460 --> 00:23:42,650 programs for them to have a the first interprofessional module was the second week of the term, 195 00:23:44,390 --> 00:23:53,450 so we had to get them to complete questionnaires in the first week and we had to get everything organised and set up, 196 00:23:53,450 --> 00:24:00,320 including some some training for some of the people doing it to actually be able to get into the groups and start observing. 197 00:24:00,860 --> 00:24:08,989 So time pressure was, was enormous in that particular project we had in that first administration, 198 00:24:08,990 --> 00:24:16,969 we had over 850 questionnaires, two to I think there were about 900 students in the in that intake. 199 00:24:16,970 --> 00:24:20,570 And we got a very, very good response rate, which was brilliant. 200 00:24:21,080 --> 00:24:25,010 It's amazing what happens if you go to room when you go here, fill it in, please. 201 00:24:25,190 --> 00:24:30,200 And they get all right, especially in the first year, it was great. And we did have ethical approval. 202 00:24:30,200 --> 00:24:33,649 We'd gone to that through the Ethics Ethics Committee beforehand. 203 00:24:33,650 --> 00:24:40,100 So we did have ethical approval. But, you know, it was a real logistical exercise. 204 00:24:40,100 --> 00:24:50,450 And of course, the other thing was that we wanted to, you know, be able to do comparisons right the way through. 205 00:24:51,470 --> 00:24:56,350 And so we had to work out a way of linking individual questionnaires. 206 00:24:56,750 --> 00:25:05,300 So we needed to know, you know, if I've got this person's questionnaire at the beginning and this person's questionnaire at the end, do they marry up? 207 00:25:05,840 --> 00:25:10,340 So we have to work out a way of doing that, and that's a sort of story in itself. 208 00:25:10,340 --> 00:25:18,080 But we did manage. So as I said, they were timing was was quite, quite difficult in that particular project. 209 00:25:19,370 --> 00:25:22,700 And then of course, once they because we followed them into practice, 210 00:25:22,970 --> 00:25:28,580 there was a whole issue of trying to to track them once they left the university, 211 00:25:29,270 --> 00:25:33,799 because also we wanted to do interviews, we wanted to send them questionnaires. 212 00:25:33,800 --> 00:25:38,750 And so there was a whole issue around trying to to manage that in the timescales 213 00:25:38,900 --> 00:25:47,750 that that that you know that made sense quick in project the time pressures weren't 214 00:25:47,750 --> 00:25:53,120 quite the same but they were still there because with an NIH project you your 215 00:25:53,120 --> 00:25:57,290 milestones can slip a bit within the project but you've got to deliver on time. 216 00:25:57,290 --> 00:26:03,650 I think you can probably negotiate about a fortnight if you're lucky, if, if you really slip. 217 00:26:03,770 --> 00:26:07,010 But actually you have to deliver. 218 00:26:07,190 --> 00:26:15,890 You have to deliver on time. And we had to get our survey done because we needed the, the, 219 00:26:16,400 --> 00:26:22,370 we needed to have it analysed so that actually we would have a really good idea of what was happening. 220 00:26:22,760 --> 00:26:27,140 That. So that would inform the data collection in the case sites. 221 00:26:28,100 --> 00:26:32,020 And also said we needed this pool for sampling. So we had to get it done. 222 00:26:32,030 --> 00:26:36,140 And of course, the first major hurdle was actually just getting hold of CGS. 223 00:26:37,220 --> 00:26:40,760 There is a list of them on a website with contact details. 224 00:26:41,030 --> 00:26:46,610 But you phone the numbers and you say, I want to talk to somebody who knows something about quality measurements and a community nurse. 225 00:26:46,730 --> 00:26:55,300 And it sort of well, you know, you don't actually get much in the way of, you know, I think after about an it takes ages, 226 00:26:55,310 --> 00:27:01,129 you know, because then you do get a name and then you can never get hold of that person and they don't answer emails. 227 00:27:01,130 --> 00:27:08,360 And you know, all the normal problems you have about contacting people in organisations and trying to make sure you've got the right person. 228 00:27:08,810 --> 00:27:15,290 And we eventually wound up because a few CQ said to us, Please, which you just send us a Freedom of Information request. 229 00:27:17,960 --> 00:27:21,590 So we eventually took the decision, the project team took the decision. 230 00:27:21,590 --> 00:27:27,499 That's how we would approach it. There's some issues about because that's the public purse again. 231 00:27:27,500 --> 00:27:31,840 So there are issues about that. But it was it seemed to be that was our choice. 232 00:27:31,860 --> 00:27:36,140 That or just scrap the survey and we couldn't scrap the survey. So we did do that. 233 00:27:36,150 --> 00:27:41,510 We did. We got a reasonable I think we got about a 70% response rate. 234 00:27:41,510 --> 00:27:45,410 So we were quite happy with that and got a lot of useful information. 235 00:27:47,810 --> 00:27:53,750 Our next hurdle was the case site recruitment. Now, in theory, this is all supposed to be streamlined. 236 00:27:53,780 --> 00:28:01,340 Now this is very different. We we while we were doing the survey, we got ethical approval from an NHS ethics committee. 237 00:28:02,750 --> 00:28:10,580 But then we had to get into each organisation. The CC G's weren't too difficult to actually sort of varied. 238 00:28:11,930 --> 00:28:18,680 And although once we because our host CCG had given us we had letters of permission, etc., 239 00:28:19,010 --> 00:28:24,530 that theoretically should have actually just been acceptable everywhere, not the case at all. 240 00:28:24,800 --> 00:28:29,190 Every single organisation we want to. Wanted us to go through their processes. 241 00:28:29,600 --> 00:28:34,580 So we were jumping through one well crawling by that time through one hoop after another. 242 00:28:35,090 --> 00:28:41,450 And I think we'd allowed six months for recruitment and it actually took us over a year. 243 00:28:42,830 --> 00:28:44,120 I mean, we managed to get them. 244 00:28:44,360 --> 00:28:51,499 But of course it did have a knock on effect on the time we had then for actual data collection and particularly for data analysis. 245 00:28:51,500 --> 00:28:57,260 And I'll talk about that in a minute. So I just want to make the point that actually, if you're going to do these big studies, 246 00:28:57,530 --> 00:29:06,290 it's really important to have excellent administrative support or skills and people who don't appreciate. 247 00:29:08,080 --> 00:29:13,600 Good people supporting them in administrative capacities are idiots. 248 00:29:13,600 --> 00:29:20,230 As far as I'm concerned. You can't do any of this sort of stuff without good administrative support. 249 00:29:21,610 --> 00:29:27,249 And the other thing is, you know, you need to be flexible so that you can adapt because you can't assume you're going 250 00:29:27,250 --> 00:29:32,350 to be able to do your project exactly as you thought you were going to do it. 251 00:29:39,030 --> 00:29:46,140 So the other challenge is actually putting together research teams and not just putting them together, but actually working with them. 252 00:29:48,330 --> 00:29:52,860 It so happened just it's just coincidental and in fact only realised that when I was putting 253 00:29:52,860 --> 00:29:57,990 this together some time during this week that we had the same number of people in each team on, 254 00:29:58,040 --> 00:30:03,330 on each project. So we had 14 project team members on each study. 255 00:30:04,350 --> 00:30:09,719 Now the, the other team was all based. 256 00:30:09,720 --> 00:30:17,310 I mean, one strength of it is everybody won. One thing that made it easier in terms of messiness was that everybody was based in the same place. 257 00:30:18,660 --> 00:30:28,800 And also I was the only person who didn't sort of know quite a lot about the the the topic of interprofessional education when I started. 258 00:30:29,730 --> 00:30:37,590 So but we did have interesting things happening in that in that team because we obviously 259 00:30:37,590 --> 00:30:43,620 we had they were mostly all lecturers and they were they were coming from different health, 260 00:30:43,620 --> 00:30:47,190 professional backgrounds and social care professional backgrounds. 261 00:30:48,030 --> 00:30:54,569 And it was quite interesting sitting being a fly on the wall sometimes and watching how the interprofessional working, 262 00:30:54,570 --> 00:31:00,030 what's going on within the research team which considering that was our topic was was yeah, 263 00:31:00,120 --> 00:31:03,809 it was amusing at times to say and cause some problems as well. 264 00:31:03,810 --> 00:31:08,670 Sometimes that quick in team was very different. 265 00:31:09,210 --> 00:31:14,010 It was it was there were five organisations involved in this project. 266 00:31:14,280 --> 00:31:19,350 So as a twin NIH, our project, it had to be hosted by an NHS organisation. 267 00:31:19,590 --> 00:31:25,380 So Bristol CCG was the host and then Hillary was the lead academic institution. 268 00:31:25,800 --> 00:31:35,340 But we also had input and researchers from University of Bristol, London School of Hygiene and Tropical Medicine and the University of Manchester. 269 00:31:36,300 --> 00:31:44,240 So it was a team that was drawn together of people who some knew each other, some didn't. 270 00:31:44,250 --> 00:31:51,690 To start off with, again, people with very different clinical and professional backgrounds and from all over the country. 271 00:31:51,690 --> 00:31:55,979 And we had to you know, there were times when you just needed everybody to get together. 272 00:31:55,980 --> 00:32:02,640 You could do a lot by Skype and video conferencing, etc., but that wasn't always successful. 273 00:32:03,540 --> 00:32:08,819 So so we needed to do both really. So there was quite a, there were sort of, again, 274 00:32:08,820 --> 00:32:16,830 logistics involved in making sure that you can get everybody together and both those teams you want a proper a good team meeting, 275 00:32:16,920 --> 00:32:21,270 you know, just handling people's diaries is something else again. 276 00:32:23,190 --> 00:32:30,870 So what was important to both of them was to make sure that we had people with the requisite skills and expertise. 277 00:32:31,290 --> 00:32:34,410 Right. And obviously, once you're doing mixed methods, 278 00:32:35,550 --> 00:32:42,510 this this is an issue that you've really got to think about because not only do you have do you need people with the, 279 00:32:43,800 --> 00:32:46,770 you know, the required skills, 280 00:32:47,100 --> 00:32:55,230 but you also often need to do some sort of mediation and interpretation between people coming from different research paradigms. 281 00:32:55,770 --> 00:33:03,750 And that was certainly the case in both studies. I mean, I think we managed it fine, but it was certainly an issue that actually came up. 282 00:33:04,020 --> 00:33:10,530 So, you know, trying to explain to a medical colleague that actually doing qualitative research was a good idea. 283 00:33:11,070 --> 00:33:18,240 For example, trying to explain to a social work colleague, on the other hand, that actually we really needed the quantitative work. 284 00:33:18,630 --> 00:33:22,260 So again, that just added to everything we were doing. 285 00:33:23,850 --> 00:33:37,020 The other thing especially, I mean, as I said in the in the curriculum evaluation, everybody had most people hadn't had knowledge of the topic. 286 00:33:37,830 --> 00:33:42,690 They were all educators interested in interprofessional learning in the cricket team. 287 00:33:42,690 --> 00:33:48,210 We really needed people with skills in specific areas in terms of knowledge. 288 00:33:48,390 --> 00:33:51,450 So we obviously we needed people who knew something about community nursing. 289 00:33:51,780 --> 00:33:58,620 We needed people who knew something about quality management. And we need people who knew something about commissioning. 290 00:33:59,580 --> 00:34:05,190 But and the sort of dynamics of what we were doing was quite interesting too, 291 00:34:05,490 --> 00:34:09,930 because with, as I said, with these disparate people and these large teams, 292 00:34:10,950 --> 00:34:18,899 it's and going back to what I said right at the beginning about are not being rational beings all the time, you know, 293 00:34:18,900 --> 00:34:24,140 it's can be quite difficult managing a team of that size when you interpersonal issues 294 00:34:24,150 --> 00:34:29,880 that are professional issues into paradigmatic issues if there is such a word, 295 00:34:30,810 --> 00:34:36,600 you know, so a lot needs to be thought about and planned and I'm not always the best person I'll put my hand on. 296 00:34:36,650 --> 00:34:40,700 My heart and I lose my temper every now and then and start snapping at people. 297 00:34:40,700 --> 00:34:44,839 But actually that works immensely well. The first time I did it, I was really surprised. 298 00:34:44,840 --> 00:34:48,620 I thought, Oh, everybody went, Oh, yes, oh, it was wonderful to all of us. 299 00:34:48,740 --> 00:34:53,330 I must, you know, employ that that strategy more often. 300 00:34:53,630 --> 00:34:59,510 But it is something, you know, to actually you have to do because every decision and all the activities need to be negotiated, 301 00:34:59,840 --> 00:35:07,370 especially when we need it quick and project when you need to start thinking about timing and changing protocols slightly, 302 00:35:07,520 --> 00:35:12,170 it all had to be negotiated with a full project team. So that was yeah. 303 00:35:12,860 --> 00:35:19,570 Something to do. So of course, we still have to think about how do we. 304 00:35:19,660 --> 00:35:26,170 We've got some doing these big projects we're doing, you know, we're trying to deal with all the complexity that arises. 305 00:35:27,730 --> 00:35:29,020 How are we going to ensure robustness? 306 00:35:33,320 --> 00:35:43,160 So in that curriculum evaluation, it was very important that people collecting the data were not involved in delivering the program, the curriculum, 307 00:35:43,970 --> 00:35:45,720 because otherwise, you know, 308 00:35:45,770 --> 00:35:51,620 what good is the quantity of of of data that's collected from a student who doesn't want to say something to upset a lecturer? 309 00:35:52,400 --> 00:35:55,490 So we had to make that was something we really have to be careful about. 310 00:35:56,270 --> 00:35:59,780 We had to validate our questionnaire. There were a few that weren't. 311 00:36:00,770 --> 00:36:05,899 There were some similar questionnaires out there, but there weren't any that actually were quite right for us. 312 00:36:05,900 --> 00:36:10,760 So we had to design our own and they were full attitude scales and they each needed validating. 313 00:36:10,760 --> 00:36:17,300 So that again, that's a whole separate exercise. And I learnt about psychometric testing. 314 00:36:17,840 --> 00:36:21,260 I learnt a lot on the ground in that particular project. 315 00:36:24,080 --> 00:36:29,210 Obviously doing qualitative research is important to consider into research reliability. 316 00:36:29,220 --> 00:36:37,520 So we had a lot of collaboration and discussion about the instruments we were using for collecting data and also for analysing, 317 00:36:37,520 --> 00:36:40,610 coding and analysing and the themes coming out. 318 00:36:41,750 --> 00:36:51,170 We were able to triangulate our quantitative and qualitative data and that was really useful because it actually really allowed us to say, 319 00:36:51,170 --> 00:36:56,870 Well, this is what we're seeing in the attitudes. This is what we're getting from all the interviews and the and the qualitative data. 320 00:36:57,140 --> 00:37:03,140 So we can really as I said, it gave us a great sort of comprehensive picture of what was happening. 321 00:37:03,980 --> 00:37:12,049 And I think the the the if you like, the proof of the pudding is that we've actually between 2004 and 2012, 322 00:37:12,050 --> 00:37:17,210 we published 16 peer reviewed papers from that curriculum evaluation. 323 00:37:17,570 --> 00:37:20,840 So I know it's held in good, good repute. 324 00:37:20,840 --> 00:37:24,950 You know, I mean, I still get invitations to talk about it from various places. 325 00:37:24,950 --> 00:37:33,320 I was in Norway a couple of years ago. I was asked I get I get requests for that for a questionnaire to be used on a regular basis. 326 00:37:33,680 --> 00:37:44,210 So I think we managed to actually produce a robust research programme and in the quick study, 327 00:37:45,050 --> 00:37:51,560 robustness sort of was a different it had a different flavour of trying to do of trying to ensure robustness. 328 00:37:53,300 --> 00:37:56,390 The first thing we had to do was the survey data analysis. 329 00:37:56,390 --> 00:38:02,090 We really needed relevant the relevant expertise because what was happening when 330 00:38:02,090 --> 00:38:07,130 we asked people to send us their local course indicators for community nursing, 331 00:38:08,510 --> 00:38:14,570 a lot of the indicators that were sent were were not necessarily I think eventually actually, because people didn't know. 332 00:38:14,780 --> 00:38:20,599 We eventually just started asking people for just the community indicators because often people couldn't tease out. 333 00:38:20,600 --> 00:38:25,300 The people we were talking to couldn't tease out whether actually they applied to community nurses or not. 334 00:38:25,910 --> 00:38:33,440 And we got something like over 800 uniquely worded quality indicators to analyse through the survey. 335 00:38:34,040 --> 00:38:38,900 And while you could look at some of them and go, Yep, that's definitely for community nursing. 336 00:38:39,140 --> 00:38:42,230 Without specialist expertise, you couldn't always tell. 337 00:38:43,040 --> 00:38:51,830 So we had a one of our project team members was a community nurse manager, and the other one was a commission of community services. 338 00:38:52,130 --> 00:38:57,210 So we pulled them in to look at everything that we weren't sure about the rest of the researchers, 339 00:38:57,230 --> 00:39:00,920 because they were four of us from the team actually collecting data. 340 00:39:02,510 --> 00:39:07,460 But they also came in to help analyse the survey. So that was the first thing we really had to do. 341 00:39:08,900 --> 00:39:16,850 Again, they were we had to do the normal research reliability processes in terms of the qualitative data. 342 00:39:17,090 --> 00:39:21,050 But also, as I said, we had time constraints which we hadn't anticipated. 343 00:39:21,860 --> 00:39:25,249 So we had to change our analysis plan slightly. 344 00:39:25,250 --> 00:39:36,500 We had wanted to do to produce five separate within case reports, so we had one for each case site. 345 00:39:37,250 --> 00:39:41,600 We didn't have time to do that. And fortunately we were really, really lucky. 346 00:39:41,600 --> 00:39:43,250 And this is just my luck comes into it. 347 00:39:44,360 --> 00:39:50,420 Everything that was happening and one side seemed to be happening, all the others, they were all struggling with the same sort of issues. 348 00:39:50,750 --> 00:39:56,900 So in fact, we could still do everything justice by just doing an across case analysis. 349 00:39:56,990 --> 00:40:01,400 So that's what we had to do to just be able to finish the project on time. 350 00:40:01,640 --> 00:40:04,130 So we did. We've done good cross case analysis, 351 00:40:04,220 --> 00:40:17,270 but we didn't manage to do that with in case one of the things we did about really sort of trying to make sure that we could be yeah, 352 00:40:17,330 --> 00:40:18,980 have some confidence in our findings. 353 00:40:19,280 --> 00:40:29,630 We did these stakeholder engagement events, we did ten events nationally and I think there was something like 200 and about 250 people involved. 354 00:40:30,170 --> 00:40:35,239 So we invited. People from right across the different stakeholder groups to come. 355 00:40:35,240 --> 00:40:37,070 And we shared some of our findings. 356 00:40:37,280 --> 00:40:44,720 We got them to talk about issues, and certainly everything we were doing seemed to resonate with the people who came to these events. 357 00:40:44,730 --> 00:40:48,110 So that was that was really encouraging. 358 00:40:49,520 --> 00:40:55,999 So so far we've got three peer reviewed publications out and our main study report 359 00:40:56,000 --> 00:41:01,070 is currently in press with the nature and should be available any day now. 360 00:41:01,100 --> 00:41:08,600 We just, just got a few tiny amendments tool to do sort of editorial tweaks on a few queries on 361 00:41:08,600 --> 00:41:13,250 Monday and hopefully it should be in the public domain before the end of the month. 362 00:41:17,310 --> 00:41:29,430 So some of the key findings, as I said, I want I want I want to be involved in research that gives us useful findings. 363 00:41:29,460 --> 00:41:35,110 Okay. For practice or education. So just a couple of the key findings with it. 364 00:41:36,510 --> 00:41:43,050 Under it, we have managed to show that if people go through undergraduate, it's professional working. 365 00:41:43,230 --> 00:41:48,420 It does there is an improvement in professional practice in terms of interprofessional working, 366 00:41:48,540 --> 00:41:52,170 which then does seem to have a knock on effect in terms of patient experience. 367 00:41:55,260 --> 00:42:02,160 And students don't necessarily appreciate the benefits of IPI working until they're out there. 368 00:42:02,700 --> 00:42:06,990 They all complained bitterly about the professional learning modules, so they didn't enjoy them. 369 00:42:07,380 --> 00:42:13,860 But when we interviewed them, once they were out in practice, they're saying, Oh God, I wish I'd paid more attention. 370 00:42:14,130 --> 00:42:19,890 I hadn't realised how important it was until I started working and I thought that was quite I 371 00:42:19,890 --> 00:42:27,450 think that's quite an interesting finding in terms of the current focus on on student experience, 372 00:42:27,810 --> 00:42:35,790 because sometimes I think students just actually need to learn stuff whether they realise at the time why, 373 00:42:36,570 --> 00:42:39,720 you know, so anyway that that's a whole nother issue. 374 00:42:40,350 --> 00:42:42,420 But I did think that was quite an interesting finding. 375 00:42:43,920 --> 00:42:51,540 I think the thing about the crick and study that has been really useful is that because most quality indicators are designed for study settings, 376 00:42:52,320 --> 00:42:57,690 at least for acute settings, there has been a tendency to just go, Oh, well, 377 00:42:57,690 --> 00:43:00,990 this one works in the acute setting, so we'll just roll it out to the community. 378 00:43:01,260 --> 00:43:04,680 Doesn't work situations are much too different. 379 00:43:04,920 --> 00:43:15,659 So that's a key, a key finding. And the other interesting thing for us was that they seemed to be the managers 380 00:43:15,660 --> 00:43:19,740 and the commissioners on one side saying this is what we need to know about, this is what we need to measure. 381 00:43:19,980 --> 00:43:24,000 And the other side, you've got the patients, the carers and the frontline nurses going. 382 00:43:24,180 --> 00:43:31,290 Actually, these are the things we need to know about. These are the things that are important and a lot of the key words that are currently in use 383 00:43:31,290 --> 00:43:36,240 don't address the things that the patients and the frontline staff think are important. 384 00:43:36,630 --> 00:43:39,810 So and I think that's also a really, really interesting finding. 385 00:43:41,190 --> 00:43:46,889 We go to website. If you Google Quicken, I think we're the second entry that comes up. 386 00:43:46,890 --> 00:43:51,270 So we have got a website with a lot more information about the project on that. 387 00:43:53,400 --> 00:43:58,590 So benefits of using mixed methods well and the curriculum evaluation with 388 00:43:58,590 --> 00:44:02,669 without a data questionnaire validation because we had for scale three of them, 389 00:44:02,670 --> 00:44:08,430 we could find other scales that we could use to look at for concurrent validation. 390 00:44:08,670 --> 00:44:11,400 Again, you know, establish concurrent validity. 391 00:44:11,610 --> 00:44:19,169 But one we couldn't the only way we could do it was to administer the questionnaire, interview people, compare data from both. 392 00:44:19,170 --> 00:44:28,350 So that was really useful. As I said, we were able to triangulate the questionnaire data, the scale data and the qualitative data. 393 00:44:28,740 --> 00:44:30,180 And so again, 394 00:44:30,420 --> 00:44:39,540 using mixed methods gave the program and our findings credibility that I don't think it would have had without without having to use both and the 395 00:44:39,540 --> 00:44:51,300 Quicken study it really allowed us that I just said the the mixing happened really in in the data collection and the analysis for the survey. 396 00:44:51,570 --> 00:44:58,170 And that was invaluable because it gave us a really detailed picture of what was happening in terms of quality measurement. 397 00:44:58,530 --> 00:45:02,940 And obviously it allows us to do this case site sampling, 398 00:45:03,240 --> 00:45:12,690 but it also informed the design of the case sites study data collection because we could go and see we were going into those sites 399 00:45:12,900 --> 00:45:21,750 with a a fairly comprehensive because I said we had breadth and depth overview of what was happening in the rest of the country. 400 00:45:21,870 --> 00:45:27,269 So we were able to locate the organisations in which we were actually doing the 401 00:45:27,270 --> 00:45:33,180 study within the national picture and that was really useful from our time. 402 00:45:34,110 --> 00:45:40,020 So I think really doing both, but using both really enhance the quality of our work in that study. 403 00:45:41,460 --> 00:45:48,480 So my conclusion is that yes, it's messy but it's worth it. 404 00:45:48,900 --> 00:45:51,310 So thank you. Thank you.