Welcome, everybody, to this evening's talk of mixed methods research in the real world.
I'm delighted to welcome a former colleague and friend,
Dr. Cathy Pollard from University of the West of England in Bristol, who going to to to do this talk.
She's going to be sort of talking for about 40 to 40 to 45 minutes and then we'll have a little bit of time at the end.
The question so if you can say your questions up until the end, that would be great.
Cathy Okay. Well, well, thank you very much for coming along here this evening.
And Margaret, thank you for inviting me to come here. I'm delighted to be here.
So as you will see, my title, I think says quite a lot about what I think about mixed methods research.
So for those of you who are wondering about it, I've been working in research,
mixed methods and both quantitative and qualitative now for about 20 years.
And I found that the assumptions often, particularly on the part of students often, is that research is this nice,
neat, tidy process, because of course we produce nice, neat, tidy looking findings or results.
And of course, that that may be true for lab based work.
It's certainly not true for social research.
And all my work has been in the field of health and social care.
So most health and social care with lots of health and social care research actually sort of qualifies as social research.
And of course, I mean, if one just stops to think about it for a second,
the reason one of the two main reasons I think for why research, social research is so messy.
And the one is just the vagaries of human nature, where we're always you know, it's people who are actually conducting research.
It's people who are participating in research. And we like to think of ourselves as rational beings.
But frankly, I don't think that's the case at all.
We certainly have some rational faculties, but a lot of our behaviour is driven by impulses that are very,
very far from being rational and that situation doesn't suddenly get held in abeyance because we are involved in a research project from either end.
If anything, once the sort of pressures, those sort of impulses can can be highlighted or emphasised.
So there's that aspect of it. The other aspect is actually our external environment, because of course, we're going out into the world to do research.
We are actually having to travel to go to places and I'll just say leaves on the line.
You know, you just never know when something is going to happen that's going to literally derail the process.
So we're always, always having to deal with the possibility of error when we're doing social research because it's a very complex process.
When you start thinking about doing mixed methods as part of social research, you actually then increase the possibilities of messiness exponentially.
So I suppose the question then becomes if that is the problem is why do mixed methods research?
Well, as I said, I'd been working in the in the field now for for quite a long time.
And I've been working in health and social care generally by the clinically or in as a researcher for about 25 years.
And I think the more I stay in it, the more I feel very strongly that if we are producing,
if we're doing research, we actually need to produce results that are useful and can be practically implemented.
And as far as I'm concerned, doing mixed methods research really aids us in that process.
And so I think I sort of reveal myself very firmly as a pragmatist by actually sort of laying out my stall in this way.
So there, you know, there's quite a lot of there's been a lot of theory written about mixed methods research.
The different researchers have got different ideas and attitudes towards mixed methods research.
You know, there is the idea that that's quantitative research and qualitative research are actually dichotomous.
I think that's a pretty outdated stance. If you have a lot of people now writing about mixed methods, research,
being on a continuum where you have qualitative research at one end and quantitative research at the other,
and a lot of researchers talk about using mixed methods in terms of depth and breadth
enough for me that that's that sort of chimes with my own feelings about it.
And in fact, when I think about mixed methods research, I tend to sort of, if you like, visualise it almost as a cross in that you've got a broad.
The quantitative research and quantitative approaches give you a nice broad picture of what's going on.
So you can find out what's happening if you use.
And the qualitative research for me is like the vertical arm where you're actually
really being able to get a go into depth to find out why something's happening,
to understand it. So if we're going to actually do come in this couple of seats up at the front,
so if we are going to really understand and produce work that's actually useful in health and social care,
I think it's somehow that intersection of the quantitative and qualitative,
which really gives us a it's our best chance of getting a comprehensive overview about a topic and really understanding it.
We still have to deal with the problem of messiness and particularly we need to think about how can we produce robust results.
We want to be able to trust our findings because otherwise it's much more to go home and not bother to even get started.
If you think you're going to produce findings that are not reliable, not trustworthy, not credible.
So I'm going to we'll come back to that a little bit later.
But what I do want to do is at this point is just talk through some of the processes involved in conducting this sort of research.
And to do that, I'm going to draw on my own experiences of two fairly large mixed methods, projects or programs that I've actually been involved in.
Right. So the first one I want to talk about is an evaluation of the interprofessional curriculum.
And as you can see from the slide, it went from it ran from 2001 to 2008.
And this was actually my my first post. My first research post was on this project.
I was appointed as the research associate on the project.
And there was it was funded by what was then.
There was a point some of you will remember some of you weren't that the the the way that health services were organised.
They were these workforce development confederations funded by the Department of Health in each region.
And so this project was funded by the Avon and Work even Workforce and Avon Workforce Development Corporation, all Confederation.
So that was the first one I want to talk about. The second one is one that we've just finished, as you can see, started in 2014, carried on to 2017.
The data collection analysis stopped a year ago and it's been tweaking and amending the final report for the last year.
And this was, again, an investigation of quality measurement in English community nursing,
and that's quite a mouthful, so we shouldn't get too quick in. So that's how we refer to it is Quicken the Quicken Study,
and that was funded by the National Institute of Health Research and that was a multi site, multi organisation study.
So I'm going to talk about both of those. So firstly,
what I want to do is just tell you a little bit more about each of them and then start
thinking about some of the the issues that arose when we were actually doing these projects.
So the U.S. interprofessional curriculum.
Now, at the time that this curriculum was introduced, it was the idea of giving undergraduates the opportunity for interprofessional education,
for learning together, for working together, which was brand new.
It hadn't really been done very much at all. It was it was starting to come in.
So a degree was one of the first universities to introduce this curriculum,
right throughout the whole professional program for undergraduate students.
But also, more importantly,
it was there was a module in each year and those modules were compulsory and they were assessed so students could choose whether or not to do them.
They had to do them and they couldn't progress or graduate unless they'd actually passed those modules.
And as you said, at that point, we didn't have medical students actually in the university because they're educated at the University of Bristol.
But as in a slightly later iteration of the curriculum,
we had medical students coming over from Bristol to join us in some of our interprofessional work and interprofessional curricula.
But the evaluation at the time didn't include medical students,
but it had a lot of different of the health professionals, all on separate programs, educational programs.
And in their wisdom, the faculty decided it would be a good idea to evaluate this new, new curriculum.
So that got it sort of set up quite well.
And these were the things we were interested really interesting in what the student attitudes towards learning and working into professionally,
collaboratively. We were also looking at not just the attitudes, but what how did they find the experience of learning into professionally,
both in academic settings and in placement settings.
And finally, we were interested in what was happened to the staff because again, this was very new for staff.
Staff were not used to having to work and facilitate students who actually were from a different professional discipline from themselves.
So the whole thing was pretty brand new. So what we decided or what the design for the study was, was a really realist evaluation.
So there was very much a focus on linking contexts, mechanisms and outcomes.
And one of the key questions that one asks in realist evaluation is what works for whom and why?
Because, I mean, it's fairly obvious that, you know,
you can have the same situation and bring people into it and they'll get different things from it.
So we were really trying to tease out, again, the complexities of the processes that students were actually being exposed to.
So we designed a mixed methods research program, and it was concurrent, independent equal status program, and it incorporated five distinct studies.
So it's mostly quantitative, although this study had some qualitative aspects to it, in fact was the serve.
Our survey was a mixed methods, but it was it was very heavy on the qualitative and quantitative side.
We wanted to find out about student attitudes and staff attitudes.
So you can see over the course of the of the of the program, we can we collected, you know, to just about 2700 questionnaires.
And please also do remember that this was way before online surveys.
We had to do everything with paper copies. So which had its own interesting times.
Of those 2700 rough, just over two, two and a half thousand of them were actually student surveys.
I think we had about we only had about 60 or 70 staff questionnaires.
So most of those were student. And what we did was we we had to do we had for administration.
So as they came in the doors point in their second year when they qualified and a year out once they were in practice,
practice for a year, we also did it.
We had three qualitative studies looking at sorry for qualitative studies, looking at non participant observations.
So we observed these interprofessional learning groups and we all went out and observed what was happening in the students on placement.
We did interviews, loads of interviews with students,
some interviews with staff in both placement areas and the academic staff and we interviewed did focus groups with students.
The third year of the interprofessional module was delivered online.
So we analysed discussion boards from the groups and then we also analysed some of the module evaluation forms.
So as I'm sure you can appreciate, we had a huge amount of data both to collect and to deal with.
And I'll talk about some of the issues around all that in a minute.
Once I've just told you a bit about the other project. So the quick and steady.
So as it says there, the aim was to determine, just find out how is the quality of care actually measured in community nursing services.
And the reason why this was of interest and why got funded is a couple of things are happening.
You may I don't know how many of you have got any knowledge of the community nursing services,
but the community services generally are being used more and more and more that the landscape has changed dramatically
over the last ten years or so and with a drive to get people out of hospital and not have people in hospitals.
The community nurses are having to provide extremely skilled care to people who actually
have very complex medical needs and who would previously have been cared for in hospital,
not at home. So that landscape has changed completely in terms of community nursing care.
And while it's actually quite easy in a ward or residential setting to see what's happening in terms of quality of care,
because things are happening and other people are watching in the domiciliary setting,
the only people who know what the quality of care is like is the person receiving it and the person providing it.
So obviously there is a need to make sure that there are robust measures for measuring the quality of community uptake of community nursing.
Now in again, I'll just digress slightly, but in health care generally in the UK and elsewhere,
a very common way of measuring quality of the health care being delivered is by using quality indicators.
That is simply just a measure of care quality used to judge how good the services and the sort of indicators.
An example of an indicator is something like, you know, staff should be trained, should be updated regularly about medicine,
administration or every older person should be assessed for the risk of a fall risk of falling.
So you can see the very, very specific statements.
Usually these quality indicators, which means that organisations providing care have rafts of them.
Some of them are nationally mandated, but a lot of the of the indicators that actually are news are negotiated between clinical commissioning groups,
CCGs and the organisations providing the nursing services.
And obviously this applies to all the services that this is commissioned from, from any, any health care providers, acute or community based.
Anyway, for these reasons, what we were interested in was seeing is really finding out about the selection,
how acute care quality indicators chosen and why,
how are they applied and why, what happens then and how useful they are both for,
you know, commissioners, managers, frontline nurses and patients and carers.
So how useful are these indicators? So we were looking at the ones that are currently in use for community nursing.
So it was an exploratory design. Again, it was a mixed method research project, sequential, interactive.
But this one really had a qualitative bias as opposed to the the curriculum, which was more an equal status program.
So we did a national survey to find out what quality indicators were being applied to community nursing.
At 2014,
we contacted every every CCG in the country and sort of 211 at the time to find out what indicators they were using so that we could actually map.
What was actually happening across the country.
And then also one of the other reasons for doing the survey was to provide us with a pool from which we could select.
Case sites because we wanted to do in-depth case sites as well.
And we had five case sites in England.
And each case site was a clinical commissioning group and a and the provider, the community nursing services.
So they would sort of dyads. So that actually meant that we were dealing with ten different organisations in our case sites.
We did individual interviews or group interviews and with Right.
All the stakeholders involved.
So again, that was NHS England representatives, commissioners, provider managers, nursing team leaders, frontline nurses, patients, carers.
We managed to get a sort of pretty good representation across those groups.
We also sat in on 25 organisational meetings, some of them which were huge,
as you can see by the the number there of the people who actually attended.
And so we were interested in any, any meetings where quality was on the agenda.
One of our we had a couple of registered nurses working on the project, including the PR.
I was a registered nurse and she actually went out and did and shadowed community nurses to see what was actually happening out on the ground.
And then also we were able to scrutinise some of the documentary data that was made available to us.
So again, both these projects had a huge amount of, of, of data in them.
The what I wouldn't say about the curriculum, the curriculum evaluation is that we mixed methods sort of all the way along,
both in terms of data collection, data analysis and also in the interpretation.
Whereas with the the quickened study, it was very much in the initial data collection and analysis phases that we that we mixed the methods.
So Tony, in recruitment, I think that the the challenges of doing these sort of large studies where you're mixing methods is,
is you have to take into account that actually you're going to run into problems,
otherwise you are really going to run into problems if you haven't sort of tried to anticipate them a little beforehand.
So with the IPA curriculum evaluation, given that it was an evaluation, it was incredibly important to have baseline data.
Right. You can't evaluate the effect of something if you don't know what's actually the status quo before you actually start implementing it.
So the way the and this is compounded,
our difficulties around this were compounded by the fact that the students were on ten different timetables because they were ten different programs,
they were ten different timetables. And as I said, this is my first job as a research associate.
I think I started in the beginning of the September in the job and the we started data collection in the last week of September.
And in my naivete, I just assumed that somewhere in the faculty there would be some sort of central information about all the students.
Not so because this had been because of the the the history of how, you know, that areas like health care, education have come into higher education.
You had absorbed courses from about ten different colleges, which meant that they all came in and they all had their way of doing things.
They all had their separate admin staff and their separate admin processes.
And although it was about ten years after that amalgamation, it hadn't actually then really moved much further in terms of centralising it.
So the first thing I had to do was to try and go around and find out where these students were going to be,
because we were absolutely bound by the student timetable.
There was no slippage. We could not slip because if those students were in, they were end.
And if they were up, they were out and there was nothing that could be changed to accommodate the fact that we were doing research.
And again, this was made a little bit more difficult because the only time that actually suited all the
programs for them to have a the first interprofessional module was the second week of the term,
so we had to get them to complete questionnaires in the first week and we had to get everything organised and set up,
including some some training for some of the people doing it to actually be able to get into the groups and start observing.
So time pressure was, was enormous in that particular project we had in that first administration,
we had over 850 questionnaires, two to I think there were about 900 students in the in that intake.
And we got a very, very good response rate, which was brilliant.
It's amazing what happens if you go to room when you go here, fill it in, please.
And they get all right, especially in the first year, it was great. And we did have ethical approval.
We'd gone to that through the Ethics Ethics Committee beforehand.
So we did have ethical approval. But, you know, it was a real logistical exercise.
And of course, the other thing was that we wanted to, you know, be able to do comparisons right the way through.
And so we had to work out a way of linking individual questionnaires.
So we needed to know, you know, if I've got this person's questionnaire at the beginning and this person's questionnaire at the end, do they marry up?
So we have to work out a way of doing that, and that's a sort of story in itself.
But we did manage. So as I said, they were timing was was quite, quite difficult in that particular project.
And then of course, once they because we followed them into practice,
there was a whole issue of trying to to track them once they left the university,
because also we wanted to do interviews, we wanted to send them questionnaires.
And so there was a whole issue around trying to to manage that in the timescales
that that that you know that made sense quick in project the time pressures weren't
quite the same but they were still there because with an NIH project you your
milestones can slip a bit within the project but you've got to deliver on time.
I think you can probably negotiate about a fortnight if you're lucky, if, if you really slip.
But actually you have to deliver.
You have to deliver on time. And we had to get our survey done because we needed the, the,
we needed to have it analysed so that actually we would have a really good idea of what was happening.
That. So that would inform the data collection in the case sites.
And also said we needed this pool for sampling. So we had to get it done.
And of course, the first major hurdle was actually just getting hold of CGS.
There is a list of them on a website with contact details.
But you phone the numbers and you say, I want to talk to somebody who knows something about quality measurements and a community nurse.
And it sort of well, you know, you don't actually get much in the way of, you know, I think after about an it takes ages,
you know, because then you do get a name and then you can never get hold of that person and they don't answer emails.
And you know, all the normal problems you have about contacting people in organisations and trying to make sure you've got the right person.
And we eventually wound up because a few CQ said to us, Please, which you just send us a Freedom of Information request.
So we eventually took the decision, the project team took the decision.
That's how we would approach it. There's some issues about because that's the public purse again.
So there are issues about that. But it was it seemed to be that was our choice.
That or just scrap the survey and we couldn't scrap the survey. So we did do that.
We did. We got a reasonable I think we got about a 70% response rate.
So we were quite happy with that and got a lot of useful information.
Our next hurdle was the case site recruitment. Now, in theory, this is all supposed to be streamlined.
Now this is very different. We we while we were doing the survey, we got ethical approval from an NHS ethics committee.
But then we had to get into each organisation. The CC G's weren't too difficult to actually sort of varied.
And although once we because our host CCG had given us we had letters of permission, etc.,
that theoretically should have actually just been acceptable everywhere, not the case at all.
Every single organisation we want to. Wanted us to go through their processes.
So we were jumping through one well crawling by that time through one hoop after another.
And I think we'd allowed six months for recruitment and it actually took us over a year.
I mean, we managed to get them.
But of course it did have a knock on effect on the time we had then for actual data collection and particularly for data analysis.
And I'll talk about that in a minute. So I just want to make the point that actually, if you're going to do these big studies,
it's really important to have excellent administrative support or skills and people who don't appreciate.
Good people supporting them in administrative capacities are idiots.
As far as I'm concerned. You can't do any of this sort of stuff without good administrative support.
And the other thing is, you know, you need to be flexible so that you can adapt because you can't assume you're going
to be able to do your project exactly as you thought you were going to do it.
So the other challenge is actually putting together research teams and not just putting them together, but actually working with them.
It so happened just it's just coincidental and in fact only realised that when I was putting
this together some time during this week that we had the same number of people in each team on,
on each project. So we had 14 project team members on each study.
Now the, the other team was all based.
I mean, one strength of it is everybody won. One thing that made it easier in terms of messiness was that everybody was based in the same place.
And also I was the only person who didn't sort of know quite a lot about the the the topic of interprofessional education when I started.
So but we did have interesting things happening in that in that team because we obviously
we had they were mostly all lecturers and they were they were coming from different health,
professional backgrounds and social care professional backgrounds.
And it was quite interesting sitting being a fly on the wall sometimes and watching how the interprofessional working,
what's going on within the research team which considering that was our topic was was yeah,
it was amusing at times to say and cause some problems as well.
Sometimes that quick in team was very different.
It was it was there were five organisations involved in this project.
So as a twin NIH, our project, it had to be hosted by an NHS organisation.
So Bristol CCG was the host and then Hillary was the lead academic institution.
But we also had input and researchers from University of Bristol, London School of Hygiene and Tropical Medicine and the University of Manchester.
So it was a team that was drawn together of people who some knew each other, some didn't.
To start off with, again, people with very different clinical and professional backgrounds and from all over the country.
And we had to you know, there were times when you just needed everybody to get together.
You could do a lot by Skype and video conferencing, etc., but that wasn't always successful.
So so we needed to do both really. So there was quite a, there were sort of, again,
logistics involved in making sure that you can get everybody together and both those teams you want a proper a good team meeting,
you know, just handling people's diaries is something else again.
So what was important to both of them was to make sure that we had people with the requisite skills and expertise.
Right. And obviously, once you're doing mixed methods,
this this is an issue that you've really got to think about because not only do you have do you need people with the,
you know, the required skills,
but you also often need to do some sort of mediation and interpretation between people coming from different research paradigms.
And that was certainly the case in both studies. I mean, I think we managed it fine, but it was certainly an issue that actually came up.
So, you know, trying to explain to a medical colleague that actually doing qualitative research was a good idea.
For example, trying to explain to a social work colleague, on the other hand, that actually we really needed the quantitative work.
So again, that just added to everything we were doing.
The other thing especially, I mean, as I said in the in the curriculum evaluation, everybody had most people hadn't had knowledge of the topic.
They were all educators interested in interprofessional learning in the cricket team.
We really needed people with skills in specific areas in terms of knowledge.
So we obviously we needed people who knew something about community nursing.
We needed people who knew something about quality management. And we need people who knew something about commissioning.
But and the sort of dynamics of what we were doing was quite interesting too,
because with, as I said, with these disparate people and these large teams,
it's and going back to what I said right at the beginning about are not being rational beings all the time, you know,
it's can be quite difficult managing a team of that size when you interpersonal issues
that are professional issues into paradigmatic issues if there is such a word,
you know, so a lot needs to be thought about and planned and I'm not always the best person I'll put my hand on.
My heart and I lose my temper every now and then and start snapping at people.
But actually that works immensely well. The first time I did it, I was really surprised.
I thought, Oh, everybody went, Oh, yes, oh, it was wonderful to all of us.
I must, you know, employ that that strategy more often.
But it is something, you know, to actually you have to do because every decision and all the activities need to be negotiated,
especially when we need it quick and project when you need to start thinking about timing and changing protocols slightly,
it all had to be negotiated with a full project team. So that was yeah.
Something to do. So of course, we still have to think about how do we.
We've got some doing these big projects we're doing, you know, we're trying to deal with all the complexity that arises.
How are we going to ensure robustness?
So in that curriculum evaluation, it was very important that people collecting the data were not involved in delivering the program, the curriculum,
because otherwise, you know,
what good is the quantity of of of data that's collected from a student who doesn't want to say something to upset a lecturer?
So we had to make that was something we really have to be careful about.
We had to validate our questionnaire. There were a few that weren't.
There were some similar questionnaires out there, but there weren't any that actually were quite right for us.
So we had to design our own and they were full attitude scales and they each needed validating.
So that again, that's a whole separate exercise. And I learnt about psychometric testing.
I learnt a lot on the ground in that particular project.
Obviously doing qualitative research is important to consider into research reliability.
So we had a lot of collaboration and discussion about the instruments we were using for collecting data and also for analysing,
coding and analysing and the themes coming out.
We were able to triangulate our quantitative and qualitative data and that was really useful because it actually really allowed us to say,
Well, this is what we're seeing in the attitudes. This is what we're getting from all the interviews and the and the qualitative data.
So we can really as I said, it gave us a great sort of comprehensive picture of what was happening.
And I think the the the if you like, the proof of the pudding is that we've actually between 2004 and 2012,
we published 16 peer reviewed papers from that curriculum evaluation.
So I know it's held in good, good repute.
You know, I mean, I still get invitations to talk about it from various places.
I was in Norway a couple of years ago. I was asked I get I get requests for that for a questionnaire to be used on a regular basis.
So I think we managed to actually produce a robust research programme and in the quick study,
robustness sort of was a different it had a different flavour of trying to do of trying to ensure robustness.
The first thing we had to do was the survey data analysis.
We really needed relevant the relevant expertise because what was happening when
we asked people to send us their local course indicators for community nursing,
a lot of the indicators that were sent were were not necessarily I think eventually actually, because people didn't know.
We eventually just started asking people for just the community indicators because often people couldn't tease out.
The people we were talking to couldn't tease out whether actually they applied to community nurses or not.
And we got something like over 800 uniquely worded quality indicators to analyse through the survey.
And while you could look at some of them and go, Yep, that's definitely for community nursing.
Without specialist expertise, you couldn't always tell.
So we had a one of our project team members was a community nurse manager, and the other one was a commission of community services.
So we pulled them in to look at everything that we weren't sure about the rest of the researchers,
because they were four of us from the team actually collecting data.
But they also came in to help analyse the survey. So that was the first thing we really had to do.
Again, they were we had to do the normal research reliability processes in terms of the qualitative data.
But also, as I said, we had time constraints which we hadn't anticipated.
So we had to change our analysis plan slightly.
We had wanted to do to produce five separate within case reports, so we had one for each case site.
We didn't have time to do that. And fortunately we were really, really lucky.
And this is just my luck comes into it.
Everything that was happening and one side seemed to be happening, all the others, they were all struggling with the same sort of issues.
So in fact, we could still do everything justice by just doing an across case analysis.
So that's what we had to do to just be able to finish the project on time.
So we did. We've done good cross case analysis,
but we didn't manage to do that with in case one of the things we did about really sort of trying to make sure that we could be yeah,
have some confidence in our findings.
We did these stakeholder engagement events, we did ten events nationally and I think there was something like 200 and about 250 people involved.
So we invited. People from right across the different stakeholder groups to come.
And we shared some of our findings.
We got them to talk about issues, and certainly everything we were doing seemed to resonate with the people who came to these events.
So that was that was really encouraging.
So so far we've got three peer reviewed publications out and our main study report
is currently in press with the nature and should be available any day now.
We just, just got a few tiny amendments tool to do sort of editorial tweaks on a few queries on
Monday and hopefully it should be in the public domain before the end of the month.
So some of the key findings, as I said, I want I want I want to be involved in research that gives us useful findings.
Okay. For practice or education. So just a couple of the key findings with it.
Under it, we have managed to show that if people go through undergraduate, it's professional working.
It does there is an improvement in professional practice in terms of interprofessional working,
which then does seem to have a knock on effect in terms of patient experience.
And students don't necessarily appreciate the benefits of IPI working until they're out there.
They all complained bitterly about the professional learning modules, so they didn't enjoy them.
But when we interviewed them, once they were out in practice, they're saying, Oh God, I wish I'd paid more attention.
I hadn't realised how important it was until I started working and I thought that was quite I
think that's quite an interesting finding in terms of the current focus on on student experience,
because sometimes I think students just actually need to learn stuff whether they realise at the time why,
you know, so anyway that that's a whole nother issue.
But I did think that was quite an interesting finding.
I think the thing about the crick and study that has been really useful is that because most quality indicators are designed for study settings,
at least for acute settings, there has been a tendency to just go, Oh, well,
this one works in the acute setting, so we'll just roll it out to the community.
Doesn't work situations are much too different.
So that's a key, a key finding. And the other interesting thing for us was that they seemed to be the managers
and the commissioners on one side saying this is what we need to know about, this is what we need to measure.
And the other side, you've got the patients, the carers and the frontline nurses going.
Actually, these are the things we need to know about. These are the things that are important and a lot of the key words that are currently in use
don't address the things that the patients and the frontline staff think are important.
So and I think that's also a really, really interesting finding.
We go to website. If you Google Quicken, I think we're the second entry that comes up.
So we have got a website with a lot more information about the project on that.
So benefits of using mixed methods well and the curriculum evaluation with
without a data questionnaire validation because we had for scale three of them,
we could find other scales that we could use to look at for concurrent validation.
Again, you know, establish concurrent validity.
But one we couldn't the only way we could do it was to administer the questionnaire, interview people, compare data from both.
So that was really useful. As I said, we were able to triangulate the questionnaire data, the scale data and the qualitative data.
And so again,
using mixed methods gave the program and our findings credibility that I don't think it would have had without without having to use both and the
Quicken study it really allowed us that I just said the the mixing happened really in in the data collection and the analysis for the survey.
And that was invaluable because it gave us a really detailed picture of what was happening in terms of quality measurement.
And obviously it allows us to do this case site sampling,
but it also informed the design of the case sites study data collection because we could go and see we were going into those sites
with a a fairly comprehensive because I said we had breadth and depth overview of what was happening in the rest of the country.
So we were able to locate the organisations in which we were actually doing the
study within the national picture and that was really useful from our time.
So I think really doing both, but using both really enhance the quality of our work in that study.
So my conclusion is that yes, it's messy but it's worth it.
So thank you. Thank you.