Title of the talk is how do you conduct a really synthesis when there is no evidence and I guess, you know, disclosures first?
This project that I'm going to talk about was funded by an I child.
And I'll explain what problems are in a minute. But of course, the usual disclosures apply.
And now, you know, this is not their opinion. It's mine. I also want to acknowledge the brilliant team we had as a big team.
So lots of different people from different backgrounds, so very grateful for them.
And that was that was great. So you have an eye to the inspiration.
I guess for this talk comes from the feedback we got from it.
And I child when I put in my grant to do a really synthesis of feedback of patient reported outcome measures.
And I will not assume that you will know what those they are, but I will explain them in due course as the talk goes on.
But first, I just wanted to give you some insight.
I don't know if any of you you've applied for an outside of funding, you know, you get the email and you, Oh, goodness me, what's it going to say?
And you have to calm yourself down to to read the feedback and, you know, remind yourself that it's really not that bad and you can't deal with it.
But one of the questions that they really interested me, I had that you can see here is is, you know,
the feedback that they wanted to give us was whether researchers will gain sufficient examples of
where problems have and have not led to improvements for an unbiased formulation of the model.
And I think ultimately what this reviewer was saying is how will you do this really synthesis when there isn't any or much evidence?
Because at the time we submitted the grant, well,
patient reported outcome measures were often used in trials as indicators of the outcome of an intervention or treatment.
There wasn't a lot of evidence if they if they use performance data, so is at the very beginning.
So when this project got funded, it was 2014. So I think it was probably about 2012 2013 when we got this feedback and the National
English National Problems programme had only been running really since 2010.
So this was the use of patient reported outcome measures as indicators of the quality of hospitals.
So there was a feeling from the reviewer to say, Well, this is a very new intervention and you know,
nobody's there's no evidence out there about its effectiveness or its impacts.
How are you going to do this review? And, you know, I chuckled to myself and thought, I know how to answer that question.
And I suppose the summary really of this talk and what I'm going to sort of elaborate in more detail as we go
along is that I think the review was coming very much coming from a traditional systematic review perspective.
So when you're doing a traditional systematic review, a Cochrane review, however you want to call them,
and I have to say also that I think realist reviews are systematic too, but that's another of the side discussion.
But I'm talking here, I suppose, about sort of Cochrane style reviews. And in those kind of reviews, the unit of analysis is the intervention itself.
It's the programme, it's the intervention.
And so evidence synthesis really does rely on there being evidence trials that have evaluated that specific intervention.
So it really is a problem if there aren't many trials available.
If there isn't any cohort studies or quasi experimental studies of the of the intervention,
you really do have a problem if there's not any any evidence.
And that's why, you know, an actual wisely require people to do some scoping work to look at the volume of evidence before they submit a grant.
Now really sort of a different, well,
one of the fundamental differences between realist reviews and what I call traditional systematic reviews
is the unit of analysis and the unit of analysis in Middle East reviews is the programme theory.
Now I'm sure you'll have heard Jeff talking about this at some point over the next two days,
and I am going to elaborate a little bit more about what I mean by programme theory, but just go with me here.
So evidence synthesis relies on the evaluation of interventions sharing the same programme theory.
So is the programme theory that you're testing that you're evaluating, not the intervention itself.
Now this is quite a big thing to get your head around. I'm not expecting you to automatically go, Oh, right, yeah, I get it.
But I do hope that during the talk, it will be made clearer.
And I think one of the things to say here is that different interventions can share the same programme theory,
but at the same time, the same intervention can have a number of different programme theories underlying it.
This sounds confusing, but it's also one of the great things about realist methods is that they can be flexible.
So just because there isn't any evidence about promise specifically, I think you'll find that the programme theories and.
The lying problems and their feedback and they use as a an indicator of hospital
quality is not that new is certainly not as new as as problems are themselves.
There are lots of other interventions that share that same programme theory.
So, you know, this is a really important and fundamental difference, I think,
which is sort of the basis of this talk about the difference between a realist review and a traditional systematic review.
I would say, however, that there are many similarities, and one of those includes the requirement to read lots of papers.
So that doesn't change. There are some similarities there.
But just before I dive into sort of talk a little bit more about the really synthesis,
I'm I'm not making the assumption that you will know what problems are. And for many people, when they hear the word problems,
they think of something that year 11 and year 13 students do at the end of their school year to celebrate,
as many of them are about to do at the moment. But but my my version of problems is it short for patient reported outcome measures?
And these are questionnaires that measure patients perceptions of the impact of
their condition and its treatment on their health and their the structured.
They ask patients to rate the health on various different dimensions.
So the example you can see here is from the Oxford Hip School, which was developed in Oxford, and they can be generic.
That is, they apply to anybody. So there are problems like the SF 36, which some of you may have heard of,
which are suitable for people with a variety of conditions and also people that don't have any health problems as well.
Well, they can be specifically focussed on looking at the impact of a specific condition.
So in the case of the Oxford Hip School, this is a condition specific from looking at the impact of arthritis on on people's hips.
And it's usually used to evaluate hip operations, which is part of the English Promise programme, which I'll explain in a minute.
But the big idea about prongs is that they sort of capture and quantify the patient's perspective on their health and their condition.
And, you know, when they were first developed, they were largely used as as I said before outcome measures in in trials of different treatments.
But more recently, there's been a use of them in the health service in two different ways.
So the main way, well, not the main way, but the focus of this talk is they use what I call an aggregate level.
So this is where data patients are asked.
So, for example, in the English National Problems programme here,
anybody that's undergoing either a knee replacement or hip replacement is asked to complete a
baseline patient approach that could measure the condition specific one four hips and one for knees.
Both developed in Oxford so that the data is collected before the procedure.
And then I think it's either six months or three months, depending on the condition they complete it again.
Those data are just off the case mix and the change that people experience from before the hip
replacement or the knee replacement after the knee replacement is aggregated at a hospital level.
So each hospital has a case mix, adjusted average change in the hip or knee outcome score for patients,
and this is used as an indicator of the amounts of health gain.
I guess that patients have gained from going through that procedure and is used as an indicator of the quality of care in that hospital.
They're also used to an individual level of synthesis.
Looked at this as well, but I'm not going to really talk about this for now, but just to provide a bit more context,
an individual level patients perhaps might complete a problem before they go visit the clinician.
This is feedback back to the clinician during the visit. The idea is this this enables patients to raise issues with clinicians,
or it helps clinicians understand what's going on for the patient, and there's a number of international examples.
But as I mentioned for for this tool, what we were interested in was understanding how and why problems,
data and aggregate level might improve patient care through what mechanisms.
How will context effect that? What are they? What are the outcomes? And this is the data offered that that providers are presented with.
So each of those little dots on what's known as a funnel plot.
It's the main post-operative Oxford hip score against the provides a sample size of the number of
procedures where there's an expectation that perhaps centres that do more procedures might be.
It might be slightly better at it, but there's 95 percent and 99 percent confidence intervals.
And if your hospital is one of the dots that's outside of these confidence intervals, you're labelled as an outlier.
So really, what you want to be is somewhere I think you can see my cursory somewhere up here.
These are positive out. So they're doing almost like better than is expected.
These ones are what's known as negative outliers and labelled as as hospitals they're not doing as well as as they're expected.
So you know what a provider supposed to do with this information is one of the things that we were really interested.
How is this supposed to incentivise create?
How do responders providers respond to this? You know, what are the mechanisms through this might work?
And you know how? How will context of this?
How does that, you know, perhaps the features of the intervention or the context into which it's implemented?
These are all very, really good questions. As you look at it from from Jeff.
So it's also interesting to note that's an interesting,
contextual feature about problems was that they were incentivised as part of local sequence payments for four particular years.
They're not, they're not. So now, so you can see this is quite a complex intervention requiring a lot of, I suppose, interpretation.
And you know, one of the things we were really interested in is, well, you know,
when this was set up, what were the ideas and assumptions underlying how this was supposed to work?
And in fact, this was partly why Nick Black, we were very privileged to have him on our team.
He was instrumental in setting up the National Problems programme.
And one of the reasons he joined the project was because he was really interested in thinking about, you know, what are the underlying?
Why did we set this up in the first place, which was which is very interesting.
So now I'm going to talk a little bit more about the process that we did this and also illustrator methodological point,
as I do so about the importance of programme theory and how that's used as a unit of analysis in this context.
But just as a quick reminder, and I'm not sure where on your realist journey you are,
whether Geoff's talked you through the process of doing a realist synthesis, but I just thought I'd provide a really quick slide here.
I'm not going to go into detail as to every single one of these processes,
but I'm just going to give you an overview and then talk to a couple of these issues to illustrate the idea about programme theory and how
you can use that as a unit of analysis and use evidence from interventions that share the same programme theory to to do your analysis.
So the basic idea is that the first sort of process of doing a really synthesis is you need to get an understanding of the programme theory,
the ideas and assumptions. And I'm going to talk to you a little bit more about programme theory in a minute.
As Jeff mentioned, you can't often look at everything in a real synthesis.
You have to be focussed. You have to decide which theories are we going to test?
Are we going to jettison some? Are we going to focus on particular ones?
Again, that's that's an important decision and offered a difficult one.
And then there's a second, sorry second phase of searching, and it's different to the first phase.
The first phase is much more interested in programme theory.
So you look at different sorts of evidence, and I'll talk a bit more about that in a minute.
So the second phase, you're looking for empirical evidence in order to test your programme theories, the quality appraisal.
And again, this is different to a standard, you know, systematic review or Cochrane review.
So you're not selecting studies or appraising quality against a hierarchy of evidence where randomised controlled trials sit at the top.
It's more about the study's relevance to testing your theory, so a whole range of different study designs are used.
You know, quantitative studies really good for looking at outcome patterns and perhaps the relationships that outcomes might have with context.
Qualitative studies are really useful for looking at mechanisms and perhaps
some links between particular contexts giving rise to particular mechanisms.
So one of the strengths I think of early synthesis is that you use different study designs and
their their importance in their their inclusion is based on their theory testing potential,
not about whether they're an asset or not. Then there's a process of data extraction.
I'm not going to go into detail about this.
Very happy to share our data extraction with you, but I'm going to talk more sort of an abstract level, I suppose.
But again, you know, this is not necessarily to a strict standard matrix.
You know, I've done Cochrane reviews and used a a an accent spreadsheet.
When I do various reviews, I tend to use word documents and do much more of a narrative commentary and then, you know, synthesise it in words tables.
But anyway. And then there's a process of this,
which is where you're bringing your programme theories into conversation with various different processes of empirical evidence.
And then there's this dissemination production of abstract, middle range theory.
And that's what I'm going to talk a little bit about much more about this middle aged programme theory.
So that's just a quick overview. So I want to go back to this idea of Programme three, which I've been sort of banding about with, you know,
with gay abandon, without thinking about and explaining what it is, which is which must be frustrating.
So. And this was one of the first questions I remember when I first met Ray White and a long time ago when I was putting in a night fellowship,
actually a personal one, which I didn't get, but everything worked out anyway.
And I thought, Well, what is the programme say? What do you mean?
You know, at the time I was saying, Well, is it just sort of psychological or sociological theory?
And he was like, Well, it can be, but actually, it's something more simple than that.
It's you know what? What are the ideas and assumptions underlying how this intervention will work?
What are what's the what? What was going on in the policymakers heads when they developed this intervention to to think
about how it was going to work and really think about how things work in a particular way?
So you will have heard Jeff, talk about this idea of mechanisms. So how are the various recipients or the stakeholders involved in the intervention
intended to respond to the resources that are offered by the intervention and how,
you know, how will context shape that? And I think what we find often is that interventions are often developed in almost like a context free way.
That's an assumption often that they're going to work the same way everywhere.
You know, we do this in pool by magic. Something happens.
And I think sometimes also those those ideas and assumptions are deemed to be so obvious that they're not spelled out.
But I think one of the things that we have to do when we're thinking as a real estate is really interrogate that.
And I think this was what attracted Nick Black to the project. He was saying, I'm not sure we've done that yet with problems.
So, so that was interesting. And I think when you're first developing your programme theory ideas, as I'll illustrate,
when I talk about how we did this in the project, they often stay very close at the beginning to the particular programme.
So how is this programme supposed to work? How are people supposed to respond?
But once you've done that, you'll then find actually those ideas are not new.
They're very similar to something else I've heard of or another intervention that's got the same ideas.
And I guess this is the other picture that I really like that that that we did as part of the Ramos's project.
One of the great joys of that project was working with a cartoonist who also was an evaluator.
I mean, what are the chances of finding somebody with that skill set?
But happily, they exist, and they did these wonderful cartoons for as they call Chris Lissy.
And he was absolutely brilliant, and he developed these great cartoons and these this is what I think like about this one is that they show that,
you know, often there are some very general and abstract and very familiar ideas underlying all the things we do.
And, you know, some people would argue actually all the interventions that anybody ever does just,
you know, can be narrowed down to whether they're a carrot, a stick or a sermon, you know, so you know, carrots are things like, you know,
reward stickers for children at school to financial incentives for us to do particular tests.
You know, they all share this sort of idea of an incentive.
Similarly, you know, stakes are behaviour points that children get at school or sanctions that people get for not following the rules.
Similarly, all sorts of teaching is providing people with information training equivalent to being a sermon.
And Paulson calls these sort of I you know these this idea that many interventions
share the same underlying ideas and solutions as reusable conceptual platforms,
and we really kind of try to embrace this idea as we as we did as we did the review.
So when we were beginning this review, we we really, as I mentioned before, stuck quite closely to the idea of,
you know, what are the underlying ideas about how the aggregate user from stage is going to improve care?
And we drew on various different policy documents, editorials, think pieces, letters.
So this was a study, a study.
It was more like an extended review of the literature that was undertaken by Nancy Devlin and John Appleby at the King's Fund.
There's also guidance that the NHS issued about how Proms data should be collected and how providers should respond to it,
the steps they should take if they're an outlier. Again, methodological guidance as well.
You know how how are these outlier allies identified, but also kind of editorials and analysis by particularly this one by Nick Black,
where various commentators set out, you know, this is how I think this intervention is going to work.
And what's really interesting are the kind of sometimes editors set up.
These debate articles don't know whether one person's arguing one side and another person's arguing another.
So these are really rich opportunities for getting at those underlying ideas about, well, how is it that problems are supposed to work?
And I know Jeff, when you were doing the smoking cars carrying children's review with Ray,
and they were really nice paper about how the Today programme on Radio four is a really interesting sort of programme series as well,
because there's that debate about how this programme is supposed to work.
We also did a stakeholder workshop where we presented some of these ideas to stakeholders and said, What do you think?
You know, is there anything we've missed? What does this really mean? I also did a few interviews with various stakeholders as well.
So we really tried to sort of get get a feel for what's what are the underlying ideas.
And this revealed, I guess, we narrowed it down after a long iterative process into three sort of main idea.
So the first one? And, you know, if you think that this was, you know, the Proms programme did emerge from the dance,
I report he was he was very interested in patient choice.
So one of the ideas behind the aggregate use of proms was that they were useful to evaluate the relative clinical quality providers
of elective procedures because hip and knee replacements were elective and it can be used by patients in exercising choice.
In other words, the idea was by presenting these data on proms,
on the outcomes and the different providers and where they stood in relation to each other.
This could be used by GP's and patients in choosing where to have their hip operation and where to have their knee operation.
So it was, you know, it was sort of in the context of the the policy at the time,
which was very big on expanding and supporting patient choice in the NHS.
Another key. So we sort of called this choice theory.
Another key idea was this idea that, you know, I could get from this data can be used to empower a commission,
as you can see how old this is because they talk about pets who are now,
of course, CCGs and now, you know, integrated care systems and that they can use the data,
establish the quality of services which they are contracting providers for.
In other words, they can hold providers accountable for the quality of the services that they are providing.
So commissioners can use these data to say, actually, you know, you said you were going to provide this service.
This is the quality. You know this, we're not happy with this.
So they can presumably impose sanctions or reward it, for example, with sequin payments.
And then another idea was that promise will be useful for assessing the relative clinical quality of providers of elective procedures for clinicians,
managers and commissioners to benchmark their own performance.
And I remember asking one of the persons in the Department of Health about this, and I said, Well, what do you think, clinicians?
You know, how do you think surgeons or clinicians should respond to these days?
And they said, Well, what I hope they'll do is they'll go, Oh, the hospital down the road is doing a really good job.
How can we learn from them? Can I go and talk to them?
So the eye is very much the idea of this intrinsic motivation that providers might have to improve their own practise to learn from others.
So you can see these are sort of three different ideas underlying these problems, and this is where we sort of started our synthesis from.
And then we we sat down the three of us. I'm not very good at drawing, so I did stick people at me.
Sonya Duncan, who is the research fellow on the project and was great to work with, and André, who sort of an adviser with us, you know, what is this?
What is this an example of this sound? Quite familiar ideas that many other interventions also share these ideas.
So we tried to think about what? What were these similarities?
So, you know, one of the specific things was patients will choose proms will use proms to choose a provider,
which is very reminiscent of, you know, patient choice theory.
This idea, you know, many other interventions were underpinned by this idea of promoting patient choice.
And then there were other also quality data initiatives that were also underpinned by this idea,
such as mortality report cause in the U.S. patient experience data,
as well various different forms of hospital performance data and also the My NHS website,
which was also set up with the intent of providing patients with with these data.
Similarly, this idea of proms will enable commissioners to hold providers to account again.
There were similar. The other interventions that share this idea.
So this idea of accountability, public disclosure. You know, we can see in secrecy inspections, quality accounts that hospitals have to produce.
And again, my NHS website.
And then this idea of this intrinsic motivation that clinicians want to improve their practise and learn from each other and
they will take action if they perceive that they are better or worse than their colleagues again shared ideas with benchmarking,
audit and feedback.
Ideas and similar interventions also share these ideas, such as clinical audits as the NHS Benchmarking Club so far from there being,
you know, problems being this sort of singular, unique intervention that had a particular set of programme theories.
When we looked at that, we actually saw that actually there's lots of other interventions using quality data that share these similar ideas.
And so we could draw on the evidence in our review to test our theories.
And also the other thing that we we explored was, OK, well, what are the intended?
And I'm sure Jeff has mentioned this. That context shapes the ways in which interventions are implemented,
how people respond, and sometimes interventions have a set of unintended consequences.
But they also might have a set of unintended consequences, and we could draw on this literature,
this knowledge to to help us think about, well, what might we look?
For when we're synthesising this evidence, what might we expect to find what's going on?
So in terms of the intended consequence of the patient choice idea,
the idea was that patients would use these data to choose high performing providers and consequently the
lower performing providers might respond by improving their care in order to attract more patients.
That's how it was intended to work. But actually, one of the you know, it can always go wrong.
One of the unintended consequences might be that patients are not even aware of these data,
or that clinicians might refuse to treat sicker patients because they don't want those patients
to bring down their average score because they don't improve as much in terms of accountability,
stakeholders might impose sanctions or offer rewards.
And this in turn was expected to to, you know, stimulate providers to either maintain the good, good care that they provided or improve it.
However, you know, many commentators talked about concerns that actually, if you focus, you know, if you if you focus on particular indicators,
this might actually result in what's known as tunnel vision, and that is exclusive focus on what is measured to the exclusion of other areas of care.
All providers might gain the data in terms of benchmarking providers.
You know, the idea was compiled. This will compare themselves to their peers and take steps to improve the buffalo average.
But actually one of the understand the consequences might be that providers don't even understand the data, or they might dismiss or ignore it.
So there were all these different possible outcomes, and I think that's another thing that that, you know,
working with Ray was always keen to emphasise is that interventions don't just produce one outcome.
They produce a whole range of different possible outcomes. Some of these are intended, some of them are unintended,
and contexts often has a big role in shaping whether the intervention produces the intended or the unintended ones.
So we also tried to think, Well, what are the kind of, you know, from the literature? What are they also?
What are the important contextual features that might play a role? And this suggested that whether there are any, you know,
existing concurrent incentives or sanctions associated with the the intervention itself and the credibility of the data,
whether whether providers take the data seriously, which raised the question of, well,
what features might support to constrain this and also the action ability of the data, how how easy is it that providers to act on it?
Let me see. So these are some of the examples. As you can see, I'm not going to go into these in great detail,
but they give you a flavour of we drew on evidence from many different kinds of interventions that shared the same underlying programme theories,
from clinical audits to the cardiac surgery reports in the U.S. audits a clinical audit patient experience,
plus some patients, some older indicators that were used in the UK.
And for those of you that like PRISMA diagrams, here's a present at what what.
The one thing I want you to take from this is how messy is compared to, say, a PRISMA diagram of a traditional systematic review.
You can see we did lots of different searches. There was lots of ins and outs.
There was large numbers of screening involved and basically it was a very messy and complex process.
So it's not as simple that you do one search. You have a set of criteria, you select them.
It was much more iterative than that.
And I guess if you take one thing from this rather messy and difficult to read slide is that it was a messy process.
It also took a little bit about context.
I've mentioned this a little bit as well, and this was a really important sort of thing that helped us to think about it.
And this is the idea that interventions might work differently in different contexts that you implement them in different contexts,
you end up with different outcomes. And I suppose the context, you know,
I and I recently wrote a paper on context where we really thought about that inspired by the work that we did on the remix project.
And I guess part of context, might you know for this project, we we interpret it quite broadly and thought about,
well, also what about the intervention features in terms of how you know, the intervention was implemented?
Is that part of the intervention resources or is it part of context? It might be both.
And but it's this idea that context shapes the way that the intervention works,
but also this idea that context doesn't necessarily come in neat little packages.
You know, when, when, when programmes are implemented, they're implemented into messy into messy contexts,
into messy hospitals, where there's lots of other stuff going on that having to compete with all sorts of different things.
And it was this idea that, you know, as you get more contextual features involved the outcomes that you might find a much more complex.
And so we tried to get our heads around what the characteristics of the different interventions that we
were looking at in terms of context and looked at the different outcome patterns that we found there.
So to go back to all of the different evidence, all of these different interventions.
She had different characteristics that we then tried to map and think about how
they related to the kind of contextual features that we might be important.
And then we we sort of grouped the studies relating to each particular intervention to find out, well, how do people respond what you know,
we looked at qualitative studies about surgeons opinions or surgeon's ideas and feelings about the about the intervention,
but also some of the quantitative studies looking at the outcome data to say, Well, did things actually change?
Did outcomes improve? So just to give you an idea about how we did this,
we set up some propositions in relation to our programme theories to try and understand how different contextual features might make a difference.
And you can see that we drew on. As I mentioned before, what we were testing was the programme theory,
so we drew on evidence from a whole range of these different interventions that shared the same underlying programme theories.
But we looked at the different contextual features. So a key important one was whether the data are public and reported.
Do does the public have access to them or not? You know,
it was reminiscent of kind of this naming and shaming idea in terms of whether this idea of
people were was supposed to respond because they've been outed as a badge poor provider.
And also whether they had any sanctions or incentives attached to them.
And we looked at a number of different interventions that the crack crack indicators were something that was implemented in Scotland a long time ago.
They were not publicly reported. There were no sanctions attached to them.
When we looked at the data and when we looked at the studies, what we found was that clinicians just ignored the data.
There was absolutely no change to patient care. They had no impact whatsoever.
Similarly, you know, there was a couple of studies looking at these hypothetical indicators based on national survey frameworks,
similar outcome percent, they were not public reported there were no sanctions.
And what happened was that the data was ignored. There was absolutely no changes to patient care, so it made no difference to patient care.
We looked at star ratings. These were publicly reported.
These were something that was, I think, around in the late 90s, early 2000s associated with the National Service Framework.
And we looked at studies that was carried out by Russell Manian and colleagues.
And what happened here was that the providers just, you know,
if they were a poor outlier or a poor provider, they dismissed the credibility of the data.
They said, We don't believe that we're not, you know, these data do not summarise.
They do not represent our hospital. And there was some evidence that what happened is that the providers focussed on, you know,
changing care in relation to particular indicators or changing care so that they could tick the boxes rather than actually improving the care.
Similarly, we looked at the quality and outcomes framework and found a similar pattern in terms of,
you know, the public reports they were there were sanctions,
but again, they they tended to lead to tunnel vision such that the data, the indicators that were measured improved.
But but the other ones didn't. In general, the care didn't.
So this begins to give some idea about, you know, some of the patterns in terms of some of the contextual features and some of the findings.
And then we added, we added another layer of context. So if you think back to the the the slide I had with the contextual features, you know,
in addition to whether the data were public reported and sanctions were received,
we were also interested in, well, you know,
one of the things that seemed to be in was whether providers trusted the credibility, whether they found the data credible.
This begs the question of what influences credibility in some of our reading suggested that one of the things that
might influence credibility was whether clinicians were involved in setting up the measurement system or not,
and also whether the data was the quality of the data.
So an indicator of that quality we took to be whether the data were collected from patient notes, you know,
in terms of they accurately reflected patient activity or whether they would gain from billing data.
And again, we looked at a number of different initiatives,
most at some of these that the top reports were a mortality report cards in cause yak surgery that were implemented in California.
A similar scheme was implemented in New York, but it had some important differences.
As you can see, they were both publicly reported and they had sanctions about them.
But in contrast to the California one,
the New York one was had a lot of clinician involvement and also it was based on patient data, not on billing data.
And we found a slightly different outcome in response to those the reports.
The the there were fewer quality improvement initiatives.
The shouted on those and clinicians tended to dismiss the credibility of the data, whereas in contrast, the New York one,
there was much greater acceptance of the data, but still not that many quality improvement initiatives.
The gaokao is like the American version of our quality commission, and they had a number of incentives that they wanted to implement again.
But they were not clinician led. They were based on billing data and again,
a similar outcome to two of the forms of a patient report data found previously where you just had incentives,
a public report, it tended to leave to just focus on the indicators and not thinking about the more global things about patient care.
And again, similarly with the GP patient experience data which was implemented in England.
What we found was that providers tended to dismiss the credibility of the data.
They didn't believe the credibility. However, we also found a small number of studies that were exploring quality improvement,
sort of quality collaboratives that were set up by clinicians themselves.
They had a lot of clinician involvement from the clinicians.
And what we found there was that that data was accepted by the clinicians and
there was some quality improvement initiatives initiated on the back of that.
So how do we make sense of this? Looking across all those outcomes patterns and looking at those contexts, what we what we found was that,
you know, externally mandated data was perceived as being driven by political motives.
The data was questioned because it was perceived as measuring what's important to clinicians and regulator.
Not sorry politicians and regulators, not the clinicians. So the clinicians didn't trust it, and they didn't tend to respond to it.
The public reporting of these data, especially if there were sanctions,
you know it could lead to tunnel vision or at worst gaming to protect providers,
represent reputation in the eyes of regulators or other responsible bodies.
And I think what we found from looking at the studies was that externally mandated didn't always enable providers to identify the causes of poor care,
so it might signal that they were outliers. But it didn't always tell the why.
So it was very difficult for them to act on it.
In contrast, internally collected data that was either clinician led or professionally led was much more accepted by clinicians.
They they perceived it as a as being based on a desire to improve patient care rather than just ticking boxes.
And also because they had a say in specifying the indicators it was perceived it was felt to be measuring what was important to clinicians.
So it has much more credibility. And because they had a stake in specifying the indicators, they found it was much harder to dismiss or ignore.
So there's much more pressure to act on it.
And they were motivated to respond to protect their reputation in the eyes of their peers, which was a very an interesting motivator.
And they also found that actually this internally collected data was really useful in telling them what was going on.
So obviously, externally mandated indicators wouldn't tell them what was going on,
whereas this internal data that was often generated by clinicians was able to much better pinpoint the causes of poor care.
However, it still depended on the resources and capacity, collect the data and then to act on it.
So it wasn't the end of the story. But what this enabled us to do was identify some sort of x CMO patterns, I guess,
which explains these patterns that we observed in terms of contextual features about
whether the use of quality data did motivate or lead to clinicians improving care or not.
And so, well, how does this relate to problems we can then sort of in a way,
go back down the ladder of abstraction and think about patient reported data in terms of these features that we found and, you know, problems.
Yes, they are in theory that publicly reported. Although you often have to look quite hard to find data, they're not, you know,
they are available on a database, but it's whether patients know how to find them.
Yes, there are sometimes incentives attached to them.
There wasn't a great deal of certainly not in the English National Promes programme, although the National College of Surgeons were involved in it.
Local clinicians? Not necessarily. And the data are placed on subjective report.
And when we looked at the evidence that the few studies, as you recall, there was only about four studies at the time.
And you know, we found that actually evidence that providers tended to dismiss the quality of the data,
rather not necessarily because it was based on billing data,
but this time because it was based on patients reporting and they were a little bit sceptical.
A. Outs her patients reports and their own opinions about the improvements that patients might experience in surgery.
And there is very little evidence that certainly in the early days that this had resulted,
that the promise programme has actually resulted in an improvement in any change in any patient outcomes.
So to go back and this is my sort of last slide really to answer the question of how can you do a really synthesis when there isn't any evidence?
The answer is that in a real synthesis, the unit of analysis is the programme theory, not the intervention itself.
So you can test your programme face against evidence of interventions that share the same programme theory.
And as you can see in this example, we tested our theories in relation to evidence from the range of interventions,
clinical audits, patient experience, data mortality, report cards that shared the same underlying programme theories.
And we came up with a number of CMO configurations to explain the patterns that we observed in the data and quantitative studies.
Really useful for looking at these contexts.
Outcome patterns, but also qualitative studies are useful for thinking about the mechanisms and how people respond.
So putting all of that together,
it's like doing a big jigsaw puzzle because you don't often find a perfect test of your CMO configuration in any particular study.
But one of the things that we did and was assembling the data in the way that we did, you know,
in terms of those tables where we had contextual features in the findings that enabled us to think
about and compare and contrast and come up with some explanations to explain those theories.
And if you want to read any more information,
we've we've published a few studies and there's also a Nature Journals library report, but that's very long.
So you have to have had a good cup of tea and a chocolate biscuit if you want to read it.
But thank you. That's the end of my talk.