My name is Janet Harris, and I work at the University of Sheffield.
And one of the things I'm working on is real estate reviews.
So I'm part of a team of people that set up a new international centre in Liverpool for a real estate evaluation and synthesis.
And we're organising our first international conference this autumn.
I'm also a member of the Cochrane Qualitative and Implementation Research Methods Group,
which has been working for the last 15 years to try to get qualitative evidence included in Cochrane reviews.
And there has been some success lately.
There's just a review published now that mixed methods, qualitative and quantitative and a couple more on the way.
So I have this a bit of a background, a mixed background.
I originally think it might be fair to say I was quantitative. I managed public health programs.
I worked for Centres for Disease Control. So I supervised a team of epidemiologists.
And then HIV AIDS came along.
And when that happened, we were having a terrible time trying to attract people into HIV counselling and testing services.
And at that point we became quite interested in something called community based, participatory approaches to designing interventions.
I don't know if anyone's heard of those or not, which is basically involving the people who are going to be the recipient of
the intervention and the people who provide it in the design of the research.
So I'm explaining that because we're at an interesting point now, I think with systematic reviewing where there's some issues coming up.
And I know you've been working on mixed methods this week,
so I thought it might be really interesting to hear what some of your thoughts are about what's cropping up in these mixed methods reviews.
So this is basically a story rather than a formal presentation with completed study results from from a real estate review.
But the story is kind of a pattern of what we found over the last three reviews that we've been working on.
So I'll start going through it, and this is informal.
So if you have questions, just wave your hand or jump in and and ask me.
Okay. So there is kind of a picture behind this which you might not be able to see very well, but it's the picture of a river delta,
which is going into Lake Superior, which is one of the Great Lakes, which forms the border between Canada and the United States.
And I put this picture up here because I'm going to use the research that was done in this area,
which was research done with a Native American Indian population,
to illustrate some of the challenges that we're running into when we try to use qualitative and quantitative research in a systematic review.
So that's why the picture's up there. Now, this is a really quick PowerPoint.
Just set the overview, which is originally we had effectiveness reviews.
They were mainly done using study designs of randomised controlled trials.
In the late 1990s. There was interest in looking at introducing qualitative research into effectiveness reviews,
so people started working on systematic reviews that use mixed methods,
so including quantitative studies in the review as well as qualitative studies.
And the method we're developing now is we'll assess reviews using mixed methods and explain a little bit what the difference is between those.
So if you're just doing a systematic review of effectiveness and you're including qualitative and quantitative research,
this is a typical review question which you might recognise the format of it.
So when most people do mixed methods reviews right now,
basically what they're doing is they're starting with the effectiveness side of the
question and they're saying what is the effectiveness of a particular intervention?
And for that side of the review, the selection, quantitative studies,
usually randomised controlled trials or controlled trials, and then they're adding on to that.
The second question, which is qualitative and usually has to do with something to do with behaviour, you know,
what are people's attitudes or their their perspectives towards something or the reactions towards something.
And they're extracting the quantitative studies separately from the qualitative.
They're analysing what all of the quantitative data says about effectiveness.
Then in this time,
they're analysing what all the qualitative data says about people's experiences of receiving an intervention or perhaps providing an intervention.
And after they've done the separate analysis, they're bringing them together.
And what you see up here is the cross study synthesis.
And they're using the qualitative as a way of answering the final question,
which is what factors actually influence the effectiveness of the intervention?
So that's the typical way right now of doing a mixed methods review.
And there's a number of them being published nowadays.
And the data sources for this, as I've said, are effective two studies that have been published and qualitative research that's been published.
Now, there's a couple of issues coming up with doing this, and I want to see if I can try and illustrate them today.
Like I said, this is very much a work in progress. But one of the issues that's coming up is which kind of data is dominant.
Have you talked about dominance this week in the data? And the reason that's coming up is because in some of these mixed methods reviews,
they're getting quantitative results that are saying that an intervention is effective, but it's not unequivocally effective.
It's effective some of the time, but they don't know why. It's only effective some of the time or it's effective for some people and not for others.
And when they go into the qualitative research, sometimes they're getting an explanation for why it's effective or not.
They can find factors that are influencing the effectiveness of it.
But sometimes when they go into the qualitative research,
they're finding other things that have that that are not necessarily directly related to the intervention, but people think they're really important.
Okay. So for example, in diabetes research, which is what we're looking at now in this review,
a typical effectiveness question might be what's the effectiveness of a short
course in diabetes in terms of being able to control your blood glucose level?
And so the intervention is a course, and they look at the qualitative research to see what people's experiences are of diabetes.
And a lot of the qualitative research is not necessarily the patient side valuing the education course itself.
There are other things they're saying are effective in terms of helping them control their diabetes,
which is totally outside the original intervention. So we've got this kind of mismatch.
And I think you talked a little bit about this, about paradigms and worldviews, about what's important in terms of an outcome.
And what started to come up in this research is that the people that designed the intervention are deciding that one kind of outcome is important,
but the people that are actually experiencing the condition have a different idea about what the important outcome is.
So we have an issue coming up about whether you can compare these studies or not.
And so this is where we're starting now when we go on to realist reviews, realist reviews have a different sort of question,
and it doesn't necessarily start by focusing all of the attention on the intervention.
So a realist review does ask what works, which is the intervention, but they're saying who does it work for?
Because the assumption is that the intervention might work better or not as well,
depending on what group you're in, what culture you know, whether you're male or female your age.
And it's also saying that the circumstances surrounding the intervention may influence whether it's effective or not.
So, for example, someone who stayed diagnosed with diabetes over the last month,
an educational intervention may be more or less effective for them than it is for somebody who's had diabetes for ten years.
So they're saying you need to take this factor into account of time, which if you look at individual studies, they very rarely talk about that at all.
And then the third thing that they talk about is the circumstances.
So it's not just at what point in time that you're diagnosed, but it's also what kind of family do you live in?
You know, does your family support healthy eating or not? She live in a family that exercises regularly or not.
So the circumstances are about the context that actually surrounds the fact that you may have
been diagnosed with diabetes and you're going on an education course to learn how to control it.
So a realist perspective, which is pragmatic, I think you've talked about that a little bit this week,
is coming at this from the point of view of will things be effective?
Yes, they'll be effective,
but the relative effectiveness is dependent on the surrounding context around the intervention and a number of other factors.
So they're saying it all depends. Rather than a typical effectiveness review,
which would say what works and what they're trying to come out with is what works all the time for everybody,
for the general, for the general population or the or the population of interest.
So when you're looking for information and realist reviews, yes, you look for qualitative studies that have been published.
You look for quantitative studies, too, that have been published. But because you're wanting to look at the it all depends side of it.
You're also looking for a range of other information sources, for example, evaluation reports.
Often big reports are written on programs, but they aren't necessarily published in a peer reviewed journal.
Policy papers, great literature. And those are the main sources of data in in a realist review.
Now, what I put down here at the bottom is these other sources, and I put them here because the last realist review we did was.
A realist review of does peer support work to help people in terms of increasing their health literacy?
And when we did this review, we suspected that there was another source of information, which is the providers themselves, the workers,
the volunteers, you know, unpaid people in the community that are providing peer support every day but aren't necessarily part of a research study.
And so we wanted to find out in the published literature what's effective with a peer support intervention.
But also when you go out and you ask providers and patients and clients who are receiving peer support,
what works and what circumstances is what they're saying matching up to what you see in the published literature.
And to make a long story short, like in two sentences or less, at the end of the process,
it was obvious that there was a big disconnect between what the literature said
about effective peer support and what the people said who actually provided it.
So this is about complex interventions, you know, the components of an intervention.
And one of the things that was most noticeable is that the researchers who've who've done extensive
research on peer support say that there are certain types of peer support that are used in health care.
One is giving people information what is giving them emotional support,
affirmation or kind of cheering them on, telling them that they can achieve something that's difficult.
But the reviews and health care have said that a type of peer support that's never offered in health
is the practical kind of showing people how to do things or showing them how to get somewhere.
But when we went out and we talked to patients and providers and we had over 200 involved in the review,
they said that's actually the most important type of peer support. So what's happening here is something that's really essential to the providers,
that's something important and something that's important to the recipients.
It's not considered something important when the researchers are constructing the intervention.
So we have something strange going on here. So that was the last review that was finished, that was published,
and we moved on to the next review, which is this diabetes one that we're looking at now.
So in diabetes, we're asking whether public and patient involvement is is influential in terms of designing and offering diabetes support programs.
And we're finding something which to me anyway, looks kind of similar to what we found in the peer support review.
So what's happened here and I've picked out one study out of we found 135 studies.
I've chosen one study and I've chosen this study because this wasn't just one study.
This was a big programme of work which was sat in northern Wisconsin on Lake Superior with a Native American tribe up there.
And Alexandra Adams, who's the her name's up at the top there.
She and her team have actually published, we've so far found ten publications on this on this program of work.
And what this ten publications have enabled us to do is to look at kind of the story of the
development of a large research project rather than just looking at one published paper.
So way back in 2004, the initiative started because they noticed that there was, you know, a high rate of prevalence in children.
And the children they were interested in were children that were between the age of two and five.
So they realised there was some kind of problem here and something needed to be done.
And this is where it kicked off. They went to the government, they got national funding for this.
The funding was long term, which doesn't doesn't always happen.
Now along the way, rather than researchers coming in to this community and saying, we know what works with diabetes education.
The tribes, the Native American tribes in North America are actually viewed as autonomous states.
So they aren't they aren't they don't necessarily come under the federal government and they have rights in terms of what research is done on them.
So the Community Advisory Board here is actually a group of people who are members of the tribe who decide what's going to happen in their community.
So immediately from the point of view of doing research in a community,
there's a different dynamic here than you'd have in many different communities.
So researchers from the University of Wisconsin had to work with this community
advisory board to get access to the community in order to do the research.
So this is how this is how the research started out.
Now knowing that they have this high prevalence rate.
And I know you've been doing this all week, so I don't know if you're tired of this right now, but,
you know, they have a high prevalence rate of children between the ages of two and five are at risk.
Where do you start in terms of designing an intervention? Any ideas?
Because. Because I am, I guess I, I should say the parents need to context of the children because you educated.
Okay. So adding nail clippers could be one place fight. Any other ideas?
Where to start? And you're saying with that it would be education?
Nutrition. Yeah. Yeah.
Cooking classes. It's of the show stopper.
Community food supplies. Yep.
Yep. So there's something here about the environment, right?
Which fits into the realist view. To take that into account.
Has anybody worked with Native American Indians before? Okay.
So this is one of the things that we find quite often when we first start designing a research project,
if we know very little about the topic area, we have to go back to what we know our own frame of reference, right?
And there's nothing wrong with that. But we need to keep in mind that our frame of reference is going to influence the design of the intervention.
And if we aren't including the people who are actually going to receive the intervention in the design of it,
then there's a risk that we're going to choose something that may not necessarily be appropriate for the community.
And I can see people nodding because this isn't rocket science. It's common sense.
Right. But the strange thing is that when you look at this research and I'm not saying in the just in the diabetes study,
because we found this in the peer support, the peer support study actually looked at peer support across breastfeeding, HIV,
safer sex, diabetes, nutrition, lifestyle, physical activity, cardiovascular disease.
So we had a number of different health issues that people received peer support with.
And in every single one of those health issues, we found this disjuncture.
The people who were designing the peer support programs were not the people that were going to provide the peer support,
and they weren't the people, including the peer support.
So we started to wonder whether this was important or not, which is why we're doing the review we are now.
So these are some some ways that you can come at it. And where would you start with quantitative or qualitative or.
Where would you start? You just got a clue, didn't you?
Yeah, I talked to some people, which is incredibly difficult for researchers.
And it's coming up now in the in the review that we're doing, it's tough for researchers.
It's a different different world for them.
And researchers have already have ideas about works, especially if they've been working in a field for certain length of time.
And if you go and you talk to other people and you're getting their opinions about works, what works?
Then you're having to take account of those opinions.
And in some cases you're having to change the study design.
And the research is in some of these qualitative studies are saying that is a scary thing for us.
It feels like we're losing control of the study. And if we include all these opinions about what the intervention ought to look like,
then at the end of the day, is the study going to be just as scientifically rigorous if we're including,
for example, lay people's opinions and what it should look like, as it would be if it's just if it's designed by people who are academics.
So there's this kind of tension going on here about appropriateness of study design and how do you how do you come out that.
Community. I mean, you identify people who are community leaders and often women in those roles
and have a little bit more different interests and then sometimes against men.
But it's true that rather than why, when you go to those opinions later,
I'm sure there's people who will trust in the community and try to immerse yourself a bit in what's going on.
It might take a while, but it does qualitative work that way to identify from.
You can see for yourself what's going on and superficial and get at what's going on at the community level.
Maybe they don't have fathers around and maybe there is no mother.
Yeah, it is. The other thing, is this a genetic predisposition or it's just all these kids not being active?
Yeah. On the television with a parent. Yeah.
So that might actually help you because then when they have see it as a community, it's not just about the kids.
It's seeing the broader picture. Mm hmm. Multiple programs.
Mm hmm. Yeah. Yeah. So bigger picture, right?
What's the bigger picture around this? What might cause it?
And then, depending on what causes that, maybe it's not a single intervention.
Right. Maybe it's more than one intervention, which would be a community view.
Usually toward the Himalayas in Darjeeling.
Mm hmm. Of, uh, thing up there where they were trying to get in and hit mothers, nutrition and pregnant women.
What they realised was they actually had to feed everybody, including the father.
Yeah. Was a first rate. And then they developed a program to teach women to weave.
Right, so that they can sell their products in Canada and then there a whole nutritional educational program.
And so that's how they actually got to, you know, better nutrition for moms.
Yeah, man, I'm going, baby. It's looking upstream from the intervention.
Yeah. Which is not something that we always do in academia and the kind of sense I have.
Well, I could say that I was. I came to academia very late in life.
But there's another reason we don't do it in academia, and that's because we're not funded to do it.
And we're not only not funded to do it, but if you look at the career track in academia, people don't get points for doing this kind of research.
It's not considered scientific. And also, I mean, you've just done a wonderful kind of account of doing the research.
The problem is it takes time.
And in academia, there's a pressure on you to be publishing something every single year from your research, at least one thing, if not two or three.
And this kind of research and to it, like you say, going in and becoming more of an insider and developing an understanding of the community.
A lot of times when people are doing that, it doesn't look like anything's happening because you're not actually doing the research.
You're setting up the relationship for designing an appropriate intervention.
And most of the places in the world don't find us to design appropriate interventions.
They find us to get in there and produce some results within a year.
And if you look at most of the funding and if you read most of the research articles, you'll see that the duration of the study was 12 to 18 months.
And at that, at the 18 month mark, they're starting to publish results because that's the pressure to publish results,
which is a big issue when something like diabetes,
because it takes longer than 12 to 18 months for most people to show a health outcome, which is better control of blood sugar.
So because that's like one of the last outcomes in the line, you know, after all sorts of other things, the eating, the activity, whatever.
So there's, there's a there's a, I think a big issue here about about what we're finding, to be honest.
And I think that some of what we're finding, maybe we could be producing better evidence.
I'm arguing. So I put this up here.
I think you've talked about some of these things this week, right? I'm not sure.
But just from the ideas that you've given me so far, some of the ideas here, some of the ideas that you've brought up.
When we're designing something, what usually happens at the very beginning is what you saw a couple of PowerPoints ago.
Someone publishes some statistics or they do some screening, and so they raising a red flag that something's wrong.
And the initial red flag is usually something statistical in the world of health research.
That's where it starts. So the phenomenon of interest to begin with for a lot of health research is epidemiological
because we have statistics that show that something's not working properly or people are at risk.
Now, based on what that phenomenon of interest is, people take an epistemological stance.
So in other words, they say this is the problem as we see it.
So the problem is that let's say 26% of children are showing are at risk for diabetes.
Since this is the problem that this looks like there's there's a high prevalence
rate or or a high prevalence for being at risk of diabetes in the population here.
Then the answer to this problem, the way we're framing this problem is to get the rate down, to get the prevalence rate down.
And so the solution that they design is directly linked to the definition of the problem.
Now, if you looked at this different way, like you were just describing and said the problem is community food,
healthy choices, if you start there, then your epistemological stance is different, isn't it?
Because it's about saying, I'm framing the problem as a community problem, as a problem where maybe people can't make healthy choices.
And when I look at it that way,
my intervention is going to be different than it might be if I if I come up with another problem definition, if you see what I mean.
So one of the things we're arguing for here is that people need to be aware at the very beginning
of the design of a study that there are different ways of defining what the problem is.
And the way that I might define it as the research researcher could be very
different from the way that people in the community see the problem themselves. And it's one of the first things.
We should be asking when we read a research study who was involved actually in defining what the problem was, because it's like a domino effect.
You can end up with a study design down here. That was one that that was based on on my view of the problem as a researcher, it's a problem.
So why don't we do an A realist review? One of the ways that realist reviews and realist research in general is different from other
types of research is that context is one of the most important things in the research.
So if you look at a typical quantitative effectiveness study, the emphasis is on the intervention and the population and the outcome.
And the assumption is that the intervention will always be delivered the same.
And if you deliver it in the same way to the same population, you're going to be able to achieve a similar outcome, right?
So it's all based on generalisability transferability and what the realist point of view is arguing for.
The kind of circumstantial evidence we talked about earlier is I could have an excellent intervention,
but if the context is different than the intervention is either going to be more supported
or more constrained depending on what's happening in the in the surrounding environment.
So if we take this this bit about the community food here as an example, if I do the diabetes intervention in Boston, Massachusetts,
in one of the well-off areas of the city, big urban area, tons of food choices, lots of food competition in terms of cheap prices.
And I offer the same intervention to the Bad River tribe. It looks like the same intervention.
They're both getting education. They're both being taught how to make healthy choices.
They both have cooking classes. They're both they they both learn how to read the labels on on food before they buy it.
They both show that they become very well educated. We could do a knowledge test.
They could become well-educated about what type of food they should be buying, what they should be eating.
And at the end of the day, it's going to be less successful in the Bad River area than it is going to be in the Boston area,
because there's a big context factor there, which is money. It could also be time.
The map that I showed you earlier, you may have noticed there were no roads there.
I mean, this is a really isolated, really isolated community.
It's right out there on a piece of land in the Great Lakes.
To get to that to get to anything beyond the small community store is going to be about an hour's drive.
You need to have money for petrol. You need to have transportation so they don't have a lot of choices.
So can I ask you to put in a lot of choices and there's no roads with these.
So where is the the junk food, so to speak?
I mean, is it it's it's more what sort of food is it? Is it more sort of not as nutritious and sort of pre-packaged separately?
What sort of foods are there? I mean, I would have thought more rural, perhaps there might have been a bit more subsistence in terms of,
you know, in growing some of your own stuff. That's cool.
Mm. So what are the, you know, what are they unhealthy food or they're bringing in the unhealthy food.
So that's one of the problems because there isn't a good level of awareness in the community store about what foods should be brought in the stores.
Green. Yes. The community store says that the residents like that,
they expect that the unhealthy food is cheaper than the healthy food because fresh fruit and veg can be quite expensive in terms of growing food.
There's actually a limited amount of arable land there.
I mean, it's hard to see on this on this map, but if we go back to the site, you might be able to see a little bit of this.
It says bad river stuff over here. So this is a marsh area.
In this area, they used to grow wild rice, which is actually a gourmet food.
And, you know, you can, you know, in the United States, anywhere in Canada, people pay a lot of money for it.
But they had industry come into the area, polluted the water problems, growing the wild rice so that as a cash crop started to disappear.
They've been complaining about that for a number of years. So they lost a food staple there.
They also had a number of local berries in the area, fish, same kind of thing.
Industry moved in and they've had problems with pollution, so they're trying to fix that, which is another big contextual barrier.
And they're looking at establishing cranberry bogs because cranberries is a good cash crop and it's something that would definitely grow there,
but very boggy areas. And not only boggy, but the kind of bog area shifts, if you see what I mean.
So it's like marshland. You don't know from, you know, one month to the next where it's going to be safe to be walking, for example.
So they have a real challenge in terms of the in terms of the food supply, but they could get the healthy food in.
So these kind of contextual things are the things that you need to think about when you're designing the intervention.
The other thing is people, their attitudes and their behaviour. Of course.
What are their attitudes going to be if you change the food in the community store?
Will they feel like they have any control over that and those things can influence how you actually get your outcomes at the end of the day.
And that's what was happening in this in this area.
So we talked about this, the physical environment. So they went out and they interviewed people in the Bad River community.
They interviewed parents, community leaders, children.
And they looked.
They wanted to know what are the issues, not just about food, but about other typical advice that you would give people who are at risk for diabetes.
Healthy lifestyle, staying active. And the mother said it's not safe for our kids to play outside.
And there were a couple of different reasons for that. One was that they have terrible weather, terrible blizzards in the winter.
You've seen the lack of roads, very few paved areas.
And another reason that it's unsafe to play outside is because there's now a high
level of unemployment rate in the community and there is drinking and there's drugs.
And so there's crime also.
So parents are concerned about their children being safe and being able to supervise them and see them so that they're playing nearby.
So that's an issue. There are some play areas, but they're far away, like it says up here.
They like transport or they like the money for that,
for the petrol to take their children there where they lack the time if they're working to stay with the kids.
We talked about the supermarkets, but they also had an attitude toward health and what they were saying was they
didn't define health as somebody who was the right way or physically active.
They defined healthy as their children being happy and happiness meant things like they like watching TV, they like playing media games.
You know, some of this was about inactivity, not necessarily activity.
And so their priorities were keeping the children safe and making sure that they were happy.
And this is the way that they framed it.
And it didn't occur to them to frame healthy eating or physical activity as something that would make children happy or families happy.
It wasn't part of part of the way that they conceptualised things.
So if you put all this together and you think about typical interventions for diabetes,
there were many reasons why they weren't going to be acceptable in the community, if that makes sense.
Yeah. So can I just ask? I know it must have a school there somewhere where they have a gym.
They do some kind of protected place. They do.
And the teachers and the curriculum were set up so that they had maybe 15 minutes
in the morning for physical activity and the rest of the day was spent on other,
other things. So there was a lack of awareness on the adult level that was affecting this as well.
If you see what I mean. So there wasn't a lot of playtime or a lot of a lot of activity time in general, even in the school and wasn't happening.
One interesting thing is this is not rocket science. Not to change it.
It's just so difficult because we have exactly the same problems with our own Indigenous people in Australia.
I mean, there's multiple problems, but to fix them shouldn't be that hard.
But I think. Yeah.
And it's not rocket science. Anybody with half a brain could see.
Yeah. That an individual person couldn't fix it.
But we need to do to get the political support and the finances to do this.
Yeah. So we keep putting Band-Aids on it. Mm hmm. And is it because they want to fund one intervention rather than this?
Kind of. Yeah. Yeah.
It's the same for this kind of thing. And the only reason this get funded is because there was a policy window.
A window of opportunity.
And the window of opportunity was that there's epidemiological data now on all indigenous people in North America that has to do with diabetes.
There's a really strong case for it being a risk factor across all of the tribes.
Politically, the history goes way back.
But, you know, a lot of the ill health problems in the tribe were caused or contributed to by people losing their land.
So there was a political will to put money into this as a long term solution, not just a short term Band-Aid.
And they funded these programs, not just here, but in I think if I remember, 53 locations in the United States,
and it was funding for ten years, and they basically said, we don't know how to fix this.
So you get you'll have longitudinal funding to go in and figure out what needs to be done.
So it's kind of a luxury program from that point of view that they got to do this problem,
that we being paternalistic because we do have a problem of being seen as being
paternalistic if we were trying to make some major changes like changing community.
It's such a criticism that we want people coming in and being deterministic to the Indigenous community.
Well, the first thing that happened and I didn't work in this community and I've been in England for a long time now,
like 20 years, but I did work in indigenous communities and in British Columbia.
The first thing that happened was Centres for Disease Control realised that there was a problem with this paternalism.
So Centres for Disease Control had a condition on this funding which said you have to use participatory approaches to design the research.
They also said there is no way that we are going to fund any kind of research intervention unless it's been approved by the tribe.
We want demonstrated demonstrated evidence that you've involved the tribe at
every single step of the design of the interventions the tribe gets to sign off.
They had contracts, so the tribe controlled the money, not the researchers.
And the researchers were co applicants for the money.
But the application actually came from the tribe itself with the researchers skilling them up a bit.
So they had to start co-producing an intervention before they could even just apply for the money.
And they had to set up a good enough relationship so that they were all happy about the interventions before they applied.
Otherwise it was they weren't going to get the money. So the tribe came up with some ideas for interventions.
They were initiated by them, but the researchers contributed by coming in and saying,
based on what we know about diabetes research, these are things that have worked in other places.
Now they might not work for you, but it is what's worked in other places.
So they went through this process of giving them the ideas from other places.
The tribal council getting together and saying, I think that will work. I don't think that well, I think we could do this quicker.
This might take a lot longer. And so they kind of developed a stage programme of work and they started picking off
what looked like the things that the community would agree to the most easily first.
And they also didn't couch it as an intervention, like a diabetes intervention.
So, you know, you're mentioning the school. They they did a survey and asked at the school whether the kids wanted more games and activities.
Of course, people were saying yes. So they worked with the teachers and framed it out as being, you know, having more games and activities at school.
And of course, one of the side effects was that they said measuring the levels of physical activity to see if they went up.
So I guess you could say it's kind of an indirect intervention in a way. And with the healthy eating part of it, yes, they had cooking classes,
but they were things where they got the parents and the children together and they did crafts.
They had games for kids to play where they were running around in the gymnasium while they were doing the cooking class.
So they were active and they were showing them how to cook things together that people like to eat.
So it was it was all about more of a group activity or family activity orientation rather than an intervention.
And they were things that the tribe wanted to do. Did they actually have sports teams there because they didn't have to do something that.
Yeah. And they had them.
But it was the the thing of the kids who were good were on the team and then the ones that were overweight were kind of at the sidelines.
So they had to work on changing that, too. Yeah, it's critical that.
Well know. So to deliver that, they live it up and it was a requirement of the funding.
So every time that they wrote an article or did a presentation, they had to go back to the tribal elders and say, this is what we think we're finding.
This is how we think we should disseminate the results. What do you think?
And then they got comments from the Community Advisory Board, and they had to show in writing how they'd addressed each one of the comments.
So then the reason they did that is because in participatory research there's what's called levels of participation,
and the bottom level is informing. And the second level up is consulting.
And a lot of what happens is we say that we consulted with people, but really what I've done,
I hope not me personally, but is I've asked your opinion and I've nodded and I've said Yes, thank you very much.
And I've gone off and designed a study. But when you look at it, you can't see anywhere where I've used your opinion.
But I say I consulted you and they wanted to avoid that.
So they said, you need to document in your meeting minutes what the suggestions have been from the Community Advisory Board,
and then you either need to include them in both a research design and the way you implement the intervention and the way you write it up.
Or if you're not going to include it, you need to give a reason why.
And that was about maintaining trust with the community, because if you destroy that kind of trust,
then they're not going to let the researchers in anyway. And then the whole thing is scuppered in those situations where you have.
I guess, teaches history between two populations. And you're trying to.
It's critical that. Body or person is seen as someone.
Inside of the commission. I thoroughly understand.
Because the intervention could be connected with that population. Yeah, it's coming.
Potential threat to be rejected outright. You know, it's called deliberate because it can be seen as a tainted.
Yeah. And that was one of the big concerns.
Yeah. Just about anything you can do.
Yeah, but, um, those would.
And that's what they used here. Mm hmm.
Mm hmm. A lot of it's about credibility. I think it links to what you're saying about whether you're an insider or an outsider.
So if you get the community approving the idea, then the community gives you, as the outside researcher,
more credibility than you would have normally, because they've given that their stamp of approval.
But it's even easier if you have some insiders who are on your research team.
Absolutely. That are actually in the community. And there were two doctors.
One of the doctors was from this tribe and the other one was from this tribe that had worked with the tribe for decades.
And they both of them raised the red flag in the first place about the risk and the children.
So everyone paid attention to it and then they supported the ideas for the research and it made it go faster than it would normally.
So yeah, it started with the community part and went on from there.
I wanted to put this one up because this is one of the issues if you're trying to do a review of this because a systematic review,
you're normally including individual studies. So if I was doing an effectiveness review on a diabetes intervention,
I probably would find the randomised controlled trial, this one here by Alexandria Adams and her co-authors,
and I'd include that in the review if I was doing a mixed method systematic review, not a realist one,
but just a mixed methods review, then I might find the RC T by Adams and I might find this one because it's qualitative.
And then I would include those two in the vignette.
But the question we have and there isn't an answer for this yet, but the question is how many of these studies do you include?
If it's a realist review and your question is about what works, at what point in time?
Any ideas about that? I don't have an answer. I mean, we're trying now to include them all.
And we've actually invented a different way of searching for studies as a result of this.
And as Cheryl and I are trying it out with that with someone on another project.
So and and the case that we're making is if I only see the T, I don't have the history of how it was developed.
So what if that I asked, for example,
reported that it had problems with recruitment or what if the asked said we met our recruitment target of 200 families,
but over the two years that we ran the randomised controlled trial we had a dropout rate of 21%.
You know, should I be worried about that? I mean normally when you appraise one ICG, you would, you would make one decision about it.
But you know, is there a problem in terms of that dropout rate?
Who is it that dropped out? And are they the people that are really important in terms of looking at the effectiveness of the intervention?
So we decided that we would include all of these, and it looks like it's a lot of extra work to do that,
which means it's also, from a research point of view, a lot of extra money.
So when we included them, we said to ourselves, we're going to include them,
but we're going to ask ourselves whether there's any added value in looking at the entire family of studies.
We call called a cluster of studies to reconstruct the picture of the entire research project or is there not enough added value in it?
And what we actually found was that some of the things that actually were quite important to the act weren't reported in the paper.
They were reported in this one, this one, this one.
So that's about having a limited number of words that you can you can use in a journal article.
And what are you going to include as important information and what are you going to leave out?
And this study at the end, which we thought, oh, it's just the story of how the project was set up, it's not going to tell us anything new.
But because they wrote it after ten years of working with this tribe and they were looking back and kind of reflecting what they've learned,
they actually said some things that were quite important about why some of these interventions worked well and some of them didn't.
So we've left them all in. But now that we found this, that there's only partial reporting of important things in each one of these studies,
the question that we're left with is, is this happening in other areas of research?
And if it is, then are our systematic reviews really good quality or are they just giving us a snapshot of the research?
And the snapshot might not necessarily be telling us why it works, I guess.
Could you include the grant application in your view? Because it's essentially a to.
This could be essentially reconstructing you.
Are you reconstructing a program theory?
I hope you guys talked about program theories, because just to go over that really quickly,
when people start out designing a research study, they have an assumption, a set of assumptions.
And the set of assumptions are that this ought to work.
Okay, so you offer the intervention, let's say it's diabetes education,
and you're saying I'm assuming the diabetes education is going to work, because when I give it to this group of people,
their knowledge about diabetes is going to increase about diabetes,
risk their skills in terms of managing it are going to increase and their attitudes about it are going to change.
So a change in attitude. So these are your assumptions here, right, when you design it.
But what happens is you start out with these assumptions when you write the original research protocol, right, for what you want to measure.
But sometimes while you're going through the study, you end up with a learning curve because you're learning a lot more about these people.
And as you learn more about them, you're realising that some of your original assumptions maybe weren't quite right.
So your program theory starts to change. Even during the lifetime of offering the intervention and measuring it.
There's a shift in theory of change in what works and why it ought to work.
And we're thinking that in one single study that's published, you don't get the whole picture that go to those changes.
I think it's pretty depressing the research for that kind of changes. Go back to the grant reports.
Sometimes in this case they did because they had really comprehensive annual reports.
But some funding organisations don't ask for those. So, you know, it depends.
It depends on the kind of funding. Yeah. Yeah.
So it really just reflects the fact that so much research is so much is so compartmentalised.
Yes, it is. Especially, I mean, in this sort of work.
And you can imagine umpteen dozen studies looking at separate little areas.
And you're right. Certainly. And I have certainly. In response, rice.
But I mean, your first question, your first comments really sort of I mean, keep an eye on this sort of work in that.
It's knowing academic units and are encouraged now to follow through.
You're encouraged to do the and the funding. That's right.
Really? I mean, they yeah, they do have to find, you know, grants and a lot of personal studies to.
Grants from small grant. Want and need to find a study that they want to bring up that really, you know.
It's true. And it's a waste. A waste of money. How do you change it?
Changed it? We'll be talking about it in just. I think we change it the same way that we manage to change some things with Cochran,
because the Cochran Colloquium managed to point out that I don't know how many researchers,
thousands were basically doing the same study in different places, and no one was building on what was known before.
So the idea of systematic reviews came out of that and it's become a big industry.
I think we could change this by pointing out that or going back to some systematic reviews, I'm not saying all of them,
but going back to systematic reviews where they're reviewing interventions which were about
attitudes and behaviours where attitudes and behaviours are key and cultural acceptability,
appropriateness and re analysing some of what's in the review, using some of these perspectives to see if we can try and explain the variation.
And I think if we put together enough of those cases,
we could make a convincing argument for putting some more requirements on the funding in terms of what they'll find.
And I think we'll get there because the National Institute of Health and Care Excellence here in England, now,
they want to start funding realist review and they want to start funding it because they're concerned about about some of this.
And it's not just coming from places like the National Institute of Health and Care Excellence.
There are well, I can where I work in CHAS, there's a big randomised controlled trial unit and these people just churn out trials.
They've been doing it for decades and the people in this trial unit are saying, can we please have workshops on realist methods and mixed methods?
Because we're doing our trials and we know that we have problems with them.
And one of the huge problems is recruiting people into the trial.
And everybody knows that it's a big problem and they also know that it ends up being a major expenditure on some of your some of your assets because
it's difficult to get people to join and they're starting to realise that if you use some of these methods it's quite easy or not quite easy,
but it's easier to get involvement in the trial than you would have originally.
So I think that there's going to be some opportunity to go back and look at some of these things.
And the other the other driver, I think, is and I chose diabetes as the example of on purpose.
If you go in and you search for diabetes interventions, there are thousands.
There are hundreds and thousands of them around the world.
And if you bothered to do a mapping exercise and categorise them by the type of intervention,
there are hundreds and hundreds of thousands that are all the same diabetes, education intervention.
So it's like people are taking the same educational program and reproducing it in very
different places and nada and not saying it works in some places and not working in others,
but they're not putting those results together to look at why it's working in some places and not others.
So. So it's obvious that something needs to be done in terms of managing the money more effectively, I think.
And it is now the problem that successful ones get published in the others.
Yes, it is. And that was the other. I'm glad you brought that up, because that was the other reason I put this up here,
because when we first searched for these, we use kind of traditional methods for searching for articles.
And of course, we found the ones that were published in the big peer reviewed journals like this one.
And we found this one because there's surveys here and and this one here came into the realm of quantitative clinical type research.
And so they were published in well-recognized journals. But then we started using this method of searching where you're looking for
what these articles referenced both forwards in time and backwards in time.
And we also looked for any mention that they had in any article of doing any kind of research,
you know, and in another part of this project, which again, takes time.
But when you look at the order in which they're published, they weren't published in this order, even though this was the order of activity.
And the reason they weren't is because the journals aren't interested in some of this stuff.
So this thing really should have been published a lot earlier.
You know, this was the account of how they set up the relationship with the tribal council,
how they got ideas from the Tribal Council about about what kind of interventions to have in the community,
the kind of how they establish an equal power balance in terms of relationships and working together that
should have come up here somewhere instead of way down there and the perceptions of the environment.
Again, that work was done a lot earlier and it was key down here.
It actually happened down here and helped them decide what interventions to have. But it was qualitative, so it wasn't published until years later.
So the publication pathway was was interesting and it's I think it's a form of bias and some of them were in very obscure journals.
So anything that had to do with the participatory approach,
we had to order those those articles on interlibrary loan even though we have access to a huge library.
You couldn't get them. So, yeah, it's.
The war hasn't changed.
It's gotten better. But not all the journals are open access yet, and some of them are not.
They just don't set up as open access. You know, these were in journals like The Journal of the Poor and Underserved, for example.
And, you know, it's just not a big journal, so a challenge.
So that's our challenge right now. And we are going to include other studies this time because we think that there's there's
it's a worthwhile exercise in terms of looking at whether it should be done before.
I don't know whether it would do it for everything, but we will for this.
So the kind of round up the story, once we looked at this and found a whole family of articles, we said, well, maybe we have some other families.
So we went back to the 135 articles that we thought were relevant for the review, and we found nine other families.
So there are nine other pilot projects out there that have published between three and ten articles on their work.
So we have we can reconstruct what happened in the in them.
So the point that we learned and this is something that we've learned with the last peer support review, but it's actually coming up here, too.
There's a couple of authors in health promotion called Townsend Green,
and they've said that you can find two different types of intervention in health, and you find this in public health, too.
And they call them authoritarian and negotiated. And they don't necessarily mean that in a negative way.
I mean, if you think of, for example, disaster relief situations, you wouldn't want to negotiate what you intervention was.
You just need to take an authoritarian stance and go in there and use your expertise and do something.
But basically, the authoritarian intervention, for whatever reason,
gets designed by experts outside of a community, and then it's brought into a community.
Sometimes it's called community based, but it isn't actually community based.
It's brought in and it's done to people try to reduce risk or improve health, whatever.
And the other side is the negotiated ones, which is what this bad river tribe one was, where the community itself is designing what needs to happen,
and then they're doing it on themselves or with themselves rather than having the outsiders come in and do it.
Now, I said that one wasn't necessarily worse than the other, depending on the situation, but for the peer support review,
we actually found a pattern because peer support is about getting people connected to other people,
getting them connected to the community, helping them build social networks so they can manage their own condition.
So it's very much a kind of behavioural and emotional type of support.
And in that review, the interventions that were authoritarian were less effective than the ones that were negotiated.
And we we questioned that. We said, maybe we're looking for that.
So then we looked at them by stage because first you have the design, the intervention stage,
and then you have recruits to the intervention, and then you have actually delivery and you have it.
And there were some interventions that started out really authoritarian, so outsiders designed them,
but then they went into the communities to do them to the community and they realised they were going to have a problem commuting, recruiting.
So they changed to a more negotiated stance and got the community to help them recruit and it started to become more successful.
And then they got to the point where they had the right number of people in the study and
they took the control back from the community and told them how to deliver the intervention.
The people dropped out and became less successful, so there was a pattern there, no matter how you looked at it.
So we're also suggesting that this is something you probably have to look at with some
interventions because it affects them all the way through the audience that that you mentioned.
It was Tones and green.
They wrote a health promotion book.
I can't remember the exact title of it right now, but it was actually reissued with the authors in the opposite order.
And I think that the last edition was 2010, if I remember right.
So this is one of the interventions they ended up with at the end of the day. And just to finish off the story, what did they do for interventions?
Because it was swampy land, difficult for the children to play outside.
They built boardwalks above the ground. So they had like marsh walks, forest walks, and they built them like like exercise or activity trail.
So that was one thing they did. They negotiated with the teachers.
They built more physical activity into the day. The teachers were concerned about them losing some of their study time,
so they came up with activities where they were learning at the same time as they were doing something active with the healthy eating.
They negotiated with the community store to get different foods in and the council elders talked to people in the community and decided how they.
Wanted to shift the balance between the unhealthy and the healthy food that was coming into the community.
We kind of agreed on changing some of the food that was brought in, in that kind of culture.
They really respect what the elders are saying. So if the elders said something was going to be a certain way people are, people would tend to try it.
And they started putting out newsletters with recipes in them.
And in each one of the newsletters, the elders had recommendations that said the elders are saying and they have some ideas for people to try.
They didn't call it diabetes. They called it healthy families, strong children as a more neutral kind of kind of set of words.
And the list goes on. I mean, it was multiple interventions at the end of the day.
Was there anything specifically targeted toward conception and how?
Yes. And what they did was they used this study here,
and the conclusions that came out of it was we have to talk to parents about how activity could fit in with happiness and health.
And so they came at it from the point of view of fine of being together.
So they reframed it that way and they kind of used the and they found this in the research, actually.
They said, if you're telling people children to eat healthy food because it's good for them right away, you're putting them off.
So rather than coming at it by telling people to be active because it's good for you, we need to come at it from the point of view of, you know,
what makes them happy, what are enjoyable activities for them, which again, it's not rocket science, but that was the way that they framed it.
So at the end of the day, children aged 2 to 5 are now on the way.
It went down by 8% and parents lost weight, which wasn't the original intention, but they did.
And the parents have maintained the weight loss now over five years. So it's working.
It's slow. I mean, you were saying that. But but it is working.
It's you know, it's there's a downward trend in that part of the world and.
Yeah, yeah, yeah, yeah, change it.
And the other clusters we found, the other studies that had families of studies together,
they used the same approach and they're coming up with the same thing. So if you look across them, it's working.
So that's the story of one one controversial thing about mixed methods.
And how do you and I guess what I'm saying is when you're combining them, I think we need to question where the study ideas originally came from,
because if I'm combining a study that has that authoritarian stance here,
if I'm bringing that into my review and I'm bringing in another study that seems to have the same intervention,
how am I going to know whether they both or I'm not being very clear here.
If I bring two diabetes education studies into my review, they both said they did the same educational program,
but they have nothing about the kind of stance they used. How do I know that with the same intervention?
I don't. And I think that they are there might be an issue there.
You know, one of them might be more acceptable than the other.
But we have no way of knowing, so that the intervention has other trappings that are actually more important than the intervention,
the context, because there's an interaction between the context and the intervention.
And that interaction is how was it designed in the first place and what was
the process of of offering it and implementing it and making your job easier?
To do this we need to be ensuring that is.
Insistence that contextual information is included.
Yeah, online, not necessarily printed version.
But at the moment we're so restrictive with words that really important contextual information just cannot make it into Python.
More post about which statistic can be used in the really important stuff.
Well, I would say they're both. They're both important.
But maybe there's something about changing the guidelines a little bit, the criteria for what you're reporting in your study.
Because right now, when we look at articles, you have an introduction or a background section to the article.
And that background section traditionally talks about research that's been done somewhere else,
and you use that to justify your approach and also to identify the gap in the research.
But maybe there's some way of asking people to make that fit a little shorter so that you can use some of the word count
there to describe the context so that the introduction would actually look like research has been done that says so-and-so,
there was a gap in this research and this is what I think we need to do.
But then follow that before you get to the research question with this is the context we'll be working in.
And based on this context, this is what we think we need to be doing to to adapt the intervention.
I think it needs to be. Published which journal?
All of them, because one part of that debate has started.
So Paul Glasgow and his group in Australia bond they they're doing a big push on describing the intervention.
Okay. So they're doing a big push on actually making, making explicit your interventions because they did a they did a quick,
a quick, quick and dirty systematic review and found that I think don't quote me, but it was almost half of them.
Could it be? Could it be? Couldn't be implemented just by reading the article.
Yeah. And then they went to the author and they got some more information for another sort of 20 or 30% of that.
So they're putting a lot of effort into trying to they've actually developed a framework called the Tidy a framework.
The first author is Tammy Hoffmann, and that's published this year, I think.
And so they they're saying that that should be used alongside the consort guideline for describing the actual intervention,
the behavioural intervention group does it, you know, what training they have,
where they're getting a little bit into context, but it's predominantly on the intervention,
but they're starting to think about, yeah, who does it, how much training and what tools are used, what resources.
So that that's one step, but it's not.
The universal contract negotiating stuff is really important because one of the reasons I understand
that Australia was very successful with the HIV AIDS epidemic was that the whole program for what do
we do about this was done at the level of we saw in in association with the most at risk community was
as I understand in the US it was more of an authoritative approach to an intervention and we had,
we did really well really quickly. And I think so I think that that is very important.
We do really know there is some kind of barrier to it.
I don't know if it's about too much challenging the way things are done right now.
Um, and I'm saying that cause there's something in England called the Schools for Public Health Research,
and it's a consortia that's was set up by the government of seven universities in the
UK because they said we have to start working together on public health research.
So we had a meeting in Sheffield about a month ago and I brought this up and I described what we were finding out and the people in this,
in this group are international researchers, so they have long track records.
And the only one of these people was saying we should make this a priority for the school.
And the rest of them were just, you know,
it's not not fitting in with the funding priorities or it's almost like it's it's it needs to be introduced more gradually, I guess.
I mean, none of us want to think are the interventions we set so high designing are inappropriately authoritarian, you know?
So I think it's a hard thing to get your head around and I think up to something here,
but there's been a huge push towards transparency in research and you're trying to grab that from the context.
You need very sophisticated to find out and even to measure, you know, exactly what you're going to do at any stage.
And then you get to say, well, this has to allow flexibility.
Yeah. And then the transparency becomes a very difficult thing to achieve because you're introducing complexity,
which of course we all know happens in our research project. So we're on the right way.
Yeah, the straight line. But yeah, the kind of conflict it is, it is.
And it's really hard to write the grant applications for that reason because
if you have people that are experienced in reviewing our safety protocols,
reviewing a grant application,
they come back and they tell you you're being too vague and they don't understand that there's a reason that you're doing that.
So it's hard. So yeah, I don't know what direction it will go in.
I know that more of the the community based participatory stuff is being funded than it was before.
So that's good news and more of it is making its way into mainstream health channels.
So that's also good news. But who knows, can we get funding from the actual funding bodies?
Because sometimes we'll top 10% for monitoring purposes and to talk to an academic to come out to do this kind of monitoring.
According to a framework. A new report seems to show that they're targeting the outcomes of their.
And that can lead to papers. Surely the funding would be very much in line with.
Proving. Think so.
I'm thinking about that because that's an idea, isn't it?
So I did this with a few years ago in Scotland.
Yeah. Funded by the European Social Fund. Yeah. Set a target to meet, like, youth engagement and so on.
And then we gave the money to local contacts.
Yeah. Ask for funding. And then the four projects were giving us every year.
And then they were great topic. And I never thought that I created all of these reports.
And that sounds funded by the top.
Mm hmm. Mm hmm. Successful?
Mm hmm. And we didn't have to worry so much about. Research Councils.
Yes, that's a really good idea because then you're in a position to not only it,
but review what they're doing and make some make some observations, like from a methodological point of view too.
The outputs part is interesting because what's coming up with some of these things now is that the outputs and the outcomes, like I said earlier,
the people value or that they're achieving, sometimes they're not the ones that were originally set by the funding body,
but the participants are saying they're more meaningful.
And so I think there's a piece of work that needs to be done there, particularly around social outcomes,
because I know in the peer support review they said for a lot of these health outcomes,
people need to have social outcomes first before they can get to the health outcomes.
You know, people that are isolated need to be more socially connected before they can manage to give up smoking or or be more physically active.
And why aren't we measuring those? And now some of the organisations we're working with are saying we want money
to measure those first and then we'll show how they're linked to the health.
But right now we have this kind of black hole in the middle because those intermediate outcomes are being are being collected anyhow.
So thank you for listening to the challenges that we're having.