1
00:00:00,060 --> 00:00:07,450
Basically, what have I been doing? I have like 20 years experience in global health policy and programs.

2
00:00:07,470 --> 00:00:11,790
I've worked for UNFPA, I've worked for Marie Stopes in Belgium,

3
00:00:12,240 --> 00:00:18,720
and now I'm a senior lecturer and research fellow at the Institute of Tropical Medicine in Antwerp, Belgium.

4
00:00:19,170 --> 00:00:31,680
And Jeff has been so kind to revive me and sort of we've been walking this roads of realist evaluation from different sides, but sort of together.

5
00:00:32,380 --> 00:00:34,170
Yeah. First,

6
00:00:34,770 --> 00:00:47,480
I wanted to add we were just a bit reflecting on it on that journey to to explore with you a bit why we came to such a thing like realist evaluation.

7
00:00:47,490 --> 00:00:50,680
And I don't know if, you know, maybe Jeff has said it.

8
00:00:50,700 --> 00:00:55,620
It comes originally from, I think, public policy sociology.

9
00:00:55,690 --> 00:01:06,000
Ray Boston is a sociologist in Leeds and he first used it, I think on the CCTV, the analyses of CCTV cameras,

10
00:01:06,330 --> 00:01:11,860
so completely different ballpark than what we have today and how we apply it today.

11
00:01:12,960 --> 00:01:20,850
And why did we come to it? So around the year 2000, we were a bunch of of doctors, let's say,

12
00:01:20,850 --> 00:01:28,530
and also social scientists working on health programmes in low and middle income countries, primarily sub-Saharan Africa,

13
00:01:28,530 --> 00:01:37,710
because that's where Belgian of funds, mostly healthcare programs, we were doing policy and programmes on quality of care,

14
00:01:38,040 --> 00:01:43,740
also teaching it on district health management, on the interface between programmes as services.

15
00:01:44,340 --> 00:01:55,860
And we were using methods like participatory more or less action research at no fee in an institute that was primarily biomedical,

16
00:01:55,860 --> 00:02:05,220
a bit like here biomedical oriented institutes where the main paradigm was our city, our cities were considered the golden standards.

17
00:02:05,610 --> 00:02:13,350
And we had a lot of people epidemiologist out there who didn't really recognise, let's say, social science methods.

18
00:02:13,950 --> 00:02:20,460
So at that time we had a growing unease about the methods we're using and also the effects we were having with this.

19
00:02:20,970 --> 00:02:29,340
And we sort of had the feeling that we were caught between a rock and a hard place, in a sense, against or not against.

20
00:02:29,340 --> 00:02:40,319
That's sort of between the cities, which didn't cover our expectations, and then the very soft, maybe Rock of Action Research,

21
00:02:40,320 --> 00:02:48,990
where we were doing completely something else and a big rift between those methods of quantitative side and the qualitative side.

22
00:02:49,800 --> 00:02:57,030
So we were looking for new methods at the time and looking for methods that sort of would fit our

23
00:02:57,030 --> 00:03:03,030
idea of the complex reality we were facing of health programs in low and middle income countries.

24
00:03:04,260 --> 00:03:12,600
This was that small minority group, two institutes, and we are sort of 300 people overall.

25
00:03:12,600 --> 00:03:18,989
These studies of tropical medicine and these six or seven were discovering real evaluation at the time,

26
00:03:18,990 --> 00:03:24,840
and I was a bit younger in a bit, and we weren't the only ones in clinical medicine.

27
00:03:25,320 --> 00:03:30,330
There were a bunch of people, including, of course, Richard Greenhalgh of Oxford,

28
00:03:30,810 --> 00:03:37,650
Jeff's colleague, who were looking at or vouching for the idea of complexity,

29
00:03:37,650 --> 00:03:47,760
thinking this would reflect much more of the reality of what was happening in health systems, the coevolution with other systems that was happening,

30
00:03:47,760 --> 00:03:53,400
and the idea of health systems that were basically made up of relationships in social systems.

31
00:03:54,180 --> 00:04:04,980
She wasn't the only one. Also, David Koenig was advocating for this, and the cities were increasingly being questioned for complex interventions.

32
00:04:05,910 --> 00:04:12,750
But you could argue you could argue that the uptake in public health was relatively slow.

33
00:04:13,290 --> 00:04:23,759
So in around 2002, I think was this Popularising paper funded by Canada, which makes the distinction between simple,

34
00:04:23,760 --> 00:04:29,430
complicated and complex systems, which was a big help in teaching about how systems work.

35
00:04:30,030 --> 00:04:35,370
We were teaching this to public health doctors from sub-Saharan African countries,

36
00:04:35,850 --> 00:04:39,420
and we needed to have an entry point that this was a very helpful thing.

37
00:04:40,230 --> 00:04:50,790
Then I think Margaret Chan was vouching for the idea of systems at the time at the W.H.O. and you get this this important, impressive report.

38
00:04:51,330 --> 00:04:59,850
And then also at the same time, 2008, 2009 is the MRC Council guidance, who finally admits.

39
00:04:59,910 --> 00:05:05,760
That there's something like complex interventions, but not mentioning real estate, by the way.

40
00:05:06,420 --> 00:05:12,480
Still, certain certain methods, especially from the social scientists,

41
00:05:13,080 --> 00:05:21,300
scientists have found little entry points into into public health and into global health policy and practice to this day.

42
00:05:21,690 --> 00:05:28,260
So realist evaluation in that sense is sort of almost an exception because if you would look at QCA,

43
00:05:28,560 --> 00:05:32,970
it's just based modelling is more already quantitative process tracing.

44
00:05:33,180 --> 00:05:36,240
These things are still quite under-used.

45
00:05:36,450 --> 00:05:44,410
In my opinion, you could see that COVID 19 would be a big game changer.

46
00:05:45,240 --> 00:05:51,300
Maybe the jury is still out there on that. We're still waiting a bit, I think, for that big game change.

47
00:05:51,930 --> 00:06:03,420
But if there was anything that showed that there was, for example, co-evolution between health to political and economic and social systems,

48
00:06:03,810 --> 00:06:13,290
or where there was a lot of confusion and a lot of unpredictability and emergence of bottom up action, then it's it's COVID.

49
00:06:13,680 --> 00:06:23,670
So and indeed, to chagrin has made this case together with a bunch of others, but still, I'm not sure if we are still already there.

50
00:06:24,660 --> 00:06:31,500
But if we look at ourselves as the realist evaluation team, for example, in my institutes today,

51
00:06:31,500 --> 00:06:39,300
we have really grown from those six researchers who were really in a minority to quite

52
00:06:39,900 --> 00:06:46,500
a big network of people that are working on it that have formed an organic network,

53
00:06:46,500 --> 00:06:50,970
let's say. So maybe Geoff already spoke about Rhamnosus.

54
00:06:51,600 --> 00:06:55,220
So we have also a small network next to that,

55
00:06:55,230 --> 00:07:02,790
and that's something spontaneous of postdoc researchers who are also working together on real estate reviews.

56
00:07:03,330 --> 00:07:11,220
And next to that, we we are developing currently an E course with the IPH Institute of Public Health in Bangalore, India.

57
00:07:11,790 --> 00:07:18,480
So there are more and more people out there, and it's sort of spreading like the ink blots.

58
00:07:20,100 --> 00:07:30,480
Why is that? I've wondered a lot about how is that possible that it has found such and such an audience in global health?

59
00:07:31,500 --> 00:07:44,490
I think there's a lot to it, to the fact that it uses theory and it opens the worlds of theory and a lot in discussions in our institute, for example.

60
00:07:45,150 --> 00:07:50,940
We have this false dichotomy often between action and theory.

61
00:07:51,210 --> 00:07:55,290
Either you were an activist and you were. Come on, let's do things.

62
00:07:56,340 --> 00:07:59,520
Let's do things quick. And then you had two people who say, no, no, no.

63
00:07:59,520 --> 00:08:02,520
We need to look at the state of the art. We need to look at theories.

64
00:08:02,520 --> 00:08:12,839
The lab and sort of realist is sort of dismissing this because programs are seen as theory incarnate by by person.

65
00:08:12,840 --> 00:08:18,809
And Chile and it's this also and that I think is also an important point.

66
00:08:18,810 --> 00:08:30,890
And I hope that that gets stronger with the time in methodological development of realist that it enables learning across programs through theory.

67
00:08:30,900 --> 00:08:38,280
You could link what has been happening on HIV service delivery models in South Africa to Nigeria and so on.

68
00:08:38,550 --> 00:08:43,050
And you built, you know, so what's theory now?

69
00:08:43,410 --> 00:08:52,160
Just one caveat. There are people who would argue that the and those are the medical anthropologists.

70
00:08:52,170 --> 00:08:55,440
I'm a medical anthropologist, also a political scientist.

71
00:08:55,500 --> 00:09:04,080
Really public health that this is not new like, okay, we have been doing this all along, they would say.

72
00:09:04,920 --> 00:09:12,479
But the big difference is, I think and the attraction for biomedical people or people in the biomedical world

73
00:09:12,480 --> 00:09:19,170
is that it offers clear guidance and the research cycle is being made explicit.

74
00:09:19,950 --> 00:09:30,330
Medical anthropology does take descriptions, contextually rich descriptions of situations, but there it stops.

75
00:09:31,050 --> 00:09:34,560
And anthropology is also more like the craft.

76
00:09:34,710 --> 00:09:36,180
It's a certain community,

77
00:09:36,180 --> 00:09:45,990
and it's not necessarily that open or it didn't used to be that open to to other types of researchers into interdisciplinary dialogue.

78
00:09:46,440 --> 00:09:51,300
This is an opinion that I wrote down in an anthropology journal, by the way.

79
00:09:52,470 --> 00:09:59,780
This is a bit the second part of my talk. And here I wanted to show you some examples of people who have done also the.

80
00:09:59,910 --> 00:10:03,840
Course. So Monica Martis from Belgium.

81
00:10:04,230 --> 00:10:09,450
And I think if the lawyer or you you want from Nigeria have full discourse and are

82
00:10:09,450 --> 00:10:15,780
currently doing real estate evaluations and pragati by the practice in India,

83
00:10:15,790 --> 00:10:19,470
Wellcome Trust Fellow at the Institute of Public Health in Bangalore.

84
00:10:20,250 --> 00:10:25,260
And I will show you examples from health service delivery programs too.

85
00:10:25,620 --> 00:10:36,690
And then also a bit more programs that are about upstream policy processes, which are where one does not really use real estate evaluation.

86
00:10:36,850 --> 00:10:40,919
My opinion a lot, I think these are the key principles.

87
00:10:40,920 --> 00:10:49,530
I hope they sort of correspond to what Jeff has been saying and this is the cycle that we are using.

88
00:10:49,920 --> 00:10:57,750
You will see some real cycles applied. This is sort of the start of that of that cycle.

89
00:10:57,780 --> 00:11:05,940
This is IBI lawyer or lawyer one he is currently doing in the face of tyranny, refinement and testing.

90
00:11:06,330 --> 00:11:17,160
But he has done a real estate evaluation on community based antiretroviral treatment delivery programs for specific groups in Benue State, Nigeria.

91
00:11:17,880 --> 00:11:24,000
Benue State is a hard policy environment, let's say, for this key population groups.

92
00:11:25,140 --> 00:11:31,860
It's specific for men having sex with men, female sex workers, injecting drug users and transgenders.

93
00:11:32,310 --> 00:11:38,610
And in that state, there's harassment. But it's also illegal that these activities are, of course, illegal.

94
00:11:39,270 --> 00:11:45,360
So the question is, the resource or evaluation question that he asks is what are the mechanisms

95
00:11:45,360 --> 00:11:50,100
and context conditions that drive successful community based implementation?

96
00:11:50,490 --> 00:11:55,740
And how do this lead to better retention in care treatment adherence, a viral suppression?

97
00:11:56,520 --> 00:12:06,240
This is a program that is funded by the fair, is being executed with the state government, and it's an NGO basically that is doing it.

98
00:12:06,630 --> 00:12:14,610
He has worked in that program before. He came to the institute to do a PGD and this is his cycle.

99
00:12:14,610 --> 00:12:21,300
And so he has adapted the cycle. He goes to identify the program theory based on a scoping review.

100
00:12:21,870 --> 00:12:31,470
And then you're already saying that is the seed testing with the mixed methods design and then you have data analysis using.

101
00:12:31,650 --> 00:12:35,490
I see. AMOS So some people you see among other use,

102
00:12:35,490 --> 00:12:44,460
I see a more so the CMO is context mechanism outcome I see AMLO's Intervention context actors mechanism outcome.

103
00:12:44,880 --> 00:12:51,540
It makes it sometimes easier if the program is really oriented towards different types of actors.

104
00:12:51,540 --> 00:12:56,790
That is what I would say. The idea is to evaluate the effectiveness of this program.

105
00:12:56,790 --> 00:13:06,839
It's an impact evaluation plus a realist. It understands why and how and he is also building and that is a specific thing or a very nice thing.

106
00:13:06,840 --> 00:13:10,559
I think he's building on the knowledge of other programs.

107
00:13:10,560 --> 00:13:22,080
So we had a real estate evaluation that we already followed from 30 coming from South Africa on antiretroviral treatment and adherence clubs.

108
00:13:22,080 --> 00:13:25,230
And he has used that for the development of the initial program.

109
00:13:26,280 --> 00:13:33,720
So sort of learning across this is the way he's developing his program theory, it's his graph.

110
00:13:34,170 --> 00:13:38,430
He starts us with a lot of programs still with a logic model.

111
00:13:38,430 --> 00:13:47,999
That's what they have in the beginning of the program. Then he has his own professional experience he's doing in the interviews.

112
00:13:48,000 --> 00:13:52,860
He's doing a scoping review and is looking at the state of the art.

113
00:13:53,400 --> 00:13:59,520
He then extracts the ICM configuration and then he develops initial program theory.

114
00:14:00,270 --> 00:14:04,620
I'm showing you this as an illustration just how it is done.

115
00:14:05,340 --> 00:14:08,370
This is sort of giving an overview.

116
00:14:08,970 --> 00:14:12,020
Yeah, you can show it in different ways. He Yes.

117
00:14:12,120 --> 00:14:21,240
Showing it in different ways and backed it up with evidence in a paper showing interviews, citations and different sources.

118
00:14:22,260 --> 00:14:33,360
The context, the mechanisms and the outcomes and the arrows between are sort of the relationships and configuration or hurdles that you're using.

119
00:14:33,360 --> 00:14:38,280
So the arrows are very important and are in a narrative being explained.

120
00:14:39,230 --> 00:14:46,910
You sort of synthesises this into an initial program theory, which is this, and there's a narrative also to it.

121
00:14:48,270 --> 00:14:56,670
The straight arrows are direct effects. These are the the how you say curved ones are indirect effects.

122
00:14:57,120 --> 00:15:02,950
It's looking also at different layers of context, especially zooming in.

123
00:15:03,540 --> 00:15:08,630
And that is, I think, very important that you select also what you are looking at.

124
00:15:08,640 --> 00:15:17,370
What is salient are the different types of policies and so on that have an impact on the practices of those groups.

125
00:15:18,750 --> 00:15:23,190
This is the narrative form. Here you have context.

126
00:15:23,820 --> 00:15:31,730
Here are the mechanisms. Trust and psychological safety are, of course, very important in that environment.

127
00:15:31,740 --> 00:15:36,950
That is not friendly, let's say. And you're also it's about trust.

128
00:15:36,960 --> 00:15:42,540
And here you see the seed, the intervention written up.

129
00:15:43,740 --> 00:15:54,059
Now, the phase that he's currently in is refining and just seeing the program theory with two phases a quantitative and qualitative analysis.

130
00:15:54,060 --> 00:16:05,050
He has done already a retrospective cohort study on retention, and his has done three case studies, and now he's exploring across case mechanisms.

131
00:16:05,070 --> 00:16:09,660
For example, there is the emergence, let's say, of inter-group solidarity.

132
00:16:10,020 --> 00:16:17,549
So those groups, for example, men having sex with men and the sex workers are also supporting each other.

133
00:16:17,550 --> 00:16:23,220
Let's see, then the next one is a bit more upstream.

134
00:16:23,220 --> 00:16:26,400
And this is about use of policy dialogue.

135
00:16:27,210 --> 00:16:37,710
It is an idea to scale up our scale up integrated chronic care in three countries Cambodia, Slovenia and Belgium.

136
00:16:38,310 --> 00:16:49,500
She's looking at how the use of policy dialogue would lead to a better implementation of integrated chronic care policy.

137
00:16:49,710 --> 00:16:55,620
Okay, so it's really about the policy instruments and how that would work in those three countries.

138
00:16:56,610 --> 00:17:02,549
The sources that she's using for the development of the program theory is a literature

139
00:17:02,550 --> 00:17:08,850
review on the role of policy dialogue and also theory theory coming from political science.

140
00:17:09,360 --> 00:17:13,290
So here you get a sort of totally different ballpark.

141
00:17:13,770 --> 00:17:20,730
And she's integrating really with political science theories to make sense of what is happening in those three cases.

142
00:17:21,790 --> 00:17:31,390
The challenge of this kind of program or of broader public sector reform or public policy programs, broader than just a service delivery,

143
00:17:31,900 --> 00:17:40,120
is that you're mainly very much have to look at those macro level mechanisms and how dose

144
00:17:40,840 --> 00:17:48,670
interacts with other levels of of of mechanisms at the relational organisational mechanisms.

145
00:17:49,000 --> 00:17:51,340
And it's a micro at the individual level.

146
00:17:52,030 --> 00:18:05,019
This is quite a challenge and I think a bit more difficult to do than sort of the relatively closed systems still maybe of health service delivery,

147
00:18:05,020 --> 00:18:11,290
although they're not closed. But here you in a huge thing here also.

148
00:18:11,500 --> 00:18:13,120
What is she trying to do here?

149
00:18:13,480 --> 00:18:24,070
She's trying to identify mechanisms and context conditions based on the policy face that the different countries are in.

150
00:18:24,100 --> 00:18:30,309
So here you also see I wanted to show you that to see sort of the creative use of

151
00:18:30,310 --> 00:18:36,340
different theories like policy analyses to make sense and to search for the mechanisms.

152
00:18:36,970 --> 00:18:41,060
And the idea is very important that you can do things creatively.

153
00:18:41,680 --> 00:18:45,010
This is something that she has developed.

154
00:18:45,610 --> 00:18:48,900
This is the program theory development she's using.

155
00:18:48,910 --> 00:18:59,200
If then because statements, that's a sort of practical suggestion you can use to really keep your eye on the ball in terms of causality.

156
00:18:59,860 --> 00:19:03,160
And here you have the context, the outcome and so on.

157
00:19:04,770 --> 00:19:09,900
For the refinements. She's at the same stage as Ebola in Nigeria.

158
00:19:10,740 --> 00:19:16,380
She will do in-depth interviews, documents, reviews and also workshop.

159
00:19:17,160 --> 00:19:21,930
And at this moment, she has is looking at the different country cases.

160
00:19:22,500 --> 00:19:25,880
And what you get in this type of thing.

161
00:19:26,130 --> 00:19:34,620
Yeah. A policy of program, if you evaluate you, gets quite different mechanisms, let's say,

162
00:19:34,650 --> 00:19:38,690
based on the understanding, based on the path dependency of each country.

163
00:19:38,700 --> 00:19:44,580
So each country is taking it on a completely different is on a completely different path.

164
00:19:45,060 --> 00:19:52,530
Belgium, very like the political culture that we have, is looking at or what works.

165
00:19:52,530 --> 00:19:59,399
There is a stakeholder negotiation, and negotiation between stakeholders will lead to better uptake.

166
00:19:59,400 --> 00:20:03,870
Cambodia. The experts play a very important role.

167
00:20:04,350 --> 00:20:11,880
And in Slovenia, its patients peer support groups that play an important role sort of co-creation.

168
00:20:12,150 --> 00:20:18,560
So you get totally different ways of implementing this according to the country context.

169
00:20:19,020 --> 00:20:23,400
Last one is the implementation of tobacco control.

170
00:20:23,700 --> 00:20:34,560
Also big program in terms of this is the control of tobacco control, the Tobacco Products Act in India.

171
00:20:35,670 --> 00:20:40,050
It's from 2003. And then you have the National Tobacco Control Program.

172
00:20:40,650 --> 00:20:47,970
And the idea is to understand how states who have the mandate to implement tobacco control.

173
00:20:48,270 --> 00:20:54,570
It's a federal system. It's decentralised. How are they implementing tobacco control policy?

174
00:20:55,610 --> 00:20:58,070
This is very real, this cycle.

175
00:20:58,940 --> 00:21:08,960
She has done a real synthesis to develop the IPT to look at tobacco control implementation specific for low and middle income countries.

176
00:21:09,920 --> 00:21:13,820
And she has also survey data that she had.

177
00:21:14,480 --> 00:21:17,810
And on the basis of that, she's selecting three states.

178
00:21:17,960 --> 00:21:21,830
She cannot possibly do all the Indian states and look at implementation.

179
00:21:22,790 --> 00:21:27,500
She has done interviews and focus group discussions.

180
00:21:28,280 --> 00:21:39,270
She is extracting CMO configurations and she will take these CMO configurations to the policy makers of the different states to validate them.

181
00:21:40,180 --> 00:21:48,520
This is how she portrays her CMO's, so it's all quite different.

182
00:21:49,660 --> 00:21:53,590
This is an intervention strategy, intersectoral coordination.

183
00:21:54,190 --> 00:21:58,180
Here we are. This is not an intervention strategy.

184
00:21:58,210 --> 00:22:05,750
This is tobacco industry interference and is sort of a countering implementation strategy.

185
00:22:05,770 --> 00:22:10,930
So it's an actor countering the implementation of the Tobacco Control Act in India.

186
00:22:11,200 --> 00:22:15,760
So she's not only looking at implementation but also the counter power.

187
00:22:16,330 --> 00:22:25,690
So that is quite specific or special. And the outputs here are state specific, context, rich, sick program theories.

188
00:22:26,650 --> 00:22:30,560
The issue here and I will get to that in a moment.

189
00:22:30,580 --> 00:22:39,729
For the the ideas or methodological developments is that she is giving a lot of conditions.

190
00:22:39,730 --> 00:22:49,450
If you really go into implementation per state and look at the influences, you get a lot of factors that are possibly influencing.

191
00:22:50,410 --> 00:22:59,470
The question is how we can portray this or how we can present this in a sort of way that is still sort of readable.

192
00:23:00,700 --> 00:23:03,070
Now getting out to try tribulations.

193
00:23:04,180 --> 00:23:13,120
Let's say that we have had a lot of methodological developments over recent years, so people are trying this out since,

194
00:23:13,450 --> 00:23:22,030
let's say, in health beginning of 2000 and we had a lot of discussion on how we should define mechanisms.

195
00:23:23,240 --> 00:23:29,850
We have been discussing different ways to look at the context, mechanism, outcome configuration.

196
00:23:29,870 --> 00:23:35,330
It doesn't fundamentally change, but it is sometimes easier, for example, to use this.

197
00:23:35,720 --> 00:23:40,160
It depends a bit on on the program. A program is different than a policy and so on.

198
00:23:40,610 --> 00:23:44,360
We have had some ideas on really interviewing.

199
00:23:45,600 --> 00:23:51,480
Also on the thinking logic. How do you do this kind of analyses, this also guidance?

200
00:23:52,320 --> 00:23:56,850
And also there has been some cross-fertilisation with other methods.

201
00:23:57,330 --> 00:24:01,110
Also directing soft systems cause a loop,

202
00:24:01,110 --> 00:24:10,349
diagramming all sorts of tools that are more or less complexity oriented and that can be combined is realist evaluation.

203
00:24:10,350 --> 00:24:15,780
And maybe that is because I have been doing this of trying to do this is beginning of 2000.

204
00:24:16,200 --> 00:24:21,120
Is it still hip? Is it under the radar or is it the Asia-Pacific?

205
00:24:21,270 --> 00:24:25,560
Like we say in French, it is still a young approach.

206
00:24:26,670 --> 00:24:36,870
If you compare it to other methods. So it is normal that we are still developing and evolving in terms of of of what we are doing.

207
00:24:37,230 --> 00:24:42,960
If you look at the other methods, those are very high are out there already since ethnography.

208
00:24:43,050 --> 00:24:50,850
Early 20th century accident research. It is modern case study research 84 if you check.

209
00:24:51,300 --> 00:24:56,520
But myths and web of science then you see that it is really still on the increase.

210
00:24:56,610 --> 00:25:05,670
There is a steady increase, amazingly, I think of realist evaluation in health, let's say 74 in title only.

211
00:25:05,730 --> 00:25:12,660
So not the ones that use keywords in 2015 to 211 in 2022.

212
00:25:14,220 --> 00:25:18,690
But and here I'm trying to to critique a bit, let's say.

213
00:25:20,350 --> 00:25:25,390
Some things are sometimes challenging because you see, and that is only an indicator.

214
00:25:25,400 --> 00:25:36,250
It's not showing or evidence of anything. But a lot of papers are on the development of program theories and there are a bit less on refinements.

215
00:25:36,790 --> 00:25:42,490
It's just an illustration. It's not evidence. Is this sometimes because it's time consuming?

216
00:25:42,820 --> 00:25:48,160
In global health, we have 3 to 5 year cycles of global health programs.

217
00:25:48,640 --> 00:25:56,470
Sometimes we do not have the time. And as I experienced myself, people leave at the end of the project.

218
00:25:57,100 --> 00:26:06,430
Sometimes if you do it retrospectively, you might not find all the information or the people have just left already to go to another project,

219
00:26:06,580 --> 00:26:13,830
typically for development cooperation, I would say. Realist informed versus realist review or synthesis.

220
00:26:14,880 --> 00:26:17,880
There are a couple of articles and I wrote one myself.

221
00:26:17,880 --> 00:26:21,780
Also realist in Ford's, let's say,

222
00:26:22,260 --> 00:26:35,220
didn't quite make the cut of realist and realist reviews for it's a small percentage of four on the 16 realist synthesis that 97 sometimes.

223
00:26:36,390 --> 00:26:42,120
It is not the method to go because what you're researching is quite new.

224
00:26:42,750 --> 00:26:47,880
There is not enough documentation or you don't have access to the documents.

225
00:26:49,200 --> 00:26:54,380
These are themes that we are still grappling with.

226
00:26:54,390 --> 00:27:05,880
I think in the cycle different types of knowledge, methodological eclecticism, managing context and tools to present program theory first.

227
00:27:06,630 --> 00:27:09,630
And that is something that that is still out there.

228
00:27:09,780 --> 00:27:19,920
What is our position on methodological eclecticism? We are very much in an age where we are combining all sorts of things, mixed methods.

229
00:27:20,100 --> 00:27:28,580
It can be anything. For example, you see, some would say and some realists would say, this is neo positivist.

230
00:27:29,210 --> 00:27:38,630
It's not to be combines with realist, but others would say, okay, Boston himself would say realistic quiry is a broad church.

231
00:27:39,140 --> 00:27:47,300
And there have been quite successful efforts to combine realist with others, like political realism and pragmatism.

232
00:27:48,980 --> 00:27:53,920
It's a question out there is not a there's no less of an example.

233
00:27:53,930 --> 00:27:57,290
And a lost one is from also Nigeria.

234
00:27:57,830 --> 00:28:04,310
Also, realist evaluation of a Ph.D. student of mine on youth friendly antiretroviral services.

235
00:28:05,180 --> 00:28:08,180
He is combining two or three methods.

236
00:28:08,210 --> 00:28:16,360
I will show you. First, he goes from a logic model to a T or C, T or C theory of change.

237
00:28:16,430 --> 00:28:20,059
It's like realist evaluation, light. And why does he do that?

238
00:28:20,060 --> 00:28:29,710
Because he does it together with the program managers. He wants to formulate a theory of change, which stops short from a program theory.

239
00:28:30,080 --> 00:28:36,740
It's one element. Then in the second phase, he goes to a program theory by adding a literature review.

240
00:28:37,340 --> 00:28:43,910
Then he does interviews with implementation teams, and then he refines troop, you see.

241
00:28:44,150 --> 00:28:48,920
Okay. So what is using this tier of change is using also QCA.

242
00:28:50,410 --> 00:28:56,650
It's under peer review. But this is really a sort of creative combination of different kinds of methods.

243
00:28:58,220 --> 00:29:03,080
Much larger, of course, than than real estate valuation, but still.

244
00:29:04,510 --> 00:29:09,370
We all need to engage, I think, more with the citizens, let's say.

245
00:29:09,790 --> 00:29:15,820
And the co-creation of policy is really something that is now out there and the way to go now.

246
00:29:16,780 --> 00:29:22,320
I'm currently working on co-creation and it's integration with real estate evaluation.

247
00:29:22,330 --> 00:29:30,160
It's an ongoing real estate review and also in the European Commission programme evaluation is a part of a

248
00:29:30,220 --> 00:29:39,580
youth centred participatory action programme where one co-create the activities together with adolescents.

249
00:29:39,880 --> 00:29:45,140
How do we engage with real estate on that? Even broader.

250
00:29:45,410 --> 00:29:53,750
But importance in global health is decolonising r e and some people have written about it.

251
00:29:54,620 --> 00:30:04,180
How do you engage in global health with the fact that our theories also are very much linked to Western high income countries,

252
00:30:04,190 --> 00:30:07,370
especially for policy analyses and public administration?

253
00:30:07,880 --> 00:30:14,140
There's a lot that is high income countries and of course we can use it.

254
00:30:14,190 --> 00:30:19,600
You adapt these theories, but what is our thinking about it?

255
00:30:19,610 --> 00:30:23,270
So also something that is out there as a question.

256
00:30:24,710 --> 00:30:29,680
The last one, and I'm almost there, is the managing of context.

257
00:30:30,820 --> 00:30:39,070
Context also wider than real estate is sort of something that came up, I think in the last ten years.

258
00:30:39,070 --> 00:30:48,100
Talk about context and there has been a contextual turn in a lot of sciences and also a realist evaluation.

259
00:30:48,460 --> 00:30:54,040
But how do we engage with context and the effect on agency?

260
00:30:55,210 --> 00:31:03,160
Joanna GREENE, Hawke and Nick Emma, who are also éminence realists, have set out practical challenges faced by realists.

261
00:31:03,160 --> 00:31:08,799
Methodologies is how to include these accounts of agency in explanations about

262
00:31:08,800 --> 00:31:14,860
what is going on in health systems and why a way out there might be structure,

263
00:31:14,860 --> 00:31:18,940
agency, culture. Margaret Archer It's a critical realist.

264
00:31:19,480 --> 00:31:27,820
That is something that I'm currently working on, but it stays despite, I think ten, 15 years.

265
00:31:28,180 --> 00:31:31,720
Talk about that. We need to analyse context still challenging.

266
00:31:33,660 --> 00:31:38,400
So I won't go into that, but I'll end here.

267
00:31:39,300 --> 00:31:43,620
Lasting is that idea of building theory, which I find very important.

268
00:31:44,010 --> 00:31:48,450
How do we build theory from different evaluations across?

269
00:31:49,740 --> 00:31:57,390
Yeah. Instead of always having evaluations of one program and another, how do we link this?

270
00:31:57,840 --> 00:32:03,030
It's also a question that is frequently asked by NGOs How do we learn from each other?

271
00:32:03,570 --> 00:32:14,790
And really, evaluation might be a way or a bridge to cross over and to by using social theory sort of linking these programs.

272
00:32:15,090 --> 00:32:18,510
How do we make this link with learning?

273
00:32:20,550 --> 00:32:22,350
So I will stop here.