Basically, what have I been doing? I have like 20 years experience in global health policy and programs.
I've worked for UNFPA, I've worked for Marie Stopes in Belgium,
and now I'm a senior lecturer and research fellow at the Institute of Tropical Medicine in Antwerp, Belgium.
And Jeff has been so kind to revive me and sort of we've been walking this roads of realist evaluation from different sides, but sort of together.
Yeah. First,
I wanted to add we were just a bit reflecting on it on that journey to to explore with you a bit why we came to such a thing like realist evaluation.
And I don't know if, you know, maybe Jeff has said it.
It comes originally from, I think, public policy sociology.
Ray Boston is a sociologist in Leeds and he first used it, I think on the CCTV, the analyses of CCTV cameras,
so completely different ballpark than what we have today and how we apply it today.
And why did we come to it? So around the year 2000, we were a bunch of of doctors, let's say,
and also social scientists working on health programmes in low and middle income countries, primarily sub-Saharan Africa,
because that's where Belgian of funds, mostly healthcare programs, we were doing policy and programmes on quality of care,
also teaching it on district health management, on the interface between programmes as services.
And we were using methods like participatory more or less action research at no fee in an institute that was primarily biomedical,
a bit like here biomedical oriented institutes where the main paradigm was our city, our cities were considered the golden standards.
And we had a lot of people epidemiologist out there who didn't really recognise, let's say, social science methods.
So at that time we had a growing unease about the methods we're using and also the effects we were having with this.
And we sort of had the feeling that we were caught between a rock and a hard place, in a sense, against or not against.
That's sort of between the cities, which didn't cover our expectations, and then the very soft, maybe Rock of Action Research,
where we were doing completely something else and a big rift between those methods of quantitative side and the qualitative side.
So we were looking for new methods at the time and looking for methods that sort of would fit our
idea of the complex reality we were facing of health programs in low and middle income countries.
This was that small minority group, two institutes, and we are sort of 300 people overall.
These studies of tropical medicine and these six or seven were discovering real evaluation at the time,
and I was a bit younger in a bit, and we weren't the only ones in clinical medicine.
There were a bunch of people, including, of course, Richard Greenhalgh of Oxford,
Jeff's colleague, who were looking at or vouching for the idea of complexity,
thinking this would reflect much more of the reality of what was happening in health systems, the coevolution with other systems that was happening,
and the idea of health systems that were basically made up of relationships in social systems.
She wasn't the only one. Also, David Koenig was advocating for this, and the cities were increasingly being questioned for complex interventions.
But you could argue you could argue that the uptake in public health was relatively slow.
So in around 2002, I think was this Popularising paper funded by Canada, which makes the distinction between simple,
complicated and complex systems, which was a big help in teaching about how systems work.
We were teaching this to public health doctors from sub-Saharan African countries,
and we needed to have an entry point that this was a very helpful thing.
Then I think Margaret Chan was vouching for the idea of systems at the time at the W.H.O. and you get this this important, impressive report.
And then also at the same time, 2008, 2009 is the MRC Council guidance, who finally admits.
That there's something like complex interventions, but not mentioning real estate, by the way.
Still, certain certain methods, especially from the social scientists,
scientists have found little entry points into into public health and into global health policy and practice to this day.
So realist evaluation in that sense is sort of almost an exception because if you would look at QCA,
it's just based modelling is more already quantitative process tracing.
These things are still quite under-used.
In my opinion, you could see that COVID 19 would be a big game changer.
Maybe the jury is still out there on that. We're still waiting a bit, I think, for that big game change.
But if there was anything that showed that there was, for example, co-evolution between health to political and economic and social systems,
or where there was a lot of confusion and a lot of unpredictability and emergence of bottom up action, then it's it's COVID.
So and indeed, to chagrin has made this case together with a bunch of others, but still, I'm not sure if we are still already there.
But if we look at ourselves as the realist evaluation team, for example, in my institutes today,
we have really grown from those six researchers who were really in a minority to quite
a big network of people that are working on it that have formed an organic network,
let's say. So maybe Geoff already spoke about Rhamnosus.
So we have also a small network next to that,
and that's something spontaneous of postdoc researchers who are also working together on real estate reviews.
And next to that, we we are developing currently an E course with the IPH Institute of Public Health in Bangalore, India.
So there are more and more people out there, and it's sort of spreading like the ink blots.
Why is that? I've wondered a lot about how is that possible that it has found such and such an audience in global health?
I think there's a lot to it, to the fact that it uses theory and it opens the worlds of theory and a lot in discussions in our institute, for example.
We have this false dichotomy often between action and theory.
Either you were an activist and you were. Come on, let's do things.
Let's do things quick. And then you had two people who say, no, no, no.
We need to look at the state of the art. We need to look at theories.
The lab and sort of realist is sort of dismissing this because programs are seen as theory incarnate by by person.
And Chile and it's this also and that I think is also an important point.
And I hope that that gets stronger with the time in methodological development of realist that it enables learning across programs through theory.
You could link what has been happening on HIV service delivery models in South Africa to Nigeria and so on.
And you built, you know, so what's theory now?
Just one caveat. There are people who would argue that the and those are the medical anthropologists.
I'm a medical anthropologist, also a political scientist.
Really public health that this is not new like, okay, we have been doing this all along, they would say.
But the big difference is, I think and the attraction for biomedical people or people in the biomedical world
is that it offers clear guidance and the research cycle is being made explicit.
Medical anthropology does take descriptions, contextually rich descriptions of situations, but there it stops.
And anthropology is also more like the craft.
It's a certain community,
and it's not necessarily that open or it didn't used to be that open to to other types of researchers into interdisciplinary dialogue.
This is an opinion that I wrote down in an anthropology journal, by the way.
This is a bit the second part of my talk. And here I wanted to show you some examples of people who have done also the.
Course. So Monica Martis from Belgium.
And I think if the lawyer or you you want from Nigeria have full discourse and are
currently doing real estate evaluations and pragati by the practice in India,
Wellcome Trust Fellow at the Institute of Public Health in Bangalore.
And I will show you examples from health service delivery programs too.
And then also a bit more programs that are about upstream policy processes, which are where one does not really use real estate evaluation.
My opinion a lot, I think these are the key principles.
I hope they sort of correspond to what Jeff has been saying and this is the cycle that we are using.
You will see some real cycles applied. This is sort of the start of that of that cycle.
This is IBI lawyer or lawyer one he is currently doing in the face of tyranny, refinement and testing.
But he has done a real estate evaluation on community based antiretroviral treatment delivery programs for specific groups in Benue State, Nigeria.
Benue State is a hard policy environment, let's say, for this key population groups.
It's specific for men having sex with men, female sex workers, injecting drug users and transgenders.
And in that state, there's harassment. But it's also illegal that these activities are, of course, illegal.
So the question is, the resource or evaluation question that he asks is what are the mechanisms
and context conditions that drive successful community based implementation?
And how do this lead to better retention in care treatment adherence, a viral suppression?
This is a program that is funded by the fair, is being executed with the state government, and it's an NGO basically that is doing it.
He has worked in that program before. He came to the institute to do a PGD and this is his cycle.
And so he has adapted the cycle. He goes to identify the program theory based on a scoping review.
And then you're already saying that is the seed testing with the mixed methods design and then you have data analysis using.
I see. AMOS So some people you see among other use,
I see a more so the CMO is context mechanism outcome I see AMLO's Intervention context actors mechanism outcome.
It makes it sometimes easier if the program is really oriented towards different types of actors.
That is what I would say. The idea is to evaluate the effectiveness of this program.
It's an impact evaluation plus a realist. It understands why and how and he is also building and that is a specific thing or a very nice thing.
I think he's building on the knowledge of other programs.
So we had a real estate evaluation that we already followed from 30 coming from South Africa on antiretroviral treatment and adherence clubs.
And he has used that for the development of the initial program.
So sort of learning across this is the way he's developing his program theory, it's his graph.
He starts us with a lot of programs still with a logic model.
That's what they have in the beginning of the program. Then he has his own professional experience he's doing in the interviews.
He's doing a scoping review and is looking at the state of the art.
He then extracts the ICM configuration and then he develops initial program theory.
I'm showing you this as an illustration just how it is done.
This is sort of giving an overview.
Yeah, you can show it in different ways. He Yes.
Showing it in different ways and backed it up with evidence in a paper showing interviews, citations and different sources.
The context, the mechanisms and the outcomes and the arrows between are sort of the relationships and configuration or hurdles that you're using.
So the arrows are very important and are in a narrative being explained.
You sort of synthesises this into an initial program theory, which is this, and there's a narrative also to it.
The straight arrows are direct effects. These are the the how you say curved ones are indirect effects.
It's looking also at different layers of context, especially zooming in.
And that is, I think, very important that you select also what you are looking at.
What is salient are the different types of policies and so on that have an impact on the practices of those groups.
This is the narrative form. Here you have context.
Here are the mechanisms. Trust and psychological safety are, of course, very important in that environment.
That is not friendly, let's say. And you're also it's about trust.
And here you see the seed, the intervention written up.
Now, the phase that he's currently in is refining and just seeing the program theory with two phases a quantitative and qualitative analysis.
He has done already a retrospective cohort study on retention, and his has done three case studies, and now he's exploring across case mechanisms.
For example, there is the emergence, let's say, of inter-group solidarity.
So those groups, for example, men having sex with men and the sex workers are also supporting each other.
Let's see, then the next one is a bit more upstream.
And this is about use of policy dialogue.
It is an idea to scale up our scale up integrated chronic care in three countries Cambodia, Slovenia and Belgium.
She's looking at how the use of policy dialogue would lead to a better implementation of integrated chronic care policy.
Okay, so it's really about the policy instruments and how that would work in those three countries.
The sources that she's using for the development of the program theory is a literature
review on the role of policy dialogue and also theory theory coming from political science.
So here you get a sort of totally different ballpark.
And she's integrating really with political science theories to make sense of what is happening in those three cases.
The challenge of this kind of program or of broader public sector reform or public policy programs, broader than just a service delivery,
is that you're mainly very much have to look at those macro level mechanisms and how dose
interacts with other levels of of of mechanisms at the relational organisational mechanisms.
And it's a micro at the individual level.
This is quite a challenge and I think a bit more difficult to do than sort of the relatively closed systems still maybe of health service delivery,
although they're not closed. But here you in a huge thing here also.
What is she trying to do here?
She's trying to identify mechanisms and context conditions based on the policy face that the different countries are in.
So here you also see I wanted to show you that to see sort of the creative use of
different theories like policy analyses to make sense and to search for the mechanisms.
And the idea is very important that you can do things creatively.
This is something that she has developed.
This is the program theory development she's using.
If then because statements, that's a sort of practical suggestion you can use to really keep your eye on the ball in terms of causality.
And here you have the context, the outcome and so on.
For the refinements. She's at the same stage as Ebola in Nigeria.
She will do in-depth interviews, documents, reviews and also workshop.
And at this moment, she has is looking at the different country cases.
And what you get in this type of thing.
Yeah. A policy of program, if you evaluate you, gets quite different mechanisms, let's say,
based on the understanding, based on the path dependency of each country.
So each country is taking it on a completely different is on a completely different path.
Belgium, very like the political culture that we have, is looking at or what works.
There is a stakeholder negotiation, and negotiation between stakeholders will lead to better uptake.
Cambodia. The experts play a very important role.
And in Slovenia, its patients peer support groups that play an important role sort of co-creation.
So you get totally different ways of implementing this according to the country context.
Last one is the implementation of tobacco control.
Also big program in terms of this is the control of tobacco control, the Tobacco Products Act in India.
It's from 2003. And then you have the National Tobacco Control Program.
And the idea is to understand how states who have the mandate to implement tobacco control.
It's a federal system. It's decentralised. How are they implementing tobacco control policy?
This is very real, this cycle.
She has done a real synthesis to develop the IPT to look at tobacco control implementation specific for low and middle income countries.
And she has also survey data that she had.
And on the basis of that, she's selecting three states.
She cannot possibly do all the Indian states and look at implementation.
She has done interviews and focus group discussions.
She is extracting CMO configurations and she will take these CMO configurations to the policy makers of the different states to validate them.
This is how she portrays her CMO's, so it's all quite different.
This is an intervention strategy, intersectoral coordination.
Here we are. This is not an intervention strategy.
This is tobacco industry interference and is sort of a countering implementation strategy.
So it's an actor countering the implementation of the Tobacco Control Act in India.
So she's not only looking at implementation but also the counter power.
So that is quite specific or special. And the outputs here are state specific, context, rich, sick program theories.
The issue here and I will get to that in a moment.
For the the ideas or methodological developments is that she is giving a lot of conditions.
If you really go into implementation per state and look at the influences, you get a lot of factors that are possibly influencing.
The question is how we can portray this or how we can present this in a sort of way that is still sort of readable.
Now getting out to try tribulations.
Let's say that we have had a lot of methodological developments over recent years, so people are trying this out since,
let's say, in health beginning of 2000 and we had a lot of discussion on how we should define mechanisms.
We have been discussing different ways to look at the context, mechanism, outcome configuration.
It doesn't fundamentally change, but it is sometimes easier, for example, to use this.
It depends a bit on on the program. A program is different than a policy and so on.
We have had some ideas on really interviewing.
Also on the thinking logic. How do you do this kind of analyses, this also guidance?
And also there has been some cross-fertilisation with other methods.
Also directing soft systems cause a loop,
diagramming all sorts of tools that are more or less complexity oriented and that can be combined is realist evaluation.
And maybe that is because I have been doing this of trying to do this is beginning of 2000.
Is it still hip? Is it under the radar or is it the Asia-Pacific?
Like we say in French, it is still a young approach.
If you compare it to other methods. So it is normal that we are still developing and evolving in terms of of of what we are doing.
If you look at the other methods, those are very high are out there already since ethnography.
Early 20th century accident research. It is modern case study research 84 if you check.
But myths and web of science then you see that it is really still on the increase.
There is a steady increase, amazingly, I think of realist evaluation in health, let's say 74 in title only.
So not the ones that use keywords in 2015 to 211 in 2022.
But and here I'm trying to to critique a bit, let's say.
Some things are sometimes challenging because you see, and that is only an indicator.
It's not showing or evidence of anything. But a lot of papers are on the development of program theories and there are a bit less on refinements.
It's just an illustration. It's not evidence. Is this sometimes because it's time consuming?
In global health, we have 3 to 5 year cycles of global health programs.
Sometimes we do not have the time. And as I experienced myself, people leave at the end of the project.
Sometimes if you do it retrospectively, you might not find all the information or the people have just left already to go to another project,
typically for development cooperation, I would say. Realist informed versus realist review or synthesis.
There are a couple of articles and I wrote one myself.
Also realist in Ford's, let's say,
didn't quite make the cut of realist and realist reviews for it's a small percentage of four on the 16 realist synthesis that 97 sometimes.
It is not the method to go because what you're researching is quite new.
There is not enough documentation or you don't have access to the documents.
These are themes that we are still grappling with.
I think in the cycle different types of knowledge, methodological eclecticism, managing context and tools to present program theory first.
And that is something that that is still out there.
What is our position on methodological eclecticism? We are very much in an age where we are combining all sorts of things, mixed methods.
It can be anything. For example, you see, some would say and some realists would say, this is neo positivist.
It's not to be combines with realist, but others would say, okay, Boston himself would say realistic quiry is a broad church.
And there have been quite successful efforts to combine realist with others, like political realism and pragmatism.
It's a question out there is not a there's no less of an example.
And a lost one is from also Nigeria.
Also, realist evaluation of a Ph.D. student of mine on youth friendly antiretroviral services.
He is combining two or three methods.
I will show you. First, he goes from a logic model to a T or C, T or C theory of change.
It's like realist evaluation, light. And why does he do that?
Because he does it together with the program managers. He wants to formulate a theory of change, which stops short from a program theory.
It's one element. Then in the second phase, he goes to a program theory by adding a literature review.
Then he does interviews with implementation teams, and then he refines troop, you see.
Okay. So what is using this tier of change is using also QCA.
It's under peer review. But this is really a sort of creative combination of different kinds of methods.
Much larger, of course, than than real estate valuation, but still.
We all need to engage, I think, more with the citizens, let's say.
And the co-creation of policy is really something that is now out there and the way to go now.
I'm currently working on co-creation and it's integration with real estate evaluation.
It's an ongoing real estate review and also in the European Commission programme evaluation is a part of a
youth centred participatory action programme where one co-create the activities together with adolescents.
How do we engage with real estate on that? Even broader.
But importance in global health is decolonising r e and some people have written about it.
How do you engage in global health with the fact that our theories also are very much linked to Western high income countries,
especially for policy analyses and public administration?
There's a lot that is high income countries and of course we can use it.
You adapt these theories, but what is our thinking about it?
So also something that is out there as a question.
The last one, and I'm almost there, is the managing of context.
Context also wider than real estate is sort of something that came up, I think in the last ten years.
Talk about context and there has been a contextual turn in a lot of sciences and also a realist evaluation.
But how do we engage with context and the effect on agency?
Joanna GREENE, Hawke and Nick Emma, who are also éminence realists, have set out practical challenges faced by realists.
Methodologies is how to include these accounts of agency in explanations about
what is going on in health systems and why a way out there might be structure,
agency, culture. Margaret Archer It's a critical realist.
That is something that I'm currently working on, but it stays despite, I think ten, 15 years.
Talk about that. We need to analyse context still challenging.
So I won't go into that, but I'll end here.
Lasting is that idea of building theory, which I find very important.
How do we build theory from different evaluations across?
Yeah. Instead of always having evaluations of one program and another, how do we link this?
It's also a question that is frequently asked by NGOs How do we learn from each other?
And really, evaluation might be a way or a bridge to cross over and to by using social theory sort of linking these programs.
How do we make this link with learning?
So I will stop here.