To optimise the use of resources available into improved intervention and to ensure public support,
those in charge of operations need to ask themselves two sets of questions.
The first thing is, are we doing things right? So are implementing our plans as efficiently as we can?
Or can we improve on how we do what we're doing? And the measurement for this is often the measurements that are made,
often called measurement of implementation, measurement of performance or measurement of efficiency.
That's one set of measurements that I use. The other set of questions that one needs to ask yourself is, are we doing the right things?
So is our plan actually any good?
Do we need to change what we're doing, not just how we do it, but actually do we need to do completely different things?
And this is what we call the measurement of effects.
And these two evaluation process, hopefully they should be able to do if we implement them is a of course to do.
Also, all we're doing right. Things right. But also what they they should be able to do is to increase the sophistication of our understanding and
find to the hypothesis that we have our theory of change so that we improve the impact of what we do.
And really important as well of all these, these measurement processes should generate some form recommendations again.
And since they're so central for the for the success operations, one could assume that these are very well established.
But the funny thing is they're not.
Well, it's not so funny, actually, but but they're not it's surprising that one reason is really that it's difficult.
And this paper addresses two sets of different difficulties. The first one is challenges that we produce ourselves.
And how do we make life more difficult for ourselves? To improve our evaluation processes.
And then the second set is the methodological problems that are inherent to measuring effects.
So I look at those two and I conclude with some recommendations.
And if we look at the challenges we set ourself how we make our own life more difficult.
I'm going to take a step back a bit from Stuart focussed very much on on just the UK approach.
And we have to look at a bit, step back and look at them often at multinational context as well.
I think it's quite clear that we all know that the current operations tend to be more ambitious than previous interventions.
We don't just try to work in war with or in conflicts.
Bachelor types work on complex the fight to root causes and so forth. We also see that that we heard that this morning about the normative influence,
were they the liberal peace theory that we're not really just trying to to stop symptoms, i.e. stop people from fighting, stop starvation?
We actually want to to change the way people think. And I think that's it's quite clear that there's far more ambitious.
We often hear this in the military context. We often hear this slogan about winning hearts and minds.
That's not what today's operations are about at all. It's about changing hearts, mind.
We want people to think differently. And if we want people to think differently.
Reasonable democracy. What that, of course, means is and that's quite difficult to accept for many people is that success,
operation's success is not something that we can determine with our own actions.
It actually depends on the activities of the local population and how they engage in peace processes and so forth.
But at the same time, they're not just more ambitious. These operations off of it's far more ambiguous.
They what we actually try to achieve that peacekeeper was easy.
Stopping people from fighting humanitarian operations are quite easy to understand what it's all about.
But what we're trying to do is trying to fix states. And what does that mean?
How do you do it? And this this ambiguous this is reflected in the strategic documents, which I think have a tendency,
although they say a lot, but they have a tendency to be apple pie statements.
You can look, for instance, at Afghanistan. They have to bring this together.
The goal of so many actors that they could include almost anything.
If you look at the aims are very broad that are acceptable to all,
you can just look at what nature had difficulties in in deciding whether it's conducting counterinsurgency campaign or what it's trying to achieve.
If we look at the Afghan National Development Strategy, it has a hundred and sixteen expected outcomes.
It's that they don't necessarily set a realistic timeline set.
The Afghan compact doesn't know the end even when they try.
And the aid is fair enough to be fair. And it tries to give some direction.
It tries to set some priorities. These priorities don't actually translate into national agendas of the contributors.
So they they while everybody can argue that whatever they do in this example, Afghanistan complies with the strategic document,
the A.D.s, the Afghan compact and so forth, that they actually have very, very different things.
And that, I would argue, is is a sign of or evidence of that these strategic documents tend to be apple pie statements.
The question is, of course, why? And the answer is very, very simple. It's because they all motivated by different motives to to intervene.
There is a part of altruism, of course, making life better for the Afghans.
There is the element of national security because we see a threat emerging in Afghanistan, in the region.
There is this argument about NATO's credibility. There are arguments about bilateral relationships with the US.
So each country, each contributor, each organisation decides on many a whole host of motives how much they're going to contribute.
And of course, if we look at Afghanistan in the centre,
we're going to see that we don't contribute enough to actually we're not as committed as we need to be to fix this in any short term.
Turning it around, we can see if we look at it globally,
that we actually overcommit resources to Afghanistan compared to the other 30 or so, the equally vulnerable states.
Therefore. Exactly, BP will be set up of reasons. So what we're left with here, or if this is left,
we're going to continue to have rather weak strategic direction that some people can argue that this is actually quite a good thing.
It gives us some advantages because it's it gives us a more flexible approach.
Bottom up approach, which you which is normally counterinsurgency context.
There is, however, real disadvantages as well, and that is that it risks to be fragmented.
So we don't achieve to reinforce success. And I think this is something that is really evident in regional command south in Afghanistan.
At first glance, if we look at regional command south, everything seems to be very neatly organised.
And one would assume that the evaluation process is really helpful as well.
The way it's organised, what they've done is quite simple. Arguably, the the intervention is led by ISAF Regional Command SOF, the South,
which has outlined an operational plan with a three year perspective that the current model is from 2008 to 2003.
And they then work in four landru operations.
And these four lines of operations over the course reflected in the strategic documents, the A.D.s and the, I think, conflict as well.
So for security. ISAF has responsibility for capacity building of the Afghan National Security Forces.
That is led by the U.S. that combined security transition command.
And for the third line of operation governance and the fourth development and
reconstruction are led by the parties and they are now merged together to two,
governance and development to reconstruction.
And this operational plan is quite helpful for the assessment, because what it does, it says on these line four lines of operations,
it has identified certain decisive points over time of three years that need to be achieved some parallel,
what needs to be achieved and what line of operations. So.
So they have quite a good idea of how they think that is going to work and to monitor progress towards this,
to fact figure out whether we're moving in the right direction and if we're moving as fast as we can.
The joint effects assessment,
so in Regional Command South has set up a evaluation to which is called the Regional Common Operations Picture after they report quarterly.
And they base their their assessments on opinion polls and on the questionnaires that they give to the task forces.
So the people under their command, both of the task forces and the PR team.
So all of this seems to be very well set up. The problem is it doesn't work very well.
Why? Well, it seems that that they're not getting the answers they need from these subordinate units.
At least the quality and the engagement from them vary considerably over time.
Which, of course, affects the validity of the data said they say they work on.
So why is this? And I think there's there's again, there the two main reasons for this.
The first is we can say it's structural.
What we see is that contributors don't reave their own priorities from the operational plan.
But our CSA setup, they look at national preferences, national pressures,
and that decides what they contribute with and what they want their own forces to do or their own agency to do.
RC South leverage to somehow make them complacent. Small as well, because, of course, they have no resources of their own.
And if you can't allocate resources, you actually have very little impact.
What happens? This is WhiteWave, the archtop, the evaluation tool and the operational plan.
If Nashed net nations and agencies don't miss their comply with it,
what it tries to measure progress against doesn't necessarily serve what these countries want to know.
So there is that this is secure. Let me just give an example up of Canada.
UK is another example, by the way. But I'll skip that since we talked about that a bit.
But the Canadians have decided to withdraw in 2011, just like the Dutch in two 10 in 2010.
And the governments, rather than saying we want to fix Afghanistan, we want to fix Kandahar, which is their priority.
They say that's not what we'd want to do. They have to explain to the public at home why they're there.
So they have to show that there is a progress.
So what they've done is they set six priorities and three senators projects with very concrete and districts projects.
These are concrete benchmarks as well. And these are not taken from the operational plan.
So what they do is if they get questionnaires from the Arkell, it doesn't really do the trick for them.
They don't get the measurements they need to sell their progress to to Canada.
And guess what they do if they're very busy? Well, they might not give this process high priority.
The same we can say, but as I said, that the breath, the same we can say about the days.
Add to this now that the governments of African governments at district level or provincial level or at the national level also pursues interests,
their own interests that we see straight away that this whole stabilisation effort actually is wrong and fragmented.
A lot of good things are done. But it is rather fragmented. We also see that there are different.
Each of these countries, each of the donors, each of the organisations have their own needs for evaluation.
And what we also see is that there are lot of different evaluation process that's going on in parallel,
which sometimes leads to two respondents to some evaluation fatigue because
they get very tired of responding to to everybody's different questionnaires.
The second reason I think is is equally as strong. And that's an individual reason.
Individuals, of course, what we see as we have the operational plan is time spent three months.
We then have that broken down to an operational order. So a shorter term plan.
It's that sequence is how we achieve the operational plan.
And of course, this operate the priorities and the operational order change over time as well.
So if you're said there is staff for six months or so, what you are actually working on is the operational order, not the operational plan.
So the the assessment to actually tries to assess something that you don't feel is relevant to what you are doing.
So what you do is you don't fill it in as consciously as it should.
Either anecdotal evidence suggests that that several people who are supposed to be respondents don't even bother collecting the information,
that they just do this. And the risk for all of all of this is.
Well, one is that the national evaluation processes aren't particularly well established either with we risk that national or if you want short
term tactical considerations and evaluations have more influence the long term strategic operational processes towards stabilisation.
And if we don't have a measurement process that actually sees that what we're doing is any good.
And equally, we could say we can see that that fragmentations risks to increase and therefore we might have less impact.
How many I have left to something I. I was going to.
I just did very quickly, because the second set of challenges that the the ones relating to the method,
the logic problems are actually quite straightforward.
The biggest one is attributing change somehow if we were all these processes to the measured change.
They have indicated measure change. What they can say is allocating this change to a particular activity that's really difficult to do.
First of all, the order effects is really simple to do. We dug a well, we have the well, that's that's easy to do.
Second and third order effects. Would that often appear with a time lag or much more difficult?
Let's say that we increase the local governments capacity because they dug the well or the government's legitimacy.
How do you link that to the digging of the well? That's very difficult to do.
So all these pross and none of them say that they actually do this.
They're much more modest in being assertive about that.
They say they achieved it, but they're still faced with no other challenges.
The only one I want to take up here is the collection of data people. That's that's real trouble in this volatile environment.
First of all, we have restricted access to the most geographical area.
So it's very difficult to to to develop a good theory of change if you don't know what what you're actually trying to change.
So that is a big problem often that those that we collect information from,
we we might actually put them at risk as well, because if they talk to us for too long, they might get into trouble.
Access to Femur's is very difficult. The volume of data,
someone is not really statistically valid that we have because we can't collect enough when we rely on Afghans to pull to conduct Pullings.
We don't know if they actually did it or just invented it. We don't know if if they did it in the area suggested.
We don't know if they did it in the numbers it. We don't know if they're accurately recorded.
We don't know if the sample spread as we wanted them to spread the sample.
We don't know if those respondents answered truthfully or if they were mainly worried about how does my offer affect my private security?
And what effect is it going to have on me that we don't capture tribal or family affiliations of interviewer interviews and so forth?
So we see that there's a lot of problems.
Another one is, of course, that we can just about ask somebody for 30 minutes before they get bored and stop being cooperative.
So we can't necessarily use controlled questions either.
So we have a dataset that isn't very good with difficulties interpreting this this data.
But if I just jumped to the conclusion, I don't think that the difficulties we have today disqualifies the efforts to do something like this.
I think we need some form of evaluation process where we'll work with them,
because if we don't if we just do this and we just we don't have a clue whether what we do makes us.
We need to have and develop these evaluation processes. There are simple solutions to improve the cooperation between different professions,
and it has looked with terminology and just agreeing on some terminology.
But I think the main thing is that we can do is improve the evaluation plan that we actually have right now.
I think we have to change the mindset. And in some nations, that has to change more than others.
From the strategic to the more long term awareness, I think we need to improve the utility of the evaluation process to those that actually are
supposed to fulfil it and to give information because there is a detachment right now.
And the reason there is that detachment, I think,
is that right now the evaluation processes are adults rather than being integrated and most staff work.
I know in some countries that they do a bit more than another.
They they they invent these evaluation processes, ex-post and it's in an additional cell outside somewhere.
So it's not an intrinsic part of the planning process.
And if it were, we would actually have to do what with the military that tries to do all the time with effects based approach.
More Maust it would achieve more stringent planning. We wouldn't think about activity, output, outcome, impact.
How this change actually works and the ownership of this process must be pushed down.
So the responsibility to to to collect information, do the evaluation,
should it should be for those that that actually use it so that those that collected
will put it down to lower levels and push it sidewards so that the people
in the stalls that are responsible for certain errors and to conduct both the
evaluation and produce the recommendations that we draw from this in the end.