1
00:00:00,240 --> 00:00:06,420
To optimise the use of resources available into improved intervention and to ensure public support,

2
00:00:06,420 --> 00:00:10,470
those in charge of operations need to ask themselves two sets of questions.

3
00:00:10,470 --> 00:00:16,980
The first thing is, are we doing things right? So are implementing our plans as efficiently as we can?

4
00:00:16,980 --> 00:00:26,040
Or can we improve on how we do what we're doing? And the measurement for this is often the measurements that are made,

5
00:00:26,040 --> 00:00:30,180
often called measurement of implementation, measurement of performance or measurement of efficiency.

6
00:00:30,180 --> 00:00:38,880
That's one set of measurements that I use. The other set of questions that one needs to ask yourself is, are we doing the right things?

7
00:00:38,880 --> 00:00:42,240
So is our plan actually any good?

8
00:00:42,240 --> 00:00:47,370
Do we need to change what we're doing, not just how we do it, but actually do we need to do completely different things?

9
00:00:47,370 --> 00:00:52,500
And this is what we call the measurement of effects.

10
00:00:52,500 --> 00:00:59,520
And these two evaluation process, hopefully they should be able to do if we implement them is a of course to do.

11
00:00:59,520 --> 00:01:08,370
Also, all we're doing right. Things right. But also what they they should be able to do is to increase the sophistication of our understanding and

12
00:01:08,370 --> 00:01:15,720
find to the hypothesis that we have our theory of change so that we improve the impact of what we do.

13
00:01:15,720 --> 00:01:23,850
And really important as well of all these, these measurement processes should generate some form recommendations again.

14
00:01:23,850 --> 00:01:31,380
And since they're so central for the for the success operations, one could assume that these are very well established.

15
00:01:31,380 --> 00:01:33,180
But the funny thing is they're not.

16
00:01:33,180 --> 00:01:40,070
Well, it's not so funny, actually, but but they're not it's surprising that one reason is really that it's difficult.

17
00:01:40,070 --> 00:01:46,920
And this paper addresses two sets of different difficulties. The first one is challenges that we produce ourselves.

18
00:01:46,920 --> 00:01:54,630
And how do we make life more difficult for ourselves? To improve our evaluation processes.

19
00:01:54,630 --> 00:02:05,520
And then the second set is the methodological problems that are inherent to measuring effects.

20
00:02:05,520 --> 00:02:09,870
So I look at those two and I conclude with some recommendations.

21
00:02:09,870 --> 00:02:16,860
And if we look at the challenges we set ourself how we make our own life more difficult.

22
00:02:16,860 --> 00:02:22,350
I'm going to take a step back a bit from Stuart focussed very much on on just the UK approach.

23
00:02:22,350 --> 00:02:28,890
And we have to look at a bit, step back and look at them often at multinational context as well.

24
00:02:28,890 --> 00:02:35,940
I think it's quite clear that we all know that the current operations tend to be more ambitious than previous interventions.

25
00:02:35,940 --> 00:02:39,930
We don't just try to work in war with or in conflicts.

26
00:02:39,930 --> 00:02:52,890
Bachelor types work on complex the fight to root causes and so forth. We also see that that we heard that this morning about the normative influence,

27
00:02:52,890 --> 00:03:02,550
were they the liberal peace theory that we're not really just trying to to stop symptoms, i.e. stop people from fighting, stop starvation?

28
00:03:02,550 --> 00:03:09,300
We actually want to to change the way people think. And I think that's it's quite clear that there's far more ambitious.

29
00:03:09,300 --> 00:03:15,240
We often hear this in the military context. We often hear this slogan about winning hearts and minds.

30
00:03:15,240 --> 00:03:19,080
That's not what today's operations are about at all. It's about changing hearts, mind.

31
00:03:19,080 --> 00:03:23,040
We want people to think differently. And if we want people to think differently.

32
00:03:23,040 --> 00:03:31,130
Reasonable democracy. What that, of course, means is and that's quite difficult to accept for many people is that success,

33
00:03:31,130 --> 00:03:35,820
operation's success is not something that we can determine with our own actions.

34
00:03:35,820 --> 00:03:45,720
It actually depends on the activities of the local population and how they engage in peace processes and so forth.

35
00:03:45,720 --> 00:03:51,930
But at the same time, they're not just more ambitious. These operations off of it's far more ambiguous.

36
00:03:51,930 --> 00:03:55,080
They what we actually try to achieve that peacekeeper was easy.

37
00:03:55,080 --> 00:04:01,860
Stopping people from fighting humanitarian operations are quite easy to understand what it's all about.

38
00:04:01,860 --> 00:04:06,090
But what we're trying to do is trying to fix states. And what does that mean?

39
00:04:06,090 --> 00:04:18,000
How do you do it? And this this ambiguous this is reflected in the strategic documents, which I think have a tendency,

40
00:04:18,000 --> 00:04:23,490
although they say a lot, but they have a tendency to be apple pie statements.

41
00:04:23,490 --> 00:04:30,180
You can look, for instance, at Afghanistan. They have to bring this together.

42
00:04:30,180 --> 00:04:35,980
The goal of so many actors that they could include almost anything.

43
00:04:35,980 --> 00:04:40,200
If you look at the aims are very broad that are acceptable to all,

44
00:04:40,200 --> 00:04:48,690
you can just look at what nature had difficulties in in deciding whether it's conducting counterinsurgency campaign or what it's trying to achieve.

45
00:04:48,690 --> 00:04:58,350
If we look at the Afghan National Development Strategy, it has a hundred and sixteen expected outcomes.

46
00:04:58,350 --> 00:05:03,090
It's that they don't necessarily set a realistic timeline set.

47
00:05:03,090 --> 00:05:07,110
The Afghan compact doesn't know the end even when they try.

48
00:05:07,110 --> 00:05:10,940
And the aid is fair enough to be fair. And it tries to give some direction.

49
00:05:10,940 --> 00:05:18,720
It tries to set some priorities. These priorities don't actually translate into national agendas of the contributors.

50
00:05:18,720 --> 00:05:28,620
So they they while everybody can argue that whatever they do in this example, Afghanistan complies with the strategic document,

51
00:05:28,620 --> 00:05:33,990
the A.D.s, the Afghan compact and so forth, that they actually have very, very different things.

52
00:05:33,990 --> 00:05:42,300
And that, I would argue, is is a sign of or evidence of that these strategic documents tend to be apple pie statements.

53
00:05:42,300 --> 00:05:50,550
The question is, of course, why? And the answer is very, very simple. It's because they all motivated by different motives to to intervene.

54
00:05:50,550 --> 00:05:55,170
There is a part of altruism, of course, making life better for the Afghans.

55
00:05:55,170 --> 00:06:01,260
There is the element of national security because we see a threat emerging in Afghanistan, in the region.

56
00:06:01,260 --> 00:06:08,880
There is this argument about NATO's credibility. There are arguments about bilateral relationships with the US.

57
00:06:08,880 --> 00:06:18,060
So each country, each contributor, each organisation decides on many a whole host of motives how much they're going to contribute.

58
00:06:18,060 --> 00:06:21,750
And of course, if we look at Afghanistan in the centre,

59
00:06:21,750 --> 00:06:30,570
we're going to see that we don't contribute enough to actually we're not as committed as we need to be to fix this in any short term.

60
00:06:30,570 --> 00:06:34,650
Turning it around, we can see if we look at it globally,

61
00:06:34,650 --> 00:06:40,920
that we actually overcommit resources to Afghanistan compared to the other 30 or so, the equally vulnerable states.

62
00:06:40,920 --> 00:06:47,540
Therefore. Exactly, BP will be set up of reasons. So what we're left with here, or if this is left,

63
00:06:47,540 --> 00:06:54,800
we're going to continue to have rather weak strategic direction that some people can argue that this is actually quite a good thing.

64
00:06:54,800 --> 00:06:59,090
It gives us some advantages because it's it gives us a more flexible approach.

65
00:06:59,090 --> 00:07:03,530
Bottom up approach, which you which is normally counterinsurgency context.

66
00:07:03,530 --> 00:07:09,360
There is, however, real disadvantages as well, and that is that it risks to be fragmented.

67
00:07:09,360 --> 00:07:17,930
So we don't achieve to reinforce success. And I think this is something that is really evident in regional command south in Afghanistan.

68
00:07:17,930 --> 00:07:23,020
At first glance, if we look at regional command south, everything seems to be very neatly organised.

69
00:07:23,020 --> 00:07:26,870
And one would assume that the evaluation process is really helpful as well.

70
00:07:26,870 --> 00:07:38,600
The way it's organised, what they've done is quite simple. Arguably, the the intervention is led by ISAF Regional Command SOF, the South,

71
00:07:38,600 --> 00:07:46,010
which has outlined an operational plan with a three year perspective that the current model is from 2008 to 2003.

72
00:07:46,010 --> 00:07:49,060
And they then work in four landru operations.

73
00:07:49,060 --> 00:07:56,030
And these four lines of operations over the course reflected in the strategic documents, the A.D.s and the, I think, conflict as well.

74
00:07:56,030 --> 00:08:02,900
So for security. ISAF has responsibility for capacity building of the Afghan National Security Forces.

75
00:08:02,900 --> 00:08:07,220
That is led by the U.S. that combined security transition command.

76
00:08:07,220 --> 00:08:11,150
And for the third line of operation governance and the fourth development and

77
00:08:11,150 --> 00:08:17,480
reconstruction are led by the parties and they are now merged together to two,

78
00:08:17,480 --> 00:08:20,900
governance and development to reconstruction.

79
00:08:20,900 --> 00:08:27,550
And this operational plan is quite helpful for the assessment, because what it does, it says on these line four lines of operations,

80
00:08:27,550 --> 00:08:36,090
it has identified certain decisive points over time of three years that need to be achieved some parallel,

81
00:08:36,090 --> 00:08:38,190
what needs to be achieved and what line of operations. So.

82
00:08:38,190 --> 00:08:44,720
So they have quite a good idea of how they think that is going to work and to monitor progress towards this,

83
00:08:44,720 --> 00:08:50,000
to fact figure out whether we're moving in the right direction and if we're moving as fast as we can.

84
00:08:50,000 --> 00:08:51,150
The joint effects assessment,

85
00:08:51,150 --> 00:09:02,150
so in Regional Command South has set up a evaluation to which is called the Regional Common Operations Picture after they report quarterly.

86
00:09:02,150 --> 00:09:10,100
And they base their their assessments on opinion polls and on the questionnaires that they give to the task forces.

87
00:09:10,100 --> 00:09:17,420
So the people under their command, both of the task forces and the PR team.

88
00:09:17,420 --> 00:09:22,550
So all of this seems to be very well set up. The problem is it doesn't work very well.

89
00:09:22,550 --> 00:09:30,590
Why? Well, it seems that that they're not getting the answers they need from these subordinate units.

90
00:09:30,590 --> 00:09:36,260
At least the quality and the engagement from them vary considerably over time.

91
00:09:36,260 --> 00:09:42,040
Which, of course, affects the validity of the data said they say they work on.

92
00:09:42,040 --> 00:09:50,840
So why is this? And I think there's there's again, there the two main reasons for this.

93
00:09:50,840 --> 00:09:55,640
The first is we can say it's structural.

94
00:09:55,640 --> 00:10:01,370
What we see is that contributors don't reave their own priorities from the operational plan.

95
00:10:01,370 --> 00:10:06,140
But our CSA setup, they look at national preferences, national pressures,

96
00:10:06,140 --> 00:10:14,840
and that decides what they contribute with and what they want their own forces to do or their own agency to do.

97
00:10:14,840 --> 00:10:22,200
RC South leverage to somehow make them complacent. Small as well, because, of course, they have no resources of their own.

98
00:10:22,200 --> 00:10:26,200
And if you can't allocate resources, you actually have very little impact.

99
00:10:26,200 --> 00:10:35,230
What happens? This is WhiteWave, the archtop, the evaluation tool and the operational plan.

100
00:10:35,230 --> 00:10:39,170
If Nashed net nations and agencies don't miss their comply with it,

101
00:10:39,170 --> 00:10:45,590
what it tries to measure progress against doesn't necessarily serve what these countries want to know.

102
00:10:45,590 --> 00:10:50,790
So there is that this is secure. Let me just give an example up of Canada.

103
00:10:50,790 --> 00:10:55,380
UK is another example, by the way. But I'll skip that since we talked about that a bit.

104
00:10:55,380 --> 00:11:02,820
But the Canadians have decided to withdraw in 2011, just like the Dutch in two 10 in 2010.

105
00:11:02,820 --> 00:11:09,600
And the governments, rather than saying we want to fix Afghanistan, we want to fix Kandahar, which is their priority.

106
00:11:09,600 --> 00:11:15,160
They say that's not what we'd want to do. They have to explain to the public at home why they're there.

107
00:11:15,160 --> 00:11:16,860
So they have to show that there is a progress.

108
00:11:16,860 --> 00:11:23,580
So what they've done is they set six priorities and three senators projects with very concrete and districts projects.

109
00:11:23,580 --> 00:11:30,360
These are concrete benchmarks as well. And these are not taken from the operational plan.

110
00:11:30,360 --> 00:11:37,590
So what they do is if they get questionnaires from the Arkell, it doesn't really do the trick for them.

111
00:11:37,590 --> 00:11:42,270
They don't get the measurements they need to sell their progress to to Canada.

112
00:11:42,270 --> 00:11:50,780
And guess what they do if they're very busy? Well, they might not give this process high priority.

113
00:11:50,780 --> 00:11:55,020
The same we can say, but as I said, that the breath, the same we can say about the days.

114
00:11:55,020 --> 00:12:04,890
Add to this now that the governments of African governments at district level or provincial level or at the national level also pursues interests,

115
00:12:04,890 --> 00:12:13,440
their own interests that we see straight away that this whole stabilisation effort actually is wrong and fragmented.

116
00:12:13,440 --> 00:12:20,280
A lot of good things are done. But it is rather fragmented. We also see that there are different.

117
00:12:20,280 --> 00:12:25,800
Each of these countries, each of the donors, each of the organisations have their own needs for evaluation.

118
00:12:25,800 --> 00:12:31,590
And what we also see is that there are lot of different evaluation process that's going on in parallel,

119
00:12:31,590 --> 00:12:35,400
which sometimes leads to two respondents to some evaluation fatigue because

120
00:12:35,400 --> 00:12:41,820
they get very tired of responding to to everybody's different questionnaires.

121
00:12:41,820 --> 00:12:47,460
The second reason I think is is equally as strong. And that's an individual reason.

122
00:12:47,460 --> 00:12:53,010
Individuals, of course, what we see as we have the operational plan is time spent three months.

123
00:12:53,010 --> 00:12:58,450
We then have that broken down to an operational order. So a shorter term plan.

124
00:12:58,450 --> 00:13:02,160
It's that sequence is how we achieve the operational plan.

125
00:13:02,160 --> 00:13:06,480
And of course, this operate the priorities and the operational order change over time as well.

126
00:13:06,480 --> 00:13:16,140
So if you're said there is staff for six months or so, what you are actually working on is the operational order, not the operational plan.

127
00:13:16,140 --> 00:13:23,880
So the the assessment to actually tries to assess something that you don't feel is relevant to what you are doing.

128
00:13:23,880 --> 00:13:28,950
So what you do is you don't fill it in as consciously as it should.

129
00:13:28,950 --> 00:13:36,950
Either anecdotal evidence suggests that that several people who are supposed to be respondents don't even bother collecting the information,

130
00:13:36,950 --> 00:13:44,360
that they just do this. And the risk for all of all of this is.

131
00:13:44,360 --> 00:13:54,110
Well, one is that the national evaluation processes aren't particularly well established either with we risk that national or if you want short

132
00:13:54,110 --> 00:14:02,830
term tactical considerations and evaluations have more influence the long term strategic operational processes towards stabilisation.

133
00:14:02,830 --> 00:14:08,930
And if we don't have a measurement process that actually sees that what we're doing is any good.

134
00:14:08,930 --> 00:14:20,240
And equally, we could say we can see that that fragmentations risks to increase and therefore we might have less impact.

135
00:14:20,240 --> 00:14:26,170
How many I have left to something I. I was going to.

136
00:14:26,170 --> 00:14:33,730
I just did very quickly, because the second set of challenges that the the ones relating to the method,

137
00:14:33,730 --> 00:14:35,840
the logic problems are actually quite straightforward.

138
00:14:35,840 --> 00:14:43,250
The biggest one is attributing change somehow if we were all these processes to the measured change.

139
00:14:43,250 --> 00:14:51,110
They have indicated measure change. What they can say is allocating this change to a particular activity that's really difficult to do.

140
00:14:51,110 --> 00:14:56,630
First of all, the order effects is really simple to do. We dug a well, we have the well, that's that's easy to do.

141
00:14:56,630 --> 00:15:01,250
Second and third order effects. Would that often appear with a time lag or much more difficult?

142
00:15:01,250 --> 00:15:08,240
Let's say that we increase the local governments capacity because they dug the well or the government's legitimacy.

143
00:15:08,240 --> 00:15:12,260
How do you link that to the digging of the well? That's very difficult to do.

144
00:15:12,260 --> 00:15:15,890
So all these pross and none of them say that they actually do this.

145
00:15:15,890 --> 00:15:19,820
They're much more modest in being assertive about that.

146
00:15:19,820 --> 00:15:25,340
They say they achieved it, but they're still faced with no other challenges.

147
00:15:25,340 --> 00:15:33,800
The only one I want to take up here is the collection of data people. That's that's real trouble in this volatile environment.

148
00:15:33,800 --> 00:15:39,440
First of all, we have restricted access to the most geographical area.

149
00:15:39,440 --> 00:15:46,280
So it's very difficult to to to develop a good theory of change if you don't know what what you're actually trying to change.

150
00:15:46,280 --> 00:15:52,520
So that is a big problem often that those that we collect information from,

151
00:15:52,520 --> 00:16:00,200
we we might actually put them at risk as well, because if they talk to us for too long, they might get into trouble.

152
00:16:00,200 --> 00:16:03,890
Access to Femur's is very difficult. The volume of data,

153
00:16:03,890 --> 00:16:13,940
someone is not really statistically valid that we have because we can't collect enough when we rely on Afghans to pull to conduct Pullings.

154
00:16:13,940 --> 00:16:20,270
We don't know if they actually did it or just invented it. We don't know if if they did it in the area suggested.

155
00:16:20,270 --> 00:16:25,430
We don't know if they did it in the numbers it. We don't know if they're accurately recorded.

156
00:16:25,430 --> 00:16:29,420
We don't know if the sample spread as we wanted them to spread the sample.

157
00:16:29,420 --> 00:16:36,290
We don't know if those respondents answered truthfully or if they were mainly worried about how does my offer affect my private security?

158
00:16:36,290 --> 00:16:44,500
And what effect is it going to have on me that we don't capture tribal or family affiliations of interviewer interviews and so forth?

159
00:16:44,500 --> 00:16:46,790
So we see that there's a lot of problems.

160
00:16:46,790 --> 00:16:54,350
Another one is, of course, that we can just about ask somebody for 30 minutes before they get bored and stop being cooperative.

161
00:16:54,350 --> 00:16:59,240
So we can't necessarily use controlled questions either.

162
00:16:59,240 --> 00:17:06,260
So we have a dataset that isn't very good with difficulties interpreting this this data.

163
00:17:06,260 --> 00:17:16,970
But if I just jumped to the conclusion, I don't think that the difficulties we have today disqualifies the efforts to do something like this.

164
00:17:16,970 --> 00:17:20,180
I think we need some form of evaluation process where we'll work with them,

165
00:17:20,180 --> 00:17:25,670
because if we don't if we just do this and we just we don't have a clue whether what we do makes us.

166
00:17:25,670 --> 00:17:37,120
We need to have and develop these evaluation processes. There are simple solutions to improve the cooperation between different professions,

167
00:17:37,120 --> 00:17:41,720
and it has looked with terminology and just agreeing on some terminology.

168
00:17:41,720 --> 00:17:47,900
But I think the main thing is that we can do is improve the evaluation plan that we actually have right now.

169
00:17:47,900 --> 00:17:52,670
I think we have to change the mindset. And in some nations, that has to change more than others.

170
00:17:52,670 --> 00:17:59,930
From the strategic to the more long term awareness, I think we need to improve the utility of the evaluation process to those that actually are

171
00:17:59,930 --> 00:18:04,850
supposed to fulfil it and to give information because there is a detachment right now.

172
00:18:04,850 --> 00:18:06,950
And the reason there is that detachment, I think,

173
00:18:06,950 --> 00:18:14,600
is that right now the evaluation processes are adults rather than being integrated and most staff work.

174
00:18:14,600 --> 00:18:19,050
I know in some countries that they do a bit more than another.

175
00:18:19,050 --> 00:18:27,470
They they they invent these evaluation processes, ex-post and it's in an additional cell outside somewhere.

176
00:18:27,470 --> 00:18:30,560
So it's not an intrinsic part of the planning process.

177
00:18:30,560 --> 00:18:36,950
And if it were, we would actually have to do what with the military that tries to do all the time with effects based approach.

178
00:18:36,950 --> 00:18:43,840
More Maust it would achieve more stringent planning. We wouldn't think about activity, output, outcome, impact.

179
00:18:43,840 --> 00:18:49,060
How this change actually works and the ownership of this process must be pushed down.

180
00:18:49,060 --> 00:18:54,430
So the responsibility to to to collect information, do the evaluation,

181
00:18:54,430 --> 00:19:02,440
should it should be for those that that actually use it so that those that collected

182
00:19:02,440 --> 00:19:08,530
will put it down to lower levels and push it sidewards so that the people

183
00:19:08,530 --> 00:19:12,220
in the stalls that are responsible for certain errors and to conduct both the

184
00:19:12,220 --> 00:19:17,381
evaluation and produce the recommendations that we draw from this in the end.