1
00:00:02,310 --> 00:00:04,650
Welcome to the CSA. You research podcasts.

2
00:00:04,860 --> 00:00:11,370
This is a series of conversations about projects taking place through the Centre for the Study of African Economies at the University of Oxford.

3
00:00:11,610 --> 00:00:17,850
I'm Simon Quinn. I'm an associate professor at the Department of Economics and Public Policy at Imperial College Business School.

4
00:00:18,210 --> 00:00:22,470
And until last year I was privileged to be Deputy Director of CSA.

5
00:00:23,160 --> 00:00:30,719
Today we're going to be talking about the project an adaptive, targeted field experiment, job search assistance for refugees in Jordan.

6
00:00:30,720 --> 00:00:35,970
And this was a project that was run in partnership between CSA and the International Rescue Committee.

7
00:00:36,420 --> 00:00:41,880
When we talk about this project, we're talking about what I think is a really important policy question,

8
00:00:41,910 --> 00:00:48,690
namely, how is it that different kinds of policy can help refugees and other displaced populations?

9
00:00:48,690 --> 00:00:55,350
And we all know, sadly, that we live in a time when this is a really important issue for policy in a lot of different geographical settings.

10
00:00:56,310 --> 00:01:01,799
When we talk today about this issue, we're thinking about Syrian refugees and also local jobseekers.

11
00:01:01,800 --> 00:01:07,740
In Jordan. The labour market in Jordan is characterised by very low employment rates, at least by international standards,

12
00:01:08,340 --> 00:01:12,270
and employment rates among refugees are much lower than among Jordanians.

13
00:01:12,600 --> 00:01:14,700
So this is a project, as we're going to discuss,

14
00:01:14,700 --> 00:01:19,950
in which we think about the impact of three interventions that were designed to improve formal employment outcomes,

15
00:01:20,160 --> 00:01:23,220
both for Syrian refugees and for local jobseekers.

16
00:01:23,640 --> 00:01:28,410
I'm really excited to be joined today by two of my co-authors on this project, Stefano Carrier,

17
00:01:28,560 --> 00:01:31,410
who's a professor in the Department of Economics at the University of Warwick,

18
00:01:31,890 --> 00:01:36,960
and Max Casey, a professor in the Department of Economics at the University of Oxford.

19
00:01:37,620 --> 00:01:41,790
Max and Stefano, thank you so much for joining me. It's great to be able to discuss this project to you.

20
00:01:42,000 --> 00:01:45,600
Hi Simon. Thanks so much for having me on this podcast. Thank you.

21
00:01:45,600 --> 00:01:48,690
SIMON It's really great to be here and to be able to talk about this project.

22
00:01:50,150 --> 00:01:52,670
Maybe if I could start with you, Max.

23
00:01:52,670 --> 00:02:01,520
I mean, one of the things that I found fascinating working with everyone on this project is that this is an adaptive field experiment.

24
00:02:02,180 --> 00:02:06,320
And I learned a lot about this whole idea of adaptive experiments by working on it.

25
00:02:06,770 --> 00:02:10,940
And it's something that I actually think is one of the most exciting parts of this research paper and

26
00:02:10,940 --> 00:02:15,600
hopefully something that will be relevant to a bunch of other researchers in this and other field.

27
00:02:15,630 --> 00:02:22,670
So can I ask you, first of all, to tell everyone what an adaptive experiment is and how we implemented the adaptive experiment in this context?

28
00:02:23,210 --> 00:02:30,890
So we ran this study in the form of an adaptive experiment and more specifically, an adaptive targeted experiment.

29
00:02:31,010 --> 00:02:37,460
Maybe that's why we go there. Let's back up a little bit and talk about what the purpose of the experiment is or what the goal is,

30
00:02:37,850 --> 00:02:41,929
because that really points up to a to why this might be a good idea and it might be useful

31
00:02:41,930 --> 00:02:46,549
in many other experiments over the last couple decades or so in developing the economics,

32
00:02:46,550 --> 00:02:50,230
and that become very standard around randomised controlled trials.

33
00:02:50,240 --> 00:02:58,040
Read the randomised controlled trial, you randomly assign different people to the different treatment arms if it's called different policy,

34
00:02:58,520 --> 00:03:05,870
and then you see about how things turn out and you get hopefully a credible estimate of how effective your policies are.

35
00:03:06,380 --> 00:03:09,290
That makes perfect sense, I think, for a lot of cases.

36
00:03:09,800 --> 00:03:18,950
What that does is it runs the experiment in a way where the goal is to get precise estimates of how effective a policy may be precisely to compare.

37
00:03:19,190 --> 00:03:21,599
But that might not be the only thing you care about. Right?

38
00:03:21,600 --> 00:03:30,380
So getting precise estimate is not the same thing as informing quality choices, and it's also not the same thing at helping participants.

39
00:03:30,680 --> 00:03:35,450
So those are two to rather different objectives. And then we want to pursue this other objective.

40
00:03:36,140 --> 00:03:44,540
Then they might need to run our experiment in a different way to maybe to illustrate that very clear example by picking nickel trial in Madison,

41
00:03:45,290 --> 00:03:53,389
where you have a track and a placebo or maybe some standard treatment that you compare to a new treatment, and then you run an experiment.

42
00:03:53,390 --> 00:03:56,510
And if you're sticking to a standard protocol,

43
00:03:56,870 --> 00:04:01,879
you just have like a 5050 split between the two treatment arms and you observe how well

44
00:04:01,880 --> 00:04:05,420
they're doing and you just keep running with that until the end of the experiment.

45
00:04:06,390 --> 00:04:13,020
But now what might happen? That's an experiment. Maybe halfway through, you realise that the new drugs killing everybody does something very good.

46
00:04:13,350 --> 00:04:16,980
Or it might be that the new drug is kind of a miracle curing everybody.

47
00:04:17,520 --> 00:04:20,430
And so you would want to use that information for the sake of the patient.

48
00:04:21,240 --> 00:04:25,320
You wouldn't just want to keep going with your protocol ignoring what you learned.

49
00:04:25,680 --> 00:04:29,570
Just use your your patient as a means to an end. You want to help your patient.

50
00:04:30,330 --> 00:04:34,229
And so what that means is you might want to adapt over time.

51
00:04:34,230 --> 00:04:40,080
You might want to shift your treatment or your assignment to treatment or to treatment are far better.

52
00:04:40,410 --> 00:04:46,350
So if that new drug during everybody that you want to assign the new drug more awkwardly or as time progresses,

53
00:04:46,350 --> 00:04:49,770
or if it seems to hurt people, then you might want to quickly shift away.

54
00:04:50,490 --> 00:04:55,590
And so that's that's kind of the core of the activity, and that's something that we implemented now in our experiment.

55
00:04:56,070 --> 00:05:04,020
It ran over a whole number of months. And in our experiment, the goal was to help Syrian refugees find jobs in the Indian labour market.

56
00:05:04,110 --> 00:05:09,180
In order to do that, we observed that for different treatment. And Stefano, I guess, can tell us more about this later.

57
00:05:09,690 --> 00:05:15,990
The very different amazement test and not intervention to try and help people find a job over time to

58
00:05:16,020 --> 00:05:22,140
give us shift treatment towards those interventions that are most successful at finding people job.

59
00:05:22,710 --> 00:05:28,860
Yes, absolutely. So thanks very much, Max. Maybe can I bring you in, Stefano, and tell us in brief about the different treatment arms?

60
00:05:28,860 --> 00:05:35,400
And then I want to go back to Max and think about how we implemented this in a way that tries to think about different subgroups.

61
00:05:35,730 --> 00:05:39,120
But let me hold hold off on that. And so we hear a bit about the treatments themselves.

62
00:05:39,720 --> 00:05:45,600
So we had three interventions that we were interested in, in trying out in this project.

63
00:05:45,960 --> 00:05:53,760
One was a small and conditional cash grant, which we calculated to cover the costs of job search for about a couple of months.

64
00:05:54,180 --> 00:05:57,360
So we had baseline data on how much people were spending and job search.

65
00:05:57,750 --> 00:06:01,900
And so we we calculated an amount that people would spend in about a two month period.

66
00:06:01,920 --> 00:06:07,770
We know that a job search can actually be surprisingly expensive because people have to use transport

67
00:06:07,860 --> 00:06:12,810
and travel around town and have all sorts of expenditures that when you live on a very low income,

68
00:06:13,170 --> 00:06:16,770
can actually make up a large share with a surprisingly large share of your budget.

69
00:06:17,190 --> 00:06:25,079
And so we try to support them in making these investments by making cash available upfront at the beginning of the set,

70
00:06:25,080 --> 00:06:32,220
spent the cash in the first intervention. The second intervention was an information intervention where we basically tried to

71
00:06:32,550 --> 00:06:38,430
coach people to change people in presenting themselves to employers by research

72
00:06:38,430 --> 00:06:44,850
that we've done that would use Hamilton and other researchers in Ethiopia and South

73
00:06:44,850 --> 00:06:51,839
Africa and Uganda shows that jobseekers ability to convey their employability,

74
00:06:51,840 --> 00:06:58,650
their skills and their net worth to employers, it's really a key ingredient to their success in the labour market.

75
00:06:59,030 --> 00:07:00,540
And so in the second intervention,

76
00:07:01,650 --> 00:07:08,400
jobseekers who may not necessarily be familiar with the new standards of the Jordanian labour market because they were refugees,

77
00:07:09,570 --> 00:07:14,580
where we're basically trained in approaching Jordanian employers and presenting their skills and convincing them.

78
00:07:14,580 --> 00:07:18,690
And then finally there was a second intervention, information intervention.

79
00:07:19,050 --> 00:07:25,470
And finally we had a psychological or energy dimension, which was highly motivated by the idea, which is well documented.

80
00:07:25,470 --> 00:07:34,920
That Job said it's a very difficult and frustrating process which requires you to sustain the motivation to search hard over a long period of time.

81
00:07:34,950 --> 00:07:38,339
So motivation was central to the success of job search,

82
00:07:38,340 --> 00:07:45,020
and people struggle to maintain high levels of motivation when they start getting rejection injection.

83
00:07:45,510 --> 00:07:49,020
So to help people maintain high level job search effort,

84
00:07:49,380 --> 00:07:55,200
we use the insights from recent psychological literature as well as a recent trial in South Africa,

85
00:07:55,620 --> 00:08:03,240
and devised a goal setting intervention where jobseekers would were encouraged to set themselves some jobs to achieve our goals.

86
00:08:03,540 --> 00:08:07,260
And then we would they would report on whether they attained those goals or not.

87
00:08:07,500 --> 00:08:14,910
They would be able to set the other goals for the following week. So these were the three interventions cash information and the psychological nudge.

88
00:08:15,180 --> 00:08:21,300
And of course, there was also a support group, a control group that didn't receive any of these three policies.

89
00:08:22,370 --> 00:08:27,199
Thank you, Stefano. I remember a fascinating visit that you and I had with some of our partners from the International Rescue Committee,

90
00:08:27,200 --> 00:08:34,219
where we actually went from Amman up to the northwest of Jordan to Irbid and spoke to several refugees.

91
00:08:34,220 --> 00:08:38,209
And I think I think we both felt it was like fascinating and humbling to hear about

92
00:08:38,210 --> 00:08:41,900
the whole constellation of different challenges that these respondents were facing.

93
00:08:41,900 --> 00:08:43,310
And I think it's exactly as you say.

94
00:08:43,490 --> 00:08:49,490
Partly it's about motivation, partly it's about understanding the labour market, partly it's about having the cash to actually access jobs.

95
00:08:50,960 --> 00:08:54,560
And I think that's absolutely right. This is why we went for these three treatments.

96
00:08:54,560 --> 00:08:58,280
In this context, Max, we opened by talking about targeting.

97
00:08:58,580 --> 00:09:01,460
Can I come back to you, having spoken about the treatments,

98
00:09:01,760 --> 00:09:06,320
to tell us a little bit more about the idea of targeting and how we implemented it in this setting?

99
00:09:07,210 --> 00:09:08,170
That sounds good, Simon.

100
00:09:08,410 --> 00:09:13,360
Maybe before we get to the targeting, though, let me check a little bit more about these other activities that I started out with.

101
00:09:13,870 --> 00:09:21,690
Right. To deepen your knowledge? We had three different interventions at the base of the control arm, and they talked about this idea,

102
00:09:21,700 --> 00:09:25,060
how you want to move over time towards the more successful treatments.

103
00:09:25,840 --> 00:09:30,850
This is something that no one is exploiting the information that you have and the machine learning literature.

104
00:09:31,390 --> 00:09:37,930
But there is kind of a tension there between exploitation and exploration, which is kind of beyond objective exploration,

105
00:09:37,930 --> 00:09:43,510
meaning that you keep experimenting in order to actually learn more precisely what is effective.

106
00:09:44,590 --> 00:09:49,390
The key challenge when when you do this type of adaptive trial is to do three things in balance.

107
00:09:49,630 --> 00:09:53,260
You might be greedy and just go all in like something looked bad in the beginning.

108
00:09:53,560 --> 00:09:55,480
This would only do that for the rest of time.

109
00:09:55,960 --> 00:10:01,060
But that might be a better idea because if I just get stuck on something that was just random really good in the beginning,

110
00:10:01,060 --> 00:10:06,190
but actually is not such a good idea, or you might be very conservative and slow and just keep experimenting,

111
00:10:06,190 --> 00:10:10,330
experimenting, but then you actually wasting the life chances of your participants.

112
00:10:10,990 --> 00:10:18,580
And the key thing is like how best to move from exploring towards exploiting what you've learned as that one thing you put a lot of attention to here.

113
00:10:18,760 --> 00:10:27,000
And then the other thing that you're targeting right here, the idea is not everything might work for everybody in the same way, right?

114
00:10:27,040 --> 00:10:34,300
Maybe if you're an older woman with higher education, but no prior labour market experience, who was deployed from Syria,

115
00:10:34,390 --> 00:10:38,840
the type of interventions that are useful for you might be quite different than if you're,

116
00:10:38,860 --> 00:10:43,300
say, like a young man who just dropped out of high school who grew up in Jordan.

117
00:10:43,360 --> 00:10:49,900
So important to kind of figure out not only what's effective on average, but what effective for whom.

118
00:10:50,260 --> 00:10:56,980
And your targeting is that you learn what what works for whom, and then actually give people the interventions that make sense for them.

119
00:10:57,220 --> 00:11:03,790
The second kind of innovation or experimental design that we're not only adapting over time towards the more successful interventions,

120
00:11:04,360 --> 00:11:09,939
but are all the targeting interventions that make sense for different groups of participants and do that?

121
00:11:09,940 --> 00:11:13,080
There's kind of a key challenge, which is how do you combine information, right?

122
00:11:13,090 --> 00:11:20,319
So maybe you've never in your data before had an older woman with higher education, but no prior labour market experience or something like that.

123
00:11:20,320 --> 00:11:25,360
But then you might like draw information from other maybe partially similar participants of the experiment,

124
00:11:25,720 --> 00:11:28,510
but then over time you might have more and more information for different

125
00:11:28,510 --> 00:11:32,450
groups and then you can just focus on learning from these groups particularly.

126
00:11:33,190 --> 00:11:38,110
And so that that again, something that we implemented there, it's called a patient hierarchical model.

127
00:11:38,440 --> 00:11:41,830
But the basic idea is that you combine this information in the optimal way

128
00:11:42,160 --> 00:11:45,640
between the group that you're targeting and other groups that might be similar.

129
00:11:46,210 --> 00:11:51,850
So that that in a nutshell, kind of the idea of this incremental design you're adapting over time to the better interventions,

130
00:11:51,850 --> 00:12:00,930
the targeting for the groups for which then another by a little bit complicated or daunting if you're running an experiment through itself.

131
00:12:00,970 --> 00:12:04,300
But I want to emphasise at the end of the day, it's not that complicated.

132
00:12:04,450 --> 00:12:09,239
There's a little bit of tools that we provide the code and apps that if you want to run an experiment yourself,

133
00:12:09,240 --> 00:12:15,620
the style can just download and use in the modern digital era is actually not too hard to implement an experiment like that for them.

134
00:12:15,640 --> 00:12:20,140
Then you would have in the field offices for case broker third unemployed job seekers.

135
00:12:20,740 --> 00:12:28,630
They might have some tablet or smartphone. Very little appetite for them, what intervention to assign whom to.

136
00:12:28,840 --> 00:12:34,030
And then in the background we can have a little program running that actually implements our adaptive target dog written.

137
00:12:34,420 --> 00:12:40,570
And so that worked out surprisingly smoothly, I would say. And it's actually, I think, quite adaptable to the many different settings.

138
00:12:41,500 --> 00:12:41,979
Thanks, Max.

139
00:12:41,980 --> 00:12:49,090
I remember very well the discussions in the research team that it was almost like logging in to check the scores of your favourite football team,

140
00:12:49,630 --> 00:12:51,010
but especially in the early days,

141
00:12:51,010 --> 00:12:58,510
we could actually see the assignment probabilities going up and down on a dashboard of sorts that I think you had kindly prepared.

142
00:12:58,630 --> 00:13:00,220
Can you tell us just a little bit more?

143
00:13:00,220 --> 00:13:06,460
Because I think the intuition is clear here, but the algorithm is handling all of the assignment probabilities,

144
00:13:06,670 --> 00:13:09,879
and then the algorithm is going to give assignment probabilities. And then what?

145
00:13:09,880 --> 00:13:14,110
And then the researcher then uses those to randomise or indeed the code then just

146
00:13:14,110 --> 00:13:19,149
generates a randomisation for somebody else who enters the program the next day,

147
00:13:19,150 --> 00:13:22,840
as it were. Is that a fair ensured acceptance? I mean, that's like the dashboard, right?

148
00:13:22,840 --> 00:13:25,719
That was the back end for us to kind of follow what's going on into the score.

149
00:13:25,720 --> 00:13:29,230
Interesting on the level of the people in the field were actually implementing this.

150
00:13:29,560 --> 00:13:32,920
They don't have to deal with any of that. They just go to the website or app,

151
00:13:33,400 --> 00:13:39,160
barely enter your have like a job security characteristics and that gives them like

152
00:13:39,160 --> 00:13:43,600
the information dimension or give them some cash to help them search for a job.

153
00:13:43,780 --> 00:13:48,040
Everything else is kind of run in the background, but the program that kind of collect the data over time,

154
00:13:48,040 --> 00:13:54,069
how's that for different interventions were collected in a retreat somewhere and then the program without

155
00:13:54,070 --> 00:13:59,380
the threat and people that decide how to assign participants to the program to different treatment arms.

156
00:13:59,890 --> 00:14:06,160
One of the interesting things about doing this, I guess, in a labour market context is that you don't necessarily have to wait six.

157
00:14:06,250 --> 00:14:09,880
1218 months to know whether a policy is or isn't working.

158
00:14:10,030 --> 00:14:17,980
In this context, as you alluded to, we were able to have relatively quick return information on whether someone had found a job in the short term.

159
00:14:18,520 --> 00:14:24,520
And of course, that may mask longer term effects. Some some treatments in some contexts may take a long time to materialise.

160
00:14:24,940 --> 00:14:26,620
But I guess the point is, in this case,

161
00:14:26,620 --> 00:14:32,350
we thought that that was going to be a pretty good proxy for people's longer term success on different treatments.

162
00:14:32,890 --> 00:14:39,280
That's exactly right. So a participant got a call, I think like six weeks or two months after participating in the program,

163
00:14:39,280 --> 00:14:41,710
and then they were just off the phone, off the job.

164
00:14:42,190 --> 00:14:49,230
And and that's the key outcome that we're trying to maximise is the probability of people finding a job a couple of months after participating.

165
00:14:49,510 --> 00:14:55,459
And that might be quite easy to implement and dump that can not be implemented other settings, right?

166
00:14:55,460 --> 00:15:03,010
So the type of method it used, a lot of online settings, they find that companies are maximising click right.

167
00:15:03,010 --> 00:15:05,889
That's something that you get very quick feedback like did somebody click on

168
00:15:05,890 --> 00:15:09,129
this link that we put it on the website or not report that kind of situation.

169
00:15:09,130 --> 00:15:11,530
It's very easy to adapt quickly and efficiently.

170
00:15:11,950 --> 00:15:19,149
But then there might be other settings where take a long time for programs to realise to thinking that is going to get a sample.

171
00:15:19,150 --> 00:15:25,570
I think like typical interventions that would affect Alzheimer's, it might take 30, 40 years for any results to be seen.

172
00:15:25,870 --> 00:15:32,260
And that kind of setting, it would not be very useful to think about adaptive methods because you won't have all types of quickly.

173
00:15:32,920 --> 00:15:37,149
But I think a lot of the type of policies you might be interested in the labour market or development

174
00:15:37,150 --> 00:15:41,979
economics context actually would not be unexpected within the timeframe of an experiment.

175
00:15:41,980 --> 00:15:48,180
But if something was effective and as long as you have observed outcomes for some people before the end of the experiment,

176
00:15:48,180 --> 00:15:52,690
then then you have information that you can use in order to take better care of your participants.

177
00:15:52,730 --> 00:16:01,120
And again, like I want to emphasise the ethical dimension of that, not just seeing them at the end, but incorporated depends on an end in themselves.

178
00:16:01,600 --> 00:16:06,759
Probably not just using that concern for our academic papers of our policy recommendations,

179
00:16:06,760 --> 00:16:12,010
but we're really trying to do right by them, by giving them the interventions that are most helpful to them.

180
00:16:13,070 --> 00:16:18,320
I think one thing that I would add for the people who are listening is that I think one area where I

181
00:16:18,330 --> 00:16:23,870
expect this to be particularly applicable is experiments that you're learning with the larger situations,

182
00:16:23,870 --> 00:16:31,309
and that, of course, in close guidance to, for example, schools or hospitals or courts or things like that,

183
00:16:31,310 --> 00:16:38,410
where you basically have a fixed institution that serves a population that gets the service and the needs a lot of them,

184
00:16:38,420 --> 00:16:40,549
and we do a lot of work with these kind of institutions.

185
00:16:40,550 --> 00:16:47,210
And in this case you get both fast feedback and a constant stream of new people that would benefit from the other thing you do.

186
00:16:47,990 --> 00:16:52,910
So I just wanted to plug that in. Many of the settings where we work in as applied.

187
00:16:52,910 --> 00:16:57,110
Economist Actually, this would actually be a good match. I think that's a great point.

188
00:16:57,110 --> 00:17:00,829
Stefano And I think also fascinating that you raise the point of the partner institution.

189
00:17:00,830 --> 00:17:03,970
I think this dovetails tails nicely with Max's point about ethics.

190
00:17:03,980 --> 00:17:08,360
I would find it really difficult to go into a crisis situation where you've got,

191
00:17:08,720 --> 00:17:12,170
for example, a large NGO trying to help a target population and say, Hi,

192
00:17:12,170 --> 00:17:18,469
can I do a long term fixed proportion asset and please don't do anything with the control group for two

193
00:17:18,470 --> 00:17:22,100
years because I need to write my paper out and the any researcher would feel comfortable with that.

194
00:17:22,400 --> 00:17:29,090
So I think this is where the ethical dimension that Max raises touches on the institutional dimension that you can say to a policymaker,

195
00:17:29,090 --> 00:17:30,290
Look, we're going to experiment,

196
00:17:30,290 --> 00:17:37,399
but we're going to do our very best to move respondents towards treatments that seem to be helping them and not just helping on average,

197
00:17:37,400 --> 00:17:41,540
but, as Max explained, helping for the particular kind of respondent that they are.

198
00:17:42,080 --> 00:17:47,629
I think it's a great point. In particular, oftentimes, I think when you run the risk of using kind of the market ethics way, right,

199
00:17:47,630 --> 00:17:51,530
then you have this tension between like an implementation partner organisation and the

200
00:17:51,530 --> 00:17:56,170
researchers and the implementation partners helping be believed know what works best.

201
00:17:56,180 --> 00:18:01,290
They believe they have recommended to clients their goal is not to do a research right.

202
00:18:01,310 --> 00:18:06,140
And then you have the researchers who run the raised academic papers like precisely estimated treatment effect.

203
00:18:06,650 --> 00:18:14,330
And those are two different goals. And so I think about this book, like adaptive experiments like BE are contributing to hear about that.

204
00:18:14,660 --> 00:18:19,670
But I wanted to do is like align more closely the objective of the experimenters with the objectives of implementation

205
00:18:19,670 --> 00:18:27,350
partners and the theoretical that's already committed that our goal is to help refugees by job quickly.

206
00:18:27,950 --> 00:18:32,630
You could say that's our goal to good designing our experiment to help refugees by topic.

207
00:18:33,110 --> 00:18:39,079
There is no conflict here, and so you should be happy to let us do the experiment because we are actually just helping you to

208
00:18:39,080 --> 00:18:43,550
fulfil your mission as opposed to kind of being in a tension with what you're trying to achieve.

209
00:18:44,090 --> 00:18:50,780
And so that might actually also open all kinds of new sites and venues where you could run experiments very quickly.

210
00:18:50,900 --> 00:18:54,850
But beyond that, to go back to politically feasible, I completely agree.

211
00:18:54,860 --> 00:19:00,049
In a minute, I want to ask Stefano to actually tell us about the results. Before I do that, let me let me pose one more question to you, Max.

212
00:19:00,050 --> 00:19:04,910
And and I might I might make friends and enemies in the world of Bayesian statistics here,

213
00:19:04,910 --> 00:19:12,170
but I want to throw in a slightly a slightly nerdy question. Suppose you're dealing with a policymaker who says, Alright, I'll do this adaptive rc t,

214
00:19:12,470 --> 00:19:17,290
but I have a very strong belief that treatment one is going to work and treatment two is not.

215
00:19:17,600 --> 00:19:25,040
Some people would say, Oh, we should, you should use that prior belief and we should use it to, if you like, seed the model.

216
00:19:25,700 --> 00:19:28,430
So are we going to start by putting a lot more people into the treatment arm,

217
00:19:28,610 --> 00:19:35,420
reflective not of data that we've collected in the experiment, but reflecting the strong prior that the policymaker has.

218
00:19:35,780 --> 00:19:40,490
Other people would say, No, no, no, it's interesting. It's important that the policymaker feels they know what works.

219
00:19:40,820 --> 00:19:48,170
But we should start with a much less informative prior to use the terminology and let the experiment in the data tell us what's working.

220
00:19:48,530 --> 00:19:52,339
I know there are different views out there about this in our context.

221
00:19:52,340 --> 00:19:58,280
We collected information about what policymakers thought, but we didn't feed that information directly into the algorithm.

222
00:19:58,610 --> 00:20:05,000
Max I'm just keen to draw you out just for a few more minutes on your thoughts about the respective pros and cons of those approaches.

223
00:20:05,270 --> 00:20:10,190
So I think that that makes perfect sense for policymakers, partners and clients for that matter,

224
00:20:10,520 --> 00:20:14,570
often have a lot of information about what might or might not work.

225
00:20:15,110 --> 00:20:19,129
Right? So we learned that in our experiment beds or running focus groups as refugees,

226
00:20:19,130 --> 00:20:25,610
and they had very strong opinions about the interventions which largely line up with what we found later in the experiment.

227
00:20:26,420 --> 00:20:32,239
And so that kind of information can be very useful for deciding to experiment that they like.

228
00:20:32,240 --> 00:20:37,760
If we had a strong belief beforehand that some intervention was more effective than make a lot of sense for that

229
00:20:37,760 --> 00:20:44,240
intervention to look at a larger sphere for the observation or for our participants to get assigned to that intervention,

230
00:20:44,700 --> 00:20:47,720
I think it's very important to think separate here.

231
00:20:48,170 --> 00:20:50,210
One is how you run the experiment, right?

232
00:20:50,240 --> 00:20:55,760
You might put a lot of people in the information intervention and let people in the batch intervention or something like that,

233
00:20:56,330 --> 00:21:03,620
but that doesn't prevent you from that effect. Analysing the experiment in a way that your analysis doesn't draw on that prior at all.

234
00:21:03,890 --> 00:21:06,799
Right? So if you just want to have the get scientific evaluation,

235
00:21:06,800 --> 00:21:12,440
that only depends on the data and does not draw on the policymaker or a partner or a client believes.

236
00:21:12,860 --> 00:21:16,190
That's perfectly fine. You can do that in Iran and experiment.

237
00:21:16,580 --> 00:21:23,330
But use those problems to make our partner believe in order to decide which treatment to find more properly understood.

238
00:21:23,330 --> 00:21:28,700
Understood. Thanks, Max. I feel like we've been teasing our listeners for a while now with the discussion of the actual results.

239
00:21:29,330 --> 00:21:36,770
Let me get back to you, Stefano. Why don't you tell us what we think we learned in this context about the constraints that refugees face in the labour

240
00:21:36,770 --> 00:21:42,230
markets and how we think that might be relevant for other kinds of policy interventions in similar settings in the future.

241
00:21:43,070 --> 00:21:48,920
Thank you. So, so I guess maybe I'll talk about treatment effects,

242
00:21:48,920 --> 00:21:55,190
but perhaps let me say a couple of words about the population that we are working with just just to to set this in context.

243
00:21:55,640 --> 00:22:02,990
So our key population into Syria was a population of Syrian refugees from cities around Jordan.

244
00:22:03,500 --> 00:22:10,600
These are the population that perhaps in Syria was not necessarily particularly poor, but wasn't doing very well in Jordan.

245
00:22:10,600 --> 00:22:18,020
And so employment rates were extremely low and so waiting so, so so people were very close to the poverty line, many of them below the poverty line.

246
00:22:18,470 --> 00:22:27,080
And there was quite a bit of interesting work and quite a bit of work experience as well, and but very little at least formal employment.

247
00:22:27,110 --> 00:22:34,489
We know that some people were working poorly, although they were not necessarily very cautious about saying that they would make the law.

248
00:22:34,490 --> 00:22:41,540
And so given that they are quite vulnerable as refugees, these were something that it wasn't really forthcoming in discussing with us.

249
00:22:41,990 --> 00:22:45,510
Now, we then offered these three interventions.

250
00:22:45,530 --> 00:22:47,989
We offered these interventions to the Syrian refugees,

251
00:22:47,990 --> 00:22:53,270
that they were the kind of citizen population in our study, but also to a second population of local Jordanians.

252
00:22:53,630 --> 00:22:56,780
This is actually quite important for two reasons. One,

253
00:22:56,780 --> 00:23:00,570
because we want to see whether the same interventions within the constraints that prevent this

254
00:23:00,680 --> 00:23:05,630
population from accessing the labour markets are similar or whether they need different policies.

255
00:23:06,080 --> 00:23:08,090
And two, going back to ethics,

256
00:23:08,090 --> 00:23:18,130
it's a very important commitment of IFC to always match the support that they give to the population that are going to countries hosting to an

257
00:23:18,140 --> 00:23:25,760
equivalent equivalent amount of support given to other national individuals living in the same areas that are also stagnant or in this case,

258
00:23:25,760 --> 00:23:35,450
have training climate. So it's it's a kind of requisite for U.S. to do this kind of equitable policy

259
00:23:35,450 --> 00:23:39,680
allocation to avoid indiscriminate in the perception the refugees of payments.

260
00:23:40,010 --> 00:23:44,030
So I would say we also can bring a second population of national Jordanians.

261
00:23:44,520 --> 00:23:50,830
Now, what did we find? We interviewed individuals six weeks after treatment and two and four months after treatment.

262
00:23:50,840 --> 00:23:55,550
So we have three courts entire which we can observe after six weeks after treatment.

263
00:23:55,710 --> 00:24:02,600
It was too soon. We found that there were minimal effects on employment for all groups two and four months after treatment.

264
00:24:02,600 --> 00:24:10,280
And we start with the cash intervention in particular, increase the job search and then in turn employment and earnings.

265
00:24:11,120 --> 00:24:14,090
Now, the type of effects that we are talking about,

266
00:24:14,480 --> 00:24:20,240
this is close to 4% despite increasing employment, which we should put in context in least two ways.

267
00:24:20,900 --> 00:24:24,260
Number one, purpose at this point, it's not huge.

268
00:24:24,470 --> 00:24:28,130
This is clearly not transformational in a sense is a light touch intervention.

269
00:24:28,140 --> 00:24:36,380
So perhaps we shouldn't expect them to transformation face magically getting the employment rates from about 9% to about 13%.

270
00:24:36,650 --> 00:24:40,640
So this is still a population that is fairly locked out of the labour market.

271
00:24:41,030 --> 00:24:45,120
However, in relative terms, this is a fairly large increase in employment.

272
00:24:45,140 --> 00:24:52,640
So it's a 50% increase. And also relative to the kind of treatment effects that we found in the actual labour market policy literature,

273
00:24:53,090 --> 00:24:58,580
the search literature that they decide to devise policies in other contexts in the world.

274
00:24:58,920 --> 00:25:02,250
It's actually in line with what people from elsewhere.

275
00:25:02,630 --> 00:25:07,430
In fact, it's a towards the upper part of the distribution of of the economic mix.

276
00:25:07,490 --> 00:25:14,450
Same story for the treatment effects of income. We found that income goes up by almost 60%, the absolute amount.

277
00:25:14,720 --> 00:25:19,180
It's limited in a sense because we're starting from a very low base in relative terms.

278
00:25:19,250 --> 00:25:24,350
This is a this is a large effect. One more thing that I want to see about the cash interventions,

279
00:25:24,350 --> 00:25:31,940
that this is one of the few pattern of tool and conditional cash interventions that are designed to boost search.

280
00:25:31,970 --> 00:25:36,650
There are other papers in this picture that show that if you gave people conditional cash,

281
00:25:36,650 --> 00:25:42,380
for example, this can build trust that enable people to take transport and integrity.

282
00:25:42,620 --> 00:25:51,080
So it does intervention usage of it. But it wasn't clear whether that was the case because you were kind of almost forcing people to

283
00:25:51,740 --> 00:25:56,479
search for work in order to benefit from the intervention or whether people would have boosted that,

284
00:25:56,480 --> 00:26:02,840
would have increased the job search, giving some additional resources, whether this was something that was very high priority.

285
00:26:03,240 --> 00:26:07,190
They were not doing extra investment in job search because of lack of cash.

286
00:26:07,760 --> 00:26:12,320
So it's quite interesting that we find that giving these are conditional cash, the experience.

287
00:26:13,310 --> 00:26:21,680
Job search quite a bit. And then this leads to employment. I mean, Jordanians, on the other hand, while this is still a fairly poor population,

288
00:26:21,680 --> 00:26:26,930
but they're richer than the Syrians, the French underestimate and they're generally better off.

289
00:26:27,450 --> 00:26:29,930
And we actually find the caches very limited effects.

290
00:26:30,320 --> 00:26:37,310
The NATO intervention, the permission, the natural component shows also increased job search, less so than the cash intervention.

291
00:26:37,670 --> 00:26:41,210
And they do lead to some employment effects and to two month massacre.

292
00:26:41,750 --> 00:26:45,130
But these effects are smaller and less persistent.

293
00:26:45,140 --> 00:26:53,630
So by the time of the former detainee, we see positive coefficients by quality and in general statistically significant.

294
00:26:54,020 --> 00:27:02,780
So these are the two interventions, although they seem to have at least activated some job search and generated some gains, they proved less.

295
00:27:03,270 --> 00:27:06,650
I think Stephane, I mean, it's a really interesting summary.

296
00:27:06,710 --> 00:27:11,780
This issue of how you help displaced people in urban labour markets is sadly not an issue that's going away.

297
00:27:12,170 --> 00:27:20,930
So the takeaway for a policy setting for future crises is that when we think about cash support for refugees or displaced populations,

298
00:27:21,110 --> 00:27:25,669
that this is not only valuable intrinsically in terms of increasing welfare,

299
00:27:25,670 --> 00:27:31,110
but we should actually think of this as unlocking such capacity in potentially new labour markets.

300
00:27:31,130 --> 00:27:36,440
Is that a fair takeaway? I know as researchers we always struggle to think about how to generalise our results to other contexts,

301
00:27:36,440 --> 00:27:42,410
but it feels to me like that's a sort of insight that we gain from this population that may be relevant for future crises.

302
00:27:43,070 --> 00:27:49,820
Absolutely. So I would say that basically one important takeaway, of course if you generalise to this crisis, other things will change.

303
00:27:49,820 --> 00:27:56,090
But it's probably true overall that refugees typically lose a lot of assets during the process of displacement.

304
00:27:56,150 --> 00:27:59,960
So this is a population with a little bit of cash and very high returns.

305
00:28:00,170 --> 00:28:04,850
The point stands that also in our population, employment rates and incomes remain quite low.

306
00:28:04,850 --> 00:28:09,950
So it's not it's not going to be by any means a sufficient intervention,

307
00:28:09,950 --> 00:28:16,010
but it's probably a small but highly targeted thing that one can do help refugees operate in the in the new labour markets.

308
00:28:16,330 --> 00:28:21,860
That is one more thing that I want to say, which is of course there are many things that would differ when you compare this to refugee crisis.

309
00:28:22,130 --> 00:28:26,270
But I think one factor that is of essential importance is language.

310
00:28:26,750 --> 00:28:30,290
And so some refugee crisis example, the word exodus by Jordan or Lebanon,

311
00:28:31,040 --> 00:28:36,919
you find that refugees coming to country speaking effectively, the language that is spoken in the country,

312
00:28:36,920 --> 00:28:39,050
maybe different dialect, maybe they have a bit of an accent,

313
00:28:39,350 --> 00:28:45,290
but by and large they're already able to communicate with pretty much anybody in the host country.

314
00:28:45,860 --> 00:28:49,910
The refugees are going to Turkey, for example, in a very different position.

315
00:28:49,940 --> 00:28:58,759
In fact, in Turkey we find that they tend to mostly work in self-employment and keep training with Syrians living elsewhere in the world

316
00:28:58,760 --> 00:29:04,640
rather than being getting integrated in the local economy like the Jordanian and Lebanese policymakers with their to do.

317
00:29:05,210 --> 00:29:08,840
I'm not entirely sure which of the two kinds of shocks is more common.

318
00:29:08,840 --> 00:29:12,800
For example, Venezuelans going to Colombia, and they spoke the same language.

319
00:29:12,940 --> 00:29:20,210
A That's fine, but same language. Lots of other refugees and some of those in the in Eastern Africa, for example,

320
00:29:20,690 --> 00:29:26,419
you find the populations that that are really in in those in Europe, in many cases, in fact, refugees missing Europe.

321
00:29:26,420 --> 00:29:31,580
Very often language is affected by this. So I would say that in cases where language is also an issue,

322
00:29:31,910 --> 00:29:36,229
I'm not entirely sure this is also generalised simply because refugees may not

323
00:29:36,230 --> 00:29:40,790
be immediately employable before they pick up the essential language skills.

324
00:29:41,640 --> 00:29:42,110
Do you think?

325
00:29:42,110 --> 00:29:48,499
That's a really excellent point, and let me loop back to a point that Max made earlier, which is that we did a bunch of qualitative work, of course,

326
00:29:48,500 --> 00:29:53,000
talking to this population and this narrative about the importance of cash for unlocking search

327
00:29:53,300 --> 00:29:57,830
was certainly one that seemed to resonate with a lot of the refugee respondents themselves,

328
00:29:57,830 --> 00:30:02,360
which I think is interesting, and maybe also something that would be useful in terms of learning in other contexts.

329
00:30:02,360 --> 00:30:06,829
Exactly. About the practicalities that you talk about. Well, I've really enjoyed this conversation.

330
00:30:06,830 --> 00:30:11,330
It's brought back a lot of fond memories of a project that I really enjoyed doing and where I certainly learned a lot.

331
00:30:11,540 --> 00:30:15,200
Thanks very much, Stefano and Max, and thanks to all of our listeners for tuning in.

332
00:30:15,560 --> 00:30:21,020
I'm certainly very keen to see how these these two different literatures go to see where people think about new interventions to

333
00:30:21,020 --> 00:30:28,130
help these kind of vulnerable displaced populations and also to see the emerging field of adaptive and targeted field experiments.

334
00:30:28,460 --> 00:30:31,510
Thanks, everyone. Very exciting. Likewise. Thank you so much. Thank you so much.

335
00:30:31,590 --> 00:30:32,660
I've only stepped out.