Welcome to the CSA. You research podcasts.
This is a series of conversations about projects taking place through the Centre for the Study of African Economies at the University of Oxford.
I'm Simon Quinn. I'm an associate professor at the Department of Economics and Public Policy at Imperial College Business School.
And until last year I was privileged to be Deputy Director of CSA.
Today we're going to be talking about the project an adaptive, targeted field experiment, job search assistance for refugees in Jordan.
And this was a project that was run in partnership between CSA and the International Rescue Committee.
When we talk about this project, we're talking about what I think is a really important policy question,
namely, how is it that different kinds of policy can help refugees and other displaced populations?
And we all know, sadly, that we live in a time when this is a really important issue for policy in a lot of different geographical settings.
When we talk today about this issue, we're thinking about Syrian refugees and also local jobseekers.
In Jordan. The labour market in Jordan is characterised by very low employment rates, at least by international standards,
and employment rates among refugees are much lower than among Jordanians.
So this is a project, as we're going to discuss,
in which we think about the impact of three interventions that were designed to improve formal employment outcomes,
both for Syrian refugees and for local jobseekers.
I'm really excited to be joined today by two of my co-authors on this project, Stefano Carrier,
who's a professor in the Department of Economics at the University of Warwick,
and Max Casey, a professor in the Department of Economics at the University of Oxford.
Max and Stefano, thank you so much for joining me. It's great to be able to discuss this project to you.
Hi Simon. Thanks so much for having me on this podcast. Thank you.
SIMON It's really great to be here and to be able to talk about this project.
Maybe if I could start with you, Max.
I mean, one of the things that I found fascinating working with everyone on this project is that this is an adaptive field experiment.
And I learned a lot about this whole idea of adaptive experiments by working on it.
And it's something that I actually think is one of the most exciting parts of this research paper and
hopefully something that will be relevant to a bunch of other researchers in this and other field.
So can I ask you, first of all, to tell everyone what an adaptive experiment is and how we implemented the adaptive experiment in this context?
So we ran this study in the form of an adaptive experiment and more specifically, an adaptive targeted experiment.
Maybe that's why we go there. Let's back up a little bit and talk about what the purpose of the experiment is or what the goal is,
because that really points up to a to why this might be a good idea and it might be useful
in many other experiments over the last couple decades or so in developing the economics,
and that become very standard around randomised controlled trials.
Read the randomised controlled trial, you randomly assign different people to the different treatment arms if it's called different policy,
and then you see about how things turn out and you get hopefully a credible estimate of how effective your policies are.
That makes perfect sense, I think, for a lot of cases.
What that does is it runs the experiment in a way where the goal is to get precise estimates of how effective a policy may be precisely to compare.
But that might not be the only thing you care about. Right?
So getting precise estimate is not the same thing as informing quality choices, and it's also not the same thing at helping participants.
So those are two to rather different objectives. And then we want to pursue this other objective.
Then they might need to run our experiment in a different way to maybe to illustrate that very clear example by picking nickel trial in Madison,
where you have a track and a placebo or maybe some standard treatment that you compare to a new treatment, and then you run an experiment.
And if you're sticking to a standard protocol,
you just have like a 5050 split between the two treatment arms and you observe how well
they're doing and you just keep running with that until the end of the experiment.
But now what might happen? That's an experiment. Maybe halfway through, you realise that the new drugs killing everybody does something very good.
Or it might be that the new drug is kind of a miracle curing everybody.
And so you would want to use that information for the sake of the patient.
You wouldn't just want to keep going with your protocol ignoring what you learned.
Just use your your patient as a means to an end. You want to help your patient.
And so what that means is you might want to adapt over time.
You might want to shift your treatment or your assignment to treatment or to treatment are far better.
So if that new drug during everybody that you want to assign the new drug more awkwardly or as time progresses,
or if it seems to hurt people, then you might want to quickly shift away.
And so that's that's kind of the core of the activity, and that's something that we implemented now in our experiment.
It ran over a whole number of months. And in our experiment, the goal was to help Syrian refugees find jobs in the Indian labour market.
In order to do that, we observed that for different treatment. And Stefano, I guess, can tell us more about this later.
The very different amazement test and not intervention to try and help people find a job over time to
give us shift treatment towards those interventions that are most successful at finding people job.
Yes, absolutely. So thanks very much, Max. Maybe can I bring you in, Stefano, and tell us in brief about the different treatment arms?
And then I want to go back to Max and think about how we implemented this in a way that tries to think about different subgroups.
But let me hold hold off on that. And so we hear a bit about the treatments themselves.
So we had three interventions that we were interested in, in trying out in this project.
One was a small and conditional cash grant, which we calculated to cover the costs of job search for about a couple of months.
So we had baseline data on how much people were spending and job search.
And so we we calculated an amount that people would spend in about a two month period.
We know that a job search can actually be surprisingly expensive because people have to use transport
and travel around town and have all sorts of expenditures that when you live on a very low income,
can actually make up a large share with a surprisingly large share of your budget.
And so we try to support them in making these investments by making cash available upfront at the beginning of the set,
spent the cash in the first intervention. The second intervention was an information intervention where we basically tried to
coach people to change people in presenting themselves to employers by research
that we've done that would use Hamilton and other researchers in Ethiopia and South
Africa and Uganda shows that jobseekers ability to convey their employability,
their skills and their net worth to employers, it's really a key ingredient to their success in the labour market.
And so in the second intervention,
jobseekers who may not necessarily be familiar with the new standards of the Jordanian labour market because they were refugees,
where we're basically trained in approaching Jordanian employers and presenting their skills and convincing them.
And then finally there was a second intervention, information intervention.
And finally we had a psychological or energy dimension, which was highly motivated by the idea, which is well documented.
That Job said it's a very difficult and frustrating process which requires you to sustain the motivation to search hard over a long period of time.
So motivation was central to the success of job search,
and people struggle to maintain high levels of motivation when they start getting rejection injection.
So to help people maintain high level job search effort,
we use the insights from recent psychological literature as well as a recent trial in South Africa,
and devised a goal setting intervention where jobseekers would were encouraged to set themselves some jobs to achieve our goals.
And then we would they would report on whether they attained those goals or not.
They would be able to set the other goals for the following week. So these were the three interventions cash information and the psychological nudge.
And of course, there was also a support group, a control group that didn't receive any of these three policies.
Thank you, Stefano. I remember a fascinating visit that you and I had with some of our partners from the International Rescue Committee,
where we actually went from Amman up to the northwest of Jordan to Irbid and spoke to several refugees.
And I think I think we both felt it was like fascinating and humbling to hear about
the whole constellation of different challenges that these respondents were facing.
And I think it's exactly as you say.
Partly it's about motivation, partly it's about understanding the labour market, partly it's about having the cash to actually access jobs.
And I think that's absolutely right. This is why we went for these three treatments.
In this context, Max, we opened by talking about targeting.
Can I come back to you, having spoken about the treatments,
to tell us a little bit more about the idea of targeting and how we implemented it in this setting?
That sounds good, Simon.
Maybe before we get to the targeting, though, let me check a little bit more about these other activities that I started out with.
Right. To deepen your knowledge? We had three different interventions at the base of the control arm, and they talked about this idea,
how you want to move over time towards the more successful treatments.
This is something that no one is exploiting the information that you have and the machine learning literature.
But there is kind of a tension there between exploitation and exploration, which is kind of beyond objective exploration,
meaning that you keep experimenting in order to actually learn more precisely what is effective.
The key challenge when when you do this type of adaptive trial is to do three things in balance.
You might be greedy and just go all in like something looked bad in the beginning.
This would only do that for the rest of time.
But that might be a better idea because if I just get stuck on something that was just random really good in the beginning,
but actually is not such a good idea, or you might be very conservative and slow and just keep experimenting,
experimenting, but then you actually wasting the life chances of your participants.
And the key thing is like how best to move from exploring towards exploiting what you've learned as that one thing you put a lot of attention to here.
And then the other thing that you're targeting right here, the idea is not everything might work for everybody in the same way, right?
Maybe if you're an older woman with higher education, but no prior labour market experience, who was deployed from Syria,
the type of interventions that are useful for you might be quite different than if you're,
say, like a young man who just dropped out of high school who grew up in Jordan.
So important to kind of figure out not only what's effective on average, but what effective for whom.
And your targeting is that you learn what what works for whom, and then actually give people the interventions that make sense for them.
The second kind of innovation or experimental design that we're not only adapting over time towards the more successful interventions,
but are all the targeting interventions that make sense for different groups of participants and do that?
There's kind of a key challenge, which is how do you combine information, right?
So maybe you've never in your data before had an older woman with higher education, but no prior labour market experience or something like that.
But then you might like draw information from other maybe partially similar participants of the experiment,
but then over time you might have more and more information for different
groups and then you can just focus on learning from these groups particularly.
And so that that again, something that we implemented there, it's called a patient hierarchical model.
But the basic idea is that you combine this information in the optimal way
between the group that you're targeting and other groups that might be similar.
So that that in a nutshell, kind of the idea of this incremental design you're adapting over time to the better interventions,
the targeting for the groups for which then another by a little bit complicated or daunting if you're running an experiment through itself.
But I want to emphasise at the end of the day, it's not that complicated.
There's a little bit of tools that we provide the code and apps that if you want to run an experiment yourself,
the style can just download and use in the modern digital era is actually not too hard to implement an experiment like that for them.
Then you would have in the field offices for case broker third unemployed job seekers.
They might have some tablet or smartphone. Very little appetite for them, what intervention to assign whom to.
And then in the background we can have a little program running that actually implements our adaptive target dog written.
And so that worked out surprisingly smoothly, I would say. And it's actually, I think, quite adaptable to the many different settings.
Thanks, Max.
I remember very well the discussions in the research team that it was almost like logging in to check the scores of your favourite football team,
but especially in the early days,
we could actually see the assignment probabilities going up and down on a dashboard of sorts that I think you had kindly prepared.
Can you tell us just a little bit more?
Because I think the intuition is clear here, but the algorithm is handling all of the assignment probabilities,
and then the algorithm is going to give assignment probabilities. And then what?
And then the researcher then uses those to randomise or indeed the code then just
generates a randomisation for somebody else who enters the program the next day,
as it were. Is that a fair ensured acceptance? I mean, that's like the dashboard, right?
That was the back end for us to kind of follow what's going on into the score.
Interesting on the level of the people in the field were actually implementing this.
They don't have to deal with any of that. They just go to the website or app,
barely enter your have like a job security characteristics and that gives them like
the information dimension or give them some cash to help them search for a job.
Everything else is kind of run in the background, but the program that kind of collect the data over time,
how's that for different interventions were collected in a retreat somewhere and then the program without
the threat and people that decide how to assign participants to the program to different treatment arms.
One of the interesting things about doing this, I guess, in a labour market context is that you don't necessarily have to wait six.
1218 months to know whether a policy is or isn't working.
In this context, as you alluded to, we were able to have relatively quick return information on whether someone had found a job in the short term.
And of course, that may mask longer term effects. Some some treatments in some contexts may take a long time to materialise.
But I guess the point is, in this case,
we thought that that was going to be a pretty good proxy for people's longer term success on different treatments.
That's exactly right. So a participant got a call, I think like six weeks or two months after participating in the program,
and then they were just off the phone, off the job.
And and that's the key outcome that we're trying to maximise is the probability of people finding a job a couple of months after participating.
And that might be quite easy to implement and dump that can not be implemented other settings, right?
So the type of method it used, a lot of online settings, they find that companies are maximising click right.
That's something that you get very quick feedback like did somebody click on
this link that we put it on the website or not report that kind of situation.
It's very easy to adapt quickly and efficiently.
But then there might be other settings where take a long time for programs to realise to thinking that is going to get a sample.
I think like typical interventions that would affect Alzheimer's, it might take 30, 40 years for any results to be seen.
And that kind of setting, it would not be very useful to think about adaptive methods because you won't have all types of quickly.
But I think a lot of the type of policies you might be interested in the labour market or development
economics context actually would not be unexpected within the timeframe of an experiment.
But if something was effective and as long as you have observed outcomes for some people before the end of the experiment,
then then you have information that you can use in order to take better care of your participants.
And again, like I want to emphasise the ethical dimension of that, not just seeing them at the end, but incorporated depends on an end in themselves.
Probably not just using that concern for our academic papers of our policy recommendations,
but we're really trying to do right by them, by giving them the interventions that are most helpful to them.
I think one thing that I would add for the people who are listening is that I think one area where I
expect this to be particularly applicable is experiments that you're learning with the larger situations,
and that, of course, in close guidance to, for example, schools or hospitals or courts or things like that,
where you basically have a fixed institution that serves a population that gets the service and the needs a lot of them,
and we do a lot of work with these kind of institutions.
And in this case you get both fast feedback and a constant stream of new people that would benefit from the other thing you do.
So I just wanted to plug that in. Many of the settings where we work in as applied.
Economist Actually, this would actually be a good match. I think that's a great point.
Stefano And I think also fascinating that you raise the point of the partner institution.
I think this dovetails tails nicely with Max's point about ethics.
I would find it really difficult to go into a crisis situation where you've got,
for example, a large NGO trying to help a target population and say, Hi,
can I do a long term fixed proportion asset and please don't do anything with the control group for two
years because I need to write my paper out and the any researcher would feel comfortable with that.
So I think this is where the ethical dimension that Max raises touches on the institutional dimension that you can say to a policymaker,
Look, we're going to experiment,
but we're going to do our very best to move respondents towards treatments that seem to be helping them and not just helping on average,
but, as Max explained, helping for the particular kind of respondent that they are.
I think it's a great point. In particular, oftentimes, I think when you run the risk of using kind of the market ethics way, right,
then you have this tension between like an implementation partner organisation and the
researchers and the implementation partners helping be believed know what works best.
They believe they have recommended to clients their goal is not to do a research right.
And then you have the researchers who run the raised academic papers like precisely estimated treatment effect.
And those are two different goals. And so I think about this book, like adaptive experiments like BE are contributing to hear about that.
But I wanted to do is like align more closely the objective of the experimenters with the objectives of implementation
partners and the theoretical that's already committed that our goal is to help refugees by job quickly.
You could say that's our goal to good designing our experiment to help refugees by topic.
There is no conflict here, and so you should be happy to let us do the experiment because we are actually just helping you to
fulfil your mission as opposed to kind of being in a tension with what you're trying to achieve.
And so that might actually also open all kinds of new sites and venues where you could run experiments very quickly.
But beyond that, to go back to politically feasible, I completely agree.
In a minute, I want to ask Stefano to actually tell us about the results. Before I do that, let me let me pose one more question to you, Max.
And and I might I might make friends and enemies in the world of Bayesian statistics here,
but I want to throw in a slightly a slightly nerdy question. Suppose you're dealing with a policymaker who says, Alright, I'll do this adaptive rc t,
but I have a very strong belief that treatment one is going to work and treatment two is not.
Some people would say, Oh, we should, you should use that prior belief and we should use it to, if you like, seed the model.
So are we going to start by putting a lot more people into the treatment arm,
reflective not of data that we've collected in the experiment, but reflecting the strong prior that the policymaker has.
Other people would say, No, no, no, it's interesting. It's important that the policymaker feels they know what works.
But we should start with a much less informative prior to use the terminology and let the experiment in the data tell us what's working.
I know there are different views out there about this in our context.
We collected information about what policymakers thought, but we didn't feed that information directly into the algorithm.
Max I'm just keen to draw you out just for a few more minutes on your thoughts about the respective pros and cons of those approaches.
So I think that that makes perfect sense for policymakers, partners and clients for that matter,
often have a lot of information about what might or might not work.
Right? So we learned that in our experiment beds or running focus groups as refugees,
and they had very strong opinions about the interventions which largely line up with what we found later in the experiment.
And so that kind of information can be very useful for deciding to experiment that they like.
If we had a strong belief beforehand that some intervention was more effective than make a lot of sense for that
intervention to look at a larger sphere for the observation or for our participants to get assigned to that intervention,
I think it's very important to think separate here.
One is how you run the experiment, right?
You might put a lot of people in the information intervention and let people in the batch intervention or something like that,
but that doesn't prevent you from that effect. Analysing the experiment in a way that your analysis doesn't draw on that prior at all.
Right? So if you just want to have the get scientific evaluation,
that only depends on the data and does not draw on the policymaker or a partner or a client believes.
That's perfectly fine. You can do that in Iran and experiment.
But use those problems to make our partner believe in order to decide which treatment to find more properly understood.
Understood. Thanks, Max. I feel like we've been teasing our listeners for a while now with the discussion of the actual results.
Let me get back to you, Stefano. Why don't you tell us what we think we learned in this context about the constraints that refugees face in the labour
markets and how we think that might be relevant for other kinds of policy interventions in similar settings in the future.
Thank you. So, so I guess maybe I'll talk about treatment effects,
but perhaps let me say a couple of words about the population that we are working with just just to to set this in context.
So our key population into Syria was a population of Syrian refugees from cities around Jordan.
These are the population that perhaps in Syria was not necessarily particularly poor, but wasn't doing very well in Jordan.
And so employment rates were extremely low and so waiting so, so so people were very close to the poverty line, many of them below the poverty line.
And there was quite a bit of interesting work and quite a bit of work experience as well, and but very little at least formal employment.
We know that some people were working poorly, although they were not necessarily very cautious about saying that they would make the law.
And so given that they are quite vulnerable as refugees, these were something that it wasn't really forthcoming in discussing with us.
Now, we then offered these three interventions.
We offered these interventions to the Syrian refugees,
that they were the kind of citizen population in our study, but also to a second population of local Jordanians.
This is actually quite important for two reasons. One,
because we want to see whether the same interventions within the constraints that prevent this
population from accessing the labour markets are similar or whether they need different policies.
And two, going back to ethics,
it's a very important commitment of IFC to always match the support that they give to the population that are going to countries hosting to an
equivalent equivalent amount of support given to other national individuals living in the same areas that are also stagnant or in this case,
have training climate. So it's it's a kind of requisite for U.S. to do this kind of equitable policy
allocation to avoid indiscriminate in the perception the refugees of payments.
So I would say we also can bring a second population of national Jordanians.
Now, what did we find? We interviewed individuals six weeks after treatment and two and four months after treatment.
So we have three courts entire which we can observe after six weeks after treatment.
It was too soon. We found that there were minimal effects on employment for all groups two and four months after treatment.
And we start with the cash intervention in particular, increase the job search and then in turn employment and earnings.
Now, the type of effects that we are talking about,
this is close to 4% despite increasing employment, which we should put in context in least two ways.
Number one, purpose at this point, it's not huge.
This is clearly not transformational in a sense is a light touch intervention.
So perhaps we shouldn't expect them to transformation face magically getting the employment rates from about 9% to about 13%.
So this is still a population that is fairly locked out of the labour market.
However, in relative terms, this is a fairly large increase in employment.
So it's a 50% increase. And also relative to the kind of treatment effects that we found in the actual labour market policy literature,
the search literature that they decide to devise policies in other contexts in the world.
It's actually in line with what people from elsewhere.
In fact, it's a towards the upper part of the distribution of of the economic mix.
Same story for the treatment effects of income. We found that income goes up by almost 60%, the absolute amount.
It's limited in a sense because we're starting from a very low base in relative terms.
This is a this is a large effect. One more thing that I want to see about the cash interventions,
that this is one of the few pattern of tool and conditional cash interventions that are designed to boost search.
There are other papers in this picture that show that if you gave people conditional cash,
for example, this can build trust that enable people to take transport and integrity.
So it does intervention usage of it. But it wasn't clear whether that was the case because you were kind of almost forcing people to
search for work in order to benefit from the intervention or whether people would have boosted that,
would have increased the job search, giving some additional resources, whether this was something that was very high priority.
They were not doing extra investment in job search because of lack of cash.
So it's quite interesting that we find that giving these are conditional cash, the experience.
Job search quite a bit. And then this leads to employment. I mean, Jordanians, on the other hand, while this is still a fairly poor population,
but they're richer than the Syrians, the French underestimate and they're generally better off.
And we actually find the caches very limited effects.
The NATO intervention, the permission, the natural component shows also increased job search, less so than the cash intervention.
And they do lead to some employment effects and to two month massacre.
But these effects are smaller and less persistent.
So by the time of the former detainee, we see positive coefficients by quality and in general statistically significant.
So these are the two interventions, although they seem to have at least activated some job search and generated some gains, they proved less.
I think Stephane, I mean, it's a really interesting summary.
This issue of how you help displaced people in urban labour markets is sadly not an issue that's going away.
So the takeaway for a policy setting for future crises is that when we think about cash support for refugees or displaced populations,
that this is not only valuable intrinsically in terms of increasing welfare,
but we should actually think of this as unlocking such capacity in potentially new labour markets.
Is that a fair takeaway? I know as researchers we always struggle to think about how to generalise our results to other contexts,
but it feels to me like that's a sort of insight that we gain from this population that may be relevant for future crises.
Absolutely. So I would say that basically one important takeaway, of course if you generalise to this crisis, other things will change.
But it's probably true overall that refugees typically lose a lot of assets during the process of displacement.
So this is a population with a little bit of cash and very high returns.
The point stands that also in our population, employment rates and incomes remain quite low.
So it's not it's not going to be by any means a sufficient intervention,
but it's probably a small but highly targeted thing that one can do help refugees operate in the in the new labour markets.
That is one more thing that I want to say, which is of course there are many things that would differ when you compare this to refugee crisis.
But I think one factor that is of essential importance is language.
And so some refugee crisis example, the word exodus by Jordan or Lebanon,
you find that refugees coming to country speaking effectively, the language that is spoken in the country,
maybe different dialect, maybe they have a bit of an accent,
but by and large they're already able to communicate with pretty much anybody in the host country.
The refugees are going to Turkey, for example, in a very different position.
In fact, in Turkey we find that they tend to mostly work in self-employment and keep training with Syrians living elsewhere in the world
rather than being getting integrated in the local economy like the Jordanian and Lebanese policymakers with their to do.
I'm not entirely sure which of the two kinds of shocks is more common.
For example, Venezuelans going to Colombia, and they spoke the same language.
A That's fine, but same language. Lots of other refugees and some of those in the in Eastern Africa, for example,
you find the populations that that are really in in those in Europe, in many cases, in fact, refugees missing Europe.
Very often language is affected by this. So I would say that in cases where language is also an issue,
I'm not entirely sure this is also generalised simply because refugees may not
be immediately employable before they pick up the essential language skills.
Do you think?
That's a really excellent point, and let me loop back to a point that Max made earlier, which is that we did a bunch of qualitative work, of course,
talking to this population and this narrative about the importance of cash for unlocking search
was certainly one that seemed to resonate with a lot of the refugee respondents themselves,
which I think is interesting, and maybe also something that would be useful in terms of learning in other contexts.
Exactly. About the practicalities that you talk about. Well, I've really enjoyed this conversation.
It's brought back a lot of fond memories of a project that I really enjoyed doing and where I certainly learned a lot.
Thanks very much, Stefano and Max, and thanks to all of our listeners for tuning in.
I'm certainly very keen to see how these these two different literatures go to see where people think about new interventions to
help these kind of vulnerable displaced populations and also to see the emerging field of adaptive and targeted field experiments.
Thanks, everyone. Very exciting. Likewise. Thank you so much. Thank you so much.
I've only stepped out.