The final talk that we have on experiments dead today is by IS by professors on Évoquant,
who is a associate professor of sustainable social development at the University of Bern.
So we're very excited to be able to bring her all the way from Switzerland to the UK to give a talk to us.
She was previously at the Centre for Experimental Social Sciences here at Nottingham College
and before that she was at the University of Zurich and the Department of Economics.
It is very interesting work using experimental methodologies, especially field experiments,
to study things that are tricky to measure and topics that are culturally sensitive, such as female genital cutting and prenatal sex selection.
And she studies and runs field experiments and in many different parts of the world,
including having done them in Sudan, in Armenia and in other places.
And today she's going to talk to us about eliciting data and challenging intercultural settings by using mobile labs.
OK, thank you. Well, thanks for having me.
It's been nice and pretty challenging to be on Saturday at four o'clock.
I try my best to be on time, so I, of course you can ask questions in between.
I might be a little more rigorous, and Ray was just maybe by trying to get the all out advice here.
So, yeah, it's easy. Already said I will talk basically about how we can use mobile apps.
So what's the advantage of using mobile apps? With respect to development research, I guess you are familiar with this idea that it's long ago.
Is it? The gold standard has been established in a way that you have to run an asset if you want to have
a reasonable or you would want to have an impact data collection and an impact measurement?
And there was a lot of focus on how we do that, right? How can we avoid attrition?
How do you really randomise into fear?
How do you negotiate with your partner in the field if it's a faith in design or to not redesign and all of this?
So there was lots of lots of research and lots of discussion about how do we really do that?
And that shifted in the last couple of years,
a bit more in the way that behavioural science got stronger and stronger with respect to development, science.
And what I mean research is that there was really this trend where people,
first of all, argues that we have to understand the behaviour and see it better.
So it's not just the standard about we have to give the right incentives, we have to give people information so that it was really moving into.
We have to understand is this behaviour biases? So we have a science was used as a diagnostic tool to inform the programmes.
That means the programmes are the change.
It's not just give you money for you sending your child to school or I explain that it's good to use fertiliser and then you do it.
But now it's all about role models. It's about how we change social norms and that and this is what I'm talking about that led to a pressure.
It's on the side of impact data collection. So if a lot of the interventions are now about telling you what to do,
I cannot just afterwards ask you what you're doing because it's very obvious what you're saying.
And so there is really a shift towards trying to improve measurements in the field as a consequence of using a lot of behavioural driven,
informative programme work by now. And this is a little bit of a problem, as you see here.
So that might be if you as a self-employed writer, if you don't use a certified.
The challenge is maybe that as soon as there's an intervention, a lot of interventions are its immediate interventions.
Now there is a big telenovela running that's really repeatedly telling you that it's not the right thing to circumcise the girls.
So you can then come to a situation where people are completely aware what's desirable as a socially desirable behaviour is,
and you need to be able as a scientist to disentangle just increased awareness from behavioural or attitudinal change.
And this is basically what I have to do in my research quite a lot, and this is what I'm talking about today.
It's probably true and all kind of topics in development.
So when you study poverty or something, that's really more incentive based,
but it's definitely a challenge when you have this kind of solid, socially culturally sensitive behaviours.
Like you said, already, female genital cutting is such a topic it's almost in any country you go.
Everybody has learnt that when we are coming, I say, like the aid agencies are coming,
that we do not want to hear that you support human agenda cutting.
So the first ladies of countries often, you know, talk about AIDS, but it's never really.
I mean, it's not really you're not punished if you if you keep on doing it, even though it's illegal in most countries,
it's similar with prenatal sex selection in Romania and in the southern caucuses, as everybody knows by now.
UNFPA made one kind of big campaign and then tells everybody that this is the wrong thing to do.
And as a topic that studies corruption in higher education or exclusion of people with disabilities.
So these are all topics where people are, where it's really challenging to disentangle awareness after complaint from the right attitude and change.
We also had this is, I think, with regard to research to measuring the tricky things we had to workshop it all together was a behavioural team of the
world and where we basically asked this question How do we actually know if we measure what we think we are measuring?
How do we figure that out? And I want to talk about the first three points.
So there are different points how you can avoid the social desirability bias that people tell
you basically what you want to hear that people try to anticipate what's good for them.
Maybe they think if I say I stop circumcising and have some advantages in another domain because the same programme is also providing other services.
And I would talk about mainly how privacy can help, how indirect measurement methods can help and how giving incentives can help.
And all of this is very easily done very easily, but it's done when you use mobile apps because they give you all three of them.
I'm not talking a lot about this as a tool. I have just one slides.
That's some statistical measure of measurement methods.
But since I'm coming from the lab, I only talk about the first three. But this is an example about the list experiment.
This is what was often done before you basically went this richer data collection,
and I talk about today where you would basically ask a direct question.
This is how basically all the time the DHS had asked the question about circumcising the guy very direct and often in face-to-face interviews,
it's always the same argument people who can't really read and write, What shall we do?
We can't just send surveys out like we do in Europe and hope many people respond, send it back.
So you basically sit in front of a person,
interview a person and probably two thirds of the extended family standing around and listening to the answers.
And then they develop methods legs in this experiment where you ask, let's say, five questions that are not related to the sensitive topic.
And then this is the control group. You only ask people to report how many answers are yes.
And then you compare it to a treatment group where you're at the sensitive question and then you can kind of measure some biases.
This is not very useful if you're really interested in randomised controlled trials because it's on an aggregate level.
And that's also why I'm not talking about these methods. And then there was another study that's pretty interesting by society that ran an experiment
where he basically was an experiment wanted to figure out what privacy actually means.
My privacy really had the social desirability biases. So he had to study in India, where he asked some question about the caste system,
where he knew how many questions would be desirable and how many people would answer truly.
And he did face to face interviews. Well, now I'm touching it.
So you did face to face interviews. And then he did the same interviews with an empty suite player where people were just listening to the
questions and then the head to sheet very simple where they would give an answer if they agree or disagree.
And what was nice is then he buried the empty three match sites,
so you would use a situation where you really don't give the results back to the interviewer to test an interview effect.
You change the formulation, whether it was more in the first order, one more direct or indirect question.
And then he also used earphones more as a speaker. So people around you would all listen to the question or you would only listen to it on your own.
So this is testing the bystander effect. And what was interesting is that was the most important effect.
So when, when, when, when everybody could hear the questions, a number of socially desirable answers would really go up.
And that I found interesting because this is very often true in most as a data collection, we look at the people not really on their own.
It's not just interview, it's really you ask women about domestic violence.
Maybe the husband of your wife is just next door and he can. I can I just hear what you say?
And we know this has also led to a serious backlash in some studies.
So I'm talking about basically how privacy can help when you use computerised learning to field experiments.
I don't know if I need to tell a lot about life in the field experiments. I guess you all have heard about it today already a little bit.
It's basically the same like lab experiment, but you take it into the field.
That means you probably have an informative, theoretically informed subject pool.
So maybe you're interested in. There's a famous study that showed that bankers actually not self-selecting into risky behaviour,
but they have a culture that makes them taking risky choices.
You wouldn't get to bankers in Zurich in our lab, so we had to take the lab to the bank as to not see so much of their time.
So this is definitely an advantage, but it's still very highly experimental controlled.
You know who was doing what? And this is what I talk about.
It's nice for interdependent research,
and it's nice to provide this critical privacy that what I personally think we very often just take away from certain people because we have this.
I mean, I've heard many stories of good papers, but often people say, well, they they couldn't read and write, What should I do?
So it's just a few studies I did. We wanted to see how social preferences develop in children.
There was a study done by and stare many years ago, but they basically had an experimental who would trade with the children,
gummy bears and other sweets, and we tried to replicate it.
But really, in a way, word was anonymous and people and kids were really behind the computer.
They were four to eight years old, but we programmed in a really nice way and try to see how that works.
And this is the study I'm talking to say most of the time about.
There was a study done in Sweden with people who have not used computers before most of them
and was kind of small-scale farmers and with people who also really needed audio recording.
And so, yeah, you see,
we built up a mobile app and keep it were at the end being able to do pretty complicated tasks on their own behind the computer.
And it's probably already my main take home message that this is actually possible.
It's a bit more work on our side, but I would encourage everybody to do that.
So the first study is about imagine putting this very little background here.
You see where most of the cutting takes place in the world. We work in Sudan.
The reason is that we did it in Sudan. There are different types of cutting. It's always a removal of female genital organs for non-medical reasons.
But in Sudan that use Typekit,
we that means they remove all the organs and then they they kind of stitched together and it's rolling together in that way.
So this is the one where health issues have been documented and for many other reasons, we chose to work in Sudan.
Just a little background to the study. So it's a policy basically assumes that due diligence cutting is a coordination game
in the way that people are forced to coordinate and want to leave their minds.
So everybody cuts on, nobody cuts.
I'll spare you all the details, but it's a pretty well established theoretical discussion about it is this really is a case or not.
And we wanted to test it, and it means we wanted to test it in a way that decide, OK,
if this is true and people coordinate on one or the other equilibrium you should have,
the aggregate level has this specific positive dependent dynamics is everybody cuts or nobody cuts.
So our first measurement girlfriend, we could not solve this computerised mess.
Let's just basically we wanted to estimates of cutting rates in villages in Sudan and see if we can have
this distribution that we have this discontinuity in communities it cut in communities that did not touch.
And again, I don't go into the details, but we figured out that there is actually a visible signal because in Sudan and in many other countries too,
even with medical exams, you wouldn't get that through the ethics committee or whatever you want to call it.
So as a government, I would rather call it the ethics committee. But we found out that all the and circumcised in the same week,
nearly two weeks before they go to school for the first time and that they get henna painted on their feet on the day of the circumcision.
And the henna does not grow out of it, does not wash out of the toenails, so it has to grow out and we had the window.
It was far more complicated than this, but in a nutshell,
we just counted how many guides first had had them on their feet and estimated the cutting rates.
And then what you see here is the that cutting rates of all the communities. And you see there is no discontinuity.
We have cutting rates of all time between 20 and 80 percent.
Basically, that does not surprise you.
Not so much because I didn't give you a whole lot of background, but this was actually this led to a lot of arguing with UNICEF Finance.
All of this too,
because this really goes again against all the programme works that they have built on this idea that it's a coordination game and it's just people,
someone equilibrium to the other. So we didn't find it, but now I'm coming back to the topic of the presentation.
So we had this very unusual measurement. Now is a hennesy, which only works one time a year and just two weeks.
And it's very hard to take that as an impact data collection. I mean, you have to really time very well and then you don't know how long before.
That period should have an intervention or not, so what we were thinking, how we can have a more.
Measurement that you can scale up better, that correlates hopefully with the data, and then we thought, OK,
we try to do an implicit association test against the people who are psychologists know
it's it's a pretty established test in psychology also critically discussed there,
but lately used a lot by development economists. And the idea is that you basically work with reaction time instead of asking people directly,
you present and you try stimuli and various stimuli to people, and then you try to let them categorise this and measures reaction time.
I give you an soon afterwards. So it's in a way an indirect measure.
It's rather complicated to do. And the challenge was how can we do it with a subject pool as we had where people are
really not exposed to work with the computer ever before and people can read and write.
So we decide we nevertheless try it and this is a goal.
So it was you. This is basically the idea in an it, you get first introduced to neutral stimuli and this should be neutral.
This is a circumcised girl and the girl that's not circumcised. It's very clear for Sudanese people because a piece of cottons is circumcised.
The class that circumcised daughters has accepted this pattern.
So there is a lot of qualitative, informative work that goes into this project that I'm not talking about today.
But anyway, it introduces two girls. And then you see, here's a good should be neutral.
And then we added a valid stimuli, namely a negative and the positive stimuli.
And then you categorise them here. The cut girl is combined with the negative stimuli.
Unpacked, though, is a positive one. And then these images of the girls popping up or people hear a bad or positive word.
And then you had only two buttons to press a pink one and blue one, and you had to do it as fast as possible.
And we measure reaction time. And we also measure the error rate and then you reverse the pairing and you do the same again.
So I guess the intention is clear. You're really you're really slow down if this is not how you feel about it and if this is how you feel about it,
it's bad to be circumcised and this task is much easier for you.
You can test many of those. It's a Harvard web page and about different religion, women and signs and all kinds of stuff.
And I guess with some of them, you really feel fear.
Maybe what I mean that that it's just slowing it down a millisecond and that stimulates over time.
So this is this is basically get the story that was it just just just the beginning?
Shall be these kids? Yeah. So the question is, can you?
Yeah. Would you like to? Could you again repeat the intuition be the end is slowing down?
Yeah.
So if you so the idea is basically let's assume you think it's better to be circumcised and it's good not to be circumcised and it's good to be cut.
And then those images are what's coming up. Then this is how you're aligned with it.
It's easier for you to press the same button for a positive word and for a cut.
It's harder for you to do this task because it's just again,
you're it goes against the implicit association, and this is why it's just a little harder.
There has been lots of meta analysis, and this works best with an uninformed subject,
which would be probably harder if I do something about racism with you guys now,
because somehow you might figure it out and you try and forces me to go against it because you want to even hide it in this task.
There's lots of discussion in this task really working a lot.
It seems to really works best with people that haven't done that before, that have no insight about this a whole lot.
And then you really, just with the reaction time picked up this difference in the implicit attitude.
Let's basically the idea, and it's a pretty complex task.
You know, you do it first for one pairing, then you have to do some intermediate stuff to get this out of their heads and the other one.
So, well, I wanted to present it to you because this is something everybody told us.
You can't do that. It's just doesn't work. It's it's crazy to do that in Sudan.
And at the end we went one hundred forty four communities. We had a lab with 50 computers.
We had 24 hour, seven days a week security with us. But this is really the message I want to bring here.
You can do it. It's just a bit more work on your side, but it's worth it in a way because it's first of all, it's valuing people that you're not.
Just seeing what I can ask you about your sexual behaviour and your mama sitting next to you, and I just don't care.
We wouldn't do that and we wouldn't go away. Was it in any country like Switzerland?
If you tried that, people would never let you in their house? So this is a nice way of trying to also be a bit more.
Taking privacy serious and topics that matter for people and other places.
And as a result, so you see, it's also very similar to the Hennessy's.
We don't have a discontinuity, we don't have a group that's clearly pro cutting and clearly against cutting societies.
Poor is basically going from two to minus two, and that means you're in favour of cutting and this means you're in favour of money cutting.
And what's important is that we could show that this correlates with the hidden.
So it was not the same people. It was randomly selected people from the communities.
But this did allow us to use this method basically to run it behind a randomised controlled trial because we have
developed two new methods and we were kind of confident that we can rely on it because it's picking up the same stuff.
This was important because we would, of course, criticise, not criticise, but the Hanafi, there's a nice idea.
It worked very well, but it's nothing that we can just use a whole lot.
And if you're really interested in attitudes are changing any time.
So we did this and then I make just for a bit more fun and make a little thing where I don't talk about medicines,
but just showing you why this was important that we have developed this as a method.
So, yeah. Sorry.
Just just as a question, so the implicit the scale, the I-80,
what what so what does it mean if it's zero point three means where do you where do you see to this is this is just mean.
This is basically it goes from two to minus two plus two is you're really in
favour of non putting and minus two means you're OK putting in zero second.
OK, and and you know that this is not just random because it correlates with the actual measure of that was important for us.
I mean, that's the message. You're right, if you develop new methods, you hopefully can validate and somehow.
And we also did that in this is over all 144 communities, but we did it for each community individually.
OK. And the choice in the implicit association test was either positive or negative.
There wasn't a middle. No, you can't have that there. You have basically two valid stimuli and two neutral ones.
And if you do that, you should probably where maybe you are even in favour of not cutting.
But the goods are neutral, a cut down on fat girls and it should be an interesting one.
Yeah, that's the idea. OK, thanks. All right. Thank you.
And just later explores what you can do with it, and so we had the fun of producing four different movies in Sudan,
and the idea was that we wanted to have again a very mild intervention where we basically use the results from the first paper and showed that people
who plot and don't cut next door to door so we can produce movie now where we just use their own arguments discussing it out without us coming.
Talking about something like this is a violation of human rights or something like this.
So we just produce four movies and we run two experiments.
We run one where we were very risk averse, where we just let people watch the movie on their own in the lab and then directly
afterwards runs implicit association test to see if there is an immediate effect,
depending if you see a movie where we discuss, we mentioned cutting or you see a movie where we don't talk about it at all is a controlled movie.
And they were so 75 percent identical. And so it was a really, really, really mild intervention.
And then we got these results and show them to you a minute and then we got more courageous and we said,
OK, now we can also randomly assigned the movies to communities, which is, of course, a much bigger study.
You have hundred forty four communities to make public viewing in every community.
You randomly select 60 people per community.
You measure the attitudes before you show the movie and you measure the attitudes afterwards, and he sees the results again.
So this is the i t score again, and here you have the control movie and three experimental movies.
And this is the big study where we randomised at the community level. And you see directly after the movie,
the attitudes are more in favour of non cutting in comparison to the control and movie where nothing seemed to have changed.
And then interesting though, in the field study,
we could wait one week because we didn't had spill-over effects because it was randomised at the community level.
The other one, of course, if you just wait an hour, everybody talks to each other.
And I do think spill-over effects of serious effects, and you should try to avoid them whenever you can.
That's why we randomised at community level.
And then you actually see that only one of the three experimental movies still has an effect a week afterwards,
which is also nice in the way that it's it's also always implicit association test.
So we did it maybe a week after they watched the movie, and then we see that actually only one of the movie has a lasting effect.
Yeah. Excuse me, miss you if I missed it. But in the movies, are they like the same?
And then the actors did another scene? Exactly.
Yeah, yeah. You directed something like that? Yeah, I was. I was very short on this.
This is really because it's not really a lot about the talk I'm trying to give you.
But the idea is that basically we had a Sudanese writer who would write the movie and we would experimentally constrain him a little bit.
So in the controlled movie, it was just like any good movie about love and drama and rejection and anything.
And then in the other story, we had 30 percent of the movie talking about in a non-judgemental way with the pro and cons about cutting.
So it's like you would listen to your neighbour secretly who talks about it in a very non-scientific way,
but it wasn't scientific in the way how we manipulate it to argument. So in one movie, you would talk about your private values.
In another movie, it was just about coordination, incentives. So it was basically the elements of a coordination game.
Preferences versus coordination.
Incentives were tested against each other, and the winning movie is the one where both were in so coordination incentive seems to matter.
But private values are. It's a matter that was an idea, and I would say so.
The these people told me if you would watch the movies without the sound,
you would probably not even be able to say the difference because we tried to keep it as soon as possible.
And as an example, I want to give you to show you again how you how if you really want to,
how you can actually collect very rich behavioural data under very challenging circumstances, and this is about prenatal sex selection and mania.
That's actually how you and I met because we have a shared interest in this topic.
You just see an overview, it's already from 2010, it's from the paper 2015,
where you see how postnatal and prenatal selection influence the story about two missing females,
and you see that prenatal sex infections actually going up.
And the idea is at least in the southern Caucasus ideas. So several things come together.
It's a decline in fertility. It's an improvement of methods. You can detect the gender of the child early on, earlier.
And in that way, there is this idea this UNICEF has this very is it.
It's a mess. So it's it's getting more and more in this direction and fertility goes down
and people have a sudden preference that you find it in many other places,
too, where people just try to basically affect the situation and ensure they have a son by aborting the females.
And Amaya is a very nice example of how you can have this problem of desirability.
So they basically figured out that they have a problem with it in somewhere.
In 2010, there were a lot of data published by demographer Jim Mato,
who could really show that the sex ratio of births but in some regions at more than one hundred fifty boys are born one hundred girls.
And so they could somehow they had to do something.
And then they did a lot of media campaign, not with scientists just telling everybody it's better to preference friends,
that you should not have a preference and that you should be favouring girls and boys in the same way.
And this is not really exclusive. It's not showing clear resides.
It's just out of their report. It's not a scientific report, but they basically asked families what kind of gender it's their child should have has.
If they had one child, what then should have?
And then they did the same 2011 and 2016, which means I don't have a preference and see that there was a big change.
This is what I meant before. That does mean, of course, that there is a change, there was not the change in the sex ratio adverse,
but this is how you see how I especially what we do now as a time,
how media campaigns and all of this not set up in the nation was a randomised controlled trial.
Can use can lead to situations like this where it's just a scientist extremely hard to work with it.
And we kind of started this study somewhere in the middle.
So this was very unfortunate for us to kind of stopped it then says you can run an asset in a situation
like this in a small country where everybody knows within a week what you're supposed to say.
So we said, OK, what we do now is we don't do an asset. But we really again trying to develop measurement.
So it said in this situation works, and I do think this is a really nice example for a really challenging situation.
Small country, big media continue and FDA was behind it.
European Union was behind it. How do you now really figure out what's going on?
And it's an interesting case because women are actually pretty well educated enough to participate in the labour market.
So we really wanted to try to understand why do they have this outrageous imbalance in the sex ratio for us?
So again, we decided that we would work with mobile apps here in India.
It wasn't possible to get people in a lab for several reasons, so we basically build mobile labs and people people's houses.
And that was important because we wanted to interview the wife, the husband and the husband's mother.
All the programmes focussing on the husband's mother. This is a common wisdom.
Somehow, UNICEF focuses on it that this is a driving force behind prenatal selection.
So we said we want to run a little study with all three of them in parallel to make sure they again,
not one stands behind the other when you give an answer.
That was the reason we couldn't get people in the lab and said never leaves the house or three.
At the same time, somebody has to be at home. What do we do with the kids?
So we had really many laps of three computers that we built up in nine hundred
different households in Romania and collected data where on the one hand,
we wanted to have explicit measures and detailed preferences on the sex distribution of the children and as a staff,
then we wanted to have an implicit measure again. So implicit association test.
And we also worked with incentivised misalucha, as I would tell you in a minute,
because it can also, of course, help to really go to the true attitudes or the child behaviour.
And I wanted to just talk very shortly about it again because you cannot just
test visit whether there is a positive or negative attitude toward something.
You can also test stereotypes. And that's a nice way to do development research because in the way they are coming from somewhere,
and if you do a campaign, it's maybe it's a more subtle way to see if you actually did do some changes here.
The stereotypical it we run in Romania is based on this kind of saying that they say that the boys are ours and they don't belong to the others,
and this is the idea that the boys stay with you. But if you have a girl, you marry her off.
And that's it, basically. So she's not really very useful at that point anymore.
So we use and we probably see now that for you, if you don't think that way, this should be a really weird IoT for you.
So instead of positive negative terms, we have terms that, on the one hand, show flourishing.
That means you extend your family, your still somehow there. And then we have lots of words that I found it a weird idea, to be honest.
I don't even know how to do that one where it's about abruptly ending the family line.
Find us. And then we did the same stuff again where we had the family was only boys and the family was only girls.
And then you have the words down, whether you continue the family or you don't.
And again, you have images or you hear different of those words.
Then we reverse encoding again. It's same story as before, and we measure again the score and just talk about the results.
It's again from two to two. It's the same. It's basically a programme you buy, and it's it's always working in the same way,
which is nice because it's nice to compare when other people use it tool.
And we did it for a balance. One was just positive and negative and for stereotypes, it just explained to you and you just again,
don't read this data or it's just came in to weeks ago. And I don't want to go into the details.
I really want to talk about the methods.
But what's interesting is in our data very systematically across all the different regions and also in society for violence and for stereotypes.
It looks like there has been some as a problem. So to say the husbands have a strong preference for a boy and the husband seemed
to be very much leaning into the stereotype and not the mother in law so much.
And now we feel like again, we're kind of. Ready after we understood all of this data may be doing in our city now, because now we maybe have a.
We are convinced that we can disentangle this awareness from the asbestos.
But because it's more complicated than mania, because we know there was this big campaign that reached everybody,
we also did a lot of other measurements with some preferences at present and then again trying to validate one for the other and not just,
you know, developing an it and run it afterwards. And also,
he was very useful that we worked with computerised methods because typically the DHS is measuring sound preferences in a way where it first
asked you if you could go back to the time you did not have children and which was exactly the number of children to have in your whole life.
How many would that be? And then they ask you based on this answer, how should be the sex composition?
And of course, it's a very problematic question. It's conditioning. First of all, under the family size, you say, if you say sorry,
you can't really reveal an unbalanced preference and wants you to say to and you have
this nightand preference is very extreme to say sunshine and you might say son,
daughter. And of course, it's the family size varies across different region.
The preference was a family size. You also have lots of noise there.
And again, all of this was done in privacy. We managed to do all of this in the way that people who do it on their own.
Of course, this was not as hard as in Sudan. I mean, they could not write the songs, although we audio recorded everything to at the same time.
And then we did basically hear what two shots would be used an indirect measurement.
So first of all, we asked them about the family composition of the youngest child.
So you don't have this problem that you basically have to rationalise in a way that you already have three children,
but you really think two would be best,
but you don't really want to say that because then you see that kind of, you know, unfair will mean or you say something really bad towards the child.
So we work with a hypothetical case where we ask them about basically what would be the right size and the composition for the youngest child.
And we could also, because it was computerised on the paper, you could never move all of this in the way that a person would handle it on its own.
So we could programme it in a way that we ask what's a perfect family size?
And we had lots of questions where people would forget about this question.
And then in a random order, we would ask different people about If you have one child,
what's your what's the perfect sex position, if you have to, if you have three, if you have four.
So again, this was extremely easy in a way. If you think about it hard enough, programme it in a way that these people could really do it.
Without that, they needed help for it because the programme would do everything for you.
And in that way, it was also nice again that there was no bystander effect, that no interviewer effect to go just to give you a short anecdote.
We work with the National Statistical Service of Armenia, and they do either state in all of this,
and they were telling us that we get the best interviewer.
Maybe I should visit when I'm actually I got anyway that we get the best interview as a half.
And then, yeah, when I train them, there were a couple of men that came to me and said, Hey, you know, you just tell me the answer.
You won't get any answer. Said, you tell me you want in your questionnaire.
So we saw yet another reason to have computerised methods and leave the interview out as much as possible.
And I mean, he really had had the best intention, right? I didn't even mean that as a criticism of my life,
people have other ideas about how they wanted to have of data together, I guess, just to answer very short.
And so you see, of course, this is all done on their own and you get a lot more insights because one of the key question also,
is it the preference for a son or is it a preference for the more sons, the better.
And so, for instance, we'll see if it's just one child. Everybody obviously wants a son.
This is a different regions and there's a three character, husband, wife, mother in law.
But then if you ask them two children, they all want a son and a daughter.
So it's not just the more sons better. And then I personally think getting really interesting here, when you have three children,
or even if you ask for for their interest, many combinations and people seem to be relatively clear.
This area is a little different. The sex ratio is much lower, a much more even than these two.
But what's interesting is everybody seems to be sure that the first one is to be a son,
and nobody wants something like even daughter, son, daughter, daughter or something like this.
So I think if you again, if you programme all of this isn't saying it's it's it's a much richer data and said it's
OK treating the people with more respect instead of just asking them all of this directly.
And that's the message I want to talk about. Your remaining two minutes is vignette studies where you use incentives, because if you use incentives,
of course, you can also really go for the true answer in the way that you can't if you don't give incentives.
And I guess most of you know how vignettes work, so you have a little background story.
In our case, people would listen to a story about the couple that has two daughters planning to have their say at the last child,
put the edge in a way where it was clear to people that chances are not increasing anymore.
And then they find out that they are pregnant and she has a girl.
And then in a vignette,
you very different experimental treatments are to say so you can vary the financial security that they think the daughter would be useful in the past.
In the future, are they the marries and then or are different things?
And then also various people do abort or they don't abort?
And then you ask people basically the private opinion, and you can also ask them beliefs about what other people would answer.
And if you don't do this in privacy, so we have everything from buying to it's an indirect measure from the wording.
We provide an incentive as you see now and again,
they could do it completely on their own without you recordings, without anybody listening or watching them.
So to say then what's nice about this is so you ask them, what do you think about the choice of the couple?
Let's assume you said they believe their daughter is financially secure in them and they decided to abort.
Then you ask them, Are you happy, sad or very happy? Very sad for the family?
And this is just a private opinion. And what you do is incentivise vineyards to basically nets and play a coordination game in the way that you say,
OK, now we interested in what you think as specific as a group did.
So it's like you're trying to measure perceived social norms. People say these are a focal point.
And in that way, we basically tell them we did the same study, the same question as a women.
If you correct, guess if you correctly guess what? Another woman answer to that?
I don't know. 10 U.S. dollar and like this, you disentangle this kind of stuff that you're not trying to be consistent.
You don't try to rationalise that you say what everybody thinks I do.
The incentives are pretty high. They know that to immediately get it by, I mean, it's on the phone.
We made it clear we showed that before the beginning of the study, they trusted us at this point.
And again, this is. So a development context, a pretty elaborate way of collecting data off of doing a lot of effort,
of trying to disentangle this stuff of awareness, whereas it's media behaviour and attitudinal change.
And this is, I think, what I'm really just going we don't have results and ones where I don't show results.
It's really just to summarise it. I think.
I think there's really this big trend that we really convinced now that it's just not enough to tell people what's right to do.
Information alone clearly does need to change, at least not with those topics.
We also know that just giving incentives is not always enough, it's just not enough to to to change.
The incentive structure is not enough to do conditional cash transfer and conditional cash transfers.
The idea is maybe if if the programmes are so heavily relying on a normative language, somehow telling you what's best to do.
Establishing role models and all of these things working with mental model shifts, then you do need to get a better measurement.
And I think this is a trend that you also use mobile apps and then you work as intended.
You have this privacy, you have kind of interdependent situations and not being and not saying,
we can't do that because people have never seen a computer. People can read and write, this is a really hard situation.
So I would encourage everybody to always think that this is your job to sort that out.
And that's the job of the people. Have to say thank you.
If you have any questions, I can make the last.
Thank you ever so much, I really love the roads that are great, but I wanted to ask, how did you like run pilot studies on these things?
Like how long has it taken you to set up this, this final body of work, for example?
Because I know I do field experiments and they take forever. And do you do you also run like pilots?
And I guess link to some of this is how much confusion did you generate amongst the participants in that?
Did you? Yeah, it's linked to pilots in the pilot to find out what they do and do not understand.
You then went back. Yeah, etcetera. Yeah. So yeah, they they they take long.
I wouldn't. I wouldn't do the Sudan study during my PhD.
Probably not. You also depend on other people. So you know it.
It's not just on me, but of course, there is a lot of. Piloting, but also what I would call material production.
That's really it's really important in Sudan. The movie production was was took forever,
writing the script and then arguing with a very talented writer because I wanted to become to Fisher to get my different treatments.
And all of this society looks, Oh, it's so easy once you present presented,
but it's incredible in some countries how long it takes to get an idea to a sketch that you can use in mania.
They always look like the Kardashians, and we kept telling the has to look like a family with no social, economic background and all of this.
I mean, that matters, of course.
But working with a was activist with the people, I mean, being an artist in Sudan means an activist, so you have a lot of free testing in that way.
Of course, as a girl is equally likely it starts there. Then what we did is when we had our message together,
we also really protested the procedure in a way because exactly we didn't want
what you said when we went in the final one hundred forty four communities,
I was never there. I had at that point sixty data collectors trained and they were a bit hierarchical.
There was a study of a C person and it went down and then four people were supervised by one.
And all of this because I think I can I can really say what people reported at the end of the song.
That fun. You know, we came in there, we had this big computers and data up in school.
They also got rewarded for a four day because I couldn't do what they normally would do.
So I think actually, I saw some video footage the government took and it didn't ever really look like it was really like people understood.
And this is a difference why I don't use this experiment. So I think the message was clear this might be a topic of sensitivity.
I'm interested in your opinion, but I don't really care. I don't record what you really do.
I will never ask you directly, do you continue to circumcise your daughters?
I think people understood in a way that we are interested in something and that we put a lot of effort of making them feeling OK with it.
And I think we managed to do that actually in a good way. But yeah, it's a lot of before the stunning study runs.
I think it was over one and a half year we got better. And I mean, once you do it, then you really get much faster and.
I mean, because that the so we do it next week in Albania.
It's a long procedure, first as a government. So we first presented to the government and then we present it later on, such as a community,
as a community leader and whoever they choose to join him or her family to him.
Yeah. I'm over,
I assume that you have read the public letter in the Guardian a year ago on the whole
critique on asked that their names and they got a lot of money in the last ten years.
But have they actually done anything besides being the thing to do?
So I was wondering, how do you in the time of increasing critique, how would you still validate that this is the way to go in development?
I think this is exactly already a reaction to this, that we are trying things in Maine, which is I mean,
first of all, I agree if you see how many of our cities are produced just by Jay Paul and they all piled up.
I once had an amazing research assistant who did a review for me about I had an offer to do a project on my nutrition,
and then he got everything together that was ever done.
We published all the studies that seemed to look really reliable, and it was a mix of anything.
Right. Everybody change something. He, as a package was changed. It was a different country.
And so I mean, this is, of course, it's a serious criticism.
I do think this behavioural insights, instead of using behavioural science,
is a step in the direction because what I didn't say before, but that basically means you focus on mechanism much more.
So that's why we had four movies in Sudan. I didn't just want to show a best of movie versus the control.
We really could learn if the argument is, is it a coordination issue or is it a private value issue?
And it requires a different programme because in one, I'm just trying to shift a critical mass over the threshold.
So to say and the other one, I can work on an individual basis. And I think that's my reply.
First of all, that it's probably the best if you're really interested in a blood test, but it should probably come.
And of course, that's expensive. I mean, if I want a test mechanism, I have four treatments and not just to control it end to end in a programme,
but I think this is the way where field experiments. I know what's ahead.
I would argue if you go back to the study in Sudan, the first one we did had just two hundred people.
And this is something I also figured out of. People think immediately about big stuff when they do acid cities.
But of course, you can also do it in a lab, in the field setting to get started at least and do it in a much smaller way.
But again, with a focus on AIDS and really trying to understand the mechanism behind it and not just why that works or doesn't work.
But then it's still the time dimension that's rather short. I mean, values develop over centuries if person just runs a couple of years.
Well, that's that's another question to discuss, right?
I mean, this depends very much on the topics and there are examples where maybe an incentive works.
I mean, there is no genuine answer I can give, does it?
There are cities that did one huge debt funded from cases either did one where they showed a 20 minutes movie and then six months later,
they could still measure their effect. We really were lucky we could come back one week and then Sudanese government said, Look, guys, this is it.
I mean, was really harsh conditions in Sudan.
But of course, it would be wonderful if we could now go back and measure the heads of penalty again and be working on all of this.
But of course, this is always the same challenge. If you're going to feel this is just kind of stuff.
I mean, that's not surprising to you. Some programmes take longer.
I mean, even now in Romania, it takes nine months to give birth to a child or whatever.
I mean, you shouldn't just have it two weeks. In fact, imagine afterwards, but.
I mean, that's just how life is in development research, I would say.
Thank you. I was wondering, could you go back to the distribution? Is it look like this?
I mean, there's a lot of within variation this one? Yeah.
Mm hmm. Is that right? Like, I don't I mean, it's kind of surprising.
I was wondering, are they aware that they are not like that as a lot of variation between them?
That was interesting seeing I mean, I think this is really only interesting for for you,
if I would have even provided more background when we started this project,
there was only one idea that everybody cuts and nobody knows how to do better.
This was basically the approach that goes back to this paper from Jeremy McKee,
where he really said This is a party nation issue is good for fundraising because everybody is somehow locked into this equilibrium.
And when we figured out, I mean, what you see here, where we did this now is many, more, many more communities, it's actually only four to four.
We did it with one hundred forty four now and we figured out that people basically left the door who cut and who don't thought.
And in the same months when they published is an economist published a paper using DHS data, also showing interest in this.
Its variation is at the household level, not at the community level in certain different African countries.
So this is when we started only asking this question.
We then later collected data where we ask people, Actually, do you have a problem and say, you're cutting family, do you?
Are you willing to marry your daughter to a family that doesn't touch all your son or whatever?
And about 40 percent in every community said, Yeah, we don't care.
So we were even more pleasant.
And then I just have an anecdote where we asked to six to six data collectors in the US and you point out in your community who plots and who doesn't.
And he said, Yeah, of course, I know exactly who is doing what.
So it's a lot of new information, and it would now need another study to probably understand.
Yeah, thank you. All of these seem you mentioned in passing and quite reasonably with the time constraint that you weren't going to mention
all of the qualitative work that went into being able to think about things like the patterns on the children's clothes.
But these are obviously very large projects that probably are coordinated between.
I would be interested in hearing are they coordinated between multiple groups or
scholars in the same working group or how do you kind of bring together this team?
That's obviously. Different expertise, both in and across nation.
Or what is the team? Yeah, yeah. Yeah, I should.
Anyway, I've mentioned that all of this work is done within Sanchez ever since I was a biologist and a behavioural economist for me.
So, yeah, we were not a development department and was just as real as was, of course.
I mean, it was really UNICEF who who was a few partners implementing partner and UNICEF.
Of course, they have a communication for development office. We had an amazing office there and she had lots of insights and all of these.
We did trust the Sudanese talent to write the script and we we really first let him go free
because we sometimes we need the perfect story and then we need to make it scientific.
And so in that way, she made and funding is a topic where there is a lot of great work done before, so we didn't need to do a lot.
But I don't think she's an anthropologist. She did a lot of work on this.
Trying to find out what campaign is medicalisation is about seeing if people even care about if you say this is human rights,
she studied backlash and all of this. So this is actually a topic where this was the first answer and the first serious quantitative approach.
But it's a topic where there is a lot of really great qualitative research and there was even a woman.
She knows exactly where we were in Sudan.
She published a book in early eighties about instructing an anthropologist in Sudan in exactly that location.
So this was going. So good qualitative work, this is the foundation that you already had, it wasn't like members of you.
Which might be the case in a situation where there wasn't a lot of.
Yeah, I think so, I'm ready now, rent proposals and similar topics, so.
So, yeah, qualitative research is important. That's right.
And I would also now, to be honest, I would say if there was, I would say there was.
Research done by UNICEF. A bit more I took that went into it,
I did now read several when proposals were really clearly relative qualitative person that we really have an anthropologist,
somebody on board from the beginning. I don't know.
Maybe here we just I don't know.
We were maybe have a media head from UNICEF at that time, but of course, it would be nice to have somebody on board and do that before.
Any loss of. We don't have any more questions, and I want to thank Sonya against a really insightful, very interesting talk.
To close experiments there and also for coming all the way from Switzerland for it.
Yeah, so let's give her a round of applause.