All right, we're going to start with the second half of all the seconds in morning session of our experiments there,
and I'm very pleased to introduce now Professor Ray Dutch,
who is the director of the Centre for Experimental Social Sciences at Nuffield College, as Zanka was mentioning in his morning,
talk says, is quite an extraordinary achievement and that it has its own subject pool of fifty thousand fifty thousand subjects.
It's spread across four locations in one Nuffield in the UK, but then also in Chile, in India and in China.
So it's quite it's quite an undertaking and as it really allowed experimentation or experimental social sciences to be advanced,
and Ray is going to talk a bit about some of his work, but also,
I imagine work that's being done and says Ray himself is a political scientist who's done excellent work using experiments,
digital trace data and public opinion analysis, and he's written a lot of very impressive papers.
So I think he's going to talk a lot about them, so I'm going to let him do the talking rather than myself.
Thank you. Is it on?
Now? Cool. Great. Well, thank you very much for the invitation. It's a real pleasure to be here.
From what I've heard, this has been a really interesting and rewarding week and next week should be,
I suspect, also really a cool educational experience.
I am going to talk a little bit about my work and says most of the stuff I'll
talk about is work that is being conducted and says I'm going to introduce,
Oh, now I've. Yeah, you just so I can say that.
Well. I can do it, screw it up.
OK, but we can use this stuff. You want to just use cool.
So there are three parts to the talk.
One of them will be a brief introduction to assess which I think Cenk is already really filled you in on, so I'll be very brief.
I'll introduce the talk. The talk has two modules.
One will look at micro replications, a work that I've been doing exploring experimental measurement error.
And the second module we'll talk about designing virtual experiments with stratified average treatment effects.
So without any ado, the introduction. So I think SAS went over basically who we are.
We have four centres here in Oxford, Santiago, Chile, India and China.
We have a pretty extensive online facility, which Cenk manages and we have.
As he pointed out, the subject will about 50000 people. It's actually spread over the UK, Ireland, India, China, the US and Chile.
And we're also, as you'll see in the second module, we're also involved in field experiments.
In addition to all of this, we do workshops, we have summer schools and we do visiting post and pre doc arrangements.
The talk, so the talk will be, I think, will be experimental perspectives on themes you covered this week, and I sort of guessed here.
So but I for my casual conversations, I think it's true.
Basically, it'll sort of be a mix of computational methods, large data,
social media and what I consider really important this notion of robust replications.
The second module of the first module that I'll talk about will be about how to detect experimental measurement error.
I'll talk about how experimental contexts or mode is really important for detecting experimental measurement error.
And I'll explain how you how we use machine learning as a as a key element of that exercise.
And then the last module, we'll talk about large scale experimental interventions.
This is a project that's ongoing that just that just started.
It focuses on digital digital trace, broadly defined outcomes, and it employs poor stratification in order to sort of estimate average treatments.
So that's the that's the layout of what I'll be discussing and interrupt if I'm totally obtuse at any point.
If you have any questions. Micro replication.
So I've been focussing on this pretty extensively because I think that we've sort of moved into this sort of what I call data generation,
which is one in which the costs of generating data have declined dramatically.
Access to data has been what I would call democratise.
Lots of people have access to the data of convenient samples have become the norm.
Nobody assigns Leslie kiss anymore and talks about representative samples,
and there's been just a proliferation of data what I would call data generation modes out there.
And all of this has implications for the kinds of research we're doing,
the kinds of research that gets reported and has, in my view, resulted in some sort of.
Probably negative side effects or outcomes, and the most important one is that lots of stuff that gets published can't be replicated, right?
I mean, this is this is this is this is a table, a figure from columns 19, 20, 18 nature piece,
where he replicated a number of experiments that were published in highly regarded psychology journals.
And the the yellow stuff is the stuff that couldn't get replicated right?
And these are these are these are these are studies that are published in some of the
leading psychology journals and the discipline and replications of serious problem.
But you could see this in economics. You can see this in political science.
I don't know sociology, actually, because but I suspect you have a similar problems as the oldest.
So replications an issue. And and I think part of the reason that it's an issue is because, oops, oops.
Now I've gotten. Want to go back?
That's a small. It's a smaller one, OK, great.
So the reason we have this problem, I think, is because of this democratisation of data generation.
The fact that the cost of data generation have been dramatically reduced.
The fact that people people generate data using all kinds of data generation modes.
Right. And the problem, of course, at least in the experimental world, is if you do an experiment, right?
How do you know your? In either the blue or the orange state, right, you do an experiment.
You get a result. You write up a paper you submitted to the American Economics Review and the question.
The important question here for me, in any case, is that how do you know this is not generated by experimental?
Experimental measurement error. Oops! I don't know. Right.
The. Typically, you don't have a clue, is my point.
Right, I mean, so you don't really know that this is this. No, you don't experiment with 500 people, right?
You've got this result. It's a significant average treatment effect you really have.
It's very uncertain as to whether this result is robust.
Right now, you can you can publish it. And you know, you may end up being one of the unfortunate people in the yellow sign, right?
People try to replicate your result and it doesn't replicate. Right.
And so I've been thinking about sort of how do you address this issue?
How do you how do you basically take measures to ensure that the experiment that you've conducted right is a is generating a robust result?
Right? Well, Michael, replications may be a way to address this by Michael replications being replications within the same study.
Right. Essentially, I do a study and oh, maybe a better microbe.
Replicate this to make sure it's a robust result to make sure that other people will be able to replicate it if it gets published right.
And the point with the argument I make in this, in this,
in this article basically is that multi rather rather than single mode replications are the way to approach the problem, right?
So for example, if you do an empirical study and you get a interesting result, don't simply sort of,
you know, pay 500 more empowered respondents to do the exact same experiment, right?
And then argue that you've got a robust micro replication. All right.
So rather than rather than replicate within a mono or single mode, think about replicating in multiple modes.
All right. That's the sort of punch line of this entire.
And this this particular module and there are lots of different modes or context in which you can conduct this work, right?
You can use Amazon. You can use Lucid. These are basically crowdsourced.
Or you could use responded, which is a large scale subject pool, a commercial subject pool.
You could use something like knowledge networks in the US, which has a very sort of representative sample of online respondents.
Or you could use us or you could do something like the American National Election Study or GISS,
right, which has a very robust representative in person sample, right?
So the point is that there's various kinds of modes that you can use when you're thinking about replicating a particular experiment.
And what are you trying to do while you're trying to sort of figure out whether there is measurement error associated with a particular mode, right?
So some K is basically the measurement error associated with the particular mode.
Right? And when you represent an average treatment effect, you should always.
Assume that it's going to reflect real, a real average treatment effect, plus the measurement error.
Right. And the point is, how do you sort of figure out how big this MLK is?
Well, if you only if you only do this in one mode, you don't have you don't have really any idea, right?
If you only do this with them to it right, you're not going to really be able to figure out whether there is mould related experimental error.
Right? So unless you look at different modes, you're not going to be able to you're not going to be able to sort of figure that out.
And if you.
If you're reasonably comfortable that you can identify the measurement error right then of replicating and multimodal is a dominant strategy.
That's that's that's what I'm going to try to convince to convince you with this particular presentation.
So this is a simulation that we did as part of this paper. This is a paper that is is coming out on political analysis.
And what the simulation basically suggests is that.
This this this new year basically indicates how certain a researcher is that they will be able to identify which
of the modes is actually the better mode or the mode with the low or ideally close to zero measurement error.
Right now, if you don't think you can figure this out, then replicating in multi modes is not going to help you, right?
And so what we will we will we try to illustrate in the simulation is that OK, well,
what happens if you are 50 percent likely to sort of identify which of the modes is the low error or zero measurement error mode, right?
And of course, as this rule gets higher right, you're much more likely to be able to identify which of the modes is low error versus higher.
OK. So what this horizontal axis suggests is the probability that you are you are actually right in a high measurement error context.
Right? And what the vertical axis suggests is the probability that you are in a very low zero measurement error context, right?
And what the simulation is suggesting is that as long as you.
Have a reasonably good chance of identifying which of the modes has low measurement error finds them.
The green is basically a reduction in the the reduction in the sampling error relative to the the the basically
the green indicates that you're more likely to select the right mode if you replicate in a second mode and right.
And basically,
what this is suggesting is that as long as you're reasonably reasonably adept at identifying the the mode that has low measurement error,
then replicating in a different mode is always the optimal strategy.
OK, so let me just illustrate this.
So this is an experiment that I've been doing with two other two with an economist in Russia and one of our post-docs and in Santiago.
And the experiment is just a long experiment, right? And it's similar to a public goods game.
People play this experiment.
People, people conduct or are asked to do a real effort task, which involves having some digits and then they're asked to report their income.
And they can lie about the income. And our interests, our interest really is how much lying occurs in these experiments, right?
And the treatment in this experiment is a deduction, right?
Right. So the deduction rates vary and the expectation is that the line will decline as the deduction rates increase.
So it's a it's a pretty simple experiment. I'll show you I'll give you the details of the the actual design, right?
So people come into this experiment and there's three different tax rates, either at 10 percent, 20 percent or 30 percent tax rates,
the deduction rates people play this this experiment in groups of four and the once the taxes are levied,
they're redistributed equally amongst the group members. So there's a public good.
There's no exclude ability. There's no social gains or losses. In most of these, there's no audits or fines.
And people play this 10 times. And they're paid for one of the rounds at random.
They play in groups of four and there's random matching at the beginning of his expense.
OK. So the in each round people.
Add these two digit numbers, they have one minute. The more numbers they add, the more money they make,
and they're then told how much money they've made and then they're told then they have to declare
their income and then they're told what their total profit is and their total profit equals,
how much they didn't declare. Plus the the the equally shared tax revenues in their group of four.
So the the experiment is pretty simple, right?
And our primary interest here is understanding the extent to which people lie about their income at the after they conduct this real extra task.
And then the treatments in this experiment are varying the document rates.
OK, so that's a that's a sort of simple public good public goods kind of game.
So the interesting part here is that we conduct this experiment in different modes.
So this is my mode. This is the modes element here. So I conduct this experiment.
We conducted this experiment initially in the Oxford lab, right?
We did this with the Oxford subject pool. We had total sixteen thousand decisions,
right made by one hundred and sixteen subjects and about the average rate of lying for those subjects is about fifty seven percent.
So. So subjects basically lied or declared forty forty three percent of their income and the lying rate was fifty seven percent, right?
So that's the the outcome of the Oxford lab. So that's.
Which is in person? They've come. Oh, yeah.
So, yeah, so this is a lab experiment, right? They come in person, they come to our lab, which is just next door, basically.
And they play this game. So usually about twenty five people come into the lab and play.
Twenty four come into the lab and play. It is divided into groups of four.
And then they they play this. Actually, they played in the lab. They actually played it a little bit more than 10 rounds.
But but that's that and this was the sort of foundational result.
So we get this result. And we thought, OK, well, there's a fair amount of lying.
And but this is one mode, right? And so we actually replicated this in a number of different modes.
So we also replicated. We did it with our online subject pool, right?
And so here we had about one hundred thirteen hundred and sixty seven decisions and we had one hundred and forty four subjects play this.
People played online. So they they they came into a virtual waiting room.
They waited till there were four people. They then they then played the game exactly as we played it in the lab, except they played it online, right?
And so that's a different mode, right? So people are playing this now, and it's a different subject.
It's a different subject. It's our virtual subject pool. We then had people come.
We then conducted the exact same. Let me just now that confuses.
So actually, this. This version is.
Our subjects from the lab playing the experiment online.
This version of the experiment is our virtual subject pool actually playing it online.
And the last the last column here are us Merck subjects playing the experiment online.
So the idea here is we we we very significantly change the the actual mode in which to which the experiment was being conducted.
And so this is this gives you a flavour of what I'm suggesting people ought to do when they're when they're conducting these experiments,
if they want to leverage mode, right, as a possible explanation for measurement error.
So. So, so one thing I'm concerned about is whether demographics are sort of that controlled here across all different modes.
That's that's a very good question. So I'm not concerned about I'm not so much concerned about demographics, but it is obviously an issue.
And I'll show you in the analysis how we deal with it, because obviously you're right, these people will all be young, right?
Mm-Hmm. As will these people. Right. But as we get into the online subject pool, we're going to get older subjects right.
Particularly in the in the in the in the entire world.
But we'll get a more representative sample of the population.
I'm going to focus on trying to tease out the the motor effect here.
Yes. Just a quick question, because the experiment you needed groups of four, right?
How does that work with the Turks would also have like a virtual waiting room, and this was the work to say yes.
And it's not easy. Let me put it let me let me let me look.
This is very difficult to do online, actually, because what has to happen is that the virtual subjects have to come into a virtual waiting room.
They have to wait two hours for individuals and then they basically play the game right.
So it's it's quite challenging. Right?
And I'm only reporting here the the results from the UK in the end, but we also did this in other countries might.
OK. So this is the conventional sort of estimation, right,
where basically I've simply broken up the groups into the results into these the lab, the online lab, the online UK and the internet.
Right. And the initial sort of suggestion here is that I shouldn't touch this.
OK, the the deduction rate here clearly seems to be working right as the deduction rate goes up.
Right? You're actually people are reporting less of their income.
That's the negative coefficient. There's some evidence here that the online subject pool, right, is behaving differently.
So this is the conventional strategy, right?
So what we argue in this paper is that really you want to sort of leverage machine learning to try to tease out the heterogeneity?
Right? Because essentially what we argue in the paper is that you want to be totally
indifferent about what the possible how heterogeneous treatment effects can be.
And then you want to look at all of the possible heterogeneous treatment effects
and then draw some conclusion as to whether it's mode related or whether it's,
as you pointed out, maybe related to particular demographic characteristics.
Right. So that's the that's the the the punch line of this paper.
Right.
And so we argue basically that you want to use some sort of machine learning effort to sort of identify conditional average treatment effects, right?
In other words, the average treatment effects that are conditioned on particular characteristics, either of the mode or of the sample.
Right. And so that's the first sort of stage of this estimation strategy.
So we use this broad estimation, which is sort of like a random us, which I'm assuming that Roberto talked about, OK.
And so basically, it's just sort of a machine learning strategy for teasing out heterogeneity in the data set, right?
And what this essentially allows us to do for each individual in the dataset.
So if you go back here, we have.
A lot of decisions, we have about twenty four thousand decisions here, six or seven hundred here, but they're not totally independent.
Fourteen thousand here and fifteen thousand here, so we have lots of decisions being made.
And that's really important, right? And the the the idea here is we want to be able to determine whether the average treatment
effect with respect to any particular covariates in the dataset right there is right.
That's the big. So it's not in an ideal world, right?
The average treatment effect right should be constant across for all individuals in this dataset.
Right. So the barred estimation is going to allow us to look at the extent to which the
average treatment effect is different for different individuals in the dataset,
and also will allow us to infer whether variation in conditional average treatment effect is related to mode or something else.
So that's the the punch line of this exercise. So the the bar code is available on my GitHub, and it's pretty simple to implement.
All right. And this is the result, right?
So this is the result. So what does the result tell us?
So if you look at this blue line, that's the estimated average treatment effect for all of the data, right?
And basically, it's about it's clearly it's it's about a negative point seven, right, which is the right direction.
Right. So we want we want the we want the treatment effect to be negative because we as detection rates go up.
We want people to cheat more right and report less. So that's clearly the right outcome.
The dotted red line is the zero effect, right?
And all we've done and this is the sort of attraction of the sort of the bar destination strategy is we've simply
basically organised all of the conditional average treatment effects along this horizontal horizontal axis.
And then we've looked at the. No.
OK, and then we've looked at the the magnitudes, right, so clearly,
if you sort of look at so this is what sort of really attraction attractive about this estimation strategy way, it's visually very summary, right?
It gives you a very nice summary summarisation of all of the data very quickly.
So. So this basically suggests that, well, most of the conditional average treatment effects are below zero, which is good, right?
There are some above zero, but they're not that many. Right. So that's sort of encouraging.
And then all we've done here is we presented we've simply graphically presented the data
of colour coded the four different modes because modes the modes of what we really were,
what we were really interested in in this paper. And oh, now I don't have the legend, but I think I remember.
So the the the red is the lab.
The conventional lab mode, the orange or brown, is the the lab participants online.
And the grey is them to work, and this light colour is is the sex online subject pool for the UK.
All right. So, so so right off the bat, you get a sense, right from a visual sense of weather mode matters here.
Right, right. You get a visual sense of weather mode is affecting the magnitude of the average treatment effect, right?
And clearly there is clearly some evidence here, right? Clearly, the and this was your intuition, right?
I mean, clearly, the online subject pool clearly is more likely to have a quite moderated treatment effect, right?
The lab subjects clearly are much more likely to behave behave at wrestling, which is consistent with a variety of other stuff that I've done.
So it's it's not it's not surprising now. So this is this.
This is what I argue people should do. They should do their experiments in different modes.
They should explicitly use diverse modes, right?
Because you want to you want to establish the robustness of your treatment effect, right?
And then and then visually. Explore the likelihood that mode might be might be affecting the magnitude of your sex life.
Yes, so just a quick clarification, so this is a conditional average treatment effect condition specifically only on
the mode or this is condition like on the I saw that you included gender and age,
you could do everything you can do out here. I've just presented the mode.
So the one above is so, so, so. So here's the way to think about this.
So this is all of the individuals in the data. Right? So this person here, right, might be a woman of age 19 and 19 in the UK subject pool.
That's the way to think about it. And actually, these are decisions. So that person actually shows up ten times, right?
But the her decision could vary, actually. So and I've just taken that this data here and just reorganised it, right?
So I'm only looking at the four modes, right? But you could reorganise it however you want.
Right? You could then say, I'm going to look at gender, I'm going to look at gender and whatever you want.
So I've just the average. So the average treatment effect for each, each person you, I just exactly right.
That's it. So that's the sort of attraction of these,
like the BART method explicitly estimates a an average treatment effect for each cover, each unique covariate in the dataset.
Right. And then you can sort of organise everything however you want. All right.
That's the. And so this is what I this is. My argument is that this is probably how you want to sort of approach the exercise.
Yup. Just a short question, I mean, like on the x axis X, it must be like decisions like if you have its decisions,
its decision, how can you compute like a treatment effect for a decision?
I mean, it's. Well, it's because that's a good point, well, the the person this individual will make 10 decisions, right?
And so here we've committed, we've actually computed a average treatment effect for that person 10 times.
But it must be looked like there's a lot of there's a lot of there's a lot of court.
Yes, there's a there's a high amount of correlation here. Yes, now, but it's not total, right?
I mean, because people will make, you know, for example.
You might argue that in the in the in the as a covariates, you might think when you make that decision matters.
So whether you make it in round one, whether or whether you make it in round 10.
So that can be part of the covariate dimensionality.
Right. But you're right. I mean, they're probably highly correlated.
It's a bit misleading to sort of say, I've got 5000 5000 observations here.
I can see that's. It's probably more like five hundred.
OK, so but this is sort of generally the flavour of the way I think you should approach the the the problem of a micro replication in multimode.
Find out whether, you know,
there is some heterogeneity to the conditional average treatment effect and then think about whether it's related to mode, and that's what we did.
And then we asked ourselves the question Well, why are we getting so then the hard part, of course, then is, Well, why are we getting this right?
So that's the next stage of the analysis, right? Clearly, there seems to be some measurement error here.
But why? Right? Yes, this a very well.
Because you choose different countries rights and different mode, the only specific in different countries.
Wouldn't it be better to test all the modes on one country?
That's a good point. That's a good point.
So we. So we effectively did.
The first three modes are all in the UK, but we then did what we did want to use a crowd as sort of a crowd sourced mode like M.
And the the problem is, at least when we started doing this, it was very difficult to find a crowdsourced mode in the UK.
So we did take the easy solution and we did do it in the US.
But the the but.
But you may be right, but the point is that the the UK or the Americas and the UK online subject cool right do sort of exhibit similar patterns.
But you're right, in an optimal world, we should not have we should not have gone out of the EU.
I agree. Yeah, I had a question about it,
because the modes are still within like into sample selection after people kind of go to
the Nuffield or to your interview or through the Met with their sort of virtue is to.
We out measurement error relative to some sort of general population of people who are in all the different surveys.
Yes. So we try to do that. I'll show you in a minute how we do.
Or maybe I mean, we try to address this and I'll you can see whether you think we did it reasonably
because the idea of them being that replication is so that the initial study,
which is basically the the reason for for this in-depth analysis, is this replication hasn't been successful.
But then this would not solve the replication on the same mode, right?
There are still a nano value in replication with the Mechanical Turk.
There would be a logical comparison, right? Yes. Yes, exactly. So, so if you did the Mechanical Turk study,
the I would argue that the value added of doing a replication within a Mechanical Turk is probably not a high.
Right. And that's what our little simulation showed. It's probably better to think about doing the replication in a very different context,
at a very different mode, because that will be more informative.
Yeah, but but then just to get it clear, because we could then suggest that nature replicates in a different model,
you should do it in the same mode, right? Because if they're doing the highest measurement error,
then the risk while the risk would have mobility of having a different position
than the original paper just goes up by virtue of the measurement error.
Well, or you or you could sort of think about the fact that you replicate in a much more noisy environment as being a conservative replication test.
Right? I mean, that's that's what I would argue. I mean, I would definitely argue that.
Thank you. Yeah,
I was also wondering the within my variation and how you compare that because I guess it's a lab has a bigger variation compared to like them type.
So, in other words, is the variation within the lab mode, right?
Much higher than the variation within the.
Or much higher than the variation across modes, right?
Yes, you could. You could easily you could easily do that with these data, right?
You know, easily, right? You could do a basically sort of you could compare the sort of variation across modes easily.
Right. I didn't do that, although I'll show you something in a minute that I think sort of goes in that direction.
OK, so let me. No, that's not the direction.
Well, maybe this. OK, so it is so the thing is, you see this.
You see this these differences across modes, then the question is, well, is there something,
can you establish what it is about the modes that might be contributing to this measurement error?
And so that was the next stage. That was the next step in this exercise that we went through.
We then sort of said, Well, maybe there's something wrong with him perk, right?
Maybe there's something about empiric experiments that creates measurement error.
All right. So that was the next stage. OK, so all we did here was say, OK, people made these decisions 10 times.
One possibility is the Amtrak people aren't paying attention like that.
They're just clicking through like this, which is what a lot of people sort of suggest.
All right. So here what we did is we simply looked at their performance on the real effort task and see whether they whether the the the the the.
Into the Inter class, correlations within those 10 rounds were high.
In other words, if someone did well on the first 80, did they do well on the 10th?
Right. So this was sort of a measure of stability, right?
And basically it, you know, basically people behave pretty stable across the 10 rounds, no matter what the mode was.
So. So that clearly was not didn't seem to be contributing to the problem.
So we sort of pushed on this because we wanted to sort of see what was going on.
And we. We then explored this notion, well, maybe there's an age issue, right?
In other words, maybe the if because if you look the the the M Turk and the SS online are older.
Right? And they clearly report higher levels of their income.
Right.
And the the the UK subject to all the Oxford subject pool clearly is younger, and they report they're less like they're much more likely to lie.
Right. And so we explore this. We introduce some controls and it doesn't seem to be age, right?
So if you control for age in within modes, right?
That doesn't seem to be driving the difference. Right.
So we sort of rejected age as being this,
this this other socioeconomic characteristic that might have been explaining the difference between the modes.
But we still were sort of concerned about this network effect. So we then as a possible explanation for measurement error.
So then what we did? So this was thanks to Roberta, actually.
So then what we did is they're also OK, oh, well, let's. We were at the time when I was sort of working on this paper.
We were doing these online experiments in India with Amtrak and with our CSA online subject pool in India.
And I thought, Well, maybe we can sort of explore this notion of inattention more explicitly in in with some,
some experimental work, with some additional experimental work, right? And so what we did is we did two things to summarise this.
We did. One thing we sort of explored was, Well,
what happens if we explicitly introduce measurement error into the experiment and compare people who have who have who,
who are who are subjected to the same experiment decision making experiment, except with additional explicitly added measurement error?
Right. And are are sort of conjecture here was well for the intern respondents.
It shouldn't matter because they're already inattentive and there's already lots of measurement error in their decision making.
And so that's the top part of this table. And effectively, that's true, right?
So in this experiment that I won't go into detail on, we we we got we didn't get a significant effect for the Turks,
but we did get a significant effect for the CSA online subject. All right.
And it turns out that introducing measurement error for the online subject pool did affect the decisions.
Right? But it had little effect on the network of the respondents.
And that was some initial evidence to us that the enteric respondents, unlike the online respondents, were not paying much attention.
So then we decided, OK, let's let's let's leave explore this further.
And we then sort of explored the effect of incentives.
What happens if we have these enteric respondents make the same kinds of decisions,
except in this one explicitly incentivise their decisions so that they understand that if they click through or are inattentive,
then they'll be forgoing some income. And here we get exactly the effect we expect, right?
Once we introduced incentives and made it clear to the enteric respondents that if they didn't pay attention, they would be forgoing income.
We got a significant effect. Right. So. These various levels of exploration led us to sort of conclude that it seems plausible that these effects
we're seeing here are the result of sort of inattention on the part of the participants in the experiment.
The takeaway from this, though, is simply if you're doing experiments with subjects.
Think about doing them in multiple modes, different modes, right?
Think about analysing the data with machine learning because here, here,
here where we're not imposing any kind of structure on the conditional average treatment effects,
we're basically saying just let's sort of let's see which is significant, right?
And then let's look at visually look at the conditional average treatment effects
and then see whether it's plausible that the result is robust to different modes,
right? And then once you find some patterns in the data that suggest that there may be motor effects,
then spend some time trying to figure out, right, what is it that might be plausibly contributing to the medicine there?
Yes. And this is really cool, right, but I'm a bit confused.
What the aim is exactly, is it to say that we are certain that a given study, the effect size of a certain study is true in some sort of sense.
And so you're doing these to these and do multiple reputations across different modes and see if that initial one was correct.
Is that the that's that's the that's that's the ultimate. So the ultimate aim basically idea in an ideal world, right, is that, you know,
the the condition of its treatment effects in these different modes that you estimate, right, are all sort of very close to this blue line.
Right. And regardless of what the mode is right or if they aren't that there, there's no systematic pattern here.
Right from one mode to the next, right?
But does this this relies on you getting some sort of initial effect size from a single study?
Well, this is the problem. I mean, I mean, this is the right.
This is the sort of broader issue, right? You do an experiment, OK?
And you get, you know, and it gets published in the American Economics Review.
And it's only based on mTOR subjects, which we see a lot of, right?
Then the question is, OK. Right? I mean, I'll stop there.
Right? Why not? Right. I've got a significant condition.
I've got a significant treatment effect. My point is you shouldn't stop there, right?
You should at least sort of explore the robustness of this treatment effect in these different, very different modes.
So I fully agree that people should be exploring and, you know, not presenting the results in one study.
But would it not be better from the get-go? Just the problem with some of this is that people are finding significance in studies that,
given a certain effect size are underpowered, for example, or that doing certain activities like hawkings.
So they're not registering hypotheses and therefore finding these significant results and the effect size is actually not interesting whatsoever.
I I I know this a slightly distinct issue, but if we did that more instead of replicating ad infinitum, that might be actually a better strategy.
And I think I think it's a very good strategy.
I think it's a necessary strategy, right? Preregistration, you know, holding your hand.
So there's no parking, having properly powered experiments, all of this is very important, right?
But the bottom line is you could have a pre-registered study that's highly powered on a metric that looks like this.
That's my only point. I mean, I'm not suggesting that you shouldn't do all these things.
I think they're very important, but I'm simply saying you can do all of them and still be here.
Looks like you. So I think it's fascinating to use the machine learning approach, but then giving your research aim.
I'm just wondering, we had this recurring discussion within the Summer Institute about the potential for shift changes at Mechanical Turk,
so this would be within modes measurement error in your framework.
So when you go on Monday, it's a completely different symbol when you go on Wednesday.
And so therefore it's very difficult to disentangle whether between mode,
measurement errors and effects within modes is dependent on the within mouth measurement error.
See what I mean, like situation, seasonality,
etc. So I wonder how you obviously the of things to do with to do multiple within modes as well and look at the measurement error.
But what is your intuition on that? This kind of feels that this is a pretty important components.
Right. So so you're ah, so your basic argument is that time is a confounding variable here.
Yes. Yes, it is true that you did these other sort of modes, but you did them a month later.
Yes. I mean, I'm not sure that I could escape that problem here because sort of anticipating the replication strategy,
anticipating the micro replication strategy and then doing it simultaneously.
Yes, you're right. I mean, this is it's always the possibility that now.
Some things are more likely to be confounded than others.
I don't like this study is interested in cheating and lying. It's not clear to me that this particular outcome would be susceptible to this.
But you're right, if it were political, right, and you were talking about, I don't know, some sort of framing experiment, what you're interested in,
whether people respond to some, you know, framing treatment right in a political campaign, then then this would not be appropriate.
I would say there's a fundamental difference in enough fields Nuffield fifty thousand and mechanical stroke, which is a marketplace, right?
So. So there's a demand supply if you see what it means.
So I. Yes. So what is the difference? Why I would say that that the marketplace has a lot more potential for within modes.
Temporal variation, which is not necessarily like in the classical treatment sense that it's before or after an election,
but just two a.m., four a.m., six a.m. ET. Oh, I see, I see.
I see. Because right, a time of day.
So I'm just trying to think of why that would be. So that's a lot of the people in some of as a way to support us for our best to involved.
So therefore, you would have a different subset of the population.
Oh, I don't doubt that. I mean, I mean, I don't I don't exactly know what, what, what is making, what is generating this.
Nor do I know exactly why I'm turkers or less attentive, possibly than says online.
I mean. Well, I do have. I do have a. And that that's what motivated this.
Idea here.
So the first online subject rule is non deception is highly paid, highly compensated, there's the word we have very strict ethical rules, right?
It's very infrequent, right? Someone.
I mean, it's so unlike internet, where they're professional online crowdsource workers, they're not being paid much, right?
They may be in some click farm in Venezuela, but we don't know, right?
They could be bots, right? So all of that is an issue, right?
Just to further respond to. There's also if you if you use the platform, there's ways you can screen for that.
So you can make sure that, for example, you get from Pakistan,
let's say people from every part of every hour like you have a quota for every hour and then you
stop collecting and you can also enhance that with instruments like the ones Ray and I use,
like Qualtrics and stuff where you put extra like, for example,
quotas in like people click through and then we say no thanks like we already have people like you.
And then the quota opens back up. So like, it's a question of using N-Trek is just a way to reach people you want,
and then you just have to be very, very careful about the constraints you put on the people you want.
So the only them they click through, that's the only thing that I would like to think of it as a population will be flexible enough.
I mean, because the batteries are in the fine.
Whereas for all other modes, it seems they are, I'm not sure that I don't know, most especially will not be one of the boundaries.
You know this? Yes. On the basis of feel like I no.
The boundaries of the extra population and even set aside, even if such, et cetera,
we don't know of any point in time how many people from Pakistan are in the big.
So it will be in flux or I don't know how you see I, yeah, I'm not sure I would call it.
That's interesting. I mean, yes, it's. Yes, I mean, I'm not quite sure what you mean by influx.
To be honest, every, every so often.
So you always know your list of participants that you know the broader thousand.
So the Nuffield participants aren't unlike Turk in one respect, so there's like India, there's like 15 or 20 thousand right.
They don't all participate in the experiment. We sent out an invitation.
We don't, you know, the some percentage of them will be interested and sign up.
I'm not sure how that's dramatically different. I mean, I think the subject pools are different, but I don't know how in terms of flux,
how that's dramatically different than publishing a hit and waiting for some immature person to sort of say, Oh, you know, I'll respond.
So I'd say the most important confounded potential will be competition on the marketplace.
So for example, if. Yes, you're right, we're not competing with anybody else.
And you're right, an immature crowd worker basically is looking at all of the possible hits and making a decision.
You're right that that is quite different. So I was wondering if you found, and I think maybe you mentioned previous studies found this well,
that the tension is very important in maybe explaining the difference between like experiments and online experiments.
And do you maybe know of any study that has tried short attention tests during
and in another experiment the sort of controls for that like to have a weight?
I I'm sure people have. So there is a woman who in the US who has looked at this pretty carefully sunshine hooligans, the Duke,
and she has this whole sort of research agenda explicitly looking at my work, Lucid responds.
And not just them also also looking at people who just answer SSI or responding and explicitly looking at,
you know, this whole phenomena of inattention, click throughs.
And she has a really interesting paper where she tries to explore the relationship between that behaviour, right?
And subject characteristics, right?
And you know, some people, for example, say, Oh, well, you know, we'll just get rid of these people, right?
Let's. But that's I mean, her argument is that's not what you want to do, right?
Because these are sort of an interesting population, right? So she has a very clever sort of analysis which is worth looking at.
Thanks. Thanks very much. I'm not an experiment person, I'm a demographer,
and I keep thinking whether this mode effect that you have or what you call and mode effect that this might actually be some sort of a, you know,
a difference in the composition of the people in the different modes and whether there would be a way or maybe this is not something you do,
but whether there would be a way to think.
For example, if you think that there might be educational differences in the way people respond in such an exercise,
then you could potentially have a highly educated pool of people for each for
modes and the low educated for each modes or what you could maybe also do.
I don't know if this is something that you do,
but you could take your Oxford people and take the same people and make them play the game in the lab and
make them play the game online and see they the same people if they do something different because to me,
that would be a mode. So when when I'm in the lab, I would be nice and when I'm at home and nobody sees what I'm doing,
then maybe I won't declare my income at the same rate. That's a very good point.
So there is a paper out there by a woman from UCLA, Lynn Vavreck, where she actually randomly assigns people.
I mean, she recruits people and then she randomly assigns them to mode.
She randomly assigns them to online, randomly assigns them to, I think, in person.
And she compares, it's a recent paper, so. So she does explicitly this and her result.
There are some areas in which there are differences, clearly, but I forget the but that is the that is the optimal design.
And if we could have done it, we would have been good to randomly assigned to randomly select the UK to the test
subjects and then randomly assigned because obviously they're not randomly assigned,
they get to select whether they want to do it in the lab or and so there may be selection there.
The other thing about this, I'm just thinking about the Sunshine's papers.
The other thing she looks at, which is really cool, is. There's this big phenomena in online where people give.
Ridiculous responses. And this is becoming a big problem, and so people, particularly younger people who are sort of in a social media sort of,
you know, been on social media, all our lives, right? They tend to sort of respond to these.
They just give, you know, absurd responses. And this is a big problem.
And these these experiments, right?
Because you know, you, you know, you see these people that are sort of really giving extreme responses to the question as well.
Is this true or false, right? How do you deal with it, right?
And again, she looks at the idea that people often try to just to drop them out, right?
But that might not be the optimal strategy.
So I agree it's a very I don't think I've solved this problem, but the idea is to give you a flavour of how to think about both on the design side,
think of multiple modes and then on the analytic side, think about the machine learning to sort of tease out the,
uh, the mode effects and establish the robustness of your treatment effects.
I think that that's the so the my my punch line on that, my punch line on the other.
Okay, so I have 17 minutes, so I'll go over this very quickly.
So a second thing about the second project that we've been involved in and says,
and this is sort of an extension of something that I was working on with with Roberto is news using
sort of the virtual environment post ratification methods to estimate average treatment effects.
And this is a project that is quite new that I'm working on with a post with a Ph.D. student at King's College, Philippe.
And here I'll just highlight some of what I think are the more interesting parts of the project we're interested in.
So there's this big debate in political science, but probably only in political science interested in whether.
And there's sort of a puzzle. So when you think about democracies, you think about people should be informed about their political representatives,
and there's been all of this literature out there,
these experiments that have been done in which people have been given information about their politicians or the
incumbent politicians about the fact that they're corrupt or that they're hopeless and that they're doing bad things.
And most of these experiments have suggested that it doesn't matter.
You can tell voters that are, you know, incumbent politician is a crook, is hopeless,
is stealing money from you, and it doesn't affect their decision in terms of re-election.
They don't. It doesn't affect their vote choice. Right. And of course, you know, political scientists are quite sort of disturbed by this, right?
This has got them agitated. And so this is an experiment in that vein, although we think we're doing it more rigorously than most people have done it.
So what we've done is so we have this lab in Chile and we've set up we have a collaboration
agreement with the ministry in Chile that is responsible for auditing municipal politicians.
So they audit the politicians.
And we've convinced them to cooperate with us and to to allow and to randomly assigned these audits to different municipalities in Chile.
And so that's a that's an opportunity for us to explore whether the results of these audits affect voter decisions.
And there's so we're just starting this and the elections in 2020.
So this is a unique opportunity for us to randomly assigned audits to the municipalities,
inform the voters about the results of these audits and see whether it affects their vote decision.
So that's the sort of flavour of this. And so this is Chile.
There are three hundred and forty five municipalities in Chile.
It's a weird country, like it's sort of like a, I don't know, like a worm or something.
And then it's there. Everything is sort of distributed along here.
And the idea is that we're going to randomly assigned audits to a 40 of these No.
To 60 of these municipalities. So the basic design that we think is vaguely interesting is that we're going to do a pre-treatment survey.
And we're going to the sample is going to be 60 municipalities and two hundred and fifty.
So small counties, I call them zip codes, but so the zip code, things are within the 60 municipalities.
And we're going to basically target 6500 people in the pre-treatment survey.
And we're going to use Facebook ad manager to basically identify the two to organise the the pre-treatment survey.
So we're going to we're going to identify.
So we get the basic.
We have 250 of these sort of zip code things and we're going to identify people in these 250 zip code things using Facebook ads manager, right?
So that's and then we're going to recruit them into those into the experiment.
And then we're going to conduct a pre-treatment survey which will give us covariates
and will allow us to get a sense of what their vote preferences for the municipal gov.
Right? And so the treatments then are going to be random audits of 30 of these municipalities that the ministry will do
and 30 of the municipalities will be in the audit treatment and then 30 will be in the control and treatment.
And then the information treatment is this is the thing we're going to be doing.
We're going to we're going to force for some of these 250 zip code things.
We're going to have an information treatment. We're going to inform everybody. Hopefully in that zip code of the result of the of the audit.
All right. And of course, there will be people in the control zip codes who will get no information.
That's the sort of broad idea. Then there will be after, after there is this treatment.
So there's two levels of treatment, there's an audit will be one level which is at the municipal level and then there will be
information treatments within the audits within the within the audited municipalities.
Then we'll there be a post-treatment survey in which we go back and re-interview those 6500
people in principle and then we'll get their vote preferences in the post-treatment period.
And then the final part of this is to think about post stratification, how do we estimate those?
So we'll be able to estimate treatment effects for these 250 villas, villas or these municipalities.
But then the question is how do we post stratify those estimated average treatment effect to the entire population because we want to know,
you know, the politicians and we do we want to know what the what the the the treatment effect is for the entire country.
Right? Potential treatment effect because that's what they're interested in, right? Does it does it make sense?
I mean, they're interested in whether these audits have any effect for the nation.
Right. And that's where the stratification comes in, which is quite important.
And this is the part that most people don't do, right? This is the part that most experimentalists don't think about.
They they estimate a treatment effect. And that's it. All right.
And we're sort of hoping or thinking that this might participate partially contribute to the fact that people don't get really good,
good estimates of what what the information treatment effects look like. OK, so the Facebooks, this is all.
So we're going to use the ad manager, which we've used before in Chile, which will allow us to so basically identify the particular one of these,
which of the two hundred and fifty sort of count of zip codes we want to target for recruitment into the study.
So we'll we'll do this for two hundred and fifty zip codes right within those within those 60 municipalities.
And we'll recruit people using Facebook ads, right, which we've done before.
Right. So we'll recruit them into the into the into the study. And this is a highlight for us.
This has been a very effective way of getting people to participate in these virtual experiments.
It's a bit expensive because you obviously have to pay Facebook, but it's pretty.
It's it's pretty useful and it allows us to target particular treated and untreated segments of the experiment.
So we'll do that sort of allow us to target subjects in specific counties or zip codes.
We will use banner ads to get people to come into the to come into the experiment.
And ideally, we'll do this with 6500 people. That's just the dashboards you need to see about this.
So just just to be clear, the treatments will be we'll actually have two levels of treatments.
One will be people will see the results of an audit of two by two by two factorial design.
There's two information treatments. One is the audit result, right?
And the other is what we call a report card, which is just a report on the performance of that municipal mayor.
And so some people will see just an audit. Some people will see just a report.
Some people will see a report and an audit.
And then there's a control condition where they where they get where there's no audits and no information just.
And so this is this roughly summarises the. So there's you can this is the way we.
It's not worth getting into, really, but the.
So there will be treatments, all of these are the treatments that are administered by the ministry right at the at the municipal level,
which is an audit or a no audit. And then there will be information treatments that are administered at the zip code level.
Right. So within all these audit, no audit treatments administered by the ministry,
there will be information treatments at the sort of zip code level that reflect the sort of either the report or audit or report and audit or control.
That's sort of clear. So the information treatments.
So this is sort of this is the kind of information you would they would see, right?
You know, they're, you know, this municipal mayor or, you know, six years in prison for defrauding.
That's OK, so that's the problem with the design, to be honest.
So. There are multiple levels of this experiment, but you're right, I mean, we are targeting a particular zip code using Facebook ads,
so only people in that zip code are going to sort of there are going to be targeted with the information.
But yes, there's I mean, we cannot assume that there won't be contamination or yes, I mean,
that's I mean I but we've got to be honest, we've not figured that out entirely or how we'll measure it.
I mean, so even if there is some. You know, spread the important issue is whether we can measure it, if we can measure it, then that's fine.
But the possibility is that there will be this stuff going on and we won't be able to measure it.
So I'm not sure exactly at this point how we're going to tackle that problem.
You're right. And that same time, and now, I mean, there's a stronger just right.
So you can have media outlets going to one particular party. Yes.
So yeah, so so how so one strong confounder that I can think of is media, right?
So if you have an information campaign that which is saying in a particular villa, this candidate is good or bad or whatever,
but then you have media sort of being biased in any way towards one particular candidate, one particular party and so on and so forth.
It's sort of like a strong confounder. It could actually influence that person's vote over the information campaign or vice versa.
I'm not I'm not on the topic, but I could definitely see the second sound right.
Yes, so. So you're right.
Uh, so so I guess the problem that you're identifying, you say, OK,
so we have one zip code where that's in our information trip and another zip code that's not in our information treatment.
And both of these zip codes is getting both is getting a media.
Representation. Well, that I don't think is too much of a problem because I mean,
it won't affect our ability to distinguish the in for our information treatment to the treated.
Now, in the worst case scenario, it'll totally wash out any treatment effect, right?
Which could happen in which case, yes, but it won't.
It will bias our estimate of the treatment effect because both of these both
of these units will have received in principle the same media information.
But you're right, it could totally wash out the treatment effect.
It's possible. Our assumption is that the media is not going to be focussed on specific municipalities, but we may be wrong.
So one way to make this about this is to simply ask the question is sort of like exposure to media, right papers.
How do you go on Facebook? Opposed to other types of information we could try?
Yes. Yes. And we could do that and do that both in the pre and the post treatment.
Yes, that's a positive, but that's a that's a possibility.
Now, the I won't even tell you the other part of these people's language is even more sort of.
So the other thing? So the so here we're I'm just talking about the voters, right?
The other part of the experiment, which I'm not going to talk about today because I'll tell you which is even more sort of really problematic,
but maybe more interesting is that we want to. So the problem with a lot of these political science studies is they look at this,
they look at the the the the treatment effect on the vote, right, which is what we're doing here, which is what I'm talking about here.
But the other thing we want to do is we want to treat these municipalities with information and see whether it affects the politician.
Right, because that's the more and so what happens if we do an information treatment and broadcast the result of a.
Audit to the electorate. Right. The interesting one of the most interesting thing for us is whether that affects the politicians behaviour.
Do they start engaging in more client list stuff?
And with this audit agency, we are able to monitor their daily expenditures and where they're going.
Right. So we'll be able to determine whether, you know, our information treatment,
in addition to the audit result, right, affects the specific spending priorities of the municipal mayor.
So that's the sort of interesting thing is so we'll look at their spending, but we're also this is the thing that Roberto inspired me on.
We'll also look at their Facebook pages because we've just we've noticed that all of these municipal mayors have active Facebook pages.
So the question is when they're treated, either with the audit,
the random audit and results and with our in for and the municipalities treated with our information treatment,
whether that has an effect on the content of their Facebook stuff.
Right. But also whether it has effect on digital trade or other words.
Now this is even more so hazy. Whether we can sort of look at the people, how do people respond to like a negative audit the voters do.
They also go to the Facebook page of the politician and who is going to the Fed, the Fed, the Facebook page of the politician, right?
Because it will be in principle, will be monitoring their Facebook pages and we'll see who goes to the Facebook page.
And then we'll be able to say something about the people who the voters who are
responding to the both the shock and to the content of the politicians Facebook page.
That's sort of that's the sort of design issues.
And then I have one minute I'll just wrap up. OK, we'll do the average treatment effects.
That's all. So we'll also the ideas we'll do post stratification, right, which Roberto's already talked to you about.
But you can sort of think, well, we'll have some very not too broad.
We'll have about thirty six cells in our post stratification frame, right?
Two gender, two education, three income, three age.
And then these are the number of cells right generated by each of these categories, right?
And then we'll have about thirty three individuals from the 6500 right in each of these in the audit information, cell breakdown and in the audit.
No information cell breakdown. So these are the cells to which we thought on, which will be estimating the post stratification model.
Right. And since you know all this already, we'll be doing, we'll be estimating a pre-treatment vote choice model, right?
Using some sort of machine probability estimate or right like random forest, right?
So we'll estimate pre pre-treatment vote vote preference and then we'll be estimating post-treatment vote preference.
In principle, we can post stratify that to the.
To all of the sudden, not seeing it, no. Whatever numbers now, I don't will be able to post stratify those results to all of the cells,
either in the nation or within a particular municipality, right?
And then since we post ratified the pre-treatment vote choice to all of those cells and
since we post stratified the post-treatment vote preference to all of those cells,
then we can simply by subtracting those two quantities,
get a sense of what the average treatment effect is for this very detailed breakdown of individuals within the nation.
So that's the sort of rough strategy.
So that's. Might. I'm sorry, I mean, this is a very general question in relation to forced certification in the context of experiments,
and I'm wondering if so in that I understand the rationale here to some extent, is to go from internal validity to external validity.
But if we think about what you are talking about, Maude's earlier, right, and perhaps is less relevant specifically for this example.
But how do we think about the fact of both started flying when there also might be these more defects that might be operating?
So, you know, can we hope so that I find it hard to get around and think about because in the specific context of experiments that could be elsewhere?
So this is probably the criticism that we will.
I mean, I have to deal with because of course, I'm I mean, I think people will come to us and say, OK,
well, you know, I mean, the point of departure here is to recruit subjects right using Facebook, right?
And that's the mode, right? And the question is, I mean, I know now from having done this in Chile that, for example, right?
It's very difficult to get certain segments of the population to participate.
Right. That's. Very specific to Facebook.
Right? We know now that it's hard to get younger people, younger people don't have Facebook accounts.
I mean, they're on other social media, right? So they'll be, you know? And it's hard to recruit that segment right into the study.
So you're right. But then that is a limitation.
I mean, and doing a multimode here would be optimal, but then it would be.
Well, I mean, you're right. I mean, I could do a multi-mode here.
I mean, there's no reason why I couldn't, for example, try to recruit people, explore the recruitment on different social media.
And I could probably end this assuming the budget I could explore sort of non social media recruitment into the study,
which would be optimal because clearly there are, you know, that would be now.
The one saving grace here is that. So I think mode matters quite a bit when people are in, for example,
in the cheating experiment where people are interacting with each other, making sensitive decisions like lying and cheating, right?
So I think there are some modes in which people are much more comfortable doing that in other modes in which they're less comfortable.
So I do think that mode matters here in this case.
We're asking a pretty simple. Saying basically vote preference, and so I would expect mode to be somewhat less important.
I mean, just to just to add on that, it's also the idea that as me and Ray have explored the digital trace,
you observe people you don't necessarily have to ask. And so the idea is that even though there might be some mode effects, we're not.
You don't have the same problems you have with anthrax where they are like they
they're they have an incentive to look for the treatment within your questionnaire,
whatever. Here people are just behaving as they would naturally on a politician's page,
and we can see on the politicians page whether they leave a positive or negative review.
And that sense, it's not that dissimilar to the given also the potential end on Facebook compared to like a lab experiment or an amateur.
It's not dissimilar to the Xbox study because it's not completely absurd that we would get
hundreds of thousands of individuals commenting amongst these different municipal places.
I think, yes, as always, there isn't selection of X issues, but I think less important, you don't talk for sure and are the other types of modes.
But. No.
OK. I mean, I would then I'll just wrap up.
Bye bye. Bye bye.
Elaborating on what Roberto said, I mean, so optimally, I mean, and this would be the mode thing optimally would in its budget related.
I would very much like to have this unobtrusive measure where I'm just observing who
is participating on these different web pages and then and then and then using those
digital then sort of reverse engineering from that digital activity to the individual and
then populating these cells in terms of partisan preference and socio economic stuff,
if I could get it. That would be the optimal sort of outcome.
And I think I think ideally, I would want to complement the simple survey strategy with these digital three strategies if it's possible.
But that's again, budget. As we as I learnt, it can be expensive trying to sort of collect this digital trace information.
So that question of the life of the person, the researcher, will we think that of the US in the US with accountants?
Say that again in some reputable research, that's.
It shouldn't be controversial. That would be my position because I have a Russian,
a Russian co-author who has seen what Roberto and I did in the Texas election and said, Oh, we should really do this in Russia.
I can get access to all the Russian version of the Facebook data.
So I'm wondering, yeah, I don't know. I don't know how I would be sort of perceived as sort of interfering with Russian election.
You're right, I'm sympathetic to your point.
Just a comment on the problem with the spill overs between between the treatment monitoring that observations, I mean, might also be interesting.
I don't know if you get some kind of connectivity between the users from Facebook.
Oh, yeah, but I mean, otherwise, you could also like the spatial distance and see whether I mean, maybe the treatment effect declines with distance,
which would be that would be that would be yes, that might be because this is what we're thinking about.
How can you sort of control right? And that would be an optimal that would be an optimal control.
Plus, we do have the geolocation right, so we will we will be able to sort of use that.
OK, Nancy, last question.
So just to go back to the initial part of the presentation about the reproducibility of these data studies, what exactly would your recommendation be?
Obviously, I see that you should, as a researcher, do repetition repetition, try to reproduce your results in different modes.
But how should journals be checking?
Oh, oh, so that's a very that's so journals, in my opinion, are checking for the wrong thing.
Yeah, exactly right. There's I mean, it's I mean, I just met with a bunch of journal letters.
It's ridiculous. So, yes. So the only thing they do is they want a replication file, right?
And they believe they're going to run. And then they sort of basically make sure that you can replicate Table six.
Right? Well, that, in my opinion, is useless. Right.
The problem isn't replication so much as data generation.
Right? Because we don't know. Right. I mean, that was my sort of earlier point.
I mean, you know, we don't know how these increasingly we no longer know how these data generated, right?
Like when I started many decades ago, right data generation was very expensive.
Very, you know, well-respected firms were the ones that did the data generation.
And everybody knew that this was, you know, data generated by, you know, Michigan or Gallup, right?
They knew that there were all these controls and time and associate with the data generation.
Now that doesn't exist, right? It's totally nobody knows. You know you.
You see lots of stuff published in journals now by people who said, you know, I've got a thousand triggers to do this task.
Oh, OK. I mean, there's there's absolutely no control over the data generation.
And that is that, I think is the problem. We're going to have to sort of deal with.
I'm just saying that you're basically making a call. Well,
it shows basically the difference between modes that any reproduction study done in a different mode
showing a different result than the initial result is in itself up to something great to be expected.
Right. So everyone should, by definition, be in the same mode. So any reproduction study should per definition, be at least in the same mode.
Because otherwise, no, I think I mean, I think this table that I think the the results of that barred analysis, right?
I think that's what you should report. Right? Oh yeah. Know, of course. Of course. But but I think for me, that would be much more reassuring.
OK, you do have a there is there is a there is a statistically significant treatment effect that blue line.
Right? And yes, there is some variation across the board, but it's not that, you know, it doesn't detract.
You're simply informing the readers that, you know, there clearly there's some heterogeneity here.
I'm being honest about it, right? I'm not quite sure why it's happening, right?
But that's transparent. That's what you should do, in my opinion. Yeah.
But I'm saying if there is a Mechanical Turk study and I want to do a rapid production of it, then I choose just a single mode.
So in that sense, I would prefer to do multimode, but I'm going to be a single mouse reproduction researcher.
Yeah, then I should at least take the same old. Yes, as a point of departure.
That's a good thing. That was your point. Yes.
I mean, and then if you can't if you can't replicate it within the same mode, then clearly that's a problem.
Yeah. Yeah, yeah. Great.
Yep. All right. Thank you very much.