So welcome everybody to this workshop on Bayesian prediction and pulse certification.
Roberto Chettinad AM, a sociology Ph.D. student at Nuffield College.
And in my third year and today I'm going to be talking to you about this technique, which is going to allow you to do a number of things.
So why would you want to use this technique at all? You have two goals.
One is to do small area estimation, which means if you have a sample that is representative at the national level,
but you like to find out something about the state or the municipality or the some of the small area level,
then this technique will allow you to extrapolate information from the national level and make reasonable inference at the area level.
Something else that this technique does is it allows you to make inference at the
national level for non representative samples and even at the state and the area level.
But as we will see, there is a few caveats there.
If you start trying to use it for the latter and the title, the workshop is Bayesian prediction and post certification.
And the reason why that is is because the Bayesian approach allows you to do simulations, posterior simulations,
and this has big advantages in terms of computing confidence intervals of the
area levels because you can simply simulate if you're working with elections,
for example, you can simulate multiple elections. If you're working with disease, you can simulate multiple rounds of the disease.
And this gives you confidence intervals without actually having to go through the trouble of calculating posterior variances and so on.
You can just use simulations. That's the that's the advantage of it. And so there are three parts to the prediction and plus fortification framework.
So one is sampling, which we will talk about at the end, actually. The other one is the prediction part and the other one is the certification part.
This workshop focuses mainly on the prediction and certification parts and in particular prediction we're going to be doing,
as I said, in a Bayesian way, which means that you guys are going to have to be. So how many of you are familiar with the Bayesian approach?
I mean, if you use the Bayesian approach in the past one two three?
OK, for it, not many of you.
So I think a good idea is for us to go through a Bayesian primer first, because how many of you are familiar with the concept of a Gibb's sampler?
Want to see, OK, so roughly the same people, OK, so there's going to be this is very good, actually, because I planned for a 30 minutes,
30 to 40 minutes of a primer to Bayesian modelling, and then we're going to get into the meat of the prediction and certification.
And this primer to Bayesian modelling is actually going to be very useful to you to understand the basics of machine learning because
machine learning is ultimately even though we don't have the computational power to do it in a purely Bayesian way at the moment.
It is ultimately a Bayesian endeavour in its conceptualisation. So you we will see how this links to Charles workshop later today, I think.
And so let's start with the Bayesian model in Paris.
So first of all, this formula here is based theorem you all night based here is a way to update
your belief about the occurrence of an event after you observe some data.
And so there are three parts to base theorem BRACA Theta is the probability it can be an event so it can be a parameter, any parameter.
So, for example, the probability of voting for a Republican in the population of the United States and P of Theta is our prior on that parameter.
So we may. It's a probability and it's an expression of uncertainty. So we may have a prior on Theta.
We may say OK, well, roughly between forty five and fifty five percent of people are going to vote for a Republican candidate.
So our prior is going to be some sort of uniform distribution between fifty five, forty five and fifty five.
That's a simple Black Friday. And then we have the likelihood, which is this p of Y given Fita.
So to understand that there is a likelihood,
which means it's the probability of occurrence of the data that you observed y given the parameter that we have a prior for Theta.
OK. So and as we will see in a second C that generates Y, this is the the where the Bayesian approach comes in.
So like Theta, the probability the global this like parameter that exists with uncertainty in broadly generates data.
And then as we observe data, we learn things about theta.
So if we observed why say, why is the number of individuals who in a survey vote say they would vote for Trump?
And we observed why in a survey of NW equals 1000, we observe Y equals nine hundred.
This suggests that our prior that was between forty five and fifty five percent is wrong,
or at least like there is a conflict between the data and the prior. And we should take that into account.
OK. And so this graph here I get there illustrates the Bayesian paradigm.
So there are two ways of looking at this graph. First, through the solid lines and then through the dotted lines.
So first, for the solid lines, parameter theta generates the distribution,
the likelihood of Y and out of this distribution of Y, you have observed values y.
So this this is a random quantity. This is a random quantity.
This is an observed quantity. OK, so this is realised. This has happened.
Why I have happened, why I would be, for example, there the the specific response in a survey that says I vote Republican,
whereas y is the distribution of YS across the population.
So if you were to ask the whole population without sampling, how many like what the probability distribution would be for people to express
that preference and C is the probability distribution of the hyper parameter.
So the parameter that generates the YS, does that kind of make sense to everybody, this kind of structure?
So you have a generation of data based on parameters? Yes. So yeah, theta.
Yeah, Theta is something like the parameters that you've defined to generate a distribution
that has uncertainty of what what y is and what is the actual distribution that was?
That's exactly correct. So feed is what we call a hyper parameter.
If this were a normal distribution, it would be the mean, for example, and in the Bayesian paradigm, the mean as uncertainty inherent uncertainty.
So this is where the very difference between the Bayesian and the frequent.
This approach, which is probably what you've been taught through your statistics,
comes in in that the frequent this approach thinks that Fita doesn't have any uncertainty.
It's a parameter that exists and you have to measure it, period.
Whereas in the Beijing paradigm, we think that everything has inherent uncertainty.
And therefore, as you observe data that you just change your distribution around Fita, that's it that you never quite get to a precise theta.
There's a paradox, which is that if you observe all the possible ends and for in order to inform your primary theta,
then the frequent extend the Bayesian approach will converge.
So if you observe every single value out there, you will be able to observe that with zero uncertainty.
And so you can imagine if this was a normal distribution, it would collapse on the mean.
That's the idea. OK, any more questions on? On this.
And so this white star is the predictive distribution, so before we observe any data, why and why stars are going to be quite similar.
Now here's the trick when you then suppose that you haven't had access to the first part of this of this generate data generating process.
This is, by the way, this conceptualisation is called.
The DAG directed a cyclical graph, and it just serves to show that information flow that within a data generating process.
And so if you didn't observe this first part,
but you only observe why and you just hypothesise that a data generating process of this kind would have had to exist in order to create why,
then you can use the information you gained by why to activate your priors on Theta and then generate a new predictive distributions,
which means that if you were to try to predict the vote share of like a new person and plus one person,
then that prediction would be informed by the new data that you've observed. OK, everybody clear on this.
Perfect. OK, so.
So we've spoken about these priors and posteriors, so the the way let me go back to base theorem.
The way in which base theorem is set up. Is that you have this prior,
you have this likelihood and then this updated this updated probability distribution on the hyper parameter, it's called the posterior.
And so the relationship between priors and likelihood determines the posterior.
And there are some regulatory issues in this relationship. So for example, there are sets of priors and and likelihoods that are married, let's say,
or, as we say, in their conjugates, which means that they have an analytical solution.
So for example, I have an example here. A famous one is the better binomial.
Does anybody is anybody familiar with a better distribution? I don't. Yeah, roughly.
Who is not familiar with better distribution or a better distribution is simply a very good.
Please let me know when when I'm not being clear, a better distribution is simply a distribution defined between zero and one.
And based on two parameters, alpha and beta can take any sort of.
It's very flexible, can take almost any shape between zero and one. OK.
So it's a perfect distribution to have a prior on probabilities. So, for instance, the probability of voting for Donald Trump in a given election.
And so thi that can be our prior can be well distributed with two parameters alpha and beta.
And then we observe a poll and this would be the likelihood.
So the the response of respondent y o respondent I am given conditional on the hyper
parameter theta on which we have a prior is distributed to a newly distribution.
How many of you are familiar with their newly distribution? OK, good. Um, and if that is the case, then you can simply you don't have to do any maths.
All you have to do, so you don't have to do any integrations. You not to do anything. All you have to do is update your parameters, as we show here.
I don't have a highlighter, but that's OK.
And so you just update alpha and beta by summing the number of Trump voters to the Alphas and subtracting the number of Trump voters to the beat us.
And then we'll give you your new posteriors. So your new belief over Theta doesn't make sense.
Very good. OK, well, we're so sorry. I'll talk slower.
Sorry. Very good. OK. Yes.
Yeah, yeah, yeah, yeah. That part is the equivalent of when you take the act's observation, which is why yep.
Yep, integrated with your pride. That's correct. Yes. Stuff what it.
Exactly, exactly.
So in reality, in order to compute this, you would need to multiply the probability distributions of your priors and your likelihood,
and that could be quite hefty and you'd have to divide it by the marginal probability of Y.
So that requires usually a lot of integration and a lot of multiplication, and that's quite a problem.
And the cool thing about conjugate priors is that somebody has already done that work for you,
and they know that all you need to do is add to the Alphas the Y and subtract the ways to the beta.
Yes, Chris. They. Perfectly comfortable and updating us will be a part of you as a nation.
So, for instance, in a pool, right? So if I had the probability that if my hyper parameter is the probability,
the general probability in a population of voting for Donald Trump, then I do a representative poll of the US population.
And I observe 30 people in one hundred people.
That is my end. I observe 30 people that say they vote for Donald Trump and 70 did not do not.
And then I would just go ahead and change the values for my distribution of Theda to account for this.
So, for instance, in the beta binomial or beat up or newly this example,
all I would do is subtract from the original value of alpha that I had in mind.
I will subtract the number of Trump respondents and the original value of beta.
I would actually sorry I would add and subtract. Yes.
Let's listen to that. Say it again, sorry.
Of course, so as you can see, so probability distributions have inherent uncertainty,
and the uncertainty around your observed value is going to depend on the sample size of your of your pool.
So if you have a pool that has a very, very, very high sample size and it estimates the true feed quite precisely,
then your posterior is going to be quite precise. Although remember that the Bayesian base theorem is is actually an averaging via variances.
So you have to you have this likelihood and this this prior.
And what happens is if your prior is really, really, really precise, it's not going to move much when you observe new data.
But if you're price quite loose and your data is very precise, your prior will quickly shift towards your data.
So it's in fact in situations where the prior is quite loose.
We call that a non informative prior because it's not going to affect the estimation of feet.
It's almost going to be exactly the same as a maximum likelihood estimation. Does that make sense?
OK, perfect. Yes. So why are? Yes, that's correct, yes.
Does it? Would it work for the U.S.?
So for Beta Beta is a primer on Theda.
So the distribution of wiiii, which is the zero one is going to be a Bernoulli.
And so if you wanted to observe. So if you wanted to observe, say, a continuous value for why that would be, say, a normal distribution,
there isn't a conjugate between a better and a normal, but there is a conjugate between a normal and a normal.
So if you had, for example, height or weight, then those are normally distributed in your prior reformulated,
not in terms of alpha and beta in a beta distribution. But in terms of new and sigma in a normal distribution.
And then you would then update that way. Does it make sense to take the form of it?
Yes. What you do is just because you got that.
Yes, that's right. Any more questions?
OK, great. And so.
Having familiarise yourself with this, well, you might say, OK, but I heard that these Bayesian people, they are always doing stuff with computations.
Why if there is always analytical solutions, which we can sum by hand?
Do people bother doing computations?
And the answer is well, in situations where your prior and your likelihood do not have conjugate form like, for example, famous ones.
Famous examples of this are a logistic distribution or a mixture models.
These don't have any conjugate forms, so you can find a nice combination of prior impulse here to give you an analytical solution.
Then we rely on what is called Monte Carlo Markov Chain methods, and the basic principle is that is that of a Markov chain.
So a Markov chain is just, in fact, I'm not even sure you need to know what a Markov chain is, because I'm going to give you an example of it.
So we're going to look at the Gib sampler. So the Gibb's sampler is a it's an algorithm that allows you to find your posterior
in instances where there is no analytical solution between your prior and posterior.
And the way it does this is as follows. So imagine you have a normal distribution, so you have two hyper parameters Mew and Theta.
The mean and the variance, OK, the variance in Beijing statistics is usually coded as the precision, which is one over.
So one over the variance and and the way the sample works is as follows.
First, you set some arbitrary initial values for you and Theda, you might say, OK, I'm going to start with theta equals one.
Sorry. New in town and I start with Tao equals one and new equals zero.
And then you say, OK, let's sample. So these are now distributions, right?
Remember, so like you have this normal distribution menu has a distribution tab was the distribution and you say,
let's sample from the distribution of new a new number based on the initial value that I put on Theta.
And then let's do the same on Fita with the new value that I just sampled from me.
And then I iterate, so I continue doing this process until I reach some sort of convergence.
So a typical example of this, I have this sort of whiteboard here.
Maybe I can show you this is very experimental, so bear with me should be called the experimental social science.
So, OK. Can you can you see?
Yeah. OK. So say I have. Tao, which is just the precision or one over the variance on the x axis and Mew,
which is the mean parameter of a normal distribution, I would initialise Mew and Theta to be say here.
Some low value from you and some high value for town. OK, then I would sample a value of MEU from from a value from this initial value of Fita.
And then I would do the same for a value of Tao with respect to a value of meat.
And then this will move me somewhere here and then I would continue doing this.
And if the properties of a mark of chain are reached, then eventually.
I'm going to get here where this is the meeting of the joint distribution.
And this is the the variance around it. So this is the uncertainty around it.
So the idea is that if the properties of remarkable chain are respected,
it doesn't matter that you don't have an analytical solution between your prior and your posterior.
What you can do is simply iterate between by sampling iterative sampling.
This called conditional iterative sampling so you, you, you sample from the conditional distribution of mean with the new values of sigma,
and then you re sample the new values of sigma from the conditional distribution of meat.
And so eventually this converges and we have what I showed you here.
So this is like the joint. The joint uncertainty around the two parameters and the centre would be the point estimates for the two parameters.
Is that kind of makes sense. Yeah. OK. Yes.
But this is like exploring the distribution of these two values.
Yes. You're aiming to get the distribution right and this will converge when you visited.
All sort of like the bulk of the. Absolutely. Absolutely.
Yes. And that's why it takes so long when you have, like very few data or whatever, you're trying to explore distribution, which is maybe too wide.
That's exactly right. Yes. Yes.
You might be familiar if you don't have machine learning, because if you go to machine learning before doing the Bayesian stuff,
you might be familiar with, like stochastic search or like a stochastic gradient descent.
So it's a very similar algorithm in that sense. OK, so having learnt what to give, Sampdoria's, you might be asking, well,
when do we stop sampling because like you could just keep sampling all all the time, but you don't know if you've reached convergence.
Convergence is the key here. If you don't reach convergence, then you might have problems with stability.
So it might mean that the next time you run the same model, if you don't have convergence,
you'll get different estimates for your probability, probably upstairs. And that can all happen.
Yes. OK. They that in this thing, you from three the out or.
I'm sensing from. Distribute some.
Mm. So what does that have to do? So to my base, yeah.
So your data comes in in the conditional distribution of the mean.
So in principle, you have a condition the mean, the mean that you observe from from your data, let's say,
has this prior and then the observed data when you sample from the conditional distribution of new,
you sample a new value of new that has to be coherent with your data.
That's the constraint. So there are two constraints here. Do it? Let me go back up, actually.
When you sample the new value of new, it has to be coherent with the current value of sigma and your data.
So this is a coherence problem. Like every time you sample, it has to get more and more coherent with your data and your sigma.
The reason why it doesn't converge sometimes is because your data is too sparse.
And then there is also, as we were saying in front, there are all sorts of places where this algorithm could be.
It could be over here, could be over there, could be over there.
And so there is any and there's an infinite amount or a non tractable amount of combinations
of new and sigma that are coherent with your data because your data so sparse. Does it make sense?
OK. Yes, it's a method in your son about.
But Justice, Mr Baker, you just said a respectable.
But that's correct, so the starting the properties of the Markov chain that matter are that it is only dependent on its previous value,
so quickly forgets the initial values. That's that's a key property.
And then eventually that if you sample enough, it converges to a coherent joint distribution.
Those are the two properties. So it's not that you're not making any assumptions, you're still making your prior assumption.
So that comes in because your prior distribution, this is these conditionals will get you to a posterior,
which is the average by variance of the prior and the likelihood.
But what you're not doing is you're not letting your initial value for the search algorithm to affect the result doesn't make sense.
OK. We're almost through the Bayesian prior I got from the basic the basic primer mission prior.
And. And another way of looking at this blob that I drew on the board here is it's like this,
so say this was just the the mean of a normal distribution and you were looking at its posterior.
Then there is a bunch of. So here we run to change.
So the way we spoke about a mark of change writes like, we run, we run this algorithm twice,
starting from two different points and then we wait until X number of iterations where it's forgotten the initial values,
which in this case takes around two hundred iterations between the two chains.
And then you can see the two chains mix nicely, which is this called mixing, which is which suggests that we are at this point.
Here we are at convergence because what does this mean?
This means that every time I'm sampling from the the previous value, I'm getting stable coefficients.
If we imagine that this is new, the variance here in each chain represents the actual variance of new the mean of the normal distribution.
And so if both chains have agreed on the variance of you, then you have you have reached convergence.
Does that make sense? Yes. When the chains are completely random and yeah, then you, then it's a problem, then it's a problem.
It means that your algorithms and converged.
And as we will see, there is all sorts of tricks that you can use to try to get your algorithm to converge.
But in principle, you cannot use your posterior values to make inference if the chains have not converged.
Potentially is sort of like a disagreement between your data and your priors.
And no, so if there is a disagreement within your data on your power, your algorithm will still converge.
It'll just it'll be like some sort of average in the middle,
and it suggests that you shouldn't reformulate your prior, really, because especially if you don't have much data,
because your your is affecting your data a lot and maybe you don't want that to happen, it's it's not and not informative prior.
It's a very informative prior and we try to stay away from those if you're trying to do predictive things, for example.
Any more questions? Yeah. So you do learn something that is not informative of the article to as a prior, you have two choices.
If you know your stuff very, very well and your data is very, very small,
you might want to pick something that's informative because that's going to go and augment your data.
But the downside is that that's might be biased because if it's your belief about a given parameter,
then you might have all sorts of bias in yourself. So for example, if I were to ask, I don't know, a a no minority person in America,
what's the proportion of police arrests that involve minorities in the United States?
My guess is they probably would underestimate that proportion, whereas if I asked a minority person, they might overestimate it.
And so some somewhere in between would probably be more more sensible.
So but I guess what I'm trying to say is if you know that your prior is going to add value to your prediction,
put it in if you don't choose a non informative prior and allow your data to speak by itself.
That makes sense. Fantastic. OK. And so the question comes, how do we choose when to stop?
Well, we can look at these plots and say and look at mixing. So as we just saw.
But there's also a formula that we can use, and the formula is called the Gilman Reuben Stat. It's this R hat here,
and all it is is a measure of the within chain and between chain variants.
So you need to, as I was explaining before, you need to have reached a sort of agreement between the two chains.
And so if that is the case that within the within chain variants has to be stable and the between
chain variants shouldn't be very large because you want them to be converging on the same point.
Does that make sense?
So usually values of Goldman Rubin's statistics that hard hat that tell you that your algorithms converged are any where below 1.1,
then you can sort of this is a juristic, by the way, it was developed by Andrew Gelman and Don Lemon.
Yeah, but you can pick you should pick values that are below one point one.
There's also some discussion in the literature about one point one being too large and some conflicts with frequent this stat. Once you get there,
but one point one is a good value for you guys to know.
And then the last problem you have is as you sample from from each one of these, you don't just have the the problem of convergence,
you also have the problem of auto correlation because clearly the value where you
are today will affect the value where you are tomorrow every time you sample.
And so in order to get rid of that, so and the way you can estimate how much that auto correlation is affecting
your your your jib sampler is through this thing called effective sample size.
And all that does is it says, well, if you remove if you filter for the auto correlation,
how many truly new data are you getting every time you sample from your from your conditional distribution?
And so if you're in effect, if you think like if you do 100 runs of your jib sampler and you think that you have reached convergence,
your effective sample size should be 100.
But often what you find is that a hundred thousand rounds of your jib sampler and you think you've reached convergence,
but then you look at the effect of sample size and it's like twenty eight, which means that your values are massively correlated.
And the technique you used to get rid of this auto correlation is called thinning,
which means instead of taking all the 1000, you take every fifth or every sixth or every tenth and so on.
And that should remove the auto correlation because if there's a Markov chain,
today's value is only dependent on yesterday's values, not on the days before.
And so if you pick every 10, they're going to be independent. Does it make sense?
Brilliant. OK. So with this small Bayesian primer, I try to give you like four years of studying in like 20 minutes,
but the learning objectives you should be familiar with what based theorem is, you should be you should know what a likelihood that a posterior is.
You should understand that based Bayesian inferential procedure.
So that kind of data generation and then inference and updating that I showed you before,
you should be able to use conjugate priors and know their limitations.
You should be able to understand what a Gibb's sampler is and why you use that when you don't have analytical solution from conjugate priors.
And then you should be able to check whether your sampler is reliable or that it has converged and that it's not auto correlated.
OK. Take a breather. This was Bayesian statistics.
Very good. What's coming next is a lot more reasonable in my mind.
So we're not going to look at this prediction, impose certification part.
So the way I'm going to teach you this is going to be through the example that I'm most
familiar with because that's what I use with my research and that is voting and in particular,
trying to predict the percentage of people who votes for a given candidate from a non representative sample.
And so, for instance, we may be interested in knowing in 2020 if Joe Biden was the candidate for the Democratic Party,
what would be the proportion of people in the United States who would vote for Donald Trump,
either as a whole in the United States or at the state level?
Because that's important because then you can calculate the Electoral College votes and then you can find out who actually wins the election.
OK? The classic decomposition of that problem that I just explained to you is as follows.
So what you want is you want to find out what is the probability, the probability distribution of people who vote for Choice J.
And turnout conditional on a set of characteristics.
So these characteristics could be the gender of the person.
They could be the race of the person, they could be the income of the person, the education and so on.
And what did they turnout people either turnout on Election Day or they stay home?
And so we are going to decompose this problem. This is a joint distribution between vote and turnout,
and we're going to decompose this problem into two parts just by the definition of a joint distribution.
And that's going to be the distribution of voting conditional on turning out and the distribution of turning out.
And we're going to multiply these together. OK. So that's fairly straightforward.
So one way we have to estimate the.
Distribution of turning out is the probability distribution of individuals will turn out is we do a non-representative sample,
say through Digital Trace, we obtain characteristics on individuals through this digital trace and then
we try to use these characteristics to get and we have some training data.
So we know from a survey, let's say one of these people turned out in 2016 or whether they say they will turn out in 2020.
And what we do is just we fit a model to predict turnout propensity for each individual within our non-representative representative sample.
And this model could be anything. It could be a linear regression. It could be a multilevel regression.
As we're going to see, it could be a machine learning method like random forests or a convolutional neural network and so on.
What matters is that you pick the model that given your limited data because your data is non-representative,
will allow you to make the best out of sample predictions. So this is a fairly common theme in machine learning, essentially.
And what do we mean by out of sample predictions?
Well, if we have a small sample say like eight thousand five hundred individuals, but we are interested in, you know,
hundreds of thousands of categories of voters, then we will only learn about those categories of voters from the small sample that we have.
So for example, if we have in our sample white male, the only race that we have in a sample is whites.
But we have whites of like lower education, higher education and middle education.
And then we are asked to make a guess as to the how minorities in the United States are going to vote based on the whites that we observed.
Then the best,
we're going to have to find out an algorithm that allows us to extrapolate from our limited non representative sample into that unobserved category.
Does that make sense? And so one way we have to do this is the multilevel regression.
Why do we choose multilevel regression? We could, as I said, we could have chosen any other algorithm.
And the answer is because it allows us to do this thing called shrinkage. How many of you are familiar with the term shrinkage?
Yes, very good. So it does this thing called shrinkage, which is a form of regularisation.
And what shrinkage does is when you estimate a coefficient, a random effect coefficient, for instance,
instead of the coefficient being exactly what you observed when your data for that particular category.
So for example, if one of our variables is education and we have three, we have four of them.
Let's say like low education, middle lower education, middle upper education and upper education.
Then instead of your estimate for the age, for the lower lower education effect being exactly the mean of the low educated in your sample,
it is going to be an average between the lowest you mean of the low educated in your sample and the global mean of the four categories.
And that average is going to be a variance average. So it's going to be a compromise between the variance between variance of the
four categories and the within variance of the single category that you have.
So for instance, if you observe only one individual with low education,
then your estimate for low education effect is going to be very close to the average of the four or the four effects.
Why?
Because that estimate is very precise in your sample, whereas if you estimate, if you have a lot of a lot, a lot of observations for low education,
but very few for the other three, the effects of the other three are going to be shrunk towards the effect of low education.
That kind of makes sense. So the effect that this has in practise is that it allows you not to fit your data.
And what this means is that when you look at data that you haven't seen before, your data has not been as affected by noise.
So your model has not been as affected by noise, and therefore you'll be able to make better predictions better out of sample predictions.
Very good. OK. Well, you see on the board is a hierarchical specification for a simple multilevel turnout model.
So we say this is a survey or a digital trace, and we've asked people, Are you going to turn out in 2020?
Then that distribution pie is a better distribution of pie.
These are likelihood. Yes or no? Yes. Turnout turnout?
Yes. No. One zero.
And the distribution can be assumed to be a burden, newly distributions are newly allows for a probability of a switch on and a switch off.
And this has hyper parameter Fita.
And then we can just use a logistic regression to estimate the value of either parameter theda for each individual in our sample.
Now you see the subscript g i. That's because each individual in our sample in our mind is part of a group.
And this is going to be very important when you're going to stratify because you're going to find out
that there are so many people of this particular group in a particular state or in your country,
and we will see how we will do that calculation later.
So the group is going to be defined by the specific combination of sex, age, race, education, household income and state in this particular model.
The these are all the all the details are all random effects.
So they are estimated in the way that I have that I have described previously.
The beta are state level predictors. So why would we put state level predictors in the individual level model?
Well, that's because ultimately you're interested in a state level effect and the
state level predictor can be extremely predictive of individual level choices, actually.
So for instance, if one one of the common variables that we put in the state level predictor,
if we're trying to predict turnout is last year's turnout and this ends up being quite helpful to
the out of sample predictions for categories of voters that we have not observed in our sample.
So these are categories of voters that we're interested in,
but we didn't manage to collect because our sample was imperfect or too small and so on and so forth.
What you see here are priors. Yes, Chris. Does that also help you the latest move by the state?
Yes. The one state that they are from the. Yeah.
Yeah, you're right. So the state level predictor helps you kerb some of that in inaccuracy slash
and consistency that your base estimate of the random effect would bring in.
Yes, I think that's that's correct. Yes. Yeah.
This is the standard framework. But then suddenly, Professor Stemple framework would also need to test the.
The state level. Estimates from a similarly known representative sample.
I'm. Well, you could put them in if you have them, you would definitely put them in.
Well. So in the case of voting, what we usually do is so you're saying, for example, if we have a representative,
a representative sample of voters in the Midwest,
would we use the point estimate from that representative sample as part of our state level predictors?
Is that right? That's a good question, I guess.
There is a there is a trade off, yeah, I think you would want to use a compromise between the latest available data and the preciseness of the data.
The cool thing about having last election results is that they are precise, they are the value.
Whereas a sample that is generated via telephone calling, for example, et cetera, has, as we have seen we saw yesterday with the total survey error,
has many, many steps in which it could go wrong and so you might end up introducing noise to your sample.
But in principle, I would say for countries that haven't had election, say, in eight years or something.
So I was thinking of working on a project in Afghanistan, and in that case,
you would pop in survey data because that would be a lot more accurate, you know, demo level.
This is a model to. Go towards some sort of problem.
From the North and the South Pole, the only work was that there was a lot of.
This is a model to go to the best possible prediction we can have for a category.
And then that's it. This is not doing any certification work.
The result of this model is not going to give you the result at the state level. We need to do extra work to do that.
Yes, I mean, the data you do is. Like the people in the face, the more markets.
Yes, that's correct. Yes. Absolutely yes. The specific characteristic of north and central.
It would work on representative samples as well. In fact, what people do sometimes is they pop representative samples in here,
representative samples at the national level in order to find out how categories vote and
then stratify those categories at the state level in order to obtain area estimation.
So the fact that the sample is representative shouldn't stop you from doing multilevel regression.
Yeah, yeah. Yeah, yeah, sorry. If those things are yes, yes.
You're not asking people what they look like. They say, Yep.
You can still expand. All the still set up today are small cells, but still they'll be assembled directly into those.
Yeah, we'll have that for sure. Yes. They're like a PlayStation.
Yeah, same question with a sample for PlayStation. Yeah, yeah.
They're going to use them all the time. Well. You couldn't use the same model you'd want to adjust for that selection effect somehow.
Yes, of course, yes. But this one is tailored to the selection mechanism for this market.
And therefore would not be because it probably led to expose Muller was very much opposed, make just one personalised PlayStation.
Yeah, they severely upvote this person, wrote the model kind.
Yes. So like we did that we saw yesterday. Yes, yes. So that's it.
An orbit in the samples of PlayStation users, then you wouldn't want to up scale down downscaled.
So there was a couple of things there. So eight, the that kind of thing.
So the guy that uses the PlayStation, which we would want to upvote him, I guess you say, right?
Because that would make it more representative of all. Well, then we should have PlayStation and Xbox as part of our of our certification frame.
Yes, that's what you were saying. Yeah, for sure. We should also have.
Some kind of estimate of the population that contains these variables.
That's exactly right. For those four to five, we don't have also.
So that's exactly what Scott was saying yesterday that one of the limitations of his work is that he has to rely on census that gets done in 2011.
And as we're going to see in the next section, which is the certification part,
you do need accurate, accurate cell sizes, as we call them, in the certification frame.
So you can't you can't do this work if you don't have accurate cell sizes.
Yes, that's correct. Yeah, yeah, go.
Uh-Huh. You take the microphone. Sorry, sorry for the live stream.
I mean, I doubt that anybody in America is awake. So. So in the line of the coefficients here what we have female age, race, education, income.
So you're assigning. So this logic is the distribution that you're assigning to each of these coefficients.
So the prior distribution for all is just the same or you're assigning which we're aware of the where the priors.
Yeah, these are the priors. Right. So I try this you use cheap notation to save space.
But the idea is that each EDA has its own variance parameters.
It doesn't make sense. Yeah, I wasn't looking at the definition of end, but the definition of and is down there.
Yep, yeah, that's right. So this is these are the priors on the eda's and they are shrinkage priors because each EDA,
even though it has like five or ten categories within it, like, say, like education, has four categories.
If I remember correctly, they have a single variance parameter, which means that they'll be shrunk towards the mean of the four categories.
Yes. And this way of parameter it.
So I told you that in Bayesian statistics, we use precision, which is one over the variance as the as the specification for the normal distribution.
And this way of parameter rising with a uniform distribution,
a positive uniform distribution large enough on the logic scale is a way of including a non informative prior on the variance.
Does it make sense to people? Yes. Why would we inspire?
Because five of the logic, this on the logic scale is very, very large, so it allows for variance values between zero.
Essentially, yeah, it's not informative, I would say.
Do you have a I'm sure there exists better suggestions. OK.
Yes, sorry. So you also have some information on state level in this case, yeah.
Did this estimate at the higher level have to be in line with the estimates you are looking for things like is there a relation now that you look for
states estimates because you're interested in calculating something at the state level or could have been also gender or something in the state?
So there have been studies that have put in a gender level variables as well for the reason that Chris was mentioning,
which is that introducing the gender level predictor would help kerb some of the bias introduced by the shrinkage.
But they've shown that they haven't thought you could do that.
You could have a state level predictor or gender lower predictor and age lower predictor and so on. But studies have shown that it wasn't.
It's it doesn't increase predictive accuracy of these models. So it would just increase running time, and we don't want to do that essentially.
So basically, you should make sure that these estimates at the higher level are at the level that you're interested in.
That's correct. Yeah. Any more questions?
OK. Good. And so how can we fit this model where it's clear that this doesn't have even though?
Well, it's clear that this doesn't have. I conjugate prior, right?
It's a very complex model, a lot of nonlinearity, et cetera.
So the way we do it is through our Markov chain and in particular we use two programmes are very famous for doing this.
One is Jag's, which is what because I come from an epidemiology background or like a medical statistics background,
Jag's is what I've been trained to use,
which uses two algorithms the Gibbs sampler and then another one called Metropolis Hastings, which you don't need to know for today.
And another one is Stan, which is a more recent development which comes out of Andrew Gettleman's sort of political science background.
But also, it's better. Stan is better.
If you can learn Stan, learn Stan. And the thing about Stan is it is better because it uses this thing called Hamiltonian Monte-Carlo,
which is a way which is somewhat similar to stochastic gradient descent, in that it's a way to create shortcuts between the iterations.
So it's a way to send you closer to the convergence point faster, essentially.
But you don't need to worry too much about that. Just learn one of these two languages.
If you're starting from scratch, I recommend you learn Stan, but Jag's is a very, very good programme.
That's what we're going to see today. And actually some having spoken to some statisticians, they have a sense the Jags, if you have many,
many, many sources of information, Jag's is a better way to aggregate them than than.
The other part of our turnout model of our decomposition that we saw before, which I remember,
we're still trying to find the joint distribution for voting and turning out was the voting distribution and in particular,
so what I'm showing you here is a very simple example of a voting for Republican.
So we just have a dichotomous outcome in a poll. Do you vote for the Democrat?
We do it for the Republicans. Yes or no? And so. This looks almost identical to what you saw before, but there are two differences.
One is that one, in fact one main difference and that is that this distribution of AR is conditional on turnout.
There are many ways to make this distribution conditional on turnout. One way is to just add turnout weights to each observation.
And then. And then obtained a posterior conditional on the turnout weights and other weight,
which is what I prefer and what I'm going to show, what we did in the example.
I'm going to show you next is to actually estimate a different alpha for people who turn out and people who do not turn out.
And the way you do that is by fitting a turn vote choice model.
You simulate who turns out and then every every simulation you fit the turnout model,
the right to vote choice model to people who turnout in that particular simulation and people who don't turn out.
And then you repeat this every time. And the cool thing about this is that it has a quite nice, like intuitive way.
It's like, OK, every time I simulate, I get a bunch of people turn out and then I'm going to use them to make my estimation.
And then also, that incorporates a lot of uncertainty into our model, which is one of the Bayesian approach based advantages.
Because you you incorporate uncertainty.
If you just put the weights in, you're not going to be able to incorporate the uncertainty about those weights.
But why? By sampling people who turn out according to those weights and fitting a model in them,
then your coefficients are going to are going to incorporate the uncertainty derived from people older now and people who do not turn out.
Does that kind of make sense? And so, yeah, and so this is this is it.
You've seen this before? OK, so here we go to our example.
So the example is thanks to the organising committee, we got some money to do a survey online.
We did a survey on Amazon Mechanical Turk. Are you all familiar with what Amazon Mechanical Turk says?
Yes. Well, for those who don't. It's just a platform where you can put any what are called human intelligent tasks, human intelligence tasks.
And so, for instance, if you needed labels for a training set like so you had like cats wore, for example,
so you can feed the Turks a lot of cat wars like we saw yesterday,
and the Turks would have to click on the cutest ones and or you could do something else.
You could ask them to transcribe an audio file or you can do something else.
You can ask them to fill in a survey like we did. And that's kind of cool about the Turks.
Is that the Mechanical Turk? If you're wondering about the name Mechanical Turk, is that this small aside?
Back in, I think it was the eighteen hundreds or something they used to be the circus they used to carry around a chess player with a turbine,
and the chess player became known as the Mechanical Turk because it was a wooden figure that would play chess and beat a lot of people.
And everybody was thinking like, How the [INAUDIBLE] do they? They have robotics or something?
And it's like, No, actually, that was a little person inside a box underneath the chess board that played chess, which which is quite funny.
But that's kind of why we we call them mechanical tricks, because these are tasks that in principle,
we would like to have an AI do, but we don't have the way to do this.
So we given to humans to to do. OK, so a few details about the survey.
So the goal of this? We asked them a bunch of questions.
Some of the questions in all of the questions involved the their voting preferences
in twenty twenty and some of the questions involved their voting preferences. Twenty sixteen.
Why did we do twenty sixteen? Because we had a benchmark.
We know the results from twenty sixteen and I want to show you that through a very known representative sample, which is the Amazon Mechanical Turk.
Because how many selection effects are there amongst people who you select into being a Mechanical Turk, right?
Like, you know, this is a completely non representative sample.
But as I will show you is that from this lone representative sample, we were replicate the 2016 election results almost to the point.
Few details on the survey. We survey a thousand five hundred workers on the 11th of June.
Amazon anthrax takes twenty five percent of the total fee,
which is a bummer because we asked for a specific characteristics about the Mechanical Turk.
So we asked that they were Turks that had, on average, an accuracy of ninety five percent.
We asked that they live in the United States and we asked that they had been approved for at least a thousand human intelligent tasks.
So I just. Why did I do this? I want to screen out bots because bots are a problem in all online platform and in particular on Mechanical Turk.
There have been paper showing that there is survey bots that just randomly click throughout.
Another thing we did to screen our box was we had a CAPTCHA in the survey, so a bot wouldn't have been able to to fulfil the CAPTCHA task.
OK, do we need to do a break, by the way, at some point? What at?
OK, so we have time with time, OK, great. And so it costed us around a thousand dollars and we obtain a thousand five hundred responses.
And. Yeah, that's it. This so now we're going to get into good.
OK. This is what I figured. You know, I'll take this opportunity.
I think Cenk tomorrow is going to speak more about Mechanical Turk and teach you how to gather data from Mechanical Turk.
But I want it to give you sort of an overview of what a screenshot of a human intelligence tasks looks looks like.
And so up here you have the percentage of human intelligent tasks in your batch.
So we had two thousand five hundred human intelligent tasks AI. We wanted our survey to be taken a thousand five hundred times.
And this collective was called to batch a batch of 1500 human intelligent tasks.
And this was the description of the task.
These are the qualifications that we asked for souhaite approval rate greater with a 95 percent location in the United States.
More than sorry, it was five hundred. We asked them that they had been approved for at least five hundred intelligent tasks.
And and this was an example of what the any given Mechanical Turk saw.
So what it was was simply where it says the link will appear here.
Only if you accept the hit, the link to our survey would appear. They would click on it.
They would conclude the survey. And then at the end of the survey, a code a unique code to each individual Turk would be shown.
And then they had to input that code inside so that we could match the responses to the Turks.
Does that make sense? OK, fantastic. Yeah.
Yeah, yep. So.
So that's just a clarification. Yeah. What's his approval rate of greater than or equal to 95?
So it means that I only wanted people whose human intelligence ask approval rate,
so the proportion of approved over non-approved was ninety five percent or above.
So I wanted people who weren't bots because bots would have a very low approval rate because people sorry for you.
I would have gone to approve each and each requester.
So the person who posts a service called the requester, each requester has to approve each individual worker.
That's the thing. Yeah. And you could sort of look at the history of what they've done to sort of accept.
That's right. OK, that makes sense. Brilliant.
OK. And so we conduct this survey.
Now we go to unless you guys can read sideways. OK.
So we conduct the survey. And we ask for a bunch of categories.
We ask for a bunch of information, so we ask for, as I said, more choice and turn out behaviour in 2016.
But also some individual level characteristics.
So like, oh no, sorry, these are not the individual level characteristics.
These are the this is the state.
So this was an example of the question, by the way, so did you vote for the presidential election 2016 presidential ouster?
Yes. No. Can't remember Don. No was not eligible, but we I got emails from people.
They were very happy that I was not eligible in because they had just been naturalised.
And a lot of people only in their surveys only get two, right? No, I didn't vote.
And then they feel bad about it because like, it's almost like they didn't do their duty. But I was happy that they were.
They were happy. And we further asked, Which candidate did you vote for president in 2016 with potential answers.
Donald Trump, Hillary Clinton third party can't remember, don't know.
Did not vote.
Notice that we introduced the third party issue here, whereas in the example of vote choice that I showed you before, there were only two options.
Which means now we're going to have to do some funky stuff of our model to allow for more multiple options.
Funky stuff. I mean, it's like, am I? The from multiple sources, including the census and so on.
We also have some state level characteristics that we're interested in.
So for each state, we have the percentage Hispanic percentage, black percentage, Asian percentage, non-college whites.
That one's important because what's become important in the narrative because of Trump?
Percentage college grad college grads in general, because, as you know, Hillary Clinton was the first Democratic candidate to win.
I don't know about the first, but first in a long time to win college grads in the United States.
Median income of the state. The percentage that the Republican won in 2016.
The percentage that the Libertarian won 2016. The percentage of the Greens and the percentage of people who turned out in 2016.
Now before you ask, yes, we're predicting 2016 state level results with 2016 covariates.
So if perhaps it's cheating? But the reason why I'm showing you this is because eight,
there is no direct circularity because we're predicting we're using these models to inform an individual level model.
And B, these are going to be the variables that are going to be using to predict the 2020 result in your in your exercise.
And so, yes, part of it was just I didn't I didn't think it was correct to go back to 2012 to get these these statistics because.
Given that we've asked the question today, this is going to be a better measure of the reliability of this model being able to to correctly
adjust for bias than a model that would have asked them would have used twenty twelve.
So I think this is the closest thing we get to showing you that if it works in twenty sixteen, it's likely to work in 2020.
I don't think. OK. This is the turnout model, it's almost identical to the one you've seen before.
This is the vote choice model, so as I said, it's a little more funky because now we have to use this categorical distribution,
which is just a multinational distribution with and equals one.
So this time, instead of everybody answering being able to say I'm either a Democrat or a Republican,
they can say I'm a Democrat, a Republican or a third party person or a Anees in this election.
Good point. Missing values and basing statistics. If you have missing values in your outcome variable, that doesn't matter.
You can just feed them in because the the Bayesian model estimates the predictive distributions and then just inputs them afterwards.
So it's an automatic imputation of the output values, which is pretty useful in a lot of scenarios.
And so the categorical distribution just says, well,
instead of having one probability distribution either Democrat or Republican, you now have three probabilities.
You either there's a probability for voting for a Democrat, probably four. There is Republican probability for voting other.
And these have to sum up to one, of course.
And the other funky stuff that's happening is that, as you can see, the effects now all have these indexed new, which is representing the vote choice.
So you can estimate a state effect for each vote choice for a state effect
for the Democrat state effect of the Republicans as they effects the others.
And the other funky stuff is that there is this ID constraint, which is it's a some two zero constraint.
So the idea is that if you have party specific effects and you have multiple parties in a binomial model,
there is already a slim to zero constraint because you're either the effect that is for one party is immediately against the other.
Here we have to introduce that by hand. And so you can just have this some two zero constraint, some two.
You can also be repurposed as the mean. This alpha bar is just the mean of the choices.
Yes. Maybe there a typo here.
But any case, I think I think you get the idea. OK.
And so now there is this this code. So. Yes.
Yes. I need to get. That's the.
That we want all day. We want the effects, yes, we want the effects to some to zero.
That's right. So because if, if, if the gender effect say is positive for the Republicans,
for the Democrats and positive for the Greens, it cannot be that it's also positive for the Republicans.
Does that makes sense? So we're looking at. And what why is that?
Well, first of all, I think that's. Well, because. Interpretability.
Yes, it's difficult to think about the full extent of but.
So I don't know that to right at this point. Yeah, but there's a little bit of zero.
Yeah. Okay, so one of them would be. Yes, that's correct.
Yes, yes it is. Does that make sense? Yeah. OK.
And then you have this model, so you're going to get to work on this on your own later on.
But I'm just going to go through the motions of showing you what, how you fit a Jags model.
So this is a model to replicate 2016 results. So the first thing you do is you create a list.
This is in R, by way. So you create a list with your model data and we introduce the data.
Notice that I introduce the vote choice as ones and zeroes.
And so instead of being a single vector with one two three, it's three columns with ones, ones, ones.
The reason I do that is because there is a fitting a categorical model on Jags is quite expensive in computational time,
and there's a well known equivalency, which is the plus on models.
So you can you can approximate a categorical model with OPOs on model and a
plus some other requires that you have three different choices on the output.
Yes. So this is one of the authors. Yeah, you're making a model and based on.
Yep, yep. The the selected resolution that's based at midnight or 812, etc. We get a different selection rule.
Yep, yep, that works the moment you make a model to estimate.
Thanks, you know? Yep, yep. What is then the virtue of that model?
So the virtual the models that we're going to find out how accurate we can get
to 2016 and then we're going to use the same infrastructure to predict 2020.
Yeah. But then if you predicted it would need to set similar selected under the model, may just the.
It's sorry this to the same 1100 I have asked who they voted for in 2016 and who they're going to vote for in 2020, right?
Yeah. What do you mean you want to create? You want to create different models for different selection rules.
I'm just saying the only reason I predict the context in if you cannot obtain any insights which will generate you,
I would to say that if the coefficient for gender is positive, for instance, yeah, the general consensus sits inside and the system itself.
Yeah, for sure. For sure, yes.
But the idea, but the idea is that through the shrinkage effects, you hope that by bringing your you don't fit the data too closely.
And so you can do better out of sample predictions. Yeah, but the system zeroes.
Well, if the selection effect is super strong and you have an account, so there is this assumption called ignore ability,
which is that I can ignore the remaining variables that I have in introducing the model as part of the residual.
If that assumption is broken as you suggest that through some very heavy selection effect,
then I can't ignore it, so I would need to introduce it in the model and then the certification frame.
So yeah, it seems that the Facebook example works because within cells.
Being imposed on Democrats isn't just a protest where you buy an explosive. Yeah, that's right.
Yeah. Google and Facebook. Yeah. Well, they still have some selection because they end up overestimating, if I remember correctly, the Obama vote.
But but not by much so they did well, but they did a little bit.
It wasn't perfect, right? So therefore, it would seem that that's basically whatever the thing you're asking for the section a little bit.
Yeah. The assumption here is that the selection into the specific time and day in which we asked the survey is
not heavy and that the selection of being an empty worker after you control for all these variables,
here is there is none. Essentially, there's obviously an assumption that's obviously false, but it allows us to make.
And as always, we will see the results of a pretty decent. So it suggests that it's not too far off.
Let's say we can agree to the beginning of the third only relevant and predictive context and not concerns.
Oh, for sure. But I mean, the yeah, this is true of the entire endeavour that I'm teaching you today.
This is a predictive context. Is not an influential tool. Yeah. Anybody else?
No. OK. So we continue we specified these variables and then we specify so Jag's,
the way it works is almost exactly as I showed you with those hierarchical models up there.
So literally, you say predicted distribution distributor of a brand only distribution with parameter pie at the individual level.
We add this to the Oh, this is an interesting thing.
So when people respond to their turnout question, there's a well known phenomenon of overreporting.
And people have found that in the axe. Sorry, the AP and the American election study can remember now an American national election study.
The overreporting factor is roughly thirteen point five percent in pew.
So online surveys there overreporting factor has been found to be 17 percent and above.
So what we do here, we do kind of a rough correction, so we estimate the model and then we say whatever,
whatever distribution you have for each individual, whatever distribution you have thus far estimated, subtract 17 percent to it.
So if the distribution is simply shifting,
if the distribution was imagine like a normal with like a mean of like 70 percent probability, then now a shift shifted 17 points down.
And this actually ends up having a very positive effect on the estimates. Yeah, you should remember if you do vote choice and vote turnout stuff,
you should remember to account for turnout over reporting because really important. The obviously this assumption is not perfect.
In principle, we would have done another model from another information source that would have told us who is more likely to out of
these categories to over report than the report and then created a predictive value for each individual in the sample.
But we didn't do that. We just assumed that people are across the board likely to over report in the same way.
So this is like a uniform over reporting kind of model. Yeah. So in this context, what it's like doing this every time?
No, so it's simply if I answer a thousand people.
And in reality, only five hundred of them voted in my sample.
It's going to be seven hundred and fifty. Just because people lie.
Yes, well, but they lie for all sorts of understandable motives, right?
Ideas like social desirability. There is like they feel like perhaps a little ashamed that they didn't turn up to vote.
All sorts of things. Yeah. OK.
And then the and then the we outline this turnout model.
These are random effects at the state level, region effects, age effect, race effect and so on.
This is the state level predictor. Specified the priors that we specified above.
Again, you'll get a chance to play with this code yourself, so you can you can start. You can start slow and then continue.
So there's some cheeky stuff that is happening here, which I will explain in a minute. You might be wondering what is this auxiliary parameter better?
And what happens is in search algorithms kind of like the jib sampler.
There's this weird phenomenon that if you over parameter rise and then monitor.
So if you if say you're interested in parameter beta and then you specify this weird big model for beta,
which is like beta is actually equal to like alpha times x plus zero and so on and so forth.
Then what you find is that by specifying these sub models and not monitoring them, you're still just monitoring beta.
Your model converges faster. And the reason for this is a mathematical reason.
There's there's been papers showing this is just that.
The search algorithm finds the correct the correct convergence point faster, essentially, Ivan explained that at all.
But like, I'll tell you the paper and you can you can look at it.
So here we like over parameter size are state level predictor by multiplying it for a new variable auxiliary beta,
which is distributed by a normal distribution. But we don't monitor it.
We don't care about it. We care about this beta here, which is the multiplication of beta star and auxiliary variable beta.
OK. And then these are, as I stated before, and these are all the you see a random effect with its random effect distribution
and auxiliary parameter multiplying by the the original random effect.
Yeah, you see a difference. Yes. Massive, massive convergence.
When you take that off. Yeah. Yeah. So the model that I a fit for this is run for seven thousand iterations and now converges almost perfectly.
There's a few parameters that could do tiny, tiny, little bit better.
But before I was using the auxiliary parameter, I needed like fifteen thousand iterations to do it.
So actually, it's very useful to know these tricks. Yeah, very, very useful.
Can these traits? OK.
Choice models, so notice again, we're specifying essentially three different Poisson regressions that are bound together by some two zero constraint.
This the this trick works so long as you introduce a individual level effect that has a uninformative distribution.
This is well known. I have. If you look at the PDF, there is a citation for this and you can go and look at that paper.
And it's quite neat because the pass on distribution is very stable based on sampler story on Jag's, and Stan is very stable.
The categorical sampler is quite unstable, and so this stuff converges a lot faster.
OK, and then so these are the facts, as I said before. Notice that the way we specify the conditionality is by specifying estimating a
different parameter for whether people turn out a pretty good turnout or not.
And so essentially, we have two models here. One is a model for vote choice for people who do not turn out.
One is a model for vote choice for people who do turn out. It's kind of interesting.
I don't bother monitoring the vote choice model for people who do not turn out
because it wasn't part of the of the inference project and we were going to do here.
We predict your project, but it's kind of cool because if you have a if you want to find out which party would benefit more from getting people to
turn out a model of vote choice where people who are predicted not to turn out will tell you that we just kind of need.
OK. And this stuff below is almost exactly the same as before with the some two zero constraint that I described before, and it's close to zero.
Yes, so the plus is this custom distribution is just defined over positive values, but the values we fed it are zeros and ones.
And then we have to put that that random effect that sorry,
the individual level effect in in order to have the random effect of efficiency to be estimated in
exactly the same number that it would be from the categorical distribution to the plus on distribution.
And therefore, this only works if the distance of the way the gross zeros and ones that the you know something.
Yes, you eat. Yeah, you're right. If you're sat, if some of your categories are very small, you run into problems.
Yes. But this is a prediction that the.
Develop like, no, we didn't that we we didn't, because what we are interested in ultimately,
which is the prediction of the 2020 results, 2020 results haven't happened yet.
And so we we're happy with.
Building the model with all the information that we have and then putting out the best possible prediction that we can right now and.
The other reason is that a thousand five hundred people for this kind of exercise is not much.
So you don't have the luxury really to do twenty eighty or seventy 30,
because even removing 30 percent of the individuals from this sample would mean that many categories would actually go empty.
And that would massively decrease the predictive accuracy of your model.
So yeah, you're right in in a ideal world, we would take fifteen thousand people.
Five thousand would leave out fit the model. See how it did in 2016.
Do again three or four times to see whether the model coefficients are stable, then fitted for the whole thing.
But the formal performance? We don't have an out of sample prediction metrics.
Not. But we do have. Yeah.
So for the predictive model, you're right, we don't have a formal prediction mechanism, sorry, a formal like cross validation happening.
But the fact that you do have the the hard validation of whether it works or not with respect to respect to the 2016 results that.
You you need to see results. Yes. Right.
Yeah, it was. I think for sure, you're right. So you're right that we could do better with predictions.
So one thing that we could do is fitted with 2012 data.
But even that is weird because the question that we asked them about their 2016 behaviour, we asked them today.
So they responded to that survey knowing what the average behaviour was as well.
So I mean, it's a weird question. In principle, the ideal case would have been we would have done a serving 20 60,
would have done a survey today and then we would have checked at each point and out of sample metric to test the exact model.
I want to say this is something that the literature doesn't do at all.
So like most models, I think because you don't want to run into the problem of having empty cells most and also because data so scarce again,
like a sample of ten thousand is considered to be massive in this literature,
whereas like in reality for most sample so gellman at all, they don't do an out of sample metric,
but they could have because they had three hundred fifty thousand people. So. So they are.
They could have done that. Yeah, but that's a point. Well taken. I think there some criticism in general about the difficulty level of suspicion.
Yeah. That by adding, right? Yeah. Being just in perspective.
Yeah. You have so much degrees of freedom basically feeding data that are sort of goodness of fit measure within data.
But we are tweeting right away.
So so my question would be, do you have, if you evaluate it, the variance of the random facts and it becomes more like an algorithm?
We didn't evaluate these. Are you saying, did we do a level specific R-Squared?
We didn't do that. No, because. Yeah.
Sorry. You're right, sorry.
So the question has been about predictive accuracy of this model outside of the sample in which the model has been trained on and in particular,
whether we did any cross validation. And so I don't know if I take that point about.
So if you compare, I agree that any model that is tested on the training set, let's say, is going to have overfitting problems.
But I don't take the point that multilevel regression is particularly war of aggression is better at dealing with that than linear regression.
It's worse than dealing with that than a random forest, for instance.
So, so because because the shrinkage, because the shrinkage effect are a form of regularisation.
So they're meant to be there in order to help you help you filter out the noise so that the thing
I'm referring to basically that multilevel effects are basically the same as a random error,
like a normal linear regression, right? Yes, in a way, I rated it like an additional error term.
Yeah. If you would think of it in the classical sense. Yeah.
And therefore, it will be quite strange to find performance of a model by two fixed effects estimates the coefficients plus the residuals, right?
Because then for definition forms is one percent. No, for sure.
No, I get what you're saying. Yes, I get what you're saying.
Yeah, we can discuss we can discuss it more later, but I'm pretty sure your point is well taken, there is no cross-pollination measure here.
Yeah. OK. And so, yeah, we end up we finish specifying the model as and you'll get to play around with this model with the code.
So don't worry if it looks daunting at the moment, we tell Jags which parameters we want to monitor by through this, this thing here.
And we then tell Jags to run four chains for seven thousand iteration burning.
The first 6000 burning is like you remember when I showed you that image of the of the chains and a conversion after the first three hundred,
then burning is the first three. We just throw them away because we know that they are not independent and they're not convergent.
Whereas the posterior sorry, the last one thousand are assumed to have converged.
And so it's like as if we are every sample it is, if we're taking a new value from the joint distribution.
So the effective sample size is a thousand. Essentially, we thought that was very good.
Yeah, true to. So two questions, two answers to that one.
Two is good, but you can never be too sure.
And two is you can run Jags in parallel on air and the way the parallel is personalisation works is that each chain is run in parallel.
So instead of running a single chain with, oh yeah, sorry, we should have this mike thing is a bit hard, sorry.
And the question was why do we run four chains instead of two chains?
And the answer is because you can't be sure enough and b, because of the way you can run chains in parallel on air,
which means that you can run four chains with a foul with two hundred and fifty useful.
So if you think that your model is going to converge after six thousand five hundred iterations,
then you can either run the two chains for if you want a thousand values at the end,
you can run the two chains for another 1000 in each chain, or you can run four chains two hundred and fifty each.
And because you can run it in parallel, running for chains 250 each is faster. That makes sense.
Yes. Yes. We run in for the packaging. Yeah, yeah.
We run in four different genes, but each chain sort of is like a mark of genes, like a distribution.
So where are we from? Which of these genes are we draw in our values?
That's a good question. So what you do is you have these four chain. So the yeah, the question was recorded.
So I'm paranoid of others. So the the the four chains reach convergence after the 6000 iteration, at which point from each chain we remove every four.
So there's a thinning factor of four.
The leaves two hundred and fifty observations for each chain that we stack them up as if they were from all from a single chain.
And then we just use the the stack of 1000 observations because think about it, given that they have converged,
or at least we assume that they have converge, they should be from the same distribution so we can just stack them up, right?
Yeah, that's the idea. Anymore questions.
But. OK. This model takes about four hours to converge, so yeah, it's a bit painful.
Used to take a lot longer. I stressed out about this and but what you're going to play with in class is going to be something a lot simpler, I think.
So we're going to pick three or four categories and we're going to let you
fit a model only for vote choice and only for those three or four categories.
So that should be quite fast. I would expect it to take about 10 to 15 minutes or so and Goldman Reuben statistics is showing here.
So notice that some parameter these these guys here are a bit are having a hard time to converge,
but actually they're very close to one point one, which means that if we run a few more iterations, they probably converge.
Yes. A just quick question on your opinion. Yeah, lately they've been using INLA.
Yeah, yeah. Great. Really, really good. Because of the speed especially is it's super slow, enlace super integrated,
nested le plus approximation is a new Bayesian software which uses integrated nested LA Pass approximation
and assumes that instead of using the Gibb's sampler uses this this technique that I just mentioned.
And but it has the answer to this event as an advantage, which is speed, massive, massive gains and speed.
These advantages are that it's hard to bring in data from multiple sources in LA,
whereas here I can literally like have two models linking to each other, which estimate data from two completely different data sources.
I could have the British election study to inform my turnout model, which is actually what is done,
and then I could have my survey to inform the vote choice model, which is actually what people do all the time.
And you can stack those into the same model, which is super cool, and you can find nonlinear ways to join the two models.
So that's really, really fun that you can do with Jack.
Super flexible as much then as well in LA, they're not quite there yet, so you can if you're like a genius,
like you can figure out the way I like to introduce this new information by priors and penalise complexly priors and so on, so forth.
But at the moment, it's it works more similar to like Glenarm.
So like just like it has like a set of like pretty standard models that you can run with it and also has the disadvantage
that you will never be able to run mixture models on it because they defy the assumption of Gaussian latent field,
which is the underlying assumption of integrating national device approximation. So yeah, but it's a great.
Yeah. Very good.
Nine Laser is a great option, I use it for for a lot of stuff, but not for this because I needed to bring in a lot of nonlinear lesia.
But actually, you could use it for this. We can talk about it later like you.
OK, so we have this model, a converged and now it's time to show we're going to the fact that a converging means that now we have this predictive
machine and we can use this predictive machine to make predictions about individual categories or categories of interest.
How can we? Which means that, you know, we need to set up these categories of interest.
We need to find out how many people from a specific category lie in the state of Texas, for which we want to make the prediction for.
And we take this number from the American Community Survey Micro Data, which is available online.
And at the end of this talk, I will show you how to download it. Yes, just no.
Sorry, I put it as a question. Sorry. And we breakdown the population into the following characteristics.
So we break it down into two categories age six categories race five categories Education
four categories household income three categories and state fifty one categories.
So this amounts to a total of thirty six thousand seven hundred cells,
of which only twenty nine thousand we actually find that are populated in the micro data.
So that means that there are some cells that are so rare that a sample of I think the
micro data is about three point five million Americans don't contain any of them.
So these are very rare cells.
This is kind of a neat plot, which is actually something that from my own research, so on the involved in the bold line, you see.
So this is an ordered. You take the cells, each cell, so each voter category and you order them by the largest cell to the smallest cell,
where the largest is represents the largest proportion of the population in the smallest, the smallest proportion.
So some of these cells in the around the 30000 index, I don't have one person in them, literally.
In fact, on the 3000, there's one person out of three million that belongs to that particular voter category,
whereas some of the cells on top, these have like three hundred a thousand people and so on and so forth.
On the y axis is the cell probability in the population. So if you were to sample at random from the population represented in the microbiota
at the largest category after you cut it up in the way that we describe,
the largest category has about a two in a thousand probability of being sampled.
OK, so these are very small categories. This is very, very defined target stratification.
The dotted line represents the cumulative distribution.
So that means that as you sum the size of these cells, you eventually get to 100 percent of the population in.
The kind of neat thing about it is that even though we we have thirty thousand cells,
the the top the largest 5000 make up about 80 percent of the population,
which means that actually in terms of power is if you start thinking about power dynamics, how powerful should your sample be?
Well, if your sample is powerful enough to capture dynamic for the top eight five thousand
so long as there is not too much heterogeneity in vote choice in the other,
in the remainder 20 20 percent, you actually fine. So that's why some of these,
even though if you did the maths in order for it to have a sample that's powerful
at the 90 percent level and have it work on an MRP approach or prediction,
poor stratification approach you might need, like a hundred thousand people.
The reason why it works so well with smaller samples like ten thousand and one thousand
five hundred is because you only need to get right the top five five thousand cells.
The rest you can kind of ignore so long, obviously like this is not true.
If you happen to live in a federation where each state is completely either a genius from the other,
then you know you cannot do this kind of work there. Yeah.
OK, so this is kind of sorry, this is a kind of like a descriptive of how the certification cells look like,
essentially, if you have to like order them and rank them, et cetera. And then.
This is actually what they looked at. Look like in practise.
So this is the lady on the left.
These are the variables that represent a given category.
So this the top category is. Females of Hispanic origin between the age of forty four and fifty four,
which are college graduates who earn between zero and forty nine thousand fifty thousand dollars, who live in the state of Florida.
And there's two hundred and forty one such people in the micro data sample.
And and if you look at the very bottom, you have these female older race category.
College graduates earn the same amount and there's only one of them in the whole sample.
So these counts, these cell counts are what you're going to use to stratify your predictions from the multilevel regression model.
That makes sense. And we're going to look at how exactly this certification is going to happen.
Before we do that, I want you to have a look at how one representative the anthrax sample actually is.
And if you look at this or on the x axis,
you have the population proportions and on the y axis you have the sample proportions and you can see that our our frame is quite rare.
So our. Sorry.
I may have swapped the labels here hold on the same scale, right? No, but it doesn't matter so much.
Well, in any case, let's look, they're different. That's what matters. That the age is different.
Presidential election is different. Yeah, I think. Sorry, guys.
I think I swapped the labels so. Yet in reality, in our sample over predicted Hillary Clinton, so our sample had more.
We have more. Yes, that's right. Do we have young people here?
In any case, that doesn't matter too much, so I'm a bit fried. But the important thing is that the samples are different.
That's the main that's the main takeaway here. If they were not different, they would be on the same y axis.
So the the point the plots would be the points would be exactly on the y axis.
Yes, that is the amount of the absence.
Sort of is remains, though, that that there are other reasons that were not based on certification variables.
Yes. And normally you would say, OK,
we do a random sample at the certification and therefore anything else that's left which might affect whatever incident is randomly distributed.
Yes. Which obviously, this doesn't help or doesn't do anything with it.
So I'm just wondering why the certification attempt is made.
When this sort of reason why it was set on fire, which is basically what incident sets start up is a random sample.
Anything that we missed, but then yeah, it would be presumably randomly distributed, right?
Yeah.
So therefore, it seems a bit difficult to make the step in between, right, because just go straight for the prediction, for instance, predicting.
This is the inference is obviously invalid, right, because. You know, like an excellent example.
Yeah. Mm hmm. In this case? Well, but it works.
So why OK? I think I get your point. I think it's a matter of how much error you're willing to tolerate.
So like, if you think that your certification result is going to be as good at is going to be of the same error as a random sample.
So there's there's two there's two ideas, right?
You can either do a random sample and just take the point estimate of the random sample and that's it.
Or you can do a non random sample stratify and take the point estimate of the stratification.
Both are going to be wrong. But if you can tolerate the level of error of the of the stratified sample, you can do it a lot cheaper.
And so for the same level of error, you need a lot less money to conduct the the non-representative sample because there's one one follow up on that.
Yeah. Let's say you're a normal person the sample comes from. At Trump rally supporters, yeah, you go there.
Well, yeah, you at some point find a white male female.
Yeah, fine. Yeah, my immigrant, etc.
Yeah, you can. You can wait it what everybody wants. You shouldn't sample dependent variable.
You should never sample. You shouldn't sample anything which is associated with the defence.
All right. Well, yeah, in principle, yes, but but most.
So the residual effects are quite small. And the reason why?
So that you mean like the residual selection effects end up being quite small because and you you know this because these work,
these these stratification effect, certification mechanisms work.
But you're right in principle, the there is an issue with the residual selection effects.
There is an issue. Yes. Yeah. Yeah. It's not just observation if we go up.
Yeah, if we go up to this table, right?
You find that in the last, there's one people we to tolerate that there is indeed no selection effect for that group.
But this is not. This is a sample of the population, not of the Turk.
All right. Yeah. So this is if in the end there was only one person.
Right. It's a problem. But what happens there is that the shrinkage of that comes in.
So if you have only one white person in your sample but you have a thousand five hundred non-white people,
then the effect of the white person will be shrunk towards that of the of the others.
So it's still a problem because maybe there is a strong white effect,
but it's a lot reduced because the effect has shrunk because there's the the white
effect is estimated with very little precision because there's only one person,
right? But if the selection effect is done on, let's say, like a swing states, OK.
So, for example, yes, if you only have one person from West Virginia, yeah, you're stuffed in that case,
yeah, right, because the selection procedure within that within that states is not random.
That's right. So if you were trying to do a prediction on the area level prediction of the state
level and you were lacking individuals who make that state different from others.
So individuals who are particular to the state and you don't have them in your sample,
you're not going to make good predictions in this case, it doesn't matter because you're not going to make a prediction.
So for District Columbia within south region, within. No, no, no.
Right, exactly. But yes, exactly. So that's a key thing here.
Like we are. Stratification means you are average weighted averaging.
Right? And when you average forecasts, you always or, well, there is a rule that like if the forecasts are capturing different levels of noise,
you are always going to get a better forecast in the end. And the weights improve that dramatically.
So this idea that even if if you're averaging over 10000 categories for a given state and five of those categories are kind of like dodgy predicted.
But the important top 1000 are properly predicted.
You're actually going to get a pretty good estimate for that state, even though some of the categories were missed predicted.
That makes sense. Yes. I guess I'm not understanding how this is applied.
Yeah, they're kind of sorted. But then why do you start with 240 in advanced age?
No, the salad is just I took the certification frame as it was, and I assigned it a salad.
Oh, you mean sorry, you mean in reference to the previous court, it is not the same as.
No, sorry. No, I should have made. You're actually right. I should have made this slide the same as the other one.
You're right. Sorry? Apologies. Yes. Chris.
Thanks. It seems to be the lion. This message seems to be quite reliant on having pretty granular data like this for many millions of respondents.
Yup. He's talking about like. I mean, I'm in just the Middle East and North Africa, this just doesn't exist.
Yes, this is, but you can't get the individual level data.
So is this message of the other ways around this or is this just not appropriate for that kind of stuff?
So I once briefly spoke to Jasmine about doing this in Afghanistan,
and she said that the last census in Afghanistan had been conducted like like in the last like big survey she said had been collected.
Doesn't seven. So data are completely outdated.
God knows what has happened in the in the area specific provinces since then.
You have to do the extra work there to actually do, predict the sales counts and then apply the certification to the predicted cell counts.
Yeah, it's unlikely to work, I think in those scenarios, even when the agriculture sector loses.
Yeah. You know, for sure, you could for sure you could you could try.
So but again, like. Yeah, so the question the follow up question was if you aggregate together a lot of surveys,
does this could this potentially replace the lack of census data?
The answer is yes, to the extent that the surveys are reliable.
So like if you have a show in India, when I worked on the Indian election, for example,
we use the India Household Development Survey because the census data was limited in their crosstabs.
So again, another big problem is that you have to use micro data, usually because there's loads of crosstabs that you're interested, right?
And often census crosstabs are very limited to like two or three interactions.
And most so yeah, service can be a solution, but usually they have to be augmented by via some modelling.
Yeah, that's that's what I would say. You. Yeah.
So what do you do? So what do you do when, for example, let's say.
For example, like when you have a survey and the combination of these stratification variables leads to what you were saying,
something like two thousand cells being empty.
Yeah, in terms of calculating the true proportion from the census to be used with the much smaller data.
So what do you do you just exclude them, would you? Yeah, you said in your part.
So you have if that situation arises, there's two things that can happen.
One is they those guys lie here in which case, like as long as those exact guys form the make up of a specific state that you're interested in.
Usually this isn't the case, by the way.
Usually these, like smaller categories, are kind of like spread out at random across states, so it doesn't really matter.
But if the so in the in the positive scenario, those guys that you don't have in your census samples are here and you ignored them.
It's fine to ignore them in the negative scenario. They're here and then you're absolutely screwed.
There's no way to get around that. Yeah. OK.
And so. So, yeah, sample very unrepresentative invo choice over it, so just going by memory because I don't know if I can trust these plots,
but over represents Hillary Clinton quite dramatically under represents Donald Trump for a fair bit.
And it over represents third parties. So after we have estimated these quantities, P of G.
So the joint distribution of turn out and vote and then the conditional distribution of turnout,
we can then estimate the area level proportion of people who will vote for a given party and turnout with this formula here.
So huge is just the number of people in a specific cell.
PG is the joint distribution of voting and turning out in a specific cell and then you some over the categories up to s.
Again, I should have been a little more tension here,
but you some over the categories for the specific state and then you recall the the the area level estimate for each party.
And then you can do so not just at the party level. So this look look here.
This is conditional on the voter characteristics and the specific party.
Right. Whereas here we don't. The specific state sorry.
Whereas here we don't bother about the state because we are interested in aggregating at the national level and so
we can just some over the groups by their weight and obtain the national level results for each voter category.
Does that make sense for everybody? Yeah. This is just simple certification weighted average.
That's all. And these are the results.
So at the bottom, you see the role. Amazon and Turk estimates, and at the top you see the predicted estimates by state.
So there's a few things to note. So first of all, correlations improved dramatically. Mean, absolute error is shaved off by on average.
So 10 points from Trump, five points from Hillary, five points from the third parties turn now to shaved off by like thirty two points.
So we massively improve on the rural sample, and there is clearly this phenomenon called attenuation bias happening.
So even to the correlation is really high. The variance in the predicted state level vote shares is very small,
which means that what you're going to get is the correct order of the states, but very much shrunk towards the global mean.
And that's actually that's an effect of the shrinkage and of the of the general trend to shrink effects towards the mean.
And what else should you notice? No, I think that's it.
I think we can. So I think overall, I think we can be pretty happy with the results,
given that at the state level at least we have shaved off a lot of error from the role of sample estimates.
Now the really cool stuff comes in when you look at the national level. And so if we look at the these are the distributions of Republicans,
Democrats and third parties, the solid lines represent the predictions, the small dotted,
the thick dotted lines represent the twenty sixteen actual result,
and the small dots represent the role prediction of the the real national level prediction of the Amazon thing.
And if we look at the errors, so our error on Trump's vote share is less than a single percentage point at the national level.
Zero point five Our error on Clinton is quite large.
It's about four percentage points, so we overestimate Clinton by four percentage points.
Our error on others is we underestimate them by about four percentage points.
So what is happening there actually is we must have missed some selection effect that
would have told us about the substitution effect between Clinton and third parties. So we have to go back and think about what that was all about.
But that's what's happening here is that this is a state level result.
No, this model is is aggregating at the national level. There's also the question was, does this model use the state level result?
The answer is no. It just the aggregated aggregate using state a as another group and aggregating over as you would otherwise.
Yeah. I mean, I'm pretty chuffed that that Trump vote point like half a percentage point from the truth, the turnout.
Again, we overestimate turnout. Still, even though we had applied that uniform correction of 17 percentage points.
But that's not news. People lie a lot about their turnout and the survey data about turnout that we get from EM talks or otherwise.
It's pretty crap.
So in our one thousand five hundred sample, about 280 people said they didn't turn out, which makes for a very small proportions of people.
And therefore probably we missed a lot of cases. We, a lot of groups were predicted to vote at higher levels.
We actually do. And on the right, there's just an Electoral College projection.
So assuming there was a lot of stuff going on with the Electoral College these days,
so like a lot of states have signed to this pledge that they will actually give
the Electoral College votes to the person who wins the national level result.
But I ignored that. I just assumed that each state is going to give the Electoral College votes as they usually do.
And so we would have assigned with this model a probability of Trump winning of about twenty three percent, which is not great.
But it's also not terrible, considering that Hillary's vote share was initially overestimated.
So Trump's much was initially underestimated by 10 percentage points in the Amazon sample.
Yes. Do you know how that compares? Predictions that other other people.
Yes, yes, yes. So the question was, do I know how this prediction?
Well, this is not a prediction, by the way. So it's kind of an unfair comparison.
But how this estimate compares to some of the pre-election predictions.
And the answer is it's a little better. But for all, apart from FiveThirtyEight, so FiveThirtyEight, if I remember correctly,
had a point two seven percent Trump winning Wong and the Princeton Election Consortium
had a zero percent Trump winning Drew Linzer had a one percent Trump winning.
So I mean, you know, from survey data, given you can't really compare because this poll stock and theirs was pretty.
But given that the survey data has the same chance of being bad this time around as it did last time around.
This should be encouraging because it suggests that if you do enough modelling, you can get better results than your average predictor.
Yeah. Yeah. Yeah. Hold on. Mike, Mike, Mike, Mike, listen.
Sorry. So just to go back one more time to the virtue of the model, because if you give me the data of the actual results,
yeah, I can make a model which probably predicts better rights.
No, but but the results are not coming in at the state level, right?
They're coming in at the individual level. But if you give me all the same data. I could probably do pretty well, right?
Look at the passport effects because we asked him who he voted for. Yes, used to the state level results.
Yeah, initial model. Yeah. So. It kind of seems that maybe a bit of a work around growth this we, as I said to you before,
you could have used it instead of twenty sixteen, you could have used the Republican 2012 results.
Yeah, but so there was a number of conceptual reasons why I still think that's wrong because again,
we asked the survey today, not before the election.
So this is not a prediction exercise. It's a replication exercise.
So, but in principle, if you were to do a prediction exercise, as you will do at the workshop,
you are going to be using the 2016 results for predicting Biden's or whoever else.
But like, I agree that this is not a prediction effort. This is a re estimation replication effort for sure.
OK, yeah, but but understand this, I make this have this point come across.
Understand that at this level, the state level results from last year are not playing any role.
They played the role at the individual level model insofar as they were correlated with individual level responses.
And actually, the chances that the 2016 results would have probably been more correlated with
individual level responses today about the 2016 behaviour than 2012 responses.
But it would have been that far off because state level behaviour is actually
pretty stable across time and the correlations across states is pretty stable.
So obviously we we should have read that at using 2012 and seeing what would have happened.
But you shouldn't think that it's it would have been that far off because their results at the state level weren't that different from 2012 to 2016.
But I'm just saying that's just a hypothetical model.
I would do a very simple set of data, namely say I just take one person out of every state's assigned this person.
The state level results that 0.20 52.
If it was 52 percent, then I just exploded by the number of people living in states.
And then I get perfect results, right? Yeah, but that's not what we are doing right now.
I mean, use at an individual level. Yeah. And then you exploded, right?
So in principle, this bits. Huge individual in.
So you said you say this. Well, if yes, so if you assign the state level is perfectly.
I agree with you, but we're not assigning the state level perfectly.
We're assigning the correlation between the state level result and the individual response.
But but but then again, like the virtual verge perspective. Yeah, also from our application perspective, you could probably do better.
Easier, right? Yeah. Yes. Yes.
Yes. Yes. No. Of course you could. Of course, the fact that there's 2016 results leads to a better estimate than 2012 results 100 percent.
I don't dismiss the certification for. Yes. Yes. We're trying to show that certification matters here.
Yeah. I know the questions.
OK. And so.
We're almost there, guys. Yeah, there's a few considerations here.
So one is that we would have done a lot better as well if we would have improved the sample and we certainly would have increased the sample size.
Sample size is actually a very contentious issue at the moment in the sort of literature.
So one way to call this model, by the way, prediction and certification is also MRP multilevel regression and certification.
And some people have suggested that. So for nationally, for estimating area level result from nationally representative surveys,
MRP does a lot better with higher sample sizes, so error reduces dramatically.
If you go from a thousand people in your sample to ten thousand people for Amazon Enteric specifically,
so for non-representative samples of this kind. Some people goil at all suggested that the name of the paper is surveys are not representative.
Surveys are cheap and reliable, or are they reliable or something like that?
But they suggest that actually there is a decreasing returns on these kind of samples because of all their set.
If you look at the total survey error, some other errors start overcrowding the sampling error.
And so even if you had 10000 people, you wouldn't do much better than you did here.
But the point is, people don't know. And as part of your one potential six project, if you'd like to study it, you could come talk to me.
Sample size for non-representative surveys Impulse fortification context is very important,
and people don't really know what the [INAUDIBLE] is going on. So that's one of the things. And yeah, that's it.
So we can take a 20 minute yet 10, 15, 20 minute you guys decide break and then we'll come to the workshop.
Yes. Mike, Mike, Mike.
OK, maybe a dummy question after the end of all of this, but no other way.
So if I understand correctly, this whole exercise is to estimate a proportion.
That's correct. So I'm not just the proportion, by the way. You can also not you can also do it with like height weight.
You can do a BMI. You can do it with like any quantity that you want to be representative at the national level,
but you only have known representative estimates for what it is to estimate a single like parameter.
Yeah. Could you also use this same way of having a non representative sample if you wanted to estimate relationships between variables?
Maybe this is really a silly question, but you know what I mean? So if my aim is not to estimate who?
Yeah, the share of people who vote, but let's say I want to know if more educated people are more likely to have children outside of marriage.
And then I do like such a quick and cheap, non-representative survey.
Do I then would I then be also able to infer?
This kind of relationship, I think that's that's an intriguing question, can we apply a pulse stratification to coefficients, let's say?
I need to think about that often, like off the top of my head.
I don't see. Maybe you need to do some extra work, but I don't see why you shouldn't.
Oh, sorry, I don't see why you couldn't. Maybe there's all sorts of reasons why you shouldn't. But yeah, I can think we can.
But this is not something that is I haven't seen in the literature yet,
but also because I come I come at it from a perspective where at the moment it's being used for like, well, political science,
like predicting who votes, but also disease mapping, like predicting like percentage of influenza cases in different states,
etc. So at the moment, it's mostly been proportions, as you say.
Okay, thanks. You have no knowledge. As you well know, rich people are more likely to.
Yes, things like that. So should should.
We can we can have a conversation about it afterwards. Thank you.
Yeah, I think it's a it's a very good question. Think about.
The existing literature on trend of life or stratification has really been in forecasting or elections or things like,
you know, just trying to replicate survey questions,
I mean, the general concept here is to go from a messy, dirty sample doing to a more reliable sample by essentially waiting, right?
This is just a weeding exercise, so it could actually be an interesting question to try.
Maybe it's a group project or something else to think about, maybe doing a survey with questions that you might be interested in, right?
For, for example, taking a survey sociodemographic survey, doing it on a non-representative sample, using a platform such as Mechanical Turk.
Right? And then finding a relationship that you would be interested in.
Essentially a relationship of some kind of correlation that you will is well documented and then seeing to what
extent that works in the non-representative sample and then compare it with something that you trust more,
which is, of course, much more expensive. So taking a correlation for something like the British Household Panel study or understanding society,
doing it on a on a non representative sample and seeing if doing the rerating allows you to get within the bounds of that.
Because of course, the big sell here is that you have a thousand person survey, which is being done for a thousand dollars, which is very cheap.
And so it's it's potentially scalable if it works like that.
But so I think it's an interesting question that could even be worth lowering as a as a project.
So, yeah, I was recently looking at a deer, this survey, literature and these calibration and ranking algorithm, but I don't often see the.
Living with bears and statistics tell me a bit more.
It doesn't seem he did the survey literature calibration issues here don't seem to be using the division was the lead there.
So it's so the MRP studies that you let me be precise.
You don't need Bayesian to do this sort of stuff. You can do it with frequent.
There's absolutely no problem like feel free to use your instead of like running the model through Jag's, feel free to use LMM and put in the model.
That's number one. The advantage of Bayesian is that because you obtain simulations for your posterior, you can.
You don't need to rely on sampling assumptions in order to calculate a specific predictive distribution.
You get the predicted distribution directly from the model. You don't need to do any extra work.
And so for maybe it's being lazy, believe it.
For me, having these simulations really helps conceptualise the problem.
And also, yeah, that all would be different because it's not completely dead, but it's quite linked.
Mm-Hmm. We use, for instance, bootstrapping in France to derive calibration weights.
Yeah, how that would be different from a Bayesian simulation.
So that's a method of deriving the weights. Yeah, but also.
And uncertainty, yeah, and uncertainty around the waits, so that is not fair.
So, so I'm not like overly familiar with that specific method, but for my general understanding of bootstrapping and that kind of thing.
So you would obtain you would obtain some measure of uncertainty.
I think that the assumptions that lie behind the bootstrapping are different from the ones that come with the Bayesian models that I showed you here.
I am very familiar, like the assumptions behind the Bayesian models are very simple.
You can derive them from your four probability, five probably axioms.
So you don't need to do any extra work to specify new assumptions beyond what is a probability.
And so that, to me, is a very intuitive thing, but that doesn't mean you can't do it.
So feel free to replicate these exercises by doing the using the bootstrapping.
The bottom line is they both get. The important thing for me is that we put it in our minds that you have to have uncertainty estimates,
you have to have uncertainty estimates and like a lot of the applications often don't.
And it's a problem because then you can't do calculations like the probability of Trump winning, you know, like we saw here.
Like this point to three comes because out of a thousand simulations, Trump wins twenty three to two hundred thirty times.
But if we didn't have uncertainty over the state level estimates, you couldn't do that calculation.
So it's really important that you do have the uncertainty. But you're right. You're right. You can just do it with other ways.
There are other ways of predicting weights and finding distributions around the weights.
If that's all, maybe we'll take a break. Yeah. Thank you.