OK, you heard this. Today's the second edition of the competition and Statistics and Machine Learning Seminary
in this Trinity term 2020 twenty first and today we're very happy to have I keep a diary.
She will be speaking about practical, pre asymptotic diagnostic of Monte-Carlo estimating based on inference and machine learning.
And I just wanted to tell you more about the Who is AKI if you don't know him.
Aki in that he's an associate professor in Computational Probabilistic Modelling in Aalto University.
His research interests are based on probability theory and methodology based on thorough work,
providing realistic programming in fair methods such as patient propagation,
operational base model diagnostics, model assessment and selection processes, and hierarchical models.
He has published a very, many very relevant papers, both in the medical aspect transparency and also in applications.
And also, he is an author of Of Course, I Hope You Like a very interesting book books.
One of the things that I come here to be, which is a classic already and also which is more recent.
The Radiation and other studies, which was published last year and is also very interesting because it addresses rigour applied,
told and said very carefully using the Baoshan framework. So given this introduction, you're going to start now.
Thanks. Okay. Thanks for the invitation and thanks for the introduction.
I'm very happy to talk with you on that kind of theme in your groups.
You are doing a lot of interesting work. Also, in addition to the university, I wanted to advertise the Initiative for Artificial Intelligence,
which is a group of A.I. machine learning, probabilistic modelling researchers in Finland.
Like, we have probably like maybe 100 people altogether, and we are advertising doctoral student and postdoc positions.
So, yeah, I'm also part of an probabilistic programming framework development team and Orbis Diagnostics,
but say some back, it's been quite a developer team.
And obviously, this work, I talk this when you call our tourist listed here, and then in the end, I'll give us some more complete references.
I. Kind of put this talk in a context that I'm interested in all parts of Bayesian workflow.
And then this is figures from a recent archived paper on Based On Workflow.
And then we today then talk more about the kind of the parts that are related or useful in here in convergence diagnostics,
cross validation and and in some other parts.
But kind of that. I'm interested in all part of this, but this talk is now focussed on one piece.
So I will briefly remind you about the estimates you've got on sampling and central limitada, which is used to justify these often.
And then I go to a concrete example.
We were not cross validation, which was the concrete reason I started to think more about this pre-symptomatic behaviour and diagnostics for that.
And then that's the main part.
And then three other examples of using it in validating and improving the end result of various imprints, diagnosing the stochastic optimisation.
And then last remaining again that it can be tough to use.
Plain Monte Carlo estimates also show in Monte Carlo and Marco chain Monte Carlo.
We get. Some drugs and Keita are exempt from this, and Pete, it could be then,
like most true distribution for these parameters and we get drugs from the distribution.
We may have some function aids and then we want to know expectations of the functioning of the AIDS data integrated over that distribution plan.
We can do that with just empirical average. And so this could be just like identity and we get like poster.
We mean it could be squaring, which is needed for variance or indicator of function.
So we get probabilities and so on which are made it in like Bayesian interest.
And then this is consistent, unbiased in case of emergency, also particularly.
And then this is often justified, which central, libertarian, which then some fine experience variance of this is the expectation estimate goes down,
which respected and so divided by the sample size, implied a sampling instead of getting to cross from directly from the target distribution directly.
We instead have some other way it's easier to get and we still can't compute this expectations estimate.
We just need to now wait them. And these weights are just using the ratio of the target and the purpose of distribution.
Again, consistent, unbiased. And if if it's the case that one or two of these distributions that we have only are normalised,
so we don't know the normalisation of constants we can do self normalised.
So we just divide by the sum of these points consistent and by a decrease with the sample size.
And these are quantities, consistency, unbiased, mixed or small bias, centrally midterm have been used to justify that.
OK, now we are using these. I need your input on something.
If this. That kind of data funk, some times the weights and the weights, if this is the patient's plan at variance,
then the estimate has been at variance, which implies a kind of centrally meeting room.
And then now, instead of directly just dividing by X.
Variance in the weights decreases the performance, and then there's this effective sample size estimate.
In this case, I show you just this approximation that ignores that function.
They are also specific versions. And so we get the effective sample size just by one per sum of squared normalised weights.
So the weights some two 110. And this is actually then related to just do the variance of these normalised weights.
And then this is going to be smaller than this total sample size.
And again. Began the useful and insulting the there's also additional useful aspect is that if it happens that there would be infinite variance,
we still would have a finite bin. And then with the infinite variance these estimates, there are distribution converts to generalise.
So the general is generally material called and we would converge towards this global distribution.
It's useful. Tomorrow, later in this talk, so often it's kind of given granted that we have finite and son central and meteor and.
Sometimes we can do it so that we can.
Guaranteed by night marines, by constructions, and, for example, in case of input on sampling,
if you choose to do so that the the weight issue rates are bound it and we know that we have also experienced some central interim holes,
but it's not generally trivial. And I will also show that this pre-symptomatic behaviour can be quite different than what Trent Jolly mid-term says.
I don't know. I called this example, which started my path to thinking about these diagnostics.
We will not cross validation. Yes. True data generating mechanism is.
Linear function y depends on x linearly plus less than just normally distributed variation around the mean.
This one data really say some from that process and its linear model there that supposed to mean.
And then because it's a finite realisation, we have uncertainty which can be then represented with post triggered growth.
So I'm here. We can compute predictive distribution.
And then, you know, we are just.
This is the predictive distribution that if we would know the parameters and since we don't know parameters,
this reflects the uncertainty and then we integrate with that uncertainty into parameters to get the predictive distribution,
and then we would like to know how could these predictive distributions are for new data?
But if we don't have the luxury of waiting new data on testing, we can instead then choose.
And observation and remove that and then fit the bill again.
And now the Green Line is then showing the what would be the fit without that observation.
And we can also then computed leaving out predictive.
Distribution. So now, instead of using that full data plus three or we are using is live on outpost area where these two know that
we have left out the 18th observation and you can see then that this predictive distribution is different.
But now we can also then use that left out observation as new kind of proxy for new future data and the importance why we need to do this.
And of course, one indication is that if we were to evaluate the predictive performance just by evaluating.
Like the predictive density, given that 18th observation, but conditioning on all the data, we get some predictive density.
But usually these leave are not predictive densities often lower because it's more difficult to predict something that you did not see.
And then we can make the summary so that we repeat this process for all in this case, 20 observations computer is predicted and took place.
And it's actually useful to take a logarithm and some or advocates of these values,
and this would be the summary of how could we think that these trees or predicted future data
on this is almost an unbiased estimate of local street predictive density for new data.
And then this can be used in model comparison models that it selects.
And then, of course, we can compute other things using other utilities on cost functions than just plug score.
And again, reminder if we would not use the easily one out predictive distributions, we would get higher value like here.
The difference is a tree,
which happens to be close to this number of parameters in the model intercept slope and the standard deviation of the residual parameters,
which reflects that given in these three parameters, we wear kind of fitting to the specific data.
And then this is and then then this is a biased estimate for the performance for the future.
And then we say it's almost unbiased. And then there are these papers discussing specifically more about this level, of course.
But I guess one other issue is that we would need to compute here in this case, 20 times.
These are not posterior. If we did use NCMEC for the full posterior.
And then we would do 20 times again this CMC for more complex models that can take a considerable amount of time.
So. Now, if he already did.
Sample from the full posteriori, and we already have these drawers.
We can use this as a proposal distribution. And this is the target distribution and or in this case, we have been as many as we have no observation.
And then we can use import on sampling important results, so we can't be.
Leave on our posterior are divided by the filibuster, and if you think about the in the cases where we have factor rising likelihood.
This story has one likelihood 10 more. And this one.
And then if we ignore normally say, some terms than this, in contrast, so it's proportional to just one per one likelihood term.
And there is no indexing this place across from the full poster,
and then we can normalise this because we don't know now this normalisation constant and
use so normalised in product sampling is again the same data on the pool posterior.
And then for the full story, or we would have just the empirical average to get these predictive distribution.
But now if we choose to one observation and leave it out.
Now, notice the difference. I only see I have exactly the same.
Lines in this speaker and a previous speaker, I only change that the Alpha channel has been.
Chosen based on these weights, so if the weights is close to zero, Alpha Channel is zero and you can't see it,
and if the weight is larger than golf, it's on a list closer to one.
And so this is the only difference. Only way that the draws and these draws are now, then better reflects of the leaving out posterior.
Of course, if the lever not pull through would be very, very different and we would not have that kind of cross at all, we would.
Feiglin will this diagnostics I talk about is related to this.
And so now we can then compute this lever, not predictive density of the left up data.
So predictive density, but not the parameter values and then S.A.C. across from the pool plus three or anything,
we just wait them using the normalised weights.
So easy, but then the question is how reliable now this estimate is, is the very finite state centrally mid-term kicking in.
There is no general analytic solution or specifically for the Lebanon, of course, will it based on the artificial results or normal linear models?
And one specific other type of model, but not there's no durable solution.
And it's also useful that the Typekit can take a test on the proposal because when we leave out one observation, there's more uncertainty.
And then it's also likely that the base weights are not bounded.
So what we can do is we can look. The empirically the distribution of the weights and unofficially now to 400 trials,
so that it's a bit easier to see these largest weights in this is to here.
And so even the largest weight, it's not close to one.
So these are not these normalised weights on this.
So yeah, they live there.
And also this would be the what would this size of the weights if they would be all equal and we have a lot of weights around there.
So it seems like that this is not not that bad case.
This is four thousand troops and now it's difficult to see, but this the largest weight and now I can't even see if I have this really big.
I can see that there are also some way out there, other weights.
I mentioned the sample size estimation. Yes. So here we get.
We had total sample size like four thousand and the sample size based on this variance approximation say around 400 400.
It is not that kind of percentage efficiency attitude.
But then the question is that is the variance finite? Can we trust these effective sample size estimate?
And of course, with with any finite sample size, we will look at also, the variance estimates are also finite.
So directly looking at the variance estimate doesn't tell whether the variance is finite.
So what we can then do is use something else to estimate.
Whether they're variances finite. So what we do is that we fit in, analysed broader distribution to the largest resource.
So since so we can ignore the public and we choose some cut point.
And then from that cut point, the journalist part of the distribution is such that it has some so cut point there and then it's decreasing,
it has some scale parameter and then it has to say parameter that affects then I'll take the tennis.
And we fight it today, the army did the last traces and this theory that saying that if we choose this plot point far enough in the tail.
Then that's a very large set of distributions that these tail parties will approximate it with this already distribution.
And the nice thing about the parade of distribution is that so that it has this
scale parameter and say parameter and the safe parameter case substance now.
Floor of one per tells the number of finite moments. And then we have finite variants and so probably maternal health.
Can I ask a question? Yes. So this sort of extreme value theory usually needs for eye observations.
But if you run a mathematician, then you have a Markov chain.
You don't have I.D. observation. So is it?
They don't. Yeah, yeah. So so the usually the spectrum value, Terry.
So for the better data,
they would be the issue if we would be asking a question that what is the probability that we would observe weight which is larger than something?
But just for estimating what is that safe barometer?
That's not the issue. So it's just that in a way that we would need to take the dependency only into account if we would say something about the like,
how likely it is that we would see even extreme rates. OK.
Thank you. But now so so we don't know.
But we are not stating anything about the kind of how likely we would see more extreme waits.
We just want to know the safe parameter.
And then if this parameter would be less than half variance is finite, limit their impulse case less than one do not last time to limit wormholes.
And so in this case, we estimate the case had to be zero point five two.
So it would say that, OK, now central troubling me term doesn't help, but we will soon see that kind of this.
Whether Kate is less than half or about half, there's no, you know, behaviour, there's no sharp dress hold.
Before going to this diagnostic port, I'll add also because this diagnostic examples,
they saw results based on specifically on part of this, would it import on sampling?
So in addition to using these are actual distribution or diagnostics,
we can also use it to replace the largest weights with ordered statistics of defeated Baroda distribution.
And this is equivalent to using water to filter most noise out of which is an illustration.
It blue line is from one repetition of symbolism in one simulation,
we draw 10000 draws from some proposal distribution of some specific proposal distribution.
And this specific target distribution.
We rank the weights, so we sort of weights to increase in order, and you can see so basically it's the largest 10th largest, 100 largest weight.
And then in each simulation you can see that there's a not of worrisome. In the latest wait.
This approach also truncated important sampling, which was truncated a the of weights and, of course, and reduces variability.
But now you can see that instead of having this jumping lines, these are smooth lines because we replaced.
This kind of object. Ratios, which would at ratios based on creating this broader distribution.
And it then reduces the variability compared to plain import on sampling and reduces bias complexity truncating input on something.
And yes, an example, then again, 100 simulations,
but now with the increasing SE estimating the normal system constant and in this case, we are integrating where the case.
Like close to zero point five and then regular input on sampling has these jumps.
And so it is an unreliable even if we get more observations that it can sometimes plot truncated impulse input on something doesn't have that issue.
But you can see here it's getting below the truth about the use of this by us.
And then I just put it in what I'm sampling. Hospitals reduce variability and bias.
We also prove in the paper, thanks to Don Simpson, that this protest would have been put on some polling estimates,
asymptotically contestant on his final appearance under some mild but complex conditions.
Cool result also is. That we can connect this case to minimum sample size that gives some feel guarantees for the error.
In self normalising import on sampling and. Simple equation, which looked like this,
so let's say that we have the case where case small and zero corresponds to having and the distribution
of weights being exponential and less than two year old would correspond to having bounded weights.
In that case, let's say we have 100 drawers.
If we were to have independent draws directly from the target distribution 100 drawers is often sufficient, for example, stream in expectation.
And then we can what is that, if now would have a proposal on distribution such that the case estimated to be 2.7 for the same accuracy,
we would need more than 100000 troops. You can see that there's no sharp change at gate zero point five.
So that way. Previously, there was also some system to estimate like could make a hypothesis testing whether the variants it's fine or not.
And then abandon the hope if the hypothesis test would say that variance is infinite.
Instead, we just use the K continuously because there's still hope here.
Although then, after about 0.7 begun of the required sample size, it's start to beat that [INAUDIBLE].
That it's not sensible one like in this lever, not cross validation.
It would be better than to use something else than simple input on something, maybe even just refit, then with them CMC.
That was the charity called Result and yes, an empirical comparison to the Terri sewing root mean square error knowing the truth.
And let's focus first on just on the black lines which are and root mean square error.
Given different proposals, distributions and it devalues your aunt Gail,
tell how good these proposal distributions are because these are the card values.
So from quite small and eventually around 0.7 and 0.9, and we can see that.
So the error goes down when we get more growth.
But this is also reflecting that like here we have this 100 dross and we have the kind of good proposal distribution.
Is somewhere around here and then if we. Draw parallel on the horizontal line, we can see that if K would be around 0.7.
It would also indirectly match that we need more than 100000 drawers to get the same accuracy.
Direct flights are moved to call a standard error estimates based on pace,
the effective sample size estimate I showed you before and then that's well, they match quite closely even.
Here. This is close already to 0.7, and so this red dashed line and this black line onto the air and they are close by,
then we can see that when they get close larger than this, our estimate starts to be also too optimistic.
But it's it's not that bad because we know based on the K hat that we can't trust them anymore.
But it's also interesting you can see that the angle of this so this angle of this is actually now corresponds to that.
The speed of convergence is this. Why?
Tickets are divided by X, but then with the card is.
Clearly larger 22.5. The angle is a different.
And we can summarise this so that here this is the convergence rate, convergence rate, so that.
Usually. In the crude case, so we would have these s two minus one.
But now we get worse convergence rates when case larger.
And, for example, around 0.7 we have the convergence rate is then again, only then the like the Ouachita S to 0.6.
There's a kind of smooth curve, which is partially, of course, because of the limited simulations and limited the length of the simulations.
And then but it's also related that there's no really super smooth transition.
Otherwise, this like starting from zero point five if we draw a line, this test line.
Behaviour close quite close to here, the blue line is for the moment of the normalisation term best moment mean second moment.
So using also then these functions specific estimates and they they have been kind of similar behaviour.
So when we get to, we can say then OK, what would be the minimum sample size required?
And then we could also say that how much we could expect that the variance decreases if you get more gross.
So this is this and we do this empirical predicate using the lightest ratios we have oracle rule selecting
how we need lightest ratios to use for estimating K and then they rule is also such that it fulfils.
Then they attempt to take properties that we always go further in the tail.
But at the same time, increasing the number of draws so that the estimate gets more accurate.
Smallest amount in the experiment has been like 20 largest ratios from 100 ratios and useful, but of course has a lot of variation.
We use empirical profile based estimate by Chang and statements excellent accuracy compared to NCMEC.
So no seems that there's no need to look for more accurate estimates for this.
And see the paper for more details, and the useful question has been from Israeli Eberstadt that why bother with part of the smoothing
improving the estimate in cases where case law like the weather variances infinite's and.
Why don't you just select the proposal so the drought ratios are bound it?
Yes, that would be nice, but it's not always trivial, as you put it, on something leaving out.
Even bigger problem is that high dimensional spaces are not intuitive and they are really scary.
And so here's an example. By construction, we have found it resource finite metrics.
So the target distribution is normal distribution. And then the proposal distribution is a studenti.
It has thicker tails than normal. And so the important ratios are bounded.
Always we. Yes, 100000 draws a lot and an soft, very number of diamonds.
Yes, they affected sample size. So in.
Lou Diamond sums, we get quite good efficiency close to 100 percent, it's efficiency.
But then the effective sample size truck starts to drop and it goes and eventually the terror.
Why is that? Also, if we look at the convergence rate that how much additional draws reduce the error, the conversion rate drop.
So even if. By construction, we have bounded ratios by not variance, asymptotically central element Terim Homes.
Previously, particularly, the practical convergence rate is less.
And eventually close to even zero. The good thing is that quiet can detect diagnosed these present behaviour.
So as we have found it resolves the eye, the true case actually less than zero and in the beginning in low dimensional hat is also less than zero.
But then it goes above zero, then it goes about zero above zero point seven and are out there.
Effective sample size must be zero as as we did, so that at around zero point seven,
we would need more than 100000 troops and then it keeps growing and then competence rate compresses.
So why is why is this happening like this? Why don't we see the asymptotic central limiting their own behaviour?
So now you use a blue line for the normal distribution.
And then this proposal to distribution with Red Line and this is a marginal density, something just a distance from the mall.
So 500 dimensional distribution and then we draw 100000 cross.
And then when we compute how far these drawers are from the moat.
We get this kind of distribution for this one hundred thousand troops, and this is for the proposal and this is for the drug.
So from these 100000 trust. Most of them are closer to the moat than what would be the draws.
From the true distribution. Now, if we are here, what is the important threat is so we can say that the important race is about it.
But look. At the scale of the y axis, the largest.
Ways around that size. And also.
Before we would see that these resource are bound, it would need to see some draws around here.
And looking at this distribution, it's quite unlikely that we would get drugs around here.
In this specific case, it's possible to compute that. The number of draws we need.
To get something here, it's much, much larger than the number of atoms in the universe.
So there's no hope that we would create asymptotic routine,
and thus it is very useful that we can diagnose these pre-symptomatic behaviour if I scale this so that the light I sold, it's only up to 10 to 30.
How the important racial code behaves here, it's almost like a wall.
So in the region where we are getting draws.
With reasonable sample sizes, it looks like it's interesting sort, so interesting support from infinite variance case.
And then we can see this with this empirical approach, OK?
It is also possible like in this case, it would make it less extreme.
It would be possible that like maybe after 10000.
We it looks like aquariums could be infinite unless unless say, let's say,
after like one million drops, we might start to finally see that it's actually founded.
It is possible to see this also then when when you get more, the product eventually starts to also recognise that it could be pounded.
So it's not saying. Kind of what would be if we would give infinitely sampling it to us to say, is that OK at this moment?
It's looks like your performance really bad and it looks like you are not getting much better if you get more trust.
OK, so that was the the main part about the diagnostic. And then I'll discuss this quickly these few applications.
So what is the end result of various small inference stochastic optimisation?
For some of the barriers in Princeton, I took blame on people.
I estimate it's been very useful in forensics, often like normal distribution excuse to some unknown Worcester distribution,
and then we can use that as a proposal distribution sampling.
And then that means that we can use it both for diagnostic and improving.
In this case, we. We'd like to use the Southeast this this order diff different at various in France,
which is kind of blackbox, very similar in France and concrete UMC. AMC here we used a U-turn sampling time looks like this.
And you can see that we can run ATV in less time, but also that if we run it to short our time.
The stochastic optimism hasn't converged yet. And then this normal approximation is bad, but we can use to have to recognise that, OK,
now the optimistic sum result is actually good, and we can also see here that in addition,
not diagnosing that exclude results instead of using just the normal approximation using normal as a proposal distribution,
we can get lower root mean square error with this.
And then again, but I just wouldn't call it a little. And if the preference.
Then. It is 80 I and many other people are reasonable inference algorithms use stochastic optimism, and often in this paper it's been said that OK,
we reduced the step size and when it all appeals, Robbins Monroe condition, it will eventually converts.
And then they've used us the last iterate. But actually, then this optimisation, they stopped much earlier and then the last iterate can be very,
very kind of noisy toys, and that's to stick optimisation.
And then there's this Polyak report, operating autonomously direct operating, which instead use many of the iterations from the end.
And here's an illustration of, again, how we can use this Gearhart. To sell that, especially when the number of dimensions increase,
this optimisation problem gets harder using just lastly, tourist gets noisier gay had.
So it's that the approximation is worse while with iterate rating we get we stay near zero point seven.
So much more stable.
Then in addition of looking at this stochastic optimisation plot, there's also that the what divergence to use use of this exclusive K,
at least not too often to underestimate the uncertainty and so people have been proposing.
Other divergences, which are more mass covering, such as inclusive kale and K squared and so on.
And these many of these common divergences can be presented as expectations of the density racial.
So this is the same density, the ratio as before, and then there's just some func some empirical average, so it's the same thing we've been doing.
And here, what about these different looking, parent based divergences objectives?
So exclusive club. Look, you and we can also know this earlier paper on Terry saying how many moments of.
Well, the W we need and so here, Delta, which just needs to be larger than zero.
And then more equitable, inclusive kale instead requires to find out moments and then Delta, which needs to be lots of them zero.
So it's it's it. You can now guess that then it's more difficult to estimate inclusive kale, an exclusive kale given these Monte-Carlo estimates.
And again, high dimensional spaces so exclusive kale it's known to and underestimate.
And here also again, when we increase the dimensions to 40 and 50, it is the red one is the approximation it is underestimating.
Inclusive Kale here in Lodi instance, it is all we're estimating, and it's giving pounded bone to drizzles, which is nice.
But this overestimation in high dimensions is actually bad.
Now these two concepts are not overlapping anymore, and it's unlikely that from this inclusive K.L. proposal on distribution,
which is over estimating the scale we would get draws around here.
And if you don't get trust there, we will get bad behaviour of this all.
And here. All right, some.
If we optimise exclusive scale so that we like to underestimate and also then how well we are actually.
Being able to estimate the short time term, so this was needed for explosive gale and this was needed for inclusive gale,
and this was needed for normalisation constant. So it's easy to estimate this exclusive scale.
And also, the normalisation constant, we yeah, it goes up again, Europe wants that.
And so not that good. But if you compare that if we are optimising inclusive kale, it's good in low demand since.
But eventually, it will call much, much higher, so it's much worse both for the normalisation factor and that divergence measure itself.
And it's just reminding that from the past decade, we could see also how many draws we would need.
And in this stochastic optimisation that in each step of stochastic optimisation, we would be using these many draws.
It's infeasible. And then also, what is the effect, then, that this Monte-Carlo estimates start to fail?
But is that here? So the dust line is the true. Exclusive Castle.
And this continuous line, the dark green is then what is the estimate?
And so we are underestimating also the divergence and in this pre-symptomatic behaviour,
they look like biased just because it's so unlikely that we would ever see these are overestimates.
And so in into practise, we see this bias behaviour,
which can explain how these results that lie steal these exclusive Gayle is so much more common than these other divergences.
Why people had problems of getting other directions to work reliably and solving.
And so we took small sample size to divert this kind of second corporate political bias, which can affect our results.
And yeah, again, we may need more trust and the number of victims of the universe to make these rely.
And yes, the paper with this more results from this. And the final example.
So this. That's just reminding that what I was talking is not just important sampling and various inference,
it can be used just for plain Monte-Carlo in this paper.
But on an Oh, we did study this leave on outgrows validation, but we wanted to compare import until.
You put on sampling, leaving out cross validation also to the case where we actually then use CMC to sample from this.
And then these drawers are actually prompted directly from the Typekit distribution.
But we did get seemingly biased results because.
So this distribution here was normal distribution.
Well, I then. Given some predicted new.
And then, Sigma. And then when we have these draws of these parameters, if we go far enough in the cave.
These densities are very close to zero.
And then by chance, one of these is that much away from zero so that this distribution of these two entities is very,
very skewed and we see the same behaviour as we could see with this important sampling case,
and we can just look at it then using the positive diagnostic also directly how long tail?
These drugs have I can see that, OK?
Actually, now CMC is also failing to give give the exact look the result.
Like with the fire a reasonable amount of trust. OK.
That's it. And here are the papers I've mentioned throughout the talk and the voters.
Yeah, thanks a lot.
It's very interesting, everything like I wasn't aware of this threat or something to me, so it's good to know all the capabilities that it has,
and I don't really get a place like you can ask questions if you want, like anyone who is in the public.
But otherwise, I have a question which is related to this study in technique that you didn't talk about,
which is that it seems to be addressing or your paper.
There seems to be addressing this very pressing topic of how do we average models given that all models are wrong?
And then you also use this technique in that case, right?
So I wanted to know like what is the usefulness of use?
But it is most important something in the context of a study just because of the computational resource or.
Yes. Yeah. Yeah, that's why we we used that in.
Really, because of the computational efficiency in when we've been when we did this, stacking with M.S., M.S.,
it usually also is working so well that can we get really these high caps or only a small amount of Typekit had so that reliability is good.
We have also example in that paper with Bob variational in France,
and then we have other paper which has more results with the variational in France, in which case it's more likely that we get like hats.
But even then, this but I just wanted important sampling can.
B so that the stacking helps, even if we don't get, like, close to true estimates.
Hmm. Even if the era starts to increase, it's still in the correct direction.
And also in this. The last paper mentioned on this slide currently showing, there's also results showing that depending on the kind of what?
What we are interested like if we are interested in posturing,
mean we do get estimates that are closer to pastrami and even if the caper is quite large.
I'm saying this very small in press. I see. Mm hmm.
We have also saw this item in our paper implicitly adaptive importance sampling has been.
Algorithm that can improve. So like in this specific to this important sampling, live on a cross validation list that.
We don't have like no distribution for the proposal distribution that we just have draws from the
proposal distribution and then this implicitly adopted the input on sampling discusses how we can adopt.
Set of cross using our fine transformations so that we can match then better Typekit distribution,
but with the benefit that we don't need to be able to know the exact shape of that proposal.
And then then and then this would be also useful in in a way that if this.
Basic, but I just moved at Langley when out starts to fail without need to really run.
Mercy, Mercy, this can give better accuracy. I see.
Thank you. Yeah, so, uh, since I got there, no questions in the Chattanooga, uh, what I do think again and again for coming.
So this one hand up by saying who is sort of the government or is it wearing?
I don't know. Well, so maybe it's a clapping symbol?
Yeah, probably. OK, so I. Mr. Mr. Yeah.
Mm hmm. Thanks a lot. OK.
Thank you very much. Right.