I guess it should be sort of a different kind of setting out here tonight is to introduce our speaker documented use,
who is a fellow and senior education evaluation specialist at RTI,
when she has extensive experience in 30 different kinds of educational projects in various international contexts.
Most of the other, the agency has conducted focus on early intervention and the promotion of learning to buy your house.
He's contributing to projects in Malawi and Tanzania aimed at improving the quality of pre-primary and primary education in those countries.
More specifically, matches with Typekit Semmens culturally relevant approaches to assessments of social and.
This is in Tanzania, for instance, or improving the damages to an understanding of the cultural basis of teacher child interactions,
various methods to such reading proficiency benchmarks or frameworks to improve evidence based decision making that he will be talking about today.
How you joining al-Qaida in 2016? Met you at the.
Two reasons why she was senior director of global research, monitoring and education.
For about 40 years and before to read, he was an associate professor at Harvard Graduate School of Education,
where he taught courses on evidence based decision making and on the growth of those culture and health ineffective education.
And that's the place where I met my chief for the first time in 2006.
I would certainly try to get to tell the truth.
His professional career, and that too,
has worked closely with various international organisations like the World Bank, UNESCO, USAID and Save the Children.
But she holds a B.A. in physics from the University of Oxford and its NSC in Experimental Psychology from Sussex and DPhil from Oxford.
The Game in Psychology and Experience. So I'd really like to present the speaker today,
and I look forward to listening to be stopped on of the rational approach to evidence based decision making in education policy.
Thank you very much. Thanks very much. And takes a lot of damage, and it just reminded me that I have spent nine years in Oxford,
three and three different departments, including epidemiology and not yet here.
And it wasn't until I got lost today that I realised the jury room was actually a of life here.
I know my psychology. It was a huge both of it's friends sometimes helps to get lost.
And so I played around with the title, but I'm aware that calling this a rational approach and evidence Evidence-Based
Decision Making someone implies that everyone else's approach is irrational, which is not really the point I'm trying to make.
But I'll explain that the rationale behind such and this is what I'm doing with a political scientist at an earlier time.
So you spend a little bit of time setting up what I think the problem is and in the three two sections to be to the idea of the approach,
which is all to think about uncertainty and.
But the whole problem being that we can't be certain about about a policy action based on the set of evidence,
very graphic, we talk about how to think about certain evidence.
And then when it's appropriate to bring back memories and discuss communications.
I just want to set up a few. What is is and is not included in this.
So I think this is all kinds of evidence based decision making,
but I'm really to talk mostly today about the kind of way you're looking at the
evidence synthesis review and then deciding how to act on what it tells you.
And also, it's it's a normative framework.
And so I know that there's a lot of good work that's done on how to convince people to listen to the evidence.
And we were just chatting with everyone before we started about. When people ignore research evidence and I know that's common.
But for now,
I'm I'm resisting in the fantasy world where people do listen to the evidence and so be mainly people who are sold on the idea of evidence,
but just recognise that it's complex to tweet about and to provide some guidance on how.
So that's what we should do not to understand that the two.
I'm talking about education, but I think this is relevant to social policy in general, particularly because human behaviours involve evidence,
becomes complicated and complex and also compared to medicine, for example,
I think social policy goes a relatively fragile of assessing potential harm.
And most of all, this kind of development and international development, I think uncertainty is rife in international development,
probably because we keep setting ourselves really ambitious goals for sustainable own goals.
So we want to eradicate the sort of poverty everywhere by 2030.
We want to ensure that every child, lifelong dedication, both competitive domestic work.
I think we have less information to go on. So inevitably that some.
And I think that also, you know, the political environment, there's a lot of pressure to be effective or to reduce aid spending.
And therefore, if you're public about your insurgency, if Republicans say, well, we don't really know what we're doing,
but we'd like to still continue spending 47 percent of the GDP on international donors,
then there's going to be a lot of pressure groups, Tony, to do it. So there's a need for what he calls incredible certitude.
I like to quote, apparently from Mr Johnson, courageous of the capital. Just give me an idea.
So this is precious to be as. OK, so that's the scope I want to talk a little bit about what I see the problem is.
So I think that there's been increasing recognition in the last couple of decades of what constitutes rigorous evidence and, you know,
a big move towards basing policy on regulatory oversight of a massive expansion of
the use of trying to mass trials in an international language and in Britain too.
And so there's a high bar that the people understand the evidence needs to turn to.
To go over to disqualify. And what only one in practise that we review a lot of studies and many systematic reviews and some
of the studies to make it into a systematic review and others will be carefully considered as well.
Over time, they will be ignored and thrown in the bin and. And one of the questions I had for consumers about how much information and
useful information was most important is not going to these kind of reviews.
And so, you know, and in some cases, I think that people conduct studies aiming for this high volume increased rigorous evidence.
And he just said this binary thinking, it doesn't feel like they just say to themselves if they haven't raised the bar.
And in many cases, it's difficult to get rid of representatives about this, all right,
because education is complex in international work and there's lots of studies out there at the moment.
Also experimental studies that look at the impact of a set of interventions on early literacy,
for example, and they will tell you whether it works or not.
But we know that actually teaching literacy involves, you know, hundreds of different actions in dozens of different activities each day.
These teachers are all different from each other for different ages. Children behave differently and try to work out what I'm going to do.
That works is quite a complex question that is not amenable to a simple trial.
And similarly, this is operating in a large system and education system,
Panopto interactive components that also are tough to address with really big.
I've seen estimates that even five percent of interventions in international development are actually applicable to randomised trials.
So this is what actually qualifies for adults 18 months ago,
so what qualifies as three percent of those in the areas of education in low income countries that this priority review?
So if you imagine that you're your Ministry of Education officials, you're looking at this,
you're trying to decide what reforms you are taking to your country, the green ones of the things that are slammed.
And you know, this definitely works. We've got two to choose from.
Cash transfers, which are given to households to encouragement for families to send their kids to school and structured pedagogy.
At least two things to illustrate some of the points I was just making,
which are that cash transfers are relatively simple, discrete interventions like this.
You give some money to a bunch of farmers and you watch what happens and don't waste a lot of
evidence for cash transfers because it's simple to manage lots of people doing financial.
But it's harder to produce evidence in a more complex kind of performance.
Structured pedagogy, well, what this really tells us is that it's a good idea to structure lessons,
so there's evidence that if you your lesson plans for teachers in areas of load capacity, it has to do it's effective.
But you know, we're less completely agnostic as the whole pedagogy is.
That could be any. Lots of different approaches to teaching lots of debates with the military,
and clearly that says nothing about which approach to effective to lots of uncertainties, even about these two.
And these two findings were something else took a little bit about their basic scholarships and a few others who would look after that promise.
And the point is that if this was the only thing you base your policy on,
this is clearly that kind of pleading for certain positions to take that there's not a lot to choose from.
So you have to look beyond the slam dunk evidence and think about what things, what you could possibly reform beyond this.
But then how do you think about it? I mean, there's a couple of ways couples who contrast the thinking I would introduce with one approach.
I think a sense, you know, I think these are really positions that you get people talking about on the field.
Sometimes you could just think harder and additional studies and not actually interviewing,
to be sure that you know what you're want to do is going to work. I said this is another reason, which is and I hear this expressed in different ways.
Don't know the the. You know, usually for choice, but you can't be tied to the efforts.
We have to sort of just go ahead lowering tuition straightforward guarantees because, you know, the evidence is too restrictive.
A couple of other. This is car, so if they can't find the father of scientific scepticism.
And she's incredibly powerful idea, and this is not an idea you I'm against all,
but we only make progress by accepting findings when we can show beyond doubt to the truth.
So I, you know, wish I could have hoped she could find evidence of that kind of technology.
Thinking and potentially evidence of this has got to.
Speaking as a former.
That's a very different kind of thinking than if you are if you are a minister of education and maybe you consider that kind of 70,
you use actual evidence and make some kind of decisions. OK, so I just want to get you to speak of one thing before we continue, so this is a real.
This is an almost real example of a conversation I had with a few of the people
advising the foundation about the decision I have to make to invest in girls education,
proving justification in lower income countries.
And that plan was to implement programmes, but also to try to prevent the space of two things.
And let's say that the short listed move down to these three things.
Merit based scholarships as reasonable evidence that merit based scholarships give a go.
So the secrecy of those who said they work to improve, achieve.
And it's pretty critical, you people do it. They're also considering developing social, emotional learning.
There's good evidence that that can be effective, but not particularly for low income countries.
But it's reasonable, and they're also considering tax reform education systems in developing countries to deliver social investment,
which has no really strong evidence that is being affected and nor is it even clear and it's not abundantly clear.
But the first step is what I'd like you to think about what you would do in this situation.
What would you recommend? It's great if you all think about that.
Some of them could even shaded. I learnt large company.
Term limited definition also changed in November 2008.
Well, let's imagine that that's up for grabs. And so if you as the policy adviser to ask, we can push them to be more specific, but.
In mindsets. Moving through this sequence.
And I wonder if I gave you enough information about, know, just just based on what I've given you,
even if you have any thoughts about what would be a bad idea, it would be a good idea.
And hopefully some argue this would be just.
So the execution rate comes. There is no excuse, no education is the footprint of education, right?
So this would this wouldn't just be a sort of welfare intervention, for example, for 2002 would have a funded.
So this is that we might have other things. They definitely want us to.
Yeah, that's right. So let's let's remember that these three.
He's getting close to. Right here in.
Sexual assault, you know, it's a the of. But it was some toxic waste.
Support for people need based scholarships with some funds,
and maybe that is something that's evidence to the contrary that say, weeks ago, maybe some of the.
Financial short and treatment systems to help you set yourself up by saying from a normal to company and as important for your life and for them,
simply because you care a lot to gain some critical skills and possibly easier talking to one of your first.
Right. Which is on tape talking about some concern. That's a good strategic idea.
So the idea is that you don't just do things that are potentially fatal.
Yeah, you want to do at least something.
And Canadians looking for a moment, which is great, but he doesn't have any opportunity to stop making scholarships and things, you have to.
This is going to be much time.
So they should be saying that. Yes.
And these are some of the considerations that we thought we had around the table,
but I mean the main one because the debate is there's no evidence for this.
And there's more to some of the debates that you could throw as a result of that.
You have a pretty weak track.
And you and also you could argue that maybe just implementing a relatively limited programme to improve for so much money would be a sticking plaster,
which would which have no sustainability, whereas this really has got to be sustainable.
But there's no it. So how do you think about that as an evidence based organisation?
And actually one of the one of the people around the table said, Actually,
sometimes I think we look beyond the evidence and, you know, think about the implications.
And that confuses me a little bit. I mean, it's like a slide.
I showed a couple of weeks where I kind of like, Well, let's forget about the edges, because that's not helpful.
Let's go with some other considerations. So really what you really pretty well.
I mean, this is a problem that really seemed to be thinking like this is like, how do you then think frankly about this?
How do you how do you find a way takes into account the evidence because it means you don't know the facts?
And part of it is to speak about, in some situations, moving away from thinking I was thinking about it as a sort of sliding scale of.
Uncertainty. So this is what I mean by a rational approach.
Why would we have? OK.
It's a yes. And. I'm sorry for you.
Yes, of course. Okay, well, thank you for helping us.
So what is the rationale for several places in Mexico?
Well, part of it is we don't ignore the best use of the evidence and every time in my definition of Russian,
there's rumours like these are these are the best outcome. So that's improving on a friendship.
And so there's two of two elements.
The first is how to think about it, so I want to introduce two elements of the consequences of a policy and its course.
And before the interview, I would tell you what I mean by a certain period of time.
Those of you who spent a lot of time doing quantitative work will be familiar to most people's minds.
Go to the end when you talk about the first one.
So you look at the impact of the programme and there will be some confidence intervals around it with some uncertainty of.
So that's included in the definition, but it's not the only thing you could think about 17 methods.
So you do not close the study, you're not sure if you've really found the programme.
In fact, there are some assumptions that you just use uncertainty as uncertainty populations.
You make your study in one place and you have some uncertainty about whether it looks like somewhere else or the length time.
And even structural uncertainty that just which is just the thinking around this thing in the hope that you give it in a real neighbourhood.
So when I used the term uncertain term, I'm talking about any reason why a policy we were about to implement.
You're uncertain, but it's going to have the opportunity. So I think about this continuum of consequences.
On the one hand, we might kill people with a fever or even not tested.
And I say that because a lot of the thinking around the evidence, if it comes from a medical model, right?
And so if we're at this end of the spectrum, we want to get control of that, at least that it doesn't kill people and probably works.
But somewhere in the middle, we might not be sure of perspective, but it's what we have.
So we know that. But we can't waste the money, and you can even think of like an extreme example where even as a short policy,
these people act, a government has to decide between this textbook of this textbook.
They don't know much about either the cost roughly the same and really boils down to an arbitrary decision for something close to.
And if this is the scale, we should feel more comfortable at weighing up the less.
The government's actually on its hands, but there was some measure of making sure that as soon as we do it,
we have some idea about whether it's worth. OK, so I'm just fine,
I'm going to present a few graphs and just to get people fine speaking in this 100 percent of
the way to a couple of years that we want to test and I'm not sure about is the cure rates.
Some studies say to be as high as 70 percent of people care about us. Other studies say it's completely effective.
We need more evidence that you put somewhere in that range, but we're pretty sure it's going to be another drug sildenafil.
There's some pretty good studies that it's effective. Pretty sure.
Appreciate it does to show people that there's some uncertainty about that, too.
Now, if you're looking at this and as healthy as a healthy person, you would you would say, Well, I'm going to choose between these two.
I'm totally going to because I leases, I'm not going to die.
And maybe, maybe well, maybe one. But this is going to. You can make a different decision if you are either, you know, totally safe.
I'll increase security. So I'm going to translate these images from racially across, I'm assuming that ultimately with this,
I don't know how many people are so typically present odds ratios, those if there's an alteration of one,
it means there's no effect of something that has no impact on the likelihood that it could be as much as 10
times more likely to secure you if you take this group and its distribution between in the middle of the.
So this is going to last. We have some uncertainty about the effectiveness.
So I'm just going to start off by this example, I just a moment when one of these graphs.
So this first round has a lot of uncertainty about whether it's going to cure you,
but you're pretty sure that the patient is more likely to get behind the.
Was this one better estimate for being sure,
but the fact this comes under the gun here means that there's a chance the patient is less likely to be alive and so.
So obviously you probably got. I'm just translating that into what I hope will be some more useful developments.
The decision theory has, you know, his thoughts are a lot of this about the about what approach you take.
So the realist would say what is the most likely expected? But they're not afraid to let me just say, well, I mean, look, I mean,
I look at the politics can give you the best possible opportunity to actually be able to
approach sort of respective respective like a realist would never buy a lottery ticket.
But under this approach, you might practise the best possible approaches if you need to raise a million pounds for tomorrow lottery.
The Russian approach. No rational approach to investing in our economy.
And the pessimists, I might say when we look at the worst possible case, the oldest policies, and I'm going to make sure that I get the least worst.
So there's different approaches. I on this graph, you can imagine.
So this is an official, this is hopefully two different interventions.
If you were a refugee for a moment, you think it's the best possible outcome for the best,
expect to get off to a serious issue of infection one, because just for the of uncertainty, that could be as good as it is, right?
And if you're a pessimist or maximum approach, you would say, well, if I go with this one list,
we could possibly get this here, right, which is which is better than the. So that's kind of the phrase.
And if we contrast to the options are getting reinforces the importance of scholarships.
This is how you think about system reform.
In theory, you have a lot of potential to be beneficial and makes a lot of uncertainty, but we think it could be.
And the scholarships, we think are pretty tightly. Typekit converted on last.
So that's essentially our choice. And you could see how direction the British decided to go for the best possible.
You think differently about it, though, if this one is on the surface, potentially if you look to this?
Well, you know, I don't know what's going to happen, but I think it might be helpful.
I think this could be catching up. Refugees, we think well.
Peter, it's not worth it the same way you say different about like buying a lottery
ticket for a possible outcome to get someone to come out with a bill for $150.
And frankly, this is one more concept, which I think is bring about a skewed utility utilities.
The concept of homelessness, which is thinking of which is really just a sum of responsibility, lots of different outcomes, and how much is I?
And if it helps, I'm just going to go through a numerical example of this in the paper in evidence based medicine,
which is the treatment of women who've had one year of GDP growth,
which is at least five different, and that decision, whether or not to take photographs of gone for the next pregnancy.
So if you don't take photographs acid, you have roughly three and a half percent cut in half.
Now it's a little bit 18 to put some money to on this, but this is purely to sort an American scale.
Let's say we would be willing to spend $100000 to avoid that.
So then we expect to each other to this is 3.4 percent, 200000, which is three thousand.
You take folic acid, there's reduced chunks of 1.0 percent, which is a utility.
So then the calculations would say I could facilitate you taking folic acid.
So you compared, I guess, just. Focuses and focuses on. I mean, three times of.
So it's a rational decision. You probably don't need us corporations to realise that, but it was time to go through the motions.
So you could apply the same rules apply second, and this is just the gross entry amount as a matter of funding to the president.
The idea that some the policy would lead to a worst case of $200 return in best case.
So you could kind of put some numbers on it to help us. But let's say the policies, the intervention cost $220.
Then you say, well, look, there's a chance this is not going to be worth it.
I mean, those returns are going to be less, but there's a big chance that they are removed. So maybe that's worth going.
But in many cases, before us make decisions, we don't know a lot, we don't know much about the cost of we'd like to.
And research is going to be very interested in the impacts on U.S. The.
Typically, there's a lot of uncertainty in the policy, which is about changes the decision.
OK, so I'm I'm done with these sort of abstract threat framework.
I'm just going to a couple of predictions and I'll give you some examples.
And the question is, you know, we need to think about the worst possible outcomes for those of us who are.
And there are places in which you could be rational to proceed to pursue a policy that has an uncertain outcomes.
If you can rule out negative effects, there is nothing disastrous is going to happen.
You think it's plausible? Logical. And you must be not going to be obese, but but even better at costume.
And one implication is there's a few of the cases,
one is that I think we need to do a better job in social policy and education policy of thinking of the positive and negative outcomes.
People typically evaluate the intended outcome not so much to be unintended.
And there are plenty of people in the population looking to saying the same thing to.
Now to think about where to gather evidence, when there's uncertainty, you have here an important outcome.
That's where you should focus whoever this guy is, particularly if you have information about him and this person,
other people say you need that across in this atmosphere of these decisions.
And I have only done four prior to that section, I used to work for a lot on these things.
The district, this is a close up of the Mouse Club. The leaders of East to claim to some sort of government.
The president. And people just ten years ago, and this was the state of the evidence,
then we had a Cochrane review systematic review saying that there's no evidence that we want to improve the detection options.
At the same time, the policies of the action went on to do.
Randomised trials have policy briefs, specific deworming is the best one.
So I spent a lot of time thinking and trying to explain why. And I do believe both of these positions can be.
Same time, and the framework can help us to figure out a little bit.
So to begin with, the impact of DOMA on improved education is less certain.
So this is not just a shocking jump. There's some there's some reasonable evidence to think that it might be effective.
It is a nice theory that says you put the media and media concentration for results.
And there's a lot of evidence that the new systematic review is evidence of a mass because this is a highly complex dependent thing.
So it works in one area and not another.
So you're going to be X amount of times you look at the evidence of a lot of randomised trials suddenly having to test results,
but they say for certain subgroups that there are so few those of you could make a case for this being something worth considering.
Meanwhile, we know that at least on an individual level, three women doesn't have the serious side effects.
We know it improves your health, and it's pretty cheap. When it's delivered through schools, it's got down to 10 cents per child.
The demands of all these things would, you know, at least compared to if these were not?
That would make the system rational approaches to test out.
Context under testing, this is crucial because actually, this is relatively easy to evaluate as you test it in context, and that's.
This next section, so I want to talk about it, ideally, we want every decision to be based on 47.
To have a lot of evidence, so trying to sort of think through where it would be acceptable,
it would appear rational, reasonable to consider ourselves.
So here's the first couple of things. I think when there's an urgent decision, this could be, you know,
focussing events where policy review because every five years it's now or never even met a justification for emergencies.
So, you know, education and humanitarian crisis.
The executive decision was a moment to think about, that is what is the alternative, right?
So you shouldn't you can say, well, there's a lot of uncertainty about this cause of action.
But there's also uncertainty about what happens if we do nothing. And so it's about balanced approach to consideration.
The second thing is, how feasible is it to get better outcomes?
So, for example, our system reform reform idea that we discussed the beginning, there's not good evidence,
but you know, if you're looking for a better analysis of opportunities, you're not going to get and emphasis.
The evidence is always in complex and there may be ethical issues as well.
In doing a controlled study, so.
You may want to embrace the war on some kind of courses of action if you know that the evidence is.
So. I have a couple of things under this heading, which I will spend more time talking about,
which start with this statement first on the efficacy of future policy decisions cannot be described by single parameters.
So, you know, you have studies that produce constraints like what works and what works clearinghouse.
We sort of try and sell the idea that this is a silver bullet approach to Evidence-Based Policy to do a bunch of studies that show the impact.
Therefore, this works. But I would argue that every decision that you make is a combination of different kinds of evidence.
Maybe the main bit of evidence is an evaluation of some kind of efficacy,
some kind of effort to study, but really easy to piece together all the facts and evidence.
And the critical thing is how important fathers are.
Governments, if that more important than the mainland sort of have massive impact evaluation here, then it could be appropriate.
So let me give you a couple of examples of that. So we just started RTI.
I worked for started working two weeks ago, packed with essential but relevant and funded by citizens against foundation,
trying to ask this question about what is the. Construction mosquitoes scale.
And we're putting us down to try to understand the structural approaches for these tax learning in large scale programmes,
mainly in Africa and South Asia, and in trying to understand what aspects of the system supporting.
So you can think you can break this down in this way.
The probability that a programme is going to work at scale is a combination of the probability of a pilot programme
works time by the probability that you can replicate this type of study conditions across the whole nation,
for example. And typically, we have much better evidence here.
Know we have lots of evaluations of other programmes that tell us this.
But we really have good information on this story, right?
I don't know if I'm ready to studies to even try and isolate this from one or two, from probably from the other.
So the consideration here is what if this is much worse for the environment?
That you will say to try to get the vaccine, that you already have a delivery system,
you never have to get a vaccine candidate recruitment, at least I'm I'm only really interested in becoming the vaccine works.
This is much more. But arguably a lot of the situations we look at in and in our countries, this is at least important because of these difficulties.
I mean, there's a bunch of the structural approaches that we could take the evolution efficacy,
but the big uncertainties have we know how it works if a bunch of teachers do it, how do you get all the teachers?
And so in this case, will your attitudes are due to a solid evidence base that the metro analysis literature in the beginning is different because,
you know, the fact is not necessarily most. And what we like to do is to try and break this problem now is to try to ask first the question of
when teachers apply the methods that this instruction approach complicates the picture of learning.
That's one question. But then the second question is when we implement this nationwide programme of teacher professional development,
do teachers apply demands to think about those two questions that.
I must say, if there's a lot of uncertainty here, then being very certain about a sexual relationship, maybe multiple as.
And this is not just hypothetical. There's a great study that just evolved and just this idea that just published this year,
which looks at this is the impact of hiring contract teachers in lower grades to improve student achievement.
There was an announcement here in India which says, There's lots of you and can you show some facts?
And then they scale up in Kenya. So that was founded by an NGO, some of the scale of other governments.
And so you see, the impact is completely dependent on austerity.
So that's an example of the uncertainty. The scaling is as important as concrete.
And then the last example is itself.
So. You're looking at evidence of a programme that's worked in Massachusetts and you're trying to understand that it's going to work and
obviously you could decompose this into probability works is the probability that you put that time for some consideration of how similar to.
And again, as with the previous example, lots of money is put into this with a high level of evidence,
but we don't really know that much about how to think about the second question.
And there's certainly no longer clear studies.
And again, so I'm said the same thing that was this uncertainty for.
And again, with things like vaccines, you know, there are some population differences in response to vaccines.
Probably it's the response is the same.
The education programmes is not. And this is paper.
If everyone in the family knows the central initiative and wondering if you can actually see the impacts of the produce, the great papers.
This is a woman addressing that issue. And I read this a quote from the blogger to which is to.
So he's a real example of this. This issue of obviously, which is paid for by large parts of the country.
So they reviewed dozens of studies on the impacts of reducing trust science on the cheap.
I think BP and American countries. So we found one last Tuesday, which showed opposed to the very small effects, positive side reducing class size.
And achievement and the few other food and other causes for this progression, this was on things I've done.
And you can see there's a smattering of some kind of positive, but somehow the negative emotion.
And there's a whole slew of nuclear studies where people have just looked for correlations between class, size and achievement.
And again, findings were posted on.
Biomarkers from Boston College for the paper,
but Paterson set a maximum level from the day to come up with this, which I'm asking you to take on trust your.
And they came up with this conclusion that the finest from the right place is better suited to this particular delivery.
That issue of all the contact stimuli is more important to this issue than getting really quality, high quality, rigorous evidence from just.
OK, so. So that was the issue of when to think about.
Until this week, including this report, one and a report from.
On the stand here, so to answer this question of when is the Russians pursue a policy on?
Witnesses are urgent, but evidence is hard to improve,
and the point I was making in this last two sets of slides when uncertainty isn't mainly about the efficacy of intervention,
but in other considerations, context scale. And I think I just add a couple of ideas about how we do.
If you take this on board, how you do things differently.
And the first two ideas are really from Charles Manson to written this book on policy, and I said, Well, if you just acknowledge that,
but there's more uncertainty decisions than might be evidence from just looking at a systematic review,
you say, Well, actually, you can't really be sure about a whole lot of things.
So a good idea would be instead of making a decision about what policy are we going to adopt for the entire country,
let's adopt policy diversification.
Let's think about trying different variants of something out, different places with Facebook's or placing small bets on different ideas.
On the other hand, as we get closer to the moment, is this probably driven pictures politicians,
which is just saying that we should implement programmes under lots of iterations of testing and is actually that to make the content specific,
but also seems to have a great to. You think you need to take some sort of research and validation when evaluating
whether it makes sense to focus on the data that provide the most information?
And I was amazed at how often the conversations like this are always do.
This seems like common sense, but actually, if you look at your average age, a lot of work for USAID and USAID in particular framework,
they wanted to measure everything which makes much more sense to think about what's going on, how it impacts whatever.
As long as the companies are scaling and it's based on average rates, consumer activity based approach to breaking down.
You probably fixed the problem. And fantastic.
And then I see, as there's a lot of work to do in research methods that really systematically assess and reduce uncertainty.
And so I don't know if you were familiar with this, I've read lots of studies that may make some strong claim.
I mean, the limitations they say, well, this wasn't part of the study to everything we just said might actually be wrong in the end.
And not many studies to say, Well,
let's try and think together about how likely it is that all the assumptions I made to draw some conclusions are really wrong.
So trying to assessing the assumptions and first ability to come up with, you know, how does.
And it still have some caveats. I think that it's very important to look at where the strong evidence is simply
doesn't work because often that gets put in the same bucket passengers,
any and conclusive evidence. But I think we could make a big step forward by just saying.
This has been tested 20 times per minute and not using this,
you are not using this kind of lowering the bar on this as a whole looking very provides an opportunity to consider.
And it's not about lowering the bar, and I think that that idea of having a high bar for what constitutes certain elements is a really important,
but, you know, I'm not arguing against. But in addition to that, we need to think about ways that we can make better use of some people.
And I think a lot of the arguments is grossly presented would be more useful if we could actually
quantify the kind of the numbers of people who did this and how to weigh the amount of it.
More but more importantly, the methods that.
I just wanted to end on an analogy that the line, do you think about it, which is what you need to see is the distance and all we have is.
Kind of broken spectacles if they're not too broken.
There's a reasonable chance of putting them on is going to help you see, not perfectly, but with vision, the character.
But there is, of course, a point so perfectly. That's pretty much what, three months.
So that's the question. I think we need to ask for evidence. It's not, you know, is a perfect, almost conclusive.
But is it just because it put us in a better position than before?
And so this is, you know, it is a work in progress.
I'm deeply motivated not to put in Typekit paper and brought this up, but I'm enjoying it presented fully by this because I.