Thank you. Good afternoon, everyone, and thank you for coming.
I'm Margaret Stephens, I'm head of the economics department and I'm very honoured to have been invited
by the trustees of the Sanjay Allow Memorial Trust to chair this event.
I want to say a few words about both the trust and also the panellists very distinguished panel that we have here this evening.
First of all, just to give you the background to this event and the centennial, our trust,
it was founded in honour of to honour the memory of Sanjay Alal and his work and it,
together with Green Templeton College Green College was was his college.
It enables the university to bring to Oxford each year for one term,
a distinguished visiting professor in the WHO works in the area of business and development.
Sanjay Allow came to Oxford in the early sixties.
He was a student at the beginning and apart from a few years at the World Bank,
he spent the whole of his career as a development economist and researcher here in Oxford.
He was initially at the Institute for Economics of Statistics.
He went to Queen Elizabeth House, where he became professor of Development Economics and a fellow of Green College.
And some of you, I'm sure, are familiar with his work for those who aren't.
He was a prolific writer on all aspects of development, especially.
He was particularly interested in technology transfer and the positive role that governments,
national governments can play in terms of policy development policy to facilitate that.
He was an advisor to too many governments.
He had a formidable international reputation as as an advisor to governments and international organisations.
I knew him only a little.
We overlapped when I first came to Oxford, so I'm going to turn to to to make Ned, Deci said about him in the obituary in 2005.
He said unusually for an academic economist, he overcame the characteristic culture clashes between social scientists and scientists,
economists and engineers, economists and business managers, academics and practitioners.
He made it his business to know how things actually worked on the ground and not how they might be assumed to work from the comfort of an armchair.
And so it does seem to me that he would have particularly approved of the 2018 Sanjay Allow visiting professor Esther Duflo,
whom we welcome to Oxford today. She has just arrived as the 2018 visiting professor.
Ronnie asked me to say also that she is very,
very proud as a as a trustee and really organiser of this event to the female line-up we have this evening.
There is. So a first female head of the economics department, a first, Sanjay, allow a visiting professor,
and we can probably come up with a few other firsts as well amongst us.
Esther is the Abdul Latif Jameel, professor of poverty alleviation and development economics at MIT.
And I am sure many people here are familiar with her career and many accolades to date and the impact that she's had on development economics.
She co-founded and co-directs the outstandingly successful Poverty Action Lab Jay Jay Powell at MIT,
which is established randomised controlled trials as a primary tool of policy of evaluation, particularly in development context.
Her focus is on what evidence can tell us about how to improve the lives of the poor.
And her book, which some of you I'm sure have read poor economics in with Abhijit Banerjee,
they argue that there are no universal answers to the question, and it really depends on the context in which a policy is applied.
To quote, they say that requires us to step out of the office and look more carefully at the world.
And that's why I think Sanjay allow her to very much approved of what they have been doing.
We also have Rachel Glenister here, who's come since the beginning of this year.
She's been chief economist at the UK Department for International Development.
Like some JLL, she started as a as an Oxford post, but she's since worked as an economic advisor at the Treasury, at the IMF and the World Bank,
before also being drawn into the orbit of Jay Powell, where she was the executive director from 2004 until arriving in the UK this year.
She's worked on evaluation on policy evaluation in the areas of microfinance and governments,
governance, health and education, and sometimes indeed with with Duflo and Banerjee.
And so she's really ideally placed this evening to provide her own perspective,
her own evaluation, if you like, of the role of actors in policymaking.
And finally, I didn't really mean to leave him out earlier.
Sir Richard Peto is the distinguished professor of medical statistics and epidemiology at this university,
and he's a trustee of the Sanjay Allowed Trust. And like Sanjay, I allow.
He spent almost all his career in Oxford, and in parallel with Esther,
he founded and co-directs the Oxford Clinical Trial Service Unit and Epidemiological Studies Unit.
It doesn't have quite such a catchy name as the Poverty Action Lab, but it conducts large,
randomised trials and meta analysis what worldwide, according to the Royal Society.
Richard has shaped the way modern clinical trials and population studies are carried out worldwide.
His own research has saved countless lives by quantifying and highlighting the preventable causes of cancer and cardiovascular disease,
smoking in particular, and by identifying optimum treatment choices for both.
So he brings a different perspective to our discussion this evening and to randomised controlled trials in particular.
In another link with Esther and Rachel's work, he has recently studied malaria in Africa and India.
So please welcome all of our panellists. So we have an ambitious title.
Our focus is on education. This evening, I'm going to invite Esther to speak first.
And I thought it's first, so first of all, think, let me start by expressing my gratitude,
thinking you will very warmly for me here today, not just to to give a talk, but to be the Sun lay professor.
As Mike pointed out, this is just getting this name is in itself an honour especially far for me,
given the type of work I do and the ambition that this work has to make a difference in the world while being academically relevant.
So we are, you know, this is one of the old model that we are trying to follow,
and this is a wonderful opportunity to to, I guess, to reflect on that.
Another thing, of course, is this institution is all I think can already be called an institution for how
successful it has been in getting a hugely distinguished line up of professors before me,
and I'm even more honoured to to follow in their footsteps.
And finally, I'm delighted to be here today with the two of you.
I should add one first that I think is relevant, which is actually the first female chief economist of Difford.
So wonderful. She also, I suppose, is the first female executive director of Japan.
But that's less interesting because she's the first executive dogs on the effects of that.
So there was not a long line of males before her.
So what? I am going to try to talk about today is really one illustration of the way
that we attempt to to address of two attack issues in Japan and in particular,
all the way. I've tried to do it in my work, which is some interplay between trying to understand the nature of our policy questions.
So here is the one slice of the problem of education in the world and understand them, not just to understand them or do that.
That's quite nice, but in order to to have enough things to say about what could be done about the problem and then try
and go one more step and work with government to try and put this possible solutions into practise.
So education is a good example to do that. First of all, it's a very important and interesting problem.
So going off, we you to read a bit more mature in a sense.
We've covered it's one where we have had to tend to cover that entire route and not just do the research of it, such other things like that do.
There are other domains like that, but it's maybe the most complete.
So what's the status of the world's primary education in three interior bullets, which which is, of course,
a somewhat of a caricature, but it's worth the kind of setting the stage, the unemployment rate.
So I'm going to talk to them mostly about primary school, almost exclusively about primary school,
with a little bit of mention of pre, primary and post-primary.
So we're thinking about primary education today, and all rates are very high worldwide.
So in a sense, that battle is being won.
This is not the issue, except in some countries or some extraordinary circumstances is not to get kids to school, even gold.
The issue is to teach them something when they come in because they are on board.
The attendance rates are not that high and the learning levels are very low.
So let me elaborate on this three points quickly. So this is a map of the world with enrolment rates.
Unfortunately, the the I took it from the World Bank and it didn't see fit to put the key.
But the the dark red means above 90 percent enrolment, and you can see that basically the whole world is dark red.
And if we're looking at the low income countries here, the share of kids who are enrolled in primary school is actually just at 100 percent.
If you're wondering why they can be 100 more than 100 percent and all, that's the the way they are kids who are in primary school, who shouldn't be.
So we take the population in primary school and divide by the fraction of kids of primary school age, and typically you get a number above 100.
So of course, much lower doesn't matter.
You just have to know what this is much lower in pre-primary, much lower and secondary and tertiary, but even that is really increasing a lot as well.
And you can see that the difference between boys and girls is really a very small.
If we are looking at India in particular, which a lot of the examples I'm going to talk about today are going to focus on India,
and then I think Rachel is going to expand to the rest of the world.
We have the same finding where all the and all men rates are very high.
So the net and all men register. I'm asking every child, are you?
And this in school is almost ninety seven percent.
The attendance is much lower. The attendance on any random day in September is just seventy one percent.
So many kids are absent for whatever reason. And you can see some differences across states.
That's a problem, but the bigger problem is really not there. The problem is that while kids are in school, they don't learn very much in school.
So in India, it has been documented for years now by the organisation Pratham that you're going to hear a lot about today,
which is a wonderful movement on education in in India.
And one of the things they have been doing since the early 2010s is, in fact,
they started in 2005 is a survey of all of a representative sample of kids in every single district in India.
So it's a very large sample of kids and they're going to their home.
So it's irrelevant with all the kids in private school or public school or not in the world, et cetera.
And what they find is that for kids who are in standard five, that's the last grade of primary school are less than half of them are able
to read a simple fog off which they should be able to master by stand up to.
And even if you're looking at one level or letters level, the the achievements are correspondingly low.
The second thing that is striking in this picture is that this started in 2005 would have put the at the club since 2011 is that there is no progress.
In fact, in 2016, they got depressed and they stopped doing.
In 2017, they didn't do one. And in 2018 they did second high score.
So we don't know what's happening after that. But from 2005 to 2017, there was no new change in that number.
This is the cost. They also do a simple mathematics test.
So if you are depressed about the reading test, the mathematics test is somewhat worse,
so they have a simple subtraction, which carry over two digit subtraction with carryover.
And you can see that in standard, five 50 percent of kids are able to do that subtraction.
And then they have a division and you stand up five.
Twenty six percent are able to do that division, which is just one number divided by one by a single digit number.
So the achievement in mathematics are even even lower. So this is not just an India problem, although it's better.
It's been documented in the deeper and more systematic way in India.
They'll know it's similar, always easier for other countries or for other countries which find very, very similar things.
A lot of kids go to school, they don't attend very well. Usually they learn very little or a lot of them done very little or nothing.
So of course, given that issue, people have been thinking about what this might be due to, and there are lots of everyone has a favourable hypothesis.
What I want to stop trying to rule out is what in my view, not just my view,
but maybe it's not a consensus, but maybe to some extent, at least what the problem is not due to.
So I want to rule out some of the explanations that are in in the home.
But before that, I want to talk. I want to go one step deeper than just this diagnostic.
This this set of slides.
Oh, very almost like a you know, everybody says everybody says, and it is a problem that all the kids go to school and a little nothing.
But I think it is worth saying it is worth realising that there is an even deeper problem.
Which is that not only the kids that the kids are learning nothing but the schools actually unable to identify what the kids know anyways.
So you can say, well, look, kids are learning nothing at school,
but at least the school would fund some kind of certification rule or something like that.
And what we. But what is finding on this very scholarly tasks of doing division and reading reflect is also the fact that children.
Existing knowledge is not picked up by the score and if anything is wasted and destroyed by the schools.
So let me try to talk to be more precise on that.
So the first thing I'm going to show you is that you can tell small kids,
I'm going to describe a little bit an experiment that we did with a team of cognitive psychologists at Harvard,
which goes by the term of baby lab at Harvard, led by Liz Pelke in pre-school.
And I'm going to show you that you can take Indian pre-school and teach them a lot of mathematics.
But once they go to formal school, the school system is unable to build on it.
That's the first thing I want to do. And the second thing is, look, you know, this is going to mirror my pre-school for me.
And then my middle school phobia is going to look at slightly older kids who are working in markets.
I'm sure that the schools can actually do complex mathematics, not complex with metrics, but very quickly, very good at it.
And yet they are terrible at school maths, even though many of them are actually and old in school.
So let me spend a little bit of time on this. So the pre-school mathematician is an experiment that actually a series of experiment.
We are about to embark in the sound lab to get us a deal.
Not the first one was one with about 500 kids in the in preschoolers in Delhi.
And we we they were all in this little informal preschools in slums,
and we randomised the pre-school to three one of three conditions a mass games where I'm going to tell you what mass games were.
But they were like a curriculum of a few weeks of abstract games to teach mathematics social games,
which is very similar games, but to teach social skills.
The main purpose of that was to have an active control where there is the same kind of game play introduced in the school,
but it's not maths and the sandwich was just the normal curriculum.
So no, no change. So we we started with a pre-test, then we run the game for three months and then we added post-GST.
And then we add a second post test and then a second test until push test after the kids have joined school.
So this is an example of the type of things that we do.
And so this school spits off doing so you need to tell which has more dots, red dots or blue dots.
The interesting thing about our learning of mathematics is that every child in every culture
battling extreme disability is able to do this type of exercises despite no exposure to maths.
And then so if you know that this one so then needed to do these things.
The second thing that was known before we did the study is that this in principle can be trained and it has been trained in the lab in small,
small, a very small scale.
The self thing that was not known is whether or not, if you did that, you could actually help the learning of formal mathematics in school.
So once they have a softer account, they can turn it back and know that in fact, they were correct.
They are more red dot. In fact, they are 10 red dot and five blue dots.
Similarly, we did. The visual form analysis is also something kids can do.
They have to turn the right card. This one is the wrong card to it doesn't have a smiley is winning the fight card, so he does a smiley.
This where the social games very, very similar exercises.
Except you can see in the software, you can see who is happier and the girl or the blue girl and then which fish doesn't go with the rest.
So the idea is the rules were exactly the same. The gameplay was exactly the same, but it didn't rely on mathematics.
It relied on on social skills.
The important findings that we find is that at the first in line, we saw that kids are much better at doing this kind of non symbolic mass.
So in fact, we were good at training them and even better also at symbolic but symbolic much we
can see that they are better at point to no as well so early on in the experiment.
The experiment was effective at teaching them mathematics,
both in the non symbolic mathematics and symbolic mathematics, even though there was no symbol in the experiment.
The effects are very persistent on non symbolic.
Mathematics are very persistent over time, which is not the case for most of the education experiment where you often see a decay.
In this case, we see at the end line to an online tree, you can see the two bars, the effect persisting through 12 months later, 18 months later.
They still have this extra mathematical mathematical abilities, but there is no transfer to a school maths once they go to school.
So you have kids who have a better ability to do this kind of informal mathematics that we know all the bedrock upon which formal mathematics is run.
But despite that, they are not able to learn formal school mathematics. And the reason is that the way that we surmise is that the ways at school
mathematics are taught in school in India is closer to 2.3 than to mathematics.
You were learning that the table of multiplication and these type of things,
the fact that you have any understanding of maths is sort of neither here nor there.
So, so that's the first experiment. The second, which is not an experiment, a descriptive study of of kids, adolescents,
young adolescents in two markets in Calcutta and in Delhi, where we we did two things.
So first, we went to see the kids in the market and we sent mystery shoppers to to buy two product like,
for example, eggplant and tomatoes, some funny quantities like three of eggplant and seven kind of tomatoes.
And when we give them, say, 500 rupees and ask for the change, so the operation requires to apply the unit,
what to do your petition, to do your auditions and then produce a picture.
So we did that three times. I'll show you the result of that in a second.
And then we pulled out the kids if they were willing and if their parents were
willing to give them more like a set type for middle school mathematics questions.
That's the first thing we did. And then the second thing we did is to do the same thing.
But we schoolchildren are in very similar communities, kids in kids in school.
By the way, a lot of the market kids are also in school.
But then we went to school children who are not working in markets for the most part, and we gave them the formal mathematics.
And we also set up a market like that where they have to buy.
They have to sell items. So in the same way, you can ask for some kilos of vegetable or some you need of beans and stockland.
And what did we find? We find we found that if you look at the the formal school mass mathematic,
you can see that the the market children are much worse than the school children.
So the highest level possible is division level. And we only have a 16 percent of market kids at the division level.
But the fifty seven percent of the school children are division. No.
So the kids, the the market kids are much worse at the formal mathematics.
But if you look at the market transaction,
then it's the opposite where the the market kids performance on the mathematics question of ninety four percent.
Ninety five percent of them. Get it right. This complicated maths. Well, the school children are pretty bad between the 50.
About 50 percent of them. Get it right. And in addition.
And by the way, the school, the kids are better than the adults than the market's adults.
And so the more we left the school children unlimited amount of time in this first study, and you can see how they tried to calculate this.
So even the one who got it hard get you tried about 10 minutes of adding something productive,
something productive, which is that even if they got the right result is not like, what could you do with that?
They couldn't possibly work in a market. So from this study, you learnt two things.
And but one is that the there is we did many,
many more things with the market kids to make sure that it was not just that they knew some things by heart and it's just automatic.
They're changing the prices, et cetera.
And if any breaks down at the moment of abstraction, it breaks down when they try to apply algorithm they belong to school in school,
unless until they use their own methods, they can figure it out even with other prices, even with other quantities.
But if you ask them to bet, if you ask them a formal abstract problem, they try to do what they've learnt in school.
And that's where they get it. They get it wrong. So that's the first thing.
So the school system basically destroys the knowledge that exists in these kids.
And the second thing is that what it teaches to the two kids who are actually in school is and practical.
They can't do anything with it. What are you going to do? You can't rely on third down mathematics.
If your understanding of multiplication is to is to write something, ten times to do 13 times that.
Neither can you walk in a market, so it doesn't impact any skill that is useful and also doesn't grow.
In a sense, this is not a skill that do exist in the population.
So that's why the problem is worse than just the supplement.
It's deeper, and it's it really creates an enormous, an enormous waste.
So what's the problem? I was advertising to you that I would talk about what the problem is not.
So there are three things that you hear commonly,
and that I think it is not the first thing is not it is not because the kids cannot learn all winter long.
You have various versions of this argument. The teachers will tell you that the parents are lazy or the children.
Children are lazy. Bill Gates went to India a few months ago, and he explained that we really need to talk to to improve the nutrition in India.
It's because it's because kids are not properly noise that they can't learn in school.
That's a more politically correct way of saying the kids are stupid.
That's why they don't learn to. That's that kind of a popular explanation in a form or another.
By the way, it's an explanation that is internalised by the internalised by the children very quickly.
Well, and which contributes to worsening the problem because they say, of course, they complain, I'm stupid anyway.
So for example, when we ask our market kids to to do the do the mass operation,
they're like, Look, I've dropped out like, there is the reason I can't do this stuff.
And so that's one I'll elaborate in a minute, but I don't think that's that.
The second thing that I don't think it is is teacher salaries.
It is not because teacher salaries are too low.
In fact, I think there is a real problem of teacher salaries in the US and probably almost created in France.
But it's very different in in developed and developing countries where, on the contrary,
teacher salaries tend to be high quality to the rest of the distribution. So I don't think it's teacher salaries, either.
And finally, I also don't think it's incentive to do the job, at least as teacher understanding that can play a role, but I don't think it's a major.
So let me elaborate on these three things, I believe. So why do I think that kids get along well?
Because we studied them, we've studied them.
If you go back to the pre-school mathematician, we actually have the question of kids in this non symbolic games of our little Indian kids.
And we have the same of kids in the laboratory who are children of graduate students, of an assistant professor in Cambridge, US.
And the progression in the games and understanding of the game is similar in the two contexts.
So it's not that kids can't learn because this kids can log just fine. It is.
It sees the material.
Moreover, we find the same correlations between current and subsequent symbiotic skills and not simply skills that is found in the US,
suggesting that the very basic cognitive processes that we find in the US also exist in India.
So the Indian kids are no different. Therefore, they should be able to learn just as well.
Now, they don't have as much help. They don't have support at home often, but they they are increasing all their.
Did show salaries do, as I said, are highly paid a teacher,
salaries are much higher in public school than in private school and then and yet private schools are accused as of performing.
I'll tell you more about it in a minute. Mom, come in and teach us like contract teachers, which are very common in many countries,
are paid a fraction of teacher salaries and tend to be more more efficient.
And finally, there was a very large school experiment in Indonesia,
which basically doubled the teacher salary and produced no effect on learning whatsoever.
Teacher incentives, so I've done some work on that, and clearly you have some effect at the margin of higher teacher incentive, for example,
if you give them incentive to show up more often,
they do show up more often and they are and therefore the test scores of the kids increase a little bit.
In Andhra Pradesh, there was a study showing that if you give incentive based on test scores increase a little bit,
although in another study in Kenya, they found that there was also a lot of gaming on the test.
But the reason why I think that's not the key problem is because there is a very nice
study by Karthik Monty Don looking at private schools in India as in private schools,
as Tonga's incentive to do what parents expect to be done.
So and when you look at private schools, what your findings were, what did this study give kids scholarships to attend private schools?
And now you can compare kids who got the scholarship who are much more likely to attend private school to kids who didn't get the scholarship.
Well, not very likely to have attended private school. And those who are randomised to they are exactly comparable or those of study we
had on private school can confuse the selection effect that kids who are brighter,
whose parents are more motivated are more likely to be in private school. But this has none of that.
And what you find is that private schools are much better at teaching English, but public schools make no attempt to teach English.
So that's not the that's that's not fair composer. But there was a teaching maths and there was a teaching Telugu, which is a local language.
They're also much better at teaching Hindi because they do teach Hindi which public school don't do.
So what did we learn here? Well, it's not that private schools have a different technology that somehow works better.
They are pushing there, but they are putting that effort differently.
They are putting that effort on English instead of putting it in Telugu and maths,
which lead to actually what performance there are with better performance in English.
Presumably the response to Japan demand. So what Fundament is not really a basic skill of local language, and maths with on demand is English.
And that leads to a possible explanation of of of what the key issue is if kids can learn about it.
What could the problem be? And this is what we have been calling the tyranny of the curriculum.
The problem is not the incentives of doing our job as teacher perceives to be the job.
The problem is the perception of teach us to be a joke, a product of what is done to them.
The problem is that teachers do not think that their job is to teach kids to read or to hide.
The problem is that teachers think that their job is to complete the curriculum and they are not wrong because our job is to complete the curriculum.
In fact, if you are looking at the result, there is a small I didn't show you these classes initially with the performance from 2005 to 2010.
But what you find is actually the test scores going down a little bit.
The reading test scores going down in 2010, and that could be we don't know for a fact,
but that's consistent with a pal west side effect of the Right to Education Act,
which was best to India on that time, which said that every key to the right to education on Nice in the world,
but also said the teachers are compelled by law to complete the curriculum.
So in a sense, criminalised not doing the curriculum.
It didn't said that teachers are complete by law to teach the kids to to every kid to do we don't like.
So the the emphasis is on the curriculum and the curriculum is is is extremely ambitious.
So I put a pulled up just one little example of what Grade four kids are supposed to be doing in Haryana, which is the state we are.
Delhi is located in maths. Oops.
So they need to in October, all they learn the guts and wheels and then halves and quarters in November.
All they play with batons and stuff to do a table up to 15, who he knows they are the table of multiplication of 14.
I'm not going to do a test because then I would have to do the result of my cell phone.
And then they, once they are done this stuff to understand volumes and then there and then they need to understand perimeters and areas of shapes.
We have talking about grade four. So this is just like. By and in in the sky, but in a very, very faraway sky.
But that's that's what they do. So that's a problem, which is not the India specific.
That's a problem which you find in many countries in particular.
In France, we have kind of the same issue, but mostly also in former colonies,
partly because the education system in the sense was inherited cities from developing
from colonial powers who had created it to create a small elite of educated clerks.
And then when the system was passed on at independence, it wasn't.
I completely see. I completely get that it wasn't feasible to dumb down at the moment where you
were going to expand it politically or kind of not a very visible gesture.
But the researchers developed a mindset and a curriculum that's just completely inappropriate,
not just for these kids, but for any kid and any and any teachers.
And if you take Finland, for example, which is doing, you know,
acing the test scores across all of the international test, the curriculum is not at all ambitious.
In fact, kids do not start to learn to to to start, to learn, to read before they they they are seven.
So mind you, that's not a causal, you know, that's not a I'm not trying to infer too much from that,
but that's just to say that it's not like an Indian peculiarity.
Overambitious curriculum is is a plague, and it's something that is very, very difficult to change.
And what? The political conversation is very hard.
So what do you do then? Well, back to them.
So time is this organisation that does so.
But even before doing so, they started working on education early on in the mid 90s when they got stuck to their motto was every child in school.
And very quickly they added every child in school and learning. And finally, they added every child in school and learning well.
And now if you talk to a mother who is the former chairman of POTEM, they want to learn in school.
Is it just every child learning was because the school board doesn't seem to be all that Hollywood to him.
But so what is the what's the approach of them?
It's very simple when you think about it, given the diagnostic I just put is that you need to teach kids where they are able to to learn.
So you need to recoup the the kids.
You need to assess the kids vision, which is a very simple tool for focussing on basic skills.
Then you need to avenge them by what they know at moment.
And then they have a set of activities that can be done either by a teacher or volunteer to run that.
And then you regroup frequently because once the kid is in the right level, you could actually progress fast to the levels.
So that's the top of the core of their approach that it can be modulated in any sort of way.
You can do it with computers, you can do it with people. Teachers can do it with volunteer.
So I'm not going to spend too much time on that, partly because I have only five minutes left, but it's the evaluation of the towel,
which was not yet co-written with Balzac at the time, was my first condom is the control trial that I was involved with.
And we've been working on that for about on and off ever since we started by demonstrating to under my control trial quote.
Tom's approach works very well when it's done with a volunteer.
We started the first people show that was about was in urban areas, then with which we worked in the rural areas, in the with the.
And then so it seems to be that the result replicated for context, to context, very similar.
We are from other countries as well. So that seems to be like.
An approach that not only makes sense in theory, but happens to work in practise and believe me, that's not true for everywhere you plug.
So now an issue that the problem the programme wanted to know is that the take up of the programme is quite low because remember parents,
they don't they don't really see the learning, the key basic skills as something which is fundamentally important.
So Padam struggled with attracting kids to their camps, even though the camps was very effective for kids who did come.
About eight percent of the kids did come in the summer, for example, to attend the camp,
so the effect on the overall population was lower than it could have been.
And that really led them to think that they have to work in schools.
So the problem when you work in schools is that you have to convince the higher authorities that they have to understand that there is a problem,
which is not to which is not always easy and different Indian states and then but that they have been successful at doing that in seven states.
That's problem. One problem, too, is you have to convince teachers to actually take it on board.
The problem is the programme has to take in the organisation and that's what I've called a class of plumbing issue,
which is that it is not a it is not deep philosophical problem, but it is very hard to get something new to come in and a new organisation.
And to cut a long story short, it took us many, many different trials to find a version of the plug on two versions of the programme.
Actually, that does work in primary school,
who first tried and we failed and we understood what we failed and we did it again and then tried again and again.
And finally, we have something that works. In fact, we have two two models that does work in one model.
It is volunteer to do it, but they do it in schools, so they basically take over all the school.
For 50 days, that works hidden in places where the education system is very, very broken, so teachers don't really mind getting 50 days of vacation.
But how many cases where actually they wouldn't mind because they feel that they are serious people, they feel they have a job to do so.
Another veteran of the programme, which is easier to scale up, actually, is to work two teachers trained to teach us to implement it.
But then you really have to signal very strongly that it's their job to do that.
That requires to train also the teacher supervisor and that required to isolate a specific moment in the day,
which can only be used for these activities. And so we decided to on an experiment.
We've found good results from them. This is the how that is the teacher led model where you can see the the yellow ones saw the difference at N-Line.
Between treatment and control, you can see the treatment is higher than control.
Uttar Pradesh is a volunteer model. The effects are much larger.
So when you can do it, that's nice. You can see that they start from an extremely low level, but then they end up mostly almost that.
They have end level at N-Line, starting from a much lower place in the dark blue bar over there.
So this is the volunteer led model in schools, which was done in this state where the education system is pretty much defunct.
So the advantage of doing that is now we have kind of we have cut them has in their hand
a model of two models that can be scaled up and they they have been scaling it up.
The India scale up as of September 2017 was sucked stage twenty 28000 scores of four million kids.
The dialect learning model is much fewer schools, but continues to play a role.
There is no randomised evaluation of that, just the proposed data, which seems to suggest good adoption and some, some some progress.
What I think is might be a little bit of a start itself is a transition to twitchers intervention.
Is that now we got the chance with funding from a large new called in fact to start
responding to a demand for the expansion of this programme in various other countries,
in particular in several countries in Africa. So you can see here all of the countries that have expressed interest are where a pilot has started.
The nice thing is that it's heavily supported by Khartoum, so the African team go to India and the India team go to Africa at some point.
My sister was involved a long time ago in it and sister of that my sister was involved in.
It says that I need to stop talking. My my sister was involved in that and she lost the Kenyan director of education in the middle of India.
So it was it created a little diplomatic crisis, but that that got solved and the UN can now move on.
So if you left for me one minute, I'll conclude with some thoughts on where do we go from here to fix primary education?
So one option is to give up on school, said the school double. And this is a little bit, you know,
maybe you might have second inclination when I was saying they if he wants to drop in school from every child learning well,
there are a lot of digital efforts that are taking place mostly out of school by distributing tablets to kids.
So it has a great potential, extremely large impact of some initiatives to, for example,
mind Spark, which is the machine learning based software programme to learn.
But the fundamental problem with this approach is that kids are in school.
So there you have them, hostage for whatever you want. Everything else require them to show up.
And because the demand for basic skills is not there, they don't show up.
For example, in the US, after the very successful evaluation of mine Spark software,
which was done after school, the parents willingness to continue with the programme was zero.
They were just not interested because while it helped the kids reach basic skills, it didn't help them go to grade level.
So the parents were like, Why do we care? It's not going to get them in it. And in fact, it is not.
But it might still have. Some have some use. So I don't think we can abandon the schools.
Changing the curriculum, of course, is the is the holy grail. The problem is that the political fight is a pin.
I do think that there is a glimmer of hope in India. In Delhi, they have just decided to give away the officially said.
We are not going to finish the curriculum. And more perhaps even more striking,
the central government also the minister indicated the desire to simplify the curriculum very recently
a couple of months ago and announced a commission to to think about it now with the commission.
Who knows? You can also work on the margins.
There are some parts of the system that are more open a pre-school where we started working on the pre-school mathematics,
early grades, even in grade one. There is less ambition to do the summer tutoring.
Different schools. We can use the margin and we've been doing that.
The. But finally, I think ultimately there is not going to be one thing that is going to solve the problem very, very fast.
It's one way I mean, I would love to be quick, but it's not going to happen.
Ultimately, you need to continue to engage with the system and one by one.
And however, splitting the parties get them to adopt things like that help kids coping with the curriculum.
So that helps teachers ignore the curriculum or things like that.
So the challenge is to exploit the existing wedges and to pry open new ones at every opportunity.
This is not going. So this doesn't create like I have the solution you just adopted.
But those are still the gains that we are getting in the meantime.
And of course, any real gains you get is quickly multiplied by millions and millions of children if you're working at the level of a school system.
So that's kind of the the the sort of not maybe earth shattering evolutionary way that we are going at it,
but which has, I think, some real progress to show for itself.
For example, the median of kids that I just mentioned, Peter, thank you very much.
Thank you very much. And now Rachel Bannister is going to give us a different perspective on the education problem.
Thanks. So it's great to be here. It's great to be back with Esta sharing the stage again.
It's great to be a stone's throw from my old colleagues, Somerville right next door.
So I'm going to follow up a bit on on what Esther said and talk about three issues.
First, look at some costing estimates, so you've heard a lot about kind of what's effective in improving education.
But you know what? What are some of the costs associated with some of these interventions?
And some of that is going to tell some similar stories, but maybe from a slightly different perspective.
And then I want to delve into the question of how do you design improvements in individual countries?
So we've heard a lot about this overarching evidence about what's effective and what's wrong with a deep dive into India.
But you know what, if you're going into Somalia, say, which is, you know, I was working with the different team in Somalia recently.
And you know,
how do you think about the problems there and how do you use evidence from other places around the world to inform what you're going to do that?
And then I'll talk just a little bit about what different is doing in this area and
in trying to shape the incentives globally to think about some of these issues.
So before I start talking about costs, let me just explain a little bit about how we try and measure cost effectiveness in education.
And there's no really great way of doing it.
But the kind of the standard in the literature is to look at the standard deviation improvements in learning per $100 spent.
And that's not great. We would love to have some absolute standard of learning, but you know,
if you're trying to compare learning and a study done in India to one done in Kenya or one in the US, everyone's at a different learning level.
And how do you compare an improvement in, you know,
getting a kid to be able to read a sentence versus someone who's getting a kid to add single digits to single digit addition?
So so what we fall back on is looking at standard deviation improvements.
And, you know, so I guess I'm in the mathematical institute, so maybe I shouldn't have to explain all the standard deviation is, but,
you know, point two of a standard deviation moving from the 50th of the 50th percentile one in education terms, one standard deviation.
That's a lot of learning. That's a decent amount of learning in a year is one way to measure it.
So, you know, if you spend hundred dollars and you get a year's worth of learning and a decent system that's like phenomenally cost effective.
When I show these results to people in the US or the UK, they think I've got the decimal point wrong because it's just, you know,
blown away by how much learning you can get from some of these effective interventions for relatively little cost,
which is actually true in lots of areas of development.
So the first thing I want to say, and maybe it's a little bit of a tweak on what Esther was saying,
which is about access because it's completely true.
So are you getting kids into school? It's completely true that kids are not learning anything like as much as they should be in school.
And I think India is particularly depressing because you're seeing learning go down.
But where? But I don't think we should give up on the fact, you know, we made huge progress in getting kids in school in,
you know, in sub-Saharan Africa in particular over the last 10 and 20 years.
I don't think that was a complete waste of effort and some of it.
When you look across all the randomised trials, you see you see some trials that had big increases in access did lead to increases in learning.
Now it was increasing learning on school maths, you know,
in-school learning so that that we should take what Esther was saying about that into account when we think about that.
But it is the case that if you go into Afghanistan and create schools where there were no schools,
kids, no more in the schools where those schools were created. So, so you know, that's and it's also interesting.
There was a lot of work done on conditional cash transfers, which improved the number of kids going to school in Latin America and a lot of
depression about the fact that they saw no increase in in test scores in those schools.
Well, it turns out that if you look carefully at those studies because you were moving kids from, you know,
maybe 80 percent of kids going to school to eighty five percent,
you simply didn't have the statistical power to have picked up a reasonable increase in learning scores.
So I'm slightly more. You know, we we shouldn't assume that access is, you know, not at least a first step.
That doesn't mean to say that, you know, we got a lot to do in terms of improving learning within school.
The next thing that comes across from this, you know, cost effectiveness.
So all of the balls in these charts are results from a randomised trial.
And so this is for which we have cost information as well of as well as impact information.
And you know, the things to take away from this slide is to reinforce the point that simply more inputs is just not getting,
you know, if you teach the same way,
but you have more textbooks or you have, you know, computers, you it doesn't do anything to learning because the fundamental issue is how we teach.
Well, what's interesting is one. So one of the charts that shows an effect is school grants.
And then you look at the second year of that same programme and it goes back to zero.
The only programme that has an effect is giving textbooks when you look just at the top of the class.
So the kids who are already performing well, they're the textbooks actually did have an effect.
And it's actually a reasonably cost effective intervention because you get over a standard deviation increase in learning for those kids.
Why? Because they're the only kids who understand the textbook, so which reinforces the point that the curriculum is completely wrong.
So it's working for a very small slice of kids, but not for all of them.
Now, this is something that really surprised me when we did this cost effectiveness was
because there's a lot of work to try and improve the way schools are managed.
And many of the studies don't have an impact. But when I started looking at the cost per impact, when you when it does work,
it's phenomenally cost effective because it doesn't actually cost much to improve the way that schools are managed.
And that's about feeding back information into the system, about about whether teachers are showing up and whether the system is working.
Now this is you know what Esther talked a lot about, which is how do you changes in how teachers teach?
And that's consistently very effective. But just look at some of those results.
I mean, we're talking about, you know, 30 full standard deviation improvements for $100 dollars.
I mean, that's just phenomenally cost effective because you've got the kids,
you've got the schools, you've got the teachers, and all you're doing is changing how they teach.
So there's a fantastic opportunity to take the systems that we've built over the last 20 years and improve,
improve how we teach in the systems at relatively little additional cost.
Esther talked a bit about accountability.
You know, accountability is matters, but it's really hard to get right, I think, as my summary of this literature.
Let me go on to talk a bit about how we take this evidence to two individual contexts,
because a lot of questions get asked when we talk about evidence about.
But isn't it different in different contexts and how do you take the context into account?
Well, you know, we've had this massive increase in the number of impact evaluations in it in education,
but we still think that, you know, context matters. So how do you square that circle because you're never going to have an impact evaluation
exactly the question that you care about in exactly the context where you're working?
Well, this is my, you know, framework for thinking about how you do it.
You know, you have to start by diagnosing what is the local failure,
what's the local challenge in a given context, and that's going to be different in different places.
And then you want to draw on general lessons of behaviour about, you know, how kids learn,
which is us to talk about is actually quite similar in many ways across contexts.
And then you have to worry about the plumbing. How do you make it work in this context?
Right? And that may be very different. It may be very different in a place where you have computers already and people who know how to use
them versus ones that what you don't or where you have volunteers who could work in the system,
et cetera. So this is how I would take this talk and kind of put it into that format.
So this teaching at the white level? If the diagnosis of the problem is, as it often is, children are in school or not learning,
there's high variation in learning levels within the class.
Teachers are focussed on teaching the curricula and not on learning, and many children are below the curricula.
Then you can draw on this body of evidence, and you heard a lot about the Pratham version of teaching at the right level and a little bit at the end.
But let me just explore a bit on that. There are other approaches to trying to tailor the level of teaching to where the children are,
which is what is not, you know, which has it, which is also positive.
So, you know, using where you have, you know, lots of computers in school,
using adaptive computer software can be very effective way to adapt to the level of the child.
Right. Tracking by just taking the existing system and saying,
let's put thought children by where they are at the beginning of the year was also very effective
way of teaching at the right level and didn't actually involve changing the curriculum.
That wasn't, as you know, I think you can probably do better.
That's at least a step of teaching at the right level. So there's many different approaches.
And when we go in and talk to a country about how they can use this learning and adapt it to their environment,
it's always useful to go in with many different ideas about how they could bring that off.
And then you have to think about local implementation. What can you pull off in this environment?
Is it remedial education after school?
Is it personalised learning software or is it tracking, which is the intervention that makes sense in the local context or changing the curriculum?
I'm running out of time, but let me just say, you know, the diagnosis is not always the same in every place.
So when my conversations in Somalia, you know, 17 percent of primary age children are in school.
So there we have to start with the first challenge of how do we provide schools in a conflict affected area where the state isn't really working?
Right. So can we take the evidence from Afghanistan and think about community based schools?
What we do? And but we can still learn from the general evidence about how to get kids in school,
which is we know that convenient, cheap access is a really important way to get so.
So if you make schools free and you make them convenient, actually most kids will go.
So it's very sensitive to price and convenience and distance is really important, especially for girls.
You know, again, local implementation, you've got to think about what you can pull off in a conflict affected area.
Building permanent structures, maybe not what you want to do. You may want to use local community groups.
So briefly, what is what is the Fed doing? One of the things that we're thinking about is how to change the incentives in the system.
So we, you know, we talked about how the curriculum is overly ambitious.
How do we get people to focus on basic skills?
Right. So one way is to do something like Pressman has been doing and measure basic skills and publish data on basic
skills to try and get education systems to focus on basic reading and writing rather than at the moment.
They will focus on getting kids through that final high stakes exam at the end of secondary school,
which I knew a few people are going to get through.
So I don't know if you heard recently about Jim Kim's initiative to produce a human capital index, but that's, you know, one of the Bennett.
One of the ideas behind that is to have kind of an index of what, what people are learning, what kids are learning.
And very important to that will be measuring whether people are acquiring these basic literacy and numeracy skills.
The other thing we're doing is investing in learning about learning. What are we?
What do we know about how to improve learning in school? So some of the work that that is just talked about has been has been funded in this way.
And finally, thinking about how do we take some of these results to scale?
And I want to just briefly talk about an experiment in and scaling up in Kenya.
We did try and tackle the curricular issue. So it said, you know, working with the Kenyan government, let's test taking the curricula,
making it more appropriate to what kids know and to the lives, you know, in the education system and where the kids are at and test that small scale.
And it was very effective. And then what were the Kenyan government to change the curricula across all of the
early years and is now being scaled up to twenty three thousand primary school?
So you can only work on the curriculum where people are open to it,
but it's absolutely fundamental to changing the the incentives and getting children learning across the world.
Thank you. Thank you very much, Rachel and Richard Peto is going to be our final speaker.
OK. I would take the slides off, please.
It's hard to take that one off as well. I can't stand looking at my face. OK.
Yeah. Well. One thing I came to Oxford in about the 90s in the nineteen sixties, as well as same decade as Sanjay,
and I've mostly been doing medical research since then, and I wanted to try to think about some of the analogies and most of it's been on trial.
When I first started working on randomised trials in medicine back in 1967 and in some.
I think some of the lessons of randomised trials in medicine will carry over into the social sciences,
you know, randomising, was it due to the criminal impact?
You know, if you have this sentence or double their sentence and randomising, what does it do in terms of education?
And I found this book, the poor economics book, really citing, I read it.
I thought I'd read it about two or three years ago, and in preparation of this panel,
I reread it and I'll tell you, I find it just much, much better on rereading.
That's partly because knowing I'm going to be sitting standing next to concentrated my mind a bit,
but it's it's really good to read about taking classes and just.
Taking the class as the unit of randomisation and then taking off, say,
the bottom 40 percent and getting them just to a really cheap teaching assistant who gets paid about a tenth as much as the teacher,
try and actually teach them something that's at their level and then put them back in the class and see what the class is as a whole.
Do better if you educate, if you try and deal at very little cost with those who are really not doing well in that class,
and the answer was yes, it did. And how'd you know it did? Because they randomised, they randomised and they got results for the class as a whole.
And then they did actually discuss thoughtfully whether it was because actually just having the 40 percent of most
troublesome kids taken out of the way let the other ones learn something and decided it wasn't that that was a non-writing,
randomised analysis. I mean, it's certainly worth thinking about, but the the idea of randomising is is just very attractive.
But I want to go back and think a bit about what went wrong with randomised evidence in medicine and randomised evidence has been brilliant.
It continues to be brilliant, but it did in various respects,
go go wrong and it is continuing to go wrong in various respects and randomised evidence in medicine.
There are few early efforts to get do some randomised comparisons.
The one that really lit the fire that was the spark that actually produced the conflagration produced the massive
amount of randomised evidence was a trial done back in the 1940s of the use of streptomycin to treat tuberculosis.
A TB was a terrible disease think by the 1940s, it wasn't nearly as many people as it had been, presumably a bit better nourished.
But still, it was a major issue back in the 19th century,
and it used to be referred to as the chief of the captains of death, which is a lovely phrase.
The registrar general doesn't use words like that in the registrar general's report now,
but that was what was just how it was described actually by the registrar general's report in the 1860s.
And suddenly they got streptomycin. There was a drug that could actually really attack the Chibuku Bacillus, and they've got limited supply of it.
They allocated randomly. Half the patients with horrible TB didn't get it.
Half did, and there was a serious difference in survival mean the half who did.
And this was great. It was really exciting to have at last, a drug that could do something against TB.
And also, there was this new technique of choosing at random. Whether or not you should, you should actually use a medicine.
And so somehow the but of course, the truth is that streptomycin was so good that with or without a randomised trial,
its merits would've been recognised and probably your thing with three standard deviations
per $100 spent would be so good that we recognised for that randomised assessment.
And anyway, so then it was in the mid-century.
There were lots of antibiotics being discovered, and also there was polio vaccination that was evaluated by a randomised trial.
But again, actually, polio vaccination works so well that you could have picked up the fact that it was working without randomisation.
It's easy with randomisation,
but you could pick it up that these striking things without randomisation and say randomisation was getting famous for producing,
striking, striking cures battery. That's not what it's useful for.
I mean, yeah, I mean, you could believe that things have striking effects like the power of prayer or homeopathy and so on.
But they don't. But it well, actually, there was that.
This is not what I supposed to be talking about. There was actually randomised trial of the power of prayer,
and it's the only trial that ever actually disproved the null hypothesis by completely nonsignificant results.
And it was done by there was a guy who was president of some statistical society back in the 1970s,
and he, you know, when he was young, he could be brought up with a religious maniac as a friend.
And they, you know, their paths diverge as time has gone by.
But anyway, they still stayed friends. And, you know, he was really excited by statistics and trials and so on back in the 1960s.
And you know, then and his friends stood out, you know, you can't really affect things of life or death by medicine.
You know, these are matters of, you know, the spirits, you know, etcetera, etcetera.
And anyway, one thing led to another and this guy decided he was going to with his friend.
They were going to set up this randomised trial of the power of prayer.
And so they were going to get patients coming into the local hospital who seemed to be similarly ill comparable patients who weren't expected to die.
And the that and they'd go in on the same day, their names we give them to the statistician who would, you know, allocate heads or tails out.
And the local congregation would either prayed for free, pray for Fred to get better or Jim to get better.
And so this was and it was really randomly done. Nobody knew who was being prayed for.
And I don't suppose the parents, the patients, the doctors cared very much.
But anyway, this was going to be the randomised trial of the power of prayer.
And, you know, my friend was sort of sitting back rather smugly thinking that he was going to be good.
This is going to be a triumph of science, French and so on.
And the first and if then it was just whoever came out of hospital first, his score either plus one for prayer or minus one for pragmatism.
And so the the first one came out in favour of prayer actually just had happened.
And then sort of the second one and then the third one, rather to the irritation of my friend who was sitting it up.
And then it went on and the end.
Every time every single one came out in favour of the power of prayer and it was really being done, and it got to the point where one more.
And it would have actually hit the boundary and I had to underline the results and declare them significance.
And he said, I find myself praying it's it goes the other way and it did.
And then, you know, from that point on, it went down and went down at six in delegate.
And so this is the only trial. That's but it's a not insignificant result that disproved the null hypothesis.
But more seriously, the trials did get into trouble because they got this tradition of looking for effects that were too big.
And it was when they started that like pretty much your comment that even if you got small gains that apply to millions, actually,
you could have said tens of millions or hundreds of millions of kids because it's not only those that now,
but there's there's going to be there in the 2020s and the 2030s. So you can actually add in medicine.
The key principle is that actually you can save more lives by moderate reduction,
a big cause of death than you can by a big reduction in small cause of death. And so and we have to in medicine.
I mean, after you go to the big spectacular things done. So there's a few more big spectacular things, but mostly you're left with.
Is there a moderate gain or is there no gain? So the role of my role as a trial is over the decades and is I've got to try and distinguish
reliably between the idea that the treatment is completely useless or that it's fairly useless.
Because even if you take effective use history and if you can take 10 percent down to eight percent,
didn't you start applying that to a million people, you know? Well, that's a huge number of lives saved.
And actually, I take that as an example because that's what we got for giving aspirin in the middle of a heart attack.
And but then there are effects that are more moderate than that.
But they apply to millions, tens of millions of patients, you know, maybe hundreds of millions of patients.
If you actually look over the lifetime of the trials of a common disease and you've
really got to get moderate differences and say you've got to get huge trials,
you've got to get huge trials with if it's going to be random, if you randomising classes, then it's going to be huge numbers of classes.
If you randomising patients, it's going to be huge numbers of patients.
If you're looking for mortality, you've got to have huge numbers of deaths and we've got that for certain things,
but we don't have it for enough things. There's lots of really boring questions, but we need to sort out moderate effects.
And I think and it and it trials got into a complete, you know, after the first quarter century,
they were really coming up with completely muddled results and looking at things that ought to work and keep you on saying it was nonsignificant.
Then you were getting some trial. It was saying something didn't work and somebody else said, yes,
it has a huge effect and there's a thing called base there in which this sort of looks too good to be true, then it probably is.
And eventually, the tradition arose of trying to get all the trials that have looked a particular question
and get them all on the same piece of paper and take some sort of weighted average of them.
And you call it a meta analysis or whatever you like, but get all the trials together.
And you know, if you do get striking claims, then again, probably there's some effect there, but it's not as good as you think.
So actually the case of our study in 1980s aspirin acute heart attack, OK, maybe it doesn't actually change 10 percent to eight percent.
That was a five standard deviation difference, but maybe it could change 10 percent to nine percent or at least half as good as the data suggested.
And and it's still widely used now, but also the.
And I think the trials, if they're going to be done, they're really going to have to be done with an emphasis on.
On what are the random errors in the results and a lot of the smarts that were shown didn't actually show that at all in both of your talks.
And if you're talking about cost effectiveness, you know, we know we want to know what's the effect and what's the cost.
And in medicine, usually it's not so difficult to get some idea of what the cost would be.
But, you know, so it's very difficult to measure the effectiveness. But I think, you know, I just wonder what is the what is the cost?
What is the effectiveness of the various things? And what are the uncertainties in these costs and in the effectiveness that needs to be?
It's going to be a much longer, more tedious process than it seems when you first beautiful results.
So medical randomisation in medicine has been much it's sort of settled down into a rather tedious activity,
which, you know, I've been doing all my life. And now and then you get striking results.
And I think that the same thing's going to have to be true of educational interventions.
It's great that randomisation is happening and also these economic randomisation that you were describing in Indonesia.
I mean, it's really, really very, very nice.
And if anybody wants to really get the book, read this and it's better the second time you read it as well.
But it's going to be a really long business before you start getting really
reliable evidence of randomised assessments where you know the idea that it works,
the idea it doesn't work are both plausible, but certainly actually some of the negative things you know,
do these do various incentives work at least a no result from a trial is good
propaganda against the idea that we know it works and therefore we must do it.
I mean, medicine used to be based so much on unjustified certainty, and the trials were good for debunking that,
even though actually they couldn't really prove that there wasn't any effect.
I mean, which adopted a trial of all these.
We know if you've got a peptic ulcer, you have to go to a milk diet and you've got to sort of eat a little bit of food every two hours,
possibly a bit of boiled fish and so on.
And he just randomised them between, you know, give the all these careful, complicated diets to just eat whatever you like.
There was actually no difference in outcome. In retrospect, his trial was much too small to be informative, but it did actually make respectable.
The idea that you know that much was said to be really important could actually be of no importance, whatever.
So they're quite good as propaganda.
But if you want the most serious scientific evidence, then the much longer and more tedious you think the trial did.
In Uttar Pradesh, we randomised a million kids to get vitamin A. every five, every six months or not.
And you know, at the time, the literature said that seven million kids,
but it was randomised by development block, so it was 72 blocks, 36 blocks versus 36 blocks.
It was great. Our study area of seeing for a satellite spent something like three degrees of latitude and one degree of longitude.
But for all that, we we fed the nought point four tonnes of vitamin A. every six months,
and at the end, it was supposed to reduce child mortality by a quarter.
And we finished up the relative risk of nought point nine six for the standard of 0.03.
And you know, at the end, I suspect that if we put all the overoptimistic trials together with our over pessimistic trial,
then probably it would reduce turbo charge by about 10 percent.
But it's, you know, it's been twenty five years of work to get to that point.
And actually, if you do get a trial result like that, we haven't really changed practise because the people who hated vitamin A anyway,
they say we're just we've done a trial that proves it doesn't work and we won't
even accept the end results of the people who were certain it was wonderful,
just as certain that, you know, we must have done an incompetent trial and so they could dismiss it and say somehow or other, it didn't.
Actually, we couldn't even get a consensus on a weighted average of our results and all the other results,
even though ours was twice as big as all the other ones put together. I don't know.
It's it's a it's a bit of a long drawn out, but then you do get some things that are just really beautiful.
You look at the evidence on cholesterol lowering. There's more than 100000 people have been randomised on statins vs. no statins.
But the differences in heart disease mortality, occlusive stroke mortality.
You know, it's beautiful. You just you can just take one little pill and not take any risk of death reclusive vascular disease by about 40 percent.
It would spoil the theatrical effect of that if I dropped it at this point. But still, yeah.
And you know, the side effects also get evaluated by by by.
The randomised trials will be evaluated and the nonsense about statins.
20 years ago, bad cholesterol lowering it was supposed to increase the risk of cancer, increase the risk of suicide and make people more.
And all these things and not to have any net effect. It just simply when you get a drug that does reduce cholesterol that is easy to give and you
actually test it properly with 100000 randomised follow up for about five years with placebo,
then you get you get the right answers. But as I say, this got it turned off.
Take a long time to get from the 1980s when. We will hypothesise that this might work to the present when, yes, it definitely does.
And of course, all we've got now is the Internet Storm about accusations of side effects instead.
And people will not look will not accept the randomised evidence on side effects,
which says that most of what is being said about these massive side effects is nonsense. So I don't know.
It's just I think you've got to you're younger than I am and you've got a longer life ahead.
But it's good. If you are going to really be pushing for randomised evidence, it's going to be a much longer and slower process than you think.
Good luck. Thank you very much for it, it's good to end on a pessimistic note, a decade so optimistic.
If you say it is, the Detroit I'm going to do now got a pretty wonderful results next year, but I'm pessimistic.
But yeah, that's it's OK on average. And thank you to all our speakers for some very stimulating ideas.
We've now time for some questions. So over to you, what would you like to ask them?
Yes, I'm pleased by the military, and it's sort of an opinion piece here in recent days when you can finally get sorted.
It's the kids who are ahead already who tend to benefit most and the kids are behind who don't benefit at all.
I wonder if you to join the gains were made, which children was who were making gains?
That's one of the things he talks about in here.
It posted, So first, what we like to say achievement, not ability, because achievement is what happened.
Whatever happens to be there, what they do now,
which might or might not reflect something fundamental about diabetes, or it's actually not rhetorical.
So when I feel he is right, he is right for academics carefully. We did this trial.
I did exactly this trial in Kenya with Michael.
Come on, Peskin in Dubai, where we, we we randomised some classes in to be touched by achievement or not, for actually two years.
The kids were tracked for two years. So very different than the pattern model, which we re-organizing just treated.
But what we found in in Kenya is that both the kids benefit the low achievement kids and the high achievement kids.
So I'm actually not aware of a similar trial in a in a rich country.
So I don't I don't know the evidence you're referring to,
but it is actually plausible that it could be that the effects are quite different in in rich countries, in particular in countries where,
for example, in the US, where all the public school system is under no child left behind tends to of the opposite set
up where a lot of the attention is is is put to the lower achieving kids from the get go.
So in this in this setting, if you track,
then the kids who are the high achievement kids get suddenly all on the high achievement kids and also a curriculum that's more appropriate to them.
So they will benefit a lot. The low achievement kids that don't benefit from the change in the curriculum because the
curriculum was already appropriate to them and they've lost the high achievement kids,
which then to pull them up anyway, so then they could suffer from it.
So it's the fact that the this is related.
It's kind of when you push the local problems globally. And if you don't, you know,
the fact that the trekking in Kenya and in India benefits the Lord's kids at least as much the high achievement kids in India,
we found the low, high achievement kids who stayed with the teacher to not benefit at all from the programme, but that was not experimental.
But in Kenya, it was experimental. And so the the that reflect the emphasis that the extreme elitism of the curriculum and that may not apply
in in the UK or in the US if the curriculum is more teaching to the median child or teaching to the bottom.
It might make sense to pull questions if they. We just have an indication of who else would like to ask a question on here.
Thank you very much for your hesitation.
It is mentioned the difficulties of scaling up and the need to spread through the system, especially with interventions changing teacher training,
where sometimes it's when you actually go into the initial teaching and education systems, but also the additional challenges.
So whether that's a university system or maybe an untested university programme versus another certification.
This is a different political system of its own with its own.
Priorities, I suppose, but the fact is the possible way of of creating a more sustainable change.
And if there is any push, I don't know where I am sure any of you.
We have a great team of people. The question just up here, I think we're.
OK, bike. Yes, sir, thank you very much. Very interesting presentation.
And it seemed to me that particularly the randomised controlled trial,
that actually all the interventions that you were talking about relied very strongly on.
Essentially, you know, very external form of expertise is imposed from the top on, you know,
very often the community level school education system where you described the parents as being
different and some of the teachers is actively hostile to the units of the intervention.
So I was just wondering if any of your interventions had taken into account.
You were kind of democratising this kind of expertise from, you know, starting off,
trying to get the communities, all the teachers on board before imposing the the child intervention.
Let's take just one more than a few. Thank you very much, very much enjoyed the panel's views.
So Professor Duplo emphasised how schools can be very ineffective in developing country,
especially when it comes to imparting basic knowledge and skills.
But we still find things such as the strong effects of sex education on child mortality or other kinds of outcomes which are very positive.
Also, infertility, for example, we see the least association of relationships between improved schooling and lower fertility.
As well as that he was on interception and so on.
So I wonder how we square these, these these two things, how do we reconcile the fact that we can have positive effects of schooling on some outcomes,
especially democratic outcomes, despite schooling being so poor in these contexts?
So I'm going to start with this one.
So I say 50 percent of kids can can read that stand up to level one down stand up five, not zero and 50 percent is much more than zero.
So the the fact that the school system is doing very poorly doesn't mean that they are doing nothing.
In fact, that's what the Rachel say when you when you bring kids to school for whatever exogenous reason, they learn more.
So in fact, what you said about activity, we also have causal evidence of that.
I've done to try it on secondary school where we give kids a scholarship to go to high schools in Ghana and they went to school, nice school.
We've been following them for 10 years and after 10 years we find a day.
They know many more things, including even practical things,
other than we give them a test of ability to want to analyse data with information on their bucket list.
They are much better at doing all of that than the ones who didn't go.
The point is not that school does not seem to point. Its core does much, much, much, much less than it should be doing.
So the two are entirely squared, and I think it was useful for Rachel to win season on the second question.
Let me take strong exception to what you said. What is imposed from outside is that curriculum.
What the organisation that is pushing teaching at the right level, Peter, it's a huge community movement.
They are the president in every village when, you know, the district education officer, I've never set foot in a place.
You go to a meeting with these guys and they tell you things that are district or even in the in the ministries,
and they are talking about the reality in the villages. In a way that makes it absolutely obvious that they have never gone.
Our community by now, who is the head of Putnam, mentioned some,
some some district in like a remote corner of the upper deck, where she said only Gundy and penalty went there.
So I just like it. So it just so and tool what you just said that I think it's it's worth bearing it.
It is not imposed from the it's not outside expecting. It is Putnam's programme.
Playtime is, you know, extremely would be, let's put it this way.
And in fact, the first time that we evaluated the programme in a rural area, which was with the we reached her,
it was part of the way it was kind of brought in is that there was first the community test testing organised by the community of all the kids so
that they could see that the kids knew much more and it much less than this would because everybody says somewhat over emphasises what kids know.
And then from there on it, the first time was precisely trying to say, What did you sell your options?
You can go and complain to the district education official within the system.
And then the self treatment was like, Well, this is one possibility to send us a kid that the model is not in the school,
not within the school is run by local volunteers. They are not paid.
They are young kids who of you just devote their time to that.
And that's the one that and beautiful. So the issue is certainly not that it's coming from outside.
The issue is that indeed your right to duty us, the teacher actually like just fine.
They go to the training and they think it's very nice and they like the activities, etcetera.
But so we are done just being a kind of qualitative work around the implementation of the scale of that field,
which was really helpful to understand why I feel and it's not because the teacher didn't like it, but it also did was great.
But they were like, I don't have the time. I have to complete that curriculum. That's what's imposed from outside.
So the teachers here would say very much the same thing, and they're facing just a simple and then Richard, we'll take your question on the curve.
So I'm, you know, trying to work through teaching colleges.
I think I mean, we have thought about that. And obviously, you know, if you're trying to change the way people teach, you know,
you might think that a good place to start is teaching teaching colleges.
I think the fundamental problem, though, is what if you do that?
If the incentives within the school system are still to complete the the curriculum, or it might be slightly different in other countries.
So it's not a law to complete the curriculum in many African countries,
but it is nevertheless the incentives are all around getting kids through that final primary school exam or final secondary school exam.
You know, the entire curricula is based around this one final exam, which very, very few people will pass.
And so even if you teach a different method and you still have those incentives,
then then you're going to have problems and you certainly see examples of where you have training and then teachers go
back into the environment and sort of go back to the old ways of teaching unless you fix all the things around it,
which in that curricular example in Kenya that I talked about, it was, you know,
it was pretty comprehensive in that they spend a lot of time understanding where the children were and developing appropriate curricula,
but then went with training the teachers, producing new textbooks, having very guided lessons plan.
So you've got to shift a lot of things if you're going to change the whole system.
And if you just do teaching colleges without the rest, it may not work.
But if if we can move to a new system now, hopefully in Kenya, they're being trained on that new system when they go through teaching colleges.
I think just a general point,
you're just making this a subject of research is going to help you to really try to research whether things work, whether they don't work.
And also, you know, how do you actually get things delivered? And you know what?
What really is still working 10 years after you after you've done just observational studies,
randomised studies, but basically research is going to help balance education.
Easily measurable things are the great progress and of course, the more difficult things to measure, you know, later on in life.
Do people, you know, have very big cost differences? Are they, you know,
the split between grammar schools and secondary boarding schools here became what was mostly abolished because people
didn't like the effects on the English class system rather than on the rather than for any reasons of academic attainment.
And things like that are going to be extremely difficult to assess or virtually impossible.
But at least the things of academic achievement that they are worth assessing should be assessed that are being assessed.
And that's an improvement. Can I can I just come back on the track because I think it's really important when we talk about the role of tracking,
you know, particularly in an English context, when we talk about the role of tracking, it is not about you.
Take one test, you get put in a school and you there for the rest of your life, you know, in the lower level school, which is the English system,
you know, in the most effective programme that Esther was talking about, you get put in it, you know, in different groups, depending on what you know.
And then as soon as you learn that thing, you move up. Right. So it's a very dynamic kind of tracking, which is very different.
So I just, you know, given that whole English history, I think it's important to emphasise that you just take one and there was one person where.
What I would like to go to the first time that you showed us, which is that a biotechnology loads and loads of vegetation,
and I would like to use on the other aspects of education that we did that upon,
which is that all the bands reading want their leads to academy that new suffered in terms of low maintenance knows,
and there's lots of cases of schooling and even bidding by the parents to the children.
If they don't perform well in the tests and exams and ran on a judicial system,
I'll be making this to incorporate those levels of skills in the in the kids,
in the confidence, in terms of happiness, in terms of life skills and not just numeracy and literacy.
So in our pre-school programme, the active control, if you remember, was a social skill was a social skill intervention.
And the the reason why we picked a social skill is not only we felt that probably that it was a good,
good, good way to have the same level of games and stuff like that, but not the mass content.
So it was a good comparison, but also because we could feel that we could.
We felt confident to say with a straight face that actually this is going to be helpful for the kids.
We hope it's going to be helpful for other kids.
And in fact, we did find an effective social skills training on social skills, and there is a lot of emphasis on that on score,
as does the teaching life skills more in the in in the US and in other rich countries, in particular on the world of the work of Carol Dweck.
Thinking about the mindset,
the importance of of seeing the world as a place that you can change and that to your own point as something that can change,
which has Bozer effect on the on happiness and effect on academic skills.
But I think you are absolutely right to say that it's one of the thing that we have in a lot of developing countries education context.
We are light years away from, from incorporating these things.
And I think you are right to mention the whole of the of the balance in there,
which is to some extent it does reflect that aren't ambitious for their children, which is perfectly understandable and better than the opposite.
But where they constantly do vision, it can be counterproductive.
And another thing that we didn't touch upon is that when one of the things that is pretty
heartbreaking in that is that that also lead to a lot of divergence in the education performance,
even within the same family,
where on the balance of identifying who they think is the bright one and then putting all of their education and eggs into that one basket.
So, for example, cricket in the family, there is one bright one, by the way.
I should said it, it's not necessarily the boy.
It could be that they decide that it's the girl who is the right one, even in India, where there is always, always.
But then to the girl by boy or girl is being sent to private school.
Everybody else contributes to that. In an off, it said if you could that create pressure both on that side and on the other kids,
another thing that is a little bit disturbing in that is that there is a very nice paper from
Malawi which shows that parents do that on the basis of very shaky information about again,
I'm not even talking about ability,
but even the achievement of their children because the information is not given to them in the most transparent of ways.
So they form an opinion, which is their only somewhat correlated with how well the kids are actually doing.
So it's a work by if I had journals that Chicago's was just trying to show that she's doing that.
The parents are trying to get their day of their investment to what they think is the achievement of the child.
But they're failing because they don't really know what the achievement of the charities and when you give them information on that just grade.
So nothing gets it, but just grade the they the response to it is very, very steep.
So that's kind of adds a layer to that, which is the heating and the pressure is not even necessarily based on very, very strong evidence.
And I think it does. I did touched upon that rapidly, but it does contribute to the to the failure of the system in the sense that every boys,
the band, the children and the teacher all internalise whatever happens to be the ranking at some point and we fight.
That's why I kind of reacted into achievement vs. ability because they make it like a thing which is immutable,
even though it could just be like, you know, 10 kids age and whatnot, and they would have been able to to find a fit again.
But they don't get that sense because their fell down fell down when they finally persuaded themselves that it's not for them.
And and that's why you have kids who can calculate at the speed of light who tell you that they cannot do the subtraction.
And that's it, that's the tragedy. And I'm completely with you on that.
And I think we should draw this to a close now, and I'd like to invite everyone here to join us.
The reception, which is in the area outside the lecture theatre here, and there'll be an opportunity,
maybe two, to ask further questions of the panellists if you have them.
We started with a very big question and I'd like to thank all three panellists for actually rising to the challenge and helping us to answer it.
So thank you.