Dear all, my name is Therese Hopfenbeck.
I'm the Director of Oxford University's Centre of Educational Assessment, and it gives me a real pleasure to welcome you all here today.
I would particularly welcome Professor Art Graesser from US and Professor Sandra Milligan from Australia.
We're extremely glad to have you here. And we also have our own Professor Josh McGrane, which we're really proud of having as one of the speakers.
I will tell more about all of you when I introduce you,
but I will say a particular welcome to all our guests from different universities, nationally and also internationally,
because we have Professor Nils Gilje, Professor Sølvi Lillejord, who flew in from Norway just to be part of this celebration.
And Sølvi has been a longstanding supporter of the centre and also a Fellow of our Department for Education.
So we're really, really happy that you're here.
We would also like to particularly say we're very pleased to have our president from Kellogg College, Jonathan Michie, welcome.
We're really pleased that you're here. And we also would like to say a special welcome to all our DPhil students and master students.
Recognised students, Sarah from Australia, from OUCEA in our department, because you are what makes Oxford a very special place to be.
And we are pleased to be offering face to face events, although a bit hot,
because after the pandemic it's been a lot online and I really hope that you will have
time now in Oxford to thrive and meet each other in person and experience that too.
It is a special welcome this year as we are now celebrating the first OUCEA Annual Lecture since 2019,
before the pandemic and our centre, which was fully funded by a generous funding from Pearson in 2008,
has been a small but thriving centre and each year we have had the pleasure of hosting some of our world leading professors to give a talk.
Previous lectures have included David Andrich from Australia, Gordon Stobart, UCL UK, Pam Sammons, Oxford UK, Nancy Perry,
British Columbia, Canada and David Pearson, Berkeley, U.S. and Derek Briggs from Colorado Boulder, US.
Little did we know that we had to wait two years to celebrate again.
Therefore, we will allow us to celebrate in appreciation of the funders who have believed in us through the pandemic
And we celebrate our national and international collaborators who stood by us and achieved so much with us in the past challenging years.
I would like to thank you all. The pandemic and lockdown has been hard in different ways for all of us.
And living with uncertainty in academia has a challenge, was a challenge we had to tackle.
I'm proud to say we were able to get through it, although many will say we're still in it.
And I would like to take a few moments to particularly than the team at OUCEA for their commitment,
dedication, good spirit and positivity which led us to successfully secure grants.
Collect data, write research reports, publish articles, and present in different international conferences.
I will use as an example that here today we have Dr. Samantha-Kaye Johnston present.
She worked hard online for more than a year before we actually met in person.
And she represents this kind of team spirit, which I'm talking about.
A particular thank you and appreciation will go to Deputy Director Josh McGrane, who led OUCEA together with me during these challenging times.
We have been able to find a way where collaboration and making our team succeed no matter what was the key to all we did.
And you use the words leading with integrity and compassion, Josh.
And that is what we do and we'll continue to do.
And thank you for that. So these are values I strongly believe in.
And also one reason why Kellogg College became such an important part of our academic life.
We are very thankful to have Kelly President Jonathan Michie here today celebrating with us.
Jonathan is now a Fellow of the Department of Education here in Oxford and an important collaborator for us.
As both Josh and I are members of the college and have attended seminars and events to support their mission on sustainability and education.
You can hardly think of anything more important. And we will continue to support the college in any way we can.
And for those of you visiting Oxford, if you do have time, walk to Kellogg and see the wonderful gardens, take a cup of tea or something in the Hub.
It's beautiful there. But for now, let me introduce our distinguished guests.
Art Graesser is a Professor Emeritus in the Department of Psychology and the Institute of Intelligence
Systems at the University of Memphis and Honorary Research Fellow at the University of Oxford.
His research is in discourse processing, cognitive science and education.
And personally, I had the pleasure of meeting you the first time in Melbourne in 2010 when we were both working on PISA.
He has developed software and learning language and discourse technologies,
including systems that hold a conversation in natural language with computer agents, AutoTutor,
and that analyse text on multiple levels of language and discourse Coh-Metrix
He served as editor of Discourse, Process and Journal of Education Psychology as president of Society for Text and Discourse,
International Society for Artificial Intelligence in Education and Federation of Associations in Behavioural and Brain Sciences.
And as a member of four panels with the National Academy of Science and four OECD expert panels on problem solving.
PIAAC 2011-21, PISA 2012 2015.
He has received lifetime achievement awards from the American Psychological Association,
McGraw-Hill Education Society for Artificial Intelligence in Education Societies with Text Discourse and University of Memphis, Art.
the floor is yours. And we're so happy to have you. So 100 degree heat Fahrenheit, right?
Well, I come from Memphis and we've had 100 degree heat now for six weeks.
And so I'm used to it if you need any, you know, sort of solutions on how to survive it, just ask me during the break.
So I retired three years ago, September of 2019.
September 2019. A month later, I found out my wife had endometrial cancer.
About a couple of months after that COVID came. The next city I was supposed to go to was a place called Wuhan, China.
Have you ever heard of that? I couldn't go because of my wife's cancer, so I keep telling my wife her cancer saved me.
So, some people think I'm not going to retire.
And in fact, I'm at a phase called transitioning.
And so, during the transition, what I've been doing is participating in a number of large scale projects.
I'm not leading any of them. I'm just a team member.
But at this phase in my life, I want to help these large-scale projects succeed.
And what I want to do is just give you.
Very succinct highlights on some of some of these, all of the work is current so I'm not going to delve into the past.
There are three important themes. Technology.
Yes. Multiple disciplines.
Not only being interdisciplinary, but inter-institutional and the intercontinental and not quite interstellar yet.
And also 21st century skills in addition to the maths and the literacy and science,
because 21st century skills, as we know, are an important part of the workforce.
And so we need to change our curriculum to accommodate that.
And assessment is one way to drive that.
This is a recent article I have in the Annual Review of Psychology.
I was asked to cover Educational Psychology in 25 pages, so I decided to focus on this particular slant.
So this is an overview. Organised around really the funders.
The first is the OECD projects. I assume everybody knows what PISA and PIACC are.
Does anybody not know what PISA? Raise your hand if you don't know what PISA and PIAAC are.
Well, we'll talk later. Yeah, right. So.
Then I'm going to talk about funding for adult learning out of the Institute of Education Sciences of the US Department of Education.
That's a major research arm of the US Department of Education, and it's all adult learning.
I know for decades they concentrated on K-12,
but it's only recently that they're branching out to adult learning because there are a lot
of struggling adults who can't keep pace with the skills that are needed in the 21st century.
Then a couple of things from the Department of Defence.
The Department of Defence has really led a lot of the learning environments and
assessment in environments because they deal with adults in the military.
And so they've funded a large number of projects.
And then finally, I want to talk a little bit about the NSF funded Learning Data Institute.
So let me start with the OECD projects.
As you know, they have these assessments throughout the world.
You have a number of countries who consistently buy into this.
Others that may some years, but not others.
And the countries have to pay for it.
The whole concept is, of course, that OECD is really interested in the economies of these countries.
And education is a big part to predict the success of economies.
And. Assessment hopefully will drive the curriculum to improve the education of the people in these countries.
As you know, there are these comparisons amongst countries, but you're not supposed to worry about that.
Instead, it's supposed to be driving countries over time in their planning on education and curriculum.
The project I've been recently involved in is the adaptive problem solving for PIAAC 2021.
Actually it's the data are being collected in 2022 because of the COVID and so.
The key thing with adaptive problem solving is the problems are not static.
Traditionally in problem solving, you have a set of givens and then a goal state,
and then you try to figure out how to solve the problem to get you from the givens state to the goal state.
Right. Well, in adaptive problem solving, the problems change in the middle.
And so you have to change your strategy. So strategy change is an important part.
You have to have better metacognition in order to track what you know and how you deal with it.
And so metacognition is a big part of the assessment.
So they're in the middle of collecting data on this.
So I can't really show you any data. But Samuel Greiff at Luxembourg is the he and his group are leading the charge.
And then Educational Testing Service is playing the big role in collecting the data.
And so maybe at some point I can talk a little bit about that in the future.
So the other thing I wanted to mention is.
A project led by a couple of projects led by Stuart Elliot.
He's been part of OECD. He also is affiliated with the National Academy of Sciences, Education and Medicine in the United States.
But so what are these reports about?
Well, everybody's worried about people not having the right skills in the future.
And they're also worried about A.I. and robotics tech taking over jobs.
And so one of the projects that I was participated in had a bunch of the experts,
11 of us all about AI and robotics are actually take consider a computer taking the test the tests on literacy, numeracy and problem solving.
And imagine how well the computers would do.
And so each of us, you know, tried to make judgements on all the items and justify our decisions.
And if you have a project or a problem like this windmill problem where it's a
problem saying how many windmills would be needed to replace the nuclear reactor?
And so you have to read the text and understand it.
You have to sometimes look at pictures and interpret them.
You have to understand a table that's down below or a table of data, and you have to integrate all that.
in order to interpret the question and answer the question.
And so you had 11 experts do this and they analysed the data.
They divided the experts into the optimists, the realists and the pessimists.
Well, I was categorised as a pessimist.
Now, we recently redid this about a about four months ago, and the same experts and a few more came and analysed the data.
And I'm now a realist.
So I guess there's either more progress, either I'm better understanding AI or AI has changed.
But interesting project and you can read about it.
The other project that Stuart Elliot is leading is.
Trying to figure out what skills. Well, first of all, what are what is a good taxonomy of skills?
And so he got together people in psychology, psychometrics, people certifying people for jobs.
He also AI people in AI human factors.
They got an ensemble of people.
I was fortunate to be asked to be one of them to and there's a book on that reflects on what they think a good taxonomy of skills are.
And in addition to identifying that taxonomy and the skills of making judgements on what can computers do versus people?
What should people be doing as opposed to computers?
And these are judgements by experts. No real hard data, but kind of interesting and a lot of interesting perspectives.
You can download these from the OECD website.
As you know, the website has, I guess, thousands of documents that you can download.
Some of them you get for free, others you have to pay for now.
But it's fascinating because.
Many people have argued that in the future we want to better understand what should the computer do versus humans do?
And there's a lot of counterintuitive findings.
And exploring that is important.
And this is really the first volume.
They're going to have other volumes in the future, because this is a pressing questions question that people have.
A lot of people are terrified that they're going to lose their jobs.
And as you know, there's been a shift more towards jobs requiring reasoning and problem solving and collaboration, collaborative problem solving.
I know Sandra and I were just talking about the collaborative problem solving, and I was involved in that in PISA.
So so what do you want the computer to do as opposed to humans?
So this is just another problem of.
And, you know, just look you look at all the information here.
A computer would have to identify which columns to get the information.
It's asking which muscles would benefit most if you use the gym bench.
And so you have to look under muscles and then you have to you have to drill down on the right cell in order to answer the question.
And there's a lot of distracting information, and you have to handle that.
Okay. Let me go on to the second part with the projects with the Institute of Education Sciences and Adult Learning.
I was fortunate to be part of a large centre grant, the Centre for the Study of Adult Learning.
That was led by Daphne Greenberg at Georgia State University.
She's really one of the gurus who've been trying to help improve adult literacy throughout her career.
Those are the Co-PI’s, and of that project was a five year project.
It's now continuing under the direction of John Sabatini.
John Sabatini has, we're fortunate.
He joined us at University of Memphis. He used to be at Educational Testing Service, and in fact,
he was the PI on the assessment of comprehension skills on a very large $100 billion investment from the Institute of Education Sciences.
And so he knows a lot about assessment.
But he wanted to break away from ETFs in order to do more research on learning and combining learning with assessment.
Both of us believe that you can't really track learning without good assessment and you
can't just have assessment without considering learning and that you see people nodding.
I think you all agree. And so he if we're fortunate, he came in at the University of Memphis.
So. we had an intervention and it was a hybrid intervention to treat.
Reading skills and especially comprehension skills because 40 to 60 million people,
adults in the United States, don't read at a deep enough level to get a decent job.
They don't read at an eighth-grade level, for example. And in fact, we've collected data in colleges with psychometric tests.
And 38% of the students in college don't read at an eighth-grade level.
And so, you know, it's a problem.
And that squares a way, I think, with the PIAAC and PISA of data, too,
so now it's hard to train teachers how to teach comprehension skills, you can get.
They do very well on decoding and vocabulary, but when it comes to comprehension,
it's very variable and too often not science based and or even tangible ways to teach comprehension.
It's hard. So what we did is built a system AutoTutor with these agents.
And so imagine people reading text on a computer and having these agents, having conversations about the text with the person.
And imagine periodically asking conversation based questions, just asking a question and see how they answer.
Well, that's what we did with AutoTutor. Now.
It was not natural language input because writing is hard and typing is hard and often they don't have the digital literacy skills.
And in fact, both John Sabatini and I are convinced you have to combine the digital skill acquisition with reading comprehension.
Both have to go hand in hand. One challenge is if you look at digital skill acquisition on the computer.
Nearly all of it requires you to know how to read. Well, that's the problem you're trying to solve.
So it's a real chicken and egg problem, a real challenge.
And so we're now building digital training skills for people where you don't have to read much, you know?
So they have these agents or we created a lot of lessons.
Imagine actually we have 30 lessons.
We created some of our own words and sentences like you have how they interpret pronouns or non-literal language or of learning new words.
Then you have others on stories and texts and in many different kind of genres and subgenres.
You get the narrative and expository and persuasive texts and compare and contrast, problem, solution, and each of these modules.
It takes 20 minutes to an hour to complete.
And so it's really a 20 to 30 hour intervention that they have.
And so we built this system and and we were hoping not only to help the students learn better,
but also instructors learn better because they're often not exposed to training for comprehension skills.
Remember, in literacy centres, a lot of these people are volunteers from the community.
And in the past the until recently not a lot of investment from the US government to train adults on reading comprehension skills.
And so both instructors can learn and and also struggling adult readers can learn.
So. This was a hybrid intervention with human instructors as well as AutoTutor on the computer.
However, our part was the attitude part.
AutoTutor can store all these data in the cloud.
Whereas humans, you don't know what really what they're doing unless you had a very large budget to videotape them and transcribe it and analyse it.
That would take forever, of course, whereas the computer, it's all in the cloud, you can immediately see it.
And there's many things you can collect.
You can collect reading time, how long it takes to read the text. accuracy in answering these conversation based questions.
The time it takes to answer the question. Any learner initiatives on asking questions or choosing lessons to take.
There's also measures of the students. You can collect background measures and it's all stored there in the database, on the cloud, in the cloud.
We have psychometric tests, three of them on comprehension skills and other skills, and we analyse the lessons.
We use this Coh-Metrix system to scale text on difficulty and many levels of language.
And so. Lots of data.
And so we've been mining it. In fact, next week I go to educational data mining and artificial intelligence in education.
It's held in Durham. First time I've ever will be in Durham.
And and so we'll be presenting some of our stuff there.
We did a study and with about 253 people, these came out of Toronto and Atlanta,
and these were struggling adult readers below the eighth-grade level.
Typically, they read between the third and fourth grade level.
Okay. So we analysed the data and with AutoTutor and we found by analysing the pattern
of reading times and all of the time to answer questions and things like that.
We did some data, we did some clustering analysis and we found four types.
You have the higher performers. These are relatively that answer questions quickly and are pretty correct.
And then you have conscientious and they're real beneficiaries and looking at effect size and some post-test minus pre-test.
So you really want to leave them alone to do whatever they're doing.
Then you have struggling readers. It's beyond the zone of proximal development for them.
And our intervention was not working on them at all.
You got to do something different. You got to scrap it and do something different.
And then under engaged. Some people try to quickly do things and so they underachieve.
And so you might ask them to have a deeper read, things like that.
So what we're trying to do is mine the data now so we can have a recommender system in order to guide the learner on what to do next.
Let me shift now to the Department of Defence Projects.
Once again, the DOD, the Department of Defence,
US Department of Defence has taken the lead over the last 30 years on building these intelligent tutoring systems.
And other advanced learning environments, including virtual reality and augmented reality.
And and concentrating on adults. And so we've been involved in projects with them in the last 30 years.
Let me just mention a couple of.
One is, during the last decade, we have been developing this generalised intelligence framework for tutoring.
It's called GIFT, for short.
And each year we gather a bunch of experts on some aspect of building these systems.
And we get together and we write a book, there's a presentation, and we write a book and you can get the book for free.
If you go to gifttutoring.org
And it's really a good snapshot if you want to find out what the latest is on intelligent tutoring systems, and that involves team training, too.
You can go to this site.
Over 300 experts throughout the world have contributed to the GIFT or expert workshops and represent all sorts of advanced learning environments,
not just intelligent tutoring systems. And so the first year, their focus was on learning or modelling.
Then on instructional strategies, then on authoring tools, then domain knowledge, then assessment.
The assessment what one was held at Educational Testing Service, then one on teams.
We had people involved in the collaborative problem solving there, then self-improving systems.
These are systems where the computer improves during the evolution of building a system and as well as learners improve and educators improve,
it's kind of a co-involving learning on all parts then data visualisation and recently competency-based scenario design.
So you're free to go there.
This was a team team up between University of Memphis and the Army, and so we're real happy with that community of people building the systems.
Traditionally, these systems have been expensive to build, very expensive.
So part of the goal is to make them be created faster, cheaper, things like that.
Now you might ask, why did they team with Memphis?
That's because we're known in our Institute for Intelligent Systems for in building a lot of these systems with intelligent agents,
these conversational agents. Some of them, they hold conversation in natural language.
So you try to interpret what people say in natural language and respond to get them to productively learn.
Now, I will say this the computer does not perfectly understand the human. Most humans don't perfectly understand other humans.
So you want to compare them. But so.
But they the agents can have dialogue moves to help people learn.
That's the idea.
One of the I'm going to talk about electronics tutor briefly, but the other one we like is the personal assistant for lifelong learning.
Imagine if you had an agent that followed you throughout your career and collect all these information.
It's almost like a learning portfolio, a digital learning portfolio.
Imagine that. And then it comes up with recommendations.
And the idea is, if you have to be an expert in a topic, it will give you a problem of the day.
And that problem tries to patch misconceptions who might have had or advance your skill.
So imagine a 20 minute a day refresher.
So you stay current as an expert in your field.
And that helps people prevent skill decay.
That's a big problem, especially if you have a class where they throw all this information at you for 40 hours,
distributed over whatever period of time. And if you look at people six months later, a lot of it is forgotten.
So the goal is to have some activity to keep up your skills.
And we actually prevented skill decay in an area of electronics, which is very important in the Department of Defence.
This is one of the projects I led and it's still being used.
It's actually ElectronixTutor is being used in the nuclear power command training for people who want to go into the Navy in that area.
That's not nuclear bombs, by the way. It's nuclear power to energise the ships and so of lots of institutions.
University of Southern California, Memphis was the lead.
And then we had people from Arizona State University, WPI, we had industry involved from Raytheon and so on.
This is the most co-authors I've ever had on an article because I wanted everybody included.
And you've probably heard physics have 500 authors on some paper.
I felt I was drifting towards that that issue.
But. Imagine.
A really a federation of intelligent tutoring systems and learning environments.
Imagine just reading texts like the Navy documents or AutoTutor conversational.
And then you have learn form doing reasoning with mathematics or assessments that trains people on like Kirchhoff law, very mathematical in Ohm's Law.
And then you have dragoon a deep sort of mental models on electronics, very deep simulation sort of stuff.
So it's a collection of intelligent tutoring systems, all integrated in one environment.
And you track all the activities stored in the cloud.
And and you kind of see hopefully the long-term vision is to see what what people will learn and what environments do they choose to use.
Do they want a visual or simulation?
Of course, we already know people try to avoid hard work and so they often don't wisely choose things if they do it in a self-regulated way.
But. You can have a recommender system to guide them and that may help them stay on a good path.
You might ask. Well, you collect a lot of measures, including affect, your can track emotions and engagement.
Excuse me. The hot weather is getting to me. You can track how much initiative they take.
You can see if they follow your recommendations.
And all of that is stored in the cloud. I just want to make one point here.
A lot of what we do involves natural language interpretation.
And what we do is see experts how much they agree.
And then we see how much the computer agrees with the experts.
And then what we do is we see how how much can the computer analysis of natural language mimic an expert?
And we find we're almost there. Whether you take a stringent criteria on interpreting the things or lenient, we're almost there.
And this is in electronics. We've found it in other topics as well.
So really assessing natural language and how well it kind of compares to expected information.
Really, the field has advanced being pretty good.
The last thing is the National Science Foundation Learning Data Institute, and we've been working on this during the last three years.
The National Science Foundation is in the middle of building these larger centres, usually for about 20 to $25 million for a five-year period.
And they have to involve many institutions.
And we've been involved in a beginning part of one of these and we have a lot of groups that have to kind of.
Collaborate and analyse data. As you know, data comes in many forms and formats and it's distributed all over the place.
You need good data science on how to cobble together all those separate databases,
and then you need to figure out how to apply advanced quantitative methods to analyse the data,
whether it's machine learning or educational data, mining methods, A.I., whatever they are,
sophisticated psychometrics, whatever it is, and to get more people involved as a community doing this.
So. At this point, I'm involved with all these large groups of people leading, none of them.
And at some point, I will be transitioning from transitioning to a full retirement.
And I just don't know when that will be. My wife does have her views on that.
Well, thank you so much. The next speaker, which will give some reflection on this topic, is Enterprise Professor Sandra Milligan,
who is Director of the Assessment Research Centre at the Melbourne Graduate School of Education, University of Melbourne.
And Sandra has an unusually wide engagement with education, industry and in education and research.
Originally a teacher of science and mathematics, she's also a former Director of curriculum in an Australian state education department and has
held senior research management and governance positions in a range of educational organisations,
including government agencies, not for profits, small start-up businesses and large listed international corporations.
Sandra's current research interests focus on assessment, recognition and warranting of hard to assess learning.
She directs several research partnerships with school networks and organisations working to develop learner profiles for their students.
She is lead author of the Future Proofing Australian Students with new credential report, outlining methods to reliably,
assess and recognise the level of attainment with general capabilities and a recognition of learning success
for all, ensuring trust and utility and a new approach to recognition of learning in senior secondary education in Australia.
So Sandra, we're pleased to welcome you.
The floor is yours. And look, first of all, can I just say it is a huge honour to be here.
I just love being here in Oxford and at the Centre for Educational Assessment.
It is a really well-known organisation globally but also in Australia.
So my associations with Therese and Josh and before that with David Andrich, who is a great mentor of mine,
has made me well know he may not claim to have mentored me, but I claim that he mentored me so he might not want to take the blame.
But so I've long had in my imagination this centre and to be here today celebrating, I'm very honoured.
So thank you very much. Actually, I wanted to take off with a final question to Art finished,
and that is to do with the real importance of aligning what we value with what we teach, with what we want, learnt, with what we assess.
So I'm going to be talking about that today and how we might do that in the context of the work that is
assisting me and possibly other Australians in the assessment and certification in senior secondary.
Now I've been watching over and I've been watching the papers on the post COVID and the COVID fallout on the A-levels.
You know, do we need these exams? Should we have exams?
Do they do the right thing? Is that what we should be doing?
Those issues are being mirrored everywhere, at least in the Anglosphere, certainly in Australia.
In the Australian context, people, including my colleagues at the University of Melbourne,
who we say is the Oxford of Australia or possibly the Harvard, depending on which country we're talking.
So my colleagues who run the medical schools, the engineering schools,
what they're saying is the senior secondary examination and certification system is not doing its work.
It's not giving us the people who are going to make the great doctors who will love to be engineers.
It's giving us the high scores on examinations, and that's not who we need.
Some of the people who are high scorers are not going to make the great doctors and
some people who are low scorers could be brilliant engineers if given the chance,
etc. In addition, in our senior secondary schools, we still can't get 100% of people finishing.
Still 15 to 20% of students don't finish.
That's terrible. Of those who do finish, a fair few of them truant.
Don't turn up, don't care.
Others sit conscientiously in the class, sending themselves into stress spirals and asking themselves, is this the pinnacle?
Of learning that I'm doing in this thing.
Do you see what I mean? Now, I'm not saying Australian education is terrible.
I think it's great. But what I am saying is that there's room for improvement.
And the general consensus in Australia is that there is room for improvement.
We've had a number of reports over the last ten years or so.
Those from Australia will recognise people like to read the Gonski reports, the Shergold reports, the review of the AQF.
They're all saying we need to do something different, guys.
And one of the reasons we don't is because of the assessment system.
So I agree with Art that assessment can be an absolute killer of really good education, but it can also be the thing that can shift things.
So it's better. So I want to tell you a little bit about the work that we're doing in senior
secondary education to shift to what we value so that we get full engagement,
enthusiastic engagement of all students in senior secondary.
That's the goal. Just say something about the basis on which I'm going to make some comments, and that is Therese said, I'm an Enterprise Professor.
I don't know whether you know what that means because I head an Enterprise Unit.
What it actually means is that I have to get my own money.
The university doesn't support us in that regard.
In fact, we pay a tax really to be part of the university.
I think that's possibly the way university research centres are going.
And I am measured not only on revenue but on impact.
So I have to show that the work in our centre is actually making a difference.
The way we do that is that we problem solve for industry folk and most particularly in the area of assessment.
We're very fortunate because we get people who are passionate about education,
who hate the assessment system, who in who believes that there’s a better system.
They come to us and say, will you help us reform assessment so that we can meet our goals?
There are I call them first movers.
So we are totally privileged to work with dedicated people who want to shift the system, and it's our job to do the R&D for them.
What a pleasure. So, okay, what R & D are we doing?
I'm going to go pretty quickly through this.
And I'm assuming that these these slides can be distributed and if there's anything else you want to follow up on those and happy to chat.
But this is it. This is what our first movers over the last in senior secondary have been telling us over the last decade is the objective.
They want to get beyond just knowledge.
They want to get to the point where students have agency in what they're doing.
In other words, they learn what they think is important.
They want to get beyond knowledge to competence so that learners not only know, but know how to do things.
This is a big shift. This is a big shift. And they.
So that's what they want. That little sort of what is it?
Pentagon of hexagons. They're the categories of learning ambition that we've abstracted from our first movers.
And you can see, I don't know if, no I haven't got a pointer.
You can see that two of those on the left hand side, a fairly normal knowledge and know how.
That's the curriculum as we know it. And basic literacies, that's the curriculum as we know it.
But they're wanting to add those other three bits.
They want to teach learners to manage their own learning, not just be drilled to death and coached to the nth degree to pass the exams.
They want to teach students how to connect to sustaining communities that they will need to be part of to thrive.
Thrive is the key word. So that's communities.
Of scholarship communities of work, communities of culture, communities of community,
so that I want to teach students how to embed themselves in those sustaining communities.
And I want to give them the learning stable.
So 21st century skills or general capabilities, because that's the skill set that enables students to build depth and breadth in their learning.
That's what they want to do. And they come to us and say, Well, that's what we want to do.
Can you fix the assessment system so that we can do it?
All right. The first step, of course, is to actually define these learning ambitions in ways that they're teachable, learnable and assessable.
It's very easy for people to drift off into things that aren't really curriculum objectives.
For instance, I mean, I'll probably make enemies by saying this, but if you take the issue of student well-being,
I mean, all sorts of things can cause a student to have very poor sense of self-esteem or well-being at any given time.
It's unlikely that a school can guarantee
well-being of every student.
But they can guarantee that they are teaching the skills that students need to that they will need that will enable them to thrive.
So we spend a lot of time trying to focus people on defining what it is they’re actually wanting to teach that students can learn and that is assessable.
So it's important that this is a curriculum based, not a generalised happiness sort of thing.
Do you see what I mean? I'm not expressing that very well.
For instance, when it comes to agency and learning, I won't go through this, we’ve worked through.
What is that? What are the skill features that you need if you're going to actually establish a learning environment that will give students agency?
So we've done a detailed analysis of many of these skills so that we understand what they are as objects of teaching and learning.
I've just put this quickly up here. I probably will deny having shown you this, because this is a framework that the Federal,
our Commonwealth Government is currently consulting on to be the basic framework for post post-school post-secondary education,
post compulsory senior secondary education, the national framework that will capture what it is those curriculum elements are.
I mean, that's a very interesting document. It's not policy yet,
but I'm hoping this or something like it will soon be policy in Australia and it
says these are the skills that you need as well as the knowledge and the know-how.
If you're a school leaver. At that point, most teachers freak out.
and they say, Oh, well, it's all very well to have a nice framework, but how do you assess those things?
And so we've put a lot of effort into getting teachers to the point where they feel confident and capable in assessing these things.
And you know, this is my favourite sort of picture from Dreyfus.
You might remember the Dreyfus taxonomy of competence.
And so what we've discovered is that for each of these general capabilities or 21st century skills,
you can define a progression with five or six big leaps in the constellation of skills that people have.
And so the challenge for teachers is just to understand those big qualitative leaps,
incompetence that students go through and then learn what to look for to place them on those positions in our final ranking in Australia.
We've gotten a thousand-point scale.
It's called the ATAR, on which every senior secondary graduate is placed.
If you don't believe me, ask the other Australians in the room.
And so when we say hmm, we think that there are five levels, not a thousand levels.
People think we're being radical okay. It's sort of interesting.
This is an example of one of those progressions that show that we've used.
This was one we used in the Philippines.
It's one on collaboration. You might. It's sort of like collaborative problem solving.
But the key thing for a teacher is they recognise those levels.
They recognise that at the lowest level, a student needs to be told what to do with others.
And then in the middle, they get quite used to the idea that they're working with a group and now fit in and and go with the flow. At the top level.
They're the people who organise the environment, get, engage and motivate others to collaborate.
And once you explain that and, and explain the behavioural indicators that go with each of those,
we find that teachers fairly soon relax and say, I can do that.
And this sort of fear factor disappears.
To assess it though, you cannot use examinations.
These are not cognitive skills. You cannot use multiple choice questions.
You can't really use stuff that's written down. It is needs to be performance based, multiple raters, multiple performances,
and so that there can be a human based judgement of the level of attainment against the of the performance of a student.
So getting that organised in a school is a non-trivial task.
I'm always forever happy to talk about this ad nauseum, but I'm going to go pretty quickly here.
Now, this is the coup de grâce.
This is what a senior secondary certificate currently is, and this exists in Australia to capture the attainments of a student.
What you will notice about it, first of all, it's a digital document.
It's got links all through it. So you can see that there are links to statements by Abbie.
You can see that it's got links to her portfolio where you can see all the authentic performances that she's performed.
The chrysanthemum in the middle. We call it a chrysanthemum because no one knows how to spell that.
So it makes sense. Right? But it's a rose graph for those who are technically minded.
We call it a chrysanthemum, and that is a standards based.
highly psychometrically, reliable, judgement-based assessment of the level of attainment of Abbie on the five areas that she was assessed on,
which were quantitative reasoning, knowing how to learn, communication and whatever.
And this has been. This is currently being used in 20 schools in Australia and a number in the US and has now been accepted
by 50% of our Australian universities as the legitimate alternative to the senior secondary assessments.
The kids love it. Everyone loves the chrysanthemum.
But most importantly you can use the metrics underpinning the chrysanthemum to do the sorts of things
that would enable matching of student attainment to opportunities that exist like you might get 99 in ATAR.
But if you've got no empathy, there's no point in going into medicine, etc.
No, that's probably not true, but never mind.
So we're currently working with one of the biggest university selector groups, UAC, University Admissions Centre,
working out what sorts of uses this could be made in universal secondary selection.
This is really interesting stuff, really interesting stuff.
And it's not just the big picture one.
We're involved in a range of new credentialing projects where first movers in Australia are being given support by the University of Melbourne.
University of Melbourne is actually underwriting the quality of the credentials so that we can move this along a bit further.
I'm going to finish now, but I would just say this.
From the point of view of being the Director of the Assessment Research Centre and all those who travel in her, in the centre.
I, we see this sort of activity as a fundamental I call it D an R development and research activity.
It's not traditional research. It's actually trying to solve a problem, coming up with a solution, working out the science that's underneath it,
and then packing the research evidence around it and altering it where it doesn't stack up.
These are the kinds of empirical questions that we're working on with our first
movers and with the very interested observation of the policy people in Australia.
And I'm hopeful that fairly soon we'll see learnt profiles based on things that we value that will help students thrive,
will become the lingua franca of the senior secondary certification system to help us match people to opportunities.
And it will be brilliant. So there you go. Thank you very much.
Thank you so much. That was extremely fascinating.
So now I think we need to listen to a little bit about evaluating a method for predicting the compact
comparability of International Baccalaureate cross-lingual assessments by combining psychometrics,
machine learning and natural language processing. And Josh McGrane,
our Deputy Director of the Oxford University's Centre of Educational Assessment and Senior Research Fellow and Fellow at Kellogg College.
He completed his university medal winning Ph.D. in quantitative psychology at the University of Sydney.
He has also been a post-doctoral fellow at the University of Western Australia and worked as a psychometrician
for the Centre for Education Statistics and Evaluation in the New South Wales Department of Education.
As a result, he has extensive experience in academic and government contexts in education assessment,
including being an expert adviser for several national and international government and non-profit organisations.
His research and teaching interests span the philosophical and political and statistical aspects of psychometrics.
He's presently working on several projects, including the development of a better framework for measurement across the physical and social sciences,
combining psychometrics and AI modelling to predict and explain bias and education assessments,
educating critical thinking amongst secondary students internationally and the development of language assessments for understudied languages.
So, Josh, the floor is yours. Thank you, Therese.
So I'm now very much convinced that the weather gods of Great Britain hate me and
they are vengeful because I've now lived for going on six years and most days,
just as the people I've worked with have complained about the weather and what they've done,
they've turned around on the one day when it's supposed to be our last hurrah and celebration and
given me a day of Australian weather so that half of the British people didn't show up today.
So thank you for your vengeful irony in that regard.
Now I just want to say thank you to the two speakers that have come before me.
It's both been wonderful talks and funnily enough, a lot of overlap with what I'm about to say in sort of nuanced ways.
But I will say that we're going from sort of big thinking, big picture ideas to a very,
very small-scale research study that really is only getting underway now.
In fact, I'd say we're about two metres into the 100 metre sprint and we probably stumbled a little bit out of the blocks.
Okay, so spoiler alert, the findings aren't that amazing, but I wanted to talk about something more.
We're talking about the future of assessment, the future of research.
Well, often in these kinds of talks, you get people basically reflecting on things they've been doing for the last ten years.
That makes it seem also simple and neat and linear, and it all goes together.
Well, I can assure you, when you're on the ground actually taking risks,
having no idea whether these small but big ideas are going to work out that it's not that neat.
And I think that's an important lesson for the students in the room to learn in particular.
And on that note, I just want to thank people who have made the effort to come today despite the heat.
So, Jonathan Michie, the president of Kellogg College, thank you for coming along.
I see several of my colleagues in the room as well, so thank you for making the effort as well our centre,
both present and former and friends and colleagues and what have you.
So thank you to everyone for coming along and listening to me talk about this rather nerdy stuff that you're about to hear.
So. What we just to give you some context.
We were contracted by the International Baccalaureate Organisation to do a study looking at whether their translation
processes were working well in their Diploma Programme examinations and specifically within their science programme.
So interestingly, the Diploma Programme qualification I see has a lot of overlaps with what you were showing
that the Australian Federal Department is considering putting in as actual policy now.
So I hope that those two bodies are in a lot of conversation with one another.
But one unique aspect of the International Baccalaureate that isn't present in
many national systems is the fact that you have to make your assessments valid,
comparable, unbiased across several languages, in this case as many languages.
But in this particular research we would be looking at three of them.
And so normally when asking this sort of question,
Are examinations comparable across across languages will be doing the typical things we do within assessment we’ll first and
foremost we’ll be having a look at the translation processes and evaluating whether they meet best practise and what have you.
Also, we might have actual human experts look at the different exam scripts and make judgements about whether
they would predict based on differences and nuanced differences in the language that they see,
whether you might predict there to be differences in the performance of the items across those languages.
Now, as part of this broader research project, we did look at both of those questions.
Yasmine, sitting in the crowd here in particular.
did, but I'm not really going to be touching on that at all, just given the amount of time that I have available to me.
And so what I am going to be focussed on focussing on is the part of the research study that I was driving,
which was looking at can you actually use AI essentially to predict differences
in performances of items across multiple languages for high stakes assessment?
Now that's an awful mouthful to get through as a title, so I thought I better come up with another one.
And for those of you familiar with the work of Stanley Kubrick and Dr. Strangelove,
an alternative title would be how I learnt to stop worrying and love statistics (and AI).
And now, of course, I have my tongue firmly in my cheek here because we have a quote from Borsboom,
from the University of Amsterdam and one of his colleagues where he refers to the psychological test or now context.
We might think of it more broadly as the educational assessment as being the atomic bomb.
Now, interestingly, we had atomic bombs come up in one of the earlier speeches.
Now, clearly, he's using a degree of hyperbole there.
But what he's really getting at is that this is the one thing that assessment is giving the world both
psychological and an educational assessment that has actually radically changed global power structures.
And so moving into the future of assessment, me as a psychometricia,
I know how much power I am often asked to imbue in assessments that magically those magical terms of is it valid and reliable?
Okay, I put it through my statistical models, spit out the answers at the other end and say, yes, it's kind of valid, it's kind of reliable.
And then you say it's valid, reliable, and go forth and change people's lives with it.
So for me, I just want to make clear that although I'm mashing up a lot of contentious areas here,
psychometrics in itself can be quite contentious and reductive,
along with machine learning and all the perils of A.I. that I'm sure most of us in the room are aware of,
that it is done with consideration of the actual ethical implications involved.
I hope that this is a fairly benign application, but I'm happy to hear feedback on that.
But I would just take a quick segue from the actual topic, because I think it is important,
given that this is the last annual lecture that will be hosted by Therese and I that I do acknowledge the host herself.
And first and foremost, I should have Sam's photo up here as well because they both organised the event today.
So thank you on that behalf.
But Therese is someone who spends a lot of time acknowledging others and very much diverts acknowledgement away from herself.
So I am probably embarrassing the crap out of her right now, but that's okay.
It's something that I want to do, Therese has spoken about the achievements that
we've achieved together and collectively as a centre over the last five years.
And I think it's a testament to both us as a team, but particularly her as a leader,
that a lot of that has happened in the last two and a half years and the last two and a half years have been very difficult.
I'm not sure if anyone has noticed, you know, on several, several levels.
And so I just wanted to say to you Therese, you brought up the topics of compassion and integrity,
well I could not think of a leader who embodies those two principles more than you do.
So it's my pleasure to have you as a leader. My pleasure to have you as a colleague.
And my pleasure to have you as a friend. So thank you.
Also, just as it takes a whole village to raise a child, it takes a whole research team to do a set of involved research projects like this.
And so I just wanted to give a shout out here to Yasmine El-Masri, who's in the crowd today.
As I said, she actually conducted a large part of this broader research study that I'm not going to be talking about today.
And also, you know, it's her work and her connection to Art Graesser and the work of Coh-Metrix, which is part of the kernel of the idea in the first place.
So I just wanted to acknowledge that too. Kit Double, another former post-doc of us who's fortunate enough to already be back in Australia,
who, you know, did a lot of the dogsbody work of having to code and clean and what have you, which is like the unsexy part of research,
but it's extremely important and people don't really acknowledge just how important it is often enough.
Heather Kayton, who basically has been my right hand person, I guess I should say, throughout this particular aspect of the project in particular.
And I'm really thankful to the support and also the expertise given her background in applied linguistics,
helping me to really understand what all this natural language process, blah blah actually means in terms of real language stuff.
So she has really, really helped me along the way and I hope that you'll continue with this.
And then finally, Rebecca Hamer, who was the project coordinator from the IB side of things.
She's based in The Hague.
And I must admit, she did bust my balls a lot throughout this project, but sometimes that is required from a project manager.
But she also helped an awful lot too, in terms of providing data, processing it and providing support by way of expert judges from the IB as well.
So I just shout out to her because I do appreciate the input that she had to the project.
Now really returning to the actual substance of the talk now and to just to create a little bit of synchronicity with Sandra.
One last acknowledgement is a mentor of mine, and I think he would actually acknowledge that I was mentored by him, or at least I hope so.
He was a reference to my job interview, so I guess he said nice things.
Where is this line of research for me is coming from is David Andrich’s view of Rasch’s measurement theory and that's the
idea that we have these psychometric models that we apply because they have good and desirable measurement principles.
And when we find problems, what he often refers to as anomalies,
it's not an indication that we should go and get a different, fancier, better, more uninterpretable model.
But rather what we should do is take that as a lesson that perhaps there's something wrong with the assessment itself,
whether that be in the administration or the substance of it.
And so this is where I'm coming from here. Okay.
We've had 1,000,001 studies in the world about multilingual assessments showing cross-lingual assessments.
Sorry, Heather, I’m make you cringe in saying multilingual, cross-lingual assessments showing differential item functioning.
But why? We need to predict and explain so that we can actually take these signs of anomalies and improve the assessments going forward.
So that's the general aim of this particular study.
I'm going to have to hurry up a little bit.
But essentially what we needed to do is basically test out as a first stepping point, an innovative and I'd say, you know,
arguably one of the best aspects of it is that it's scalable,
doesn't rely on you having to go back to humans and then making sense of things and whatnot.
So if it works, it's a scalable approach to predicting language effects,
i.e. these anomalies on item functionally across different language versions of the high stakes IB Diploma Programme Examinations.
For those of you who don't know, the Diploma Programme is actually the end of secondary schooling qualification for the IB.
So the thing that determines whether you go to university or on to other relevant post-secondary pursuits and I, you know,
we have been very fortunate to have the IB as a research funder and partner because they have been open to
us doing more of this sort of blue-sky research in the context of also doing practical things for them,
like reviewing translation processes and what have you.
And so, what we've been what I propose to do is to basically mash up psychometrics,
machine learning and natural language processing to understand the extent to which the
differences in item performance are related to differences in the textual complexity
of the audience across languages and to determine which particular aspects of textual
complexity are the most predictive of any cross language differences that we find.
So in order to understand the rest of the talk, you're just going to have to understand psychometrics,
machine learning and natural language processing. Okay. No, just joking.
Just follow the gist of it. In terms of what data we were dealing with.
Pretty complex stuff when you started breaking it all down,
I must admit we went into the contract not quite realising just how complex what they were asking was, but there's another lesson for future in that.
So we're dealing with 90,000 students.
Most of them responded to the exam in English,
but we have a fair proportion who responded to it in Spanish and a much smaller number who responded to it in French.
What are we talking about here? We're talking about 12 exam papers, Biology, Chemistry and Physics.
So subjects across two calendar years that have these levels of subjects as well as at a higher level than the standard level.
And they actually have three papers within each. But we just looked at the first two because Paper 3 involves optionality,
which just creates even more complexity to deal with Paper 1 being a multiple choice exam and Paper 2 being constructed response.
So all in all, we have 495 items in total across these different factors.
So in order to build a predictive, predictive model, what do we have to do?
So we use the machine learning approach to identify optimal sets of features that predict the outcome variable,
i.e. whether there was differential item functioning or not. I realise I haven't defined that yet, but I will in a minute.
So, this is done using the ‘caret’ package in R. Now, this is just an incredible package.
It still blows my mind with the open-source software, how people create this stuff and distribute it like free of charge.
I recommend any student out there, even if you're just doing a basic linear regression or logistic regression to look
at this package because it allows you to apply what's referred to as cross validation,
which I will get to in a second, which is so rarely done in our field.
And, you know, the data scientist in the crowd is nodding his head.
It's very necessary for us to start implementing that as standard practise in our field.
And so to do this, it's first necessary to establish the outcome variable,
which is the DIF estimates from the TAM package in as well as the predictive variables.
And so where we got this from was an open source software called READERBENCH.
And also we included the test and item characteristics in there as predictive variables as well.
So detecting the DIF, basically differential item functioning is when an item displays differences in difficulty.
Across different groups, we would estimate as having the same underlying ability,
which means that they have a different probability of getting a correct response for that item.
So between groups, between, say, English responders and French responders,
it may well be that there's on average at different level of attainment on a particular item.
So where DIF is different to that, it's not average performance.
It's if we take, say, the French and English speakers who we believe have the same underlying ability level,
is the item differentially difficult for them?
And the particular model that we applied here is the random coefficient multinomial logit Rasch model.
Now you don't need to understand what that is, other than to know that this is a parameterization of the Rasch model.
That's much more like a multilevel model. The reason why that matters is because DIF can be understood in the parameterization of
the model this way as being an interaction between the item difficulty parameter.
And then you can add a group for group parameter into the model.
In addition to that, you can also add in covariance. And this is important because so often when people evaluate DIF,
it's just like this group factor is an analysis followed by this group factor as an analysis.
And the fact is that the language of the test interacts with all sorts of other different group factors.
So, for example. Are they responding to in that language which just so happens to be their native language?
Is it their language at home? Okay.
That would be one key covariate that's going to potentially show up as being language DIF when it's actually nothing to do with language DIF potentially.
And so we didn't really have the sheer amount of data required to delve into that too far.
But we did include some covariates like that, for example, whether the test languages matches the home language as well as gender,
as well as a couple of other factors to try to purify it, so to speak.
The DIF estimate that we had based on language. And so this gives us DIF as both a continuous, standardised outcome,
but it can also be categorised into basically an effect size measurement as well as small, negligible DIF, moderate and large.
And so I really don't want you to glaze over trying to read all of this because I appreciate it's too much information,
but from here on is the DIF that basically is considered to matter.
And you can see here that by and large, the percentages across the different subjects,
across the different levels, across the different languages are pretty small.
So this is a good news story for theIB, basically, that it seems that they are.
And then I'll show you for 2019 as well, it's probably even better.
So by and large, it looks like they're doing a pretty good job of the translation.
It's a bad news story for us, however, because, you know,
machine learning wants lots of observations of different kinds of categories and we don't have them here.
So, I'll come back to that at the end. So generally, the findings were very positive for the IB DP Science examinations.
And as I pointed out, unfortunately, because there are so few items showing moderate and large
DIF, we do have this highly imbalanced data set for later class predictions.
So, the second step is to establish textual complexity.
So, we've got the outcome variable. Now we need the predictive variables.
Now the way that we did that, as I said earlier, was using the textual analysis software, READERBENCH.
Any of you who know about NLP and the prediction of text complexity know that
there's a strong overlap between READERBENCH and what's offered in Art.
Graesser and his colleagues product, Coh-Metrix.
It's just that READERBENCH has been generalised to more languages than Coh-Metrix has and it's also completely open source as well.
So, it was helpful for us in that regard.
And so, I'm not going to go into too much of the technical detail here,
but basically pre-process, pre-process is the text using fairly standard core NLP procedures,
basically, tokenising the bits of the text tagging, the kinds of words, etc. fronted adverbials, and what not.
Michael Gove joke in there and dependency passing,
which is to do with grammatical structures, and it also applies language specific resources as well.
So language specific corpora, the actual body of text that it's drawing upon and the lexical ontologies and semantic models
which have to do with the sort of semantic modelling of relationships of the text as well.
So it provides you with a wide array of indices over 300 for understanding the textual complexity in multiple languages.
And, you know, I think this is still an open empirical question,
but at least they argue in their literature that it provides information that can be compared across languages.
The five categories in particular surface syntax, morphology, word, cohesion.
I'm not going to go into defining all of those in great detail, but we can talk about it in the end, if you like.
And this is how we got to our final set, basically.
It's very standard practise in machine learning.
You get rid of any indicators that don't have any variance because they're not going to predict anything for you.
We also got rid of any variables that showed more than a sort of trivial amount of missing as well.
And then we ended up with 151 indices at the end.
So in terms of what the outcome variable is, if you can picture it.
So for each of the languages, we have text complexity indices for each of the items.
So we have it in English, French and Spanish.
Now we just did this in a kind of brute force, slightly dumb way, where we just literally got all the text for a question,
bundled it together and ran the analysis on it, which again is a point I'll come back to at the end.
But it's a starting point. We have to start somewhere. And so then the outcome variable then becomes the difference between the complexity index,
for English versus French, and the difference between the complexity index for English versus Spanish.
Okay. Because we have a DIF estimate for both of those pairings.
English being the reference group in Spanish and French being the two focal groups.
So this is just to show you that across the different categories there was still
the reference in all the various indices represented the various categories quite well.
So as I said before, a machine learning model is designed to optimise optimal sets of features to predict an outcome variable.
How do you do that? How do you build the model? It's based on evaluation of different criteria.
So basically, whether this lower prediction error rather or greater explanatory power.
And in particular, I'd say what differentiates, say,
machine learning or data science from just normal applications of statistics is that there's a lot of emphasis placed
on cross-validation to avoid the overfitting of the data and to enhance the generalisability of the findings.
And so if anyone ever tries to tell you that machine learning is something different to statistics, tell them they're lying,
but just say, you know, there's a nuance in that, that there's a much stronger emphasis on this topic of cross-validation.
So how do we cross validate things? Well, first and foremost, we have a training and test sample.
So we're going to build a model that makes a certain set of predictions that's going to be based on the training set of data.
Then we get the final model and then we apply it to a test set of data.
The model has never seen before in order to get a unbiased view of whether the model performs well or not.
And when we're training the data itself, we do something very similar,
although a little bit more convoluted by way of what's referred to as K-fold cross-validation.
Dividing the training data up into, in this case, five folds each time you train it on four of the folds test it on the face.
Right? You iterate that around and then you get the final averages that gives you the model performance.
So. We’re using those 150 indices or the different indices, the five categorical test characteristics.
And we were looking at 999 standardised DIF estimates, 495 across the two languages.
What we were using, we used both random forests regression.
So this was for when DIF was a continuous outcome and also ordinal random forest regression for DIF as an audit category outcome.
Remember what I said the small, medium, large.
Now I don't think I'm going to have time to teach you what random forest regression is, but it is actually a pretty cool idea, to be honest.
It's referred to as an ensemble learning approach.
And the reason why is that what we have here,
we have different decision trees and these are set up such that they are necessarily going to be different solutions to the problem,
which then essentially get averaged at the end to give you a prediction.
The great thing about it, it's nonparametric, it's non-linear, it's able to model very complex interactions.
It can easily deal with both categorical and continuous predictors and it's robust against multicollinearity.
The sorts of problems we routinely deal with in our world,
especially when you've got 151 different text indices that are all only very subtly different from one another,
and it's commonly used in machine learning circles, and it's somewhat interpretable.
So I was going to teach you what a decision tree was, but let’s see if this works.
So essentially what you're doing is you’re recursively partitioning the data in terms of the predictor
variables so that you can purify the prediction in terms of the dependent variable as quickly as possible.
So in this case, you've got two predictive variables, X and Y right? at the top there.
It's basically looking for the partition.
It can introduce where at the next node, the groups are as purified as possible.
So let's say we want to predict whether it's a cat or a dog based on two predictor variables you've got.
Does it have pointy years or it's white and all?
Basically the algorithm will cycle through different levels of these predictive variables to essentially
give you the initial split that will purify your beginning set of cats and dogs into something that's purer.
So this one has more cats and this one has more dogs and so on and so forth.
And it stops doing that when it gets to a point where any further bifurcations don't add any information to the process.
And so why Random Forest? Well, how many trees are you going to have?
So we actually just fixed it to have a thousand trees. What it does that is that it creates new samples.
So it takes the original sample and it creates what's referred to as a bootstrapped sample.
So it randomly samples from the original sample with replacement.
So the rose from the original sample might appear more than once in the bootstrap sample, and it will do that a thousand times with a thousand trees.
So therefore every tree is unique. And then the second thing it does is it has a different number of predicted variables for each tree.
And so how do you determine those numbers?
Well, basically, again, you just brute force it and you say try every value and tell me which one works the best.
So that's what we did. The results? As I said, they're not particularly compelling, but they're also not nothing.
And to be honest, at the beginning of this research project, we were a little bit worried that this would just show up, nothing.
And so you see here in the training set with the K-fold cross-validation, the R-squared ends up being about 0.12.
And interestingly, when we then apply that to the test set, it's more like 0.20.
So normally what you're looking for between training and test is that that number for test, isn’t too much lower than for the training
Something strange is going on here where the test performance is actually much better than the training performance.
So somewhere between 12 and 20% of that variance in that continuous DIF outcome variable is being predicted by this model.
I don't, again, don't want you to read all of this other than to pay attention to the fact that
remember the sort of basic characteristics of the items that were in the model.
They don't appear here. All of these are NLP based predictors and then how does it determine whether something is important or not?
Essentially, it pulls it out of all the models and says, how much of the explained variance do we lose based on pulling out that one predictor?
In terms of predicting the ordinal outcome.
Again, not great, but not nothing. So we again have this improved performance of the model in the test data as opposed to the training data.
Somewhere between 34 and 40% of the language DIF effect size categories were predicted.
But remember, you would predict a successful prediction just based on chance alone, which is where the Kappa comes in.
And by any sort of conventional standards. That's not a very good Kappa, but it's not nothing either.
Again. These are all NLP variables that are doing the heavy lifting in the modelling.
So the models perform somewhat poorly, but they still predicted something, especially for the continuous DIF outcome.
And I want you to keep in mind that when we got the expert reviewers right based on the cApStAn model, you call it as of translation,
and we created a survey out of that and got them to compare the items, Ithink on six different criteria and predicted exactly zilch.
Okay. So relative to that, I think this is doing okay.
And also,
you can see here that the predictors that were doing the heavy lifting do come from the automated natural language processing of the text.
So the future for this work. I personally think these findings show promise; you might disagree.
What do I think we need to do to improve it? I think we need purer and larger estimates of DIF by language.
What do I mean by purer? I think we need datasets which basically remember I said we tried to decontaminate
the DIF estimates by way of having other covariates in the DIF model.
Well, we could only do so much with that. So I think that we need to get other datasets which enable us to do that even more so.
So for example, with international large scale assessments, PISA, PIRLS,within a country,
they'll have people doing the assessment in multiple languages.
And it just enables you, once you build up a dataset like that,
to partial out some of that variance attributable to other factors that contaminates the language DIF, so to speak.
Also, what's this ever really going to work that well for science while science, you know,
particularly when we're talking about physics at what we’d call a Year 12 level in Australia,
there's a lot of mathematics and formulae in there. The same goes for chemistry and biology is a little bit more language heavy but still.
So probably this wasn't the best starting place to apply this something like a reading assessment, we probably worked better.
And also coming back to this idea of us just sort of jumbling all the text together for each item and running at the NLP analysis on that.
It's not a very intelligent approach. We need a more nuanced approach to evaluating text complexity, complexity by item type.
When you think about it, a multiple-choice item is a very, very complex linguistic entity.
Okay? And just the sheer amount of text and its qualities in it only go so much of the way to explaining the complexity of the item,
and same goes for constructive response items.
And as a final point, I come back to the sort of general premise of the whole research and whether it's valid or not.
And I'm sure that there’s some people in the room have ideas on this. So is automatic cross-lingual text complexity assessment even possible?
So we need to build models of text complexity on common corpora as READERBENCH exist right now.
Each of the languages is basing itself on a different corpus, which I don't think is very good.
We need to build models of text complexity, using new language neutral tools for tagging passing so that, you know,
as Art alluded to, the world of NLP, in the research world of NLP like every week is making advances.
So, keeping keeping up to date on what's happening there,
validating the comparability of the complexity categories and these need to be psychometrically evaluated.
So, for me when I saw this that it screams out to me this is a factor analysis problem.
The idea that you have multiple indicators of these categories that are sort of latent latent categories underlying what the indices are about.
And once you approach it from that way, it also immediately leads to questions of, well,
if you get a certain factor analysis solution for one language, do you get it across other languages?
And you can actually start answering the question of whether these indices do work in an invariant way across languages.
I'd say generally it's more plausible within major language families, families than across.
So, you know, Indo European languages within might work, okay, but not necessarily when you bring in other major language family.
And generally it just remains a very interesting empirical question to us.