Trained food is wonderful.
We gave it a few minutes because obviously people are travelling from other places to get here through the grounds of the college,
so but starting a few minutes late? Good evening, everyone, and welcome to the first of a series of seminars on student access to university.
I'm Rebecca Surrender, University of the search for a vice chancellor for equality of Diversity,
and I'm delighted to have been invited by the organisers of this series to the chair tonight.
This series really couldn't be more important or timely as issues of access to higher education about
the policy agenda not just in the UK but internationally also and debates around cost quality,
but also equality of access to equality of outcomes of higher education become more prominent.
And of course, nowhere is that more true than for this institution when it was pulled from media headlines and public and government scrutiny.
So I'm really delighted that the Department of Education has put on this series.
I think it will be of enormous value, not just for the damage analysis of the issues,
but also the practical utility for the University for colleges, tutors and, of course, for the applicant and future students.
Now, the organisers have asked me to say a few words about the series, so I'm just going to do that.
It's a series of five seminars. The series was developed has been developed by Professor Jonathan and Professor Simon
Marginson from the Higher Education Research Group College of Education here in Oxford.
The starting point of the series is the premise that in order for the public discourse,
sorry for the public discourse around access to social and educational inequalities and widening participation.
In order for that debate to move forward, we need to bring to it a research based treatment of these issues.
Reasoned and data driven approaches will help us reflect on the scope for development and reform at Oxford and in the country as a whole.
We kick off tonight with a discussion on admissions, testing, preparation effects.
Obviously a very important issue for this university in particular.
Tonight's presentation views on the findings from a really exciting and actually somewhat unusual research
collaboration involving expertise from the Central University offices and academic department and college.
The Department, of course, the Department of Education and the college's brain science.
The project was facilitated by Fresno's college's access officer to Organ and Race Most senior tutors,
Simon Smith, and was part funded by Cambridge International Examinations.
I need to tell you the proceedings are public and will be recorded as speakers tonight,
Dr Samina Khan, director of Undergraduate Admissions and Outreach Professor,
Joanne Fed, director of the Department of Education and sadly,
we were to be joined also by Alison Matthews, deputy director of undergraduate admissions and outreach.
But she can't be here this evening. They're going to do a joint presentation lasting for about 45 minutes,
after which the official response from Professor Karen, Karen, Karen O'Brien, a very close colleague and friend of mine had.
She really is. And of the Carolyn's response to people that have I had plenty of time for questions and discussion comments.
Thank you all very much for. Thank you, construction.
Welcome everybody to the first of this very exciting series of meetings that were to kick off with something that I am interested in.
Assessment testing a scenario two for many years and when I joined Oxford,
I was very interested to know about the various admissions tests that we do to them since I've been here.
It's really interesting how many more applicants are taking other questions.
So we're at a point where 88 percent of our undergraduates who apply take one of the more tests.
We have 11 tests across different courses that we offer, but this study is mainly about being at the TSA.
You may ask, why do we do the tests? We have so much information on only because that's apply to us right from the
new cast forms also in to take part a level two and also do interviews as well.
Well, if I could just give you a small insight into why we do test.
Well, our application numbers are increasing as year and increasingly they're coming
from parts of the world where the education system is a very different task.
And we don't really have any segments over the interviews that looks right across the different types of performance and that potential.
So that's one of the reasons why we do do admissions tests.
We also have an issue now in the UK higher education system, where the reliability of predictive grades for A-levels is this questionable.
And we I just heard today a report that said that this is something that needs to be looked at and those
that are used to doing is asking for that and also moving to post qualifications admissions as well.
Well, before we do that, I think we need to look at what we've already got some life into a race as an issue,
particularly with those conditional offers being made by a vast number of universities.
And the other thing that we often Mercia is the U.K. folks,
especially because we have ministers who come in and tell us what is best for students in schools.
And then they move on to talking about recycling plastic bottles so we can see of the challenges of how business.
But this is not unique to Oxford.
Most competitive courses, I suppose, have some form of admissions test, and I'm not the only punch, but those emissions tests for further education.
There are other countries that do admissions tests as well.
Emissions tests that we have in the case have been around since the introduction in two thousand and three,
and those who did initially set pulses to be able to look at able to differentiate between spending and what comes next year to pay grades,
not much before it takes place. So we have had for a long time before the excitement for the science to test, which was retired in 1985.
It was concerns time because the country since independence took a seventh term in school, where it's like the state schools, the testing system.
So you could say you could question a different journey. So students from under the test to the the working party that was put together by context
looked at the tests and looked at the type of tests that the children are introducing.
And they described them as aptitude test and they were tests that really could not be coached for that.
There was no real reason for the Postal Service to help us to better understand that we did do this committee's
study and we've got some very interesting findings that we must share with you today in the final stages.
This one is still a work in progress. So there is more work for us yet to do in time to a couple of companies to hand over to each other.
So I think a ridiculous generation before you help for just the thing for me is a range of tests in which you get these application effects.
There's a whole range of studies showing this just takes a few years and we spent our life and two of them together.
So one of these is the 11 plus exam, which is a kind of selection test for grammar schools and independent schools as well.
And you see this preparation for the risk, even though they're sort of marketed as not being something that you can prepare for.
So what is interesting is that these fears by and the sense of educational measurement at Durham University two presumptive years has been automated,
with a plus test saying that this was something that you couldn't prepare for and that you could improve your scores by test generation.
And they were drawn into investigating whether this was really true,
and they actually claimed that they were preservation effects for their test, too.
That was really interesting for me to observe from the sidelines could be watching the marketing and thinking,
Really, what kind of test is it that you can prepare for, especially when you go to lunch to learn this?
But we have an international test like Pisa. They showed in Germany,
and this is a test run by the Organisation for Economic Co-operation and Development and looks at how real nations are doing educationally.
And in Germany, the obvious issues there that you can prepare students for those types of things and do better little ones in this very test of
interesting today and examples vehemence is the research study by Durham University again actually found test inflation effects.
But in terms of some of the inequality issues that are going to talk to you later.
So there's a vindication of these tests that I've got on this slide.
Some of them are curriculum related, actually publishing curricula.
You expected to prepare for the tests, and that's intended to help you do better than others.
Different kind of logic to them altogether, which are sort of getting up.
And I was talking about the first exams. But what we know, of course, is that teaching to the test works.
Of course, immigration and teaching is going to help people to better, and knowledge of the test itself is going to happen here.
And I've got an idea from the Khan Academy where they're promoting a course to help people do better on educational testing services.
Discounting scholastic aptitude test is actually no longer quote unquote imminent.
And so here's the counter kind of we say if you do a course, you're going to do better this test.
No, I'm just explaining this logic because the whole idea about teacher testing and how people used to think about.
And we have these tests. Is that what you want is a test that level the playing field a lot of table support
students have had to do well in the first two related exams that A-levels and so on.
What you want is a test that can identify talent aptitude for the future.
So there was a whole discussion around that was the set of values in which these tests were developed to show the teacher in the past.
So this kind of goes along with notions of intelligence, too,
that you can identify people's ability through intelligence, through tests and then to assess the potential.
So it goes on as well with ideas about accredits ability. You know, that's not necessary.
And part of this logic, but as well is that it goes along with ideas, is permitted to proceed.
So no matter the advantages,
people have got to do that you cannot use these aptitude tests to talk to the really clever students who might not have all the advantages.
So right back in the midst of the time, it was obvious from the data that even IQ tests would be tied to education privilege in the world.
Education drove the higher IQ scores. Where to do these things?
And so what we have here is this notion about aptitude testing this to try and promote a meritocratic approach to the selection.
So in fact, we used to call them aptitude tests heat index, which is but there's a whole literature on this that undermines that kind of logic.
And in the 1980s, it's in the states where were challenged on this to be discredited as the true test and issues for university entrance in the U.S.
And what was happening was that loads of companies were setting themselves up to serve, cramming schools and helping kids and didn't get the sense.
Given the kind of logic that there was.
This was a problem because either you could teach to the test preparation and improve your scores, or you couldn't.
But you couldn't have is two sets of commercial practise at the time to think about this in different ways.
So we're not saying I can help you develop and test the other, but she can't do that with our test.
So it spawned a huge amount of research. It's looking at this.
So he took the 1990 synthesis of four and meta analysis boards at receptionist's.
What they found, they call it coaching.
And what they found is that it's virtually to test results that you get better defence score and examples of activities like maths,
which are going to take for the variable sections of the test.
Obviously, we're coaching and programmes that are actually larger effect size and as well as that particular very complex questions.
Coaching helps more with that, too. And and it's just very gratifying that you actually had a second test preparation effects.
So I mean, this is a company that have really been at the forefront of educational assessment and testing,
but I think it's the really thought leaders in the field.
So here they are,
evaluating their own products and came to the conclusion right back in the 80s that actually aptitude testing was a viable way of thinking about this.
And so they changed the name of it to score scholastic assessment test.
But it also combined the dirty groceries. Keep forgetting to change it.
So essentially the argument then from all other states is that what you're measuring is performance is not cheap.
You don't have a special breed of actually assessing potential. All you can measure is how will people do on the day of these tests?
So there are different ways of preparing for tests that we might be interested in.
They're human and you could think about just test familiarity, actually know what the test is going to contain.
But when you get the kinds of questions are going to be asked the content of the test,
whether it's related to the curriculum that you've been studying and so on.
This is quite a nice model that there was a previous work from the University of Vienna,
so they looked at the test preparation things for a particular programme and also for their medical entrance test.
So the model was the same, which is kind of the obvious reason, but you can think about this and the destruction equation model to look at.
Which which would to protect the sort of things that are at the heart.
So another approach that you can take is to just teach them tips and tricks, but actually perform it in the test.
So they're going to be doing stuff like telling people and time management.
I know how it turns into fights, and I've never been an item on multiple choice tests and so on.
Eliminate the wrong answers and so on. Or you can actually teach people some solutions strategies.
So sure, the previous items talk to them about the kind of techniques that you would use to solve problem solving.
Items are not safe whomsoever, and that would be helpful to them, or, as you might be familiar with, the actual teaching.
OK, doing so would be a good, viable method for getting people to test.
So what differentiates between this or the these approaches and perhaps and test form is whether
the impact more broadly reducing actually introduce a skill that generalises David is here.
Another to the specific test or whether it generalise, is in fact intelligence and data.
So they found in this particular programme the repairs and test that and the to model here, and it's in the database.
And this is also keeping with other studies that we don't typically get from
use test preparation programmes as generalisability to identify intelligence.
Excuse me, General Intelligence and G for really very much to bring specific knowledge on other similar tests.
So. So what we're getting in is tax preparation introduced people for a particular taste,
and the question is how much of interest is on us when we've using selection tests in universities?
Basically, what we're going to do is try and discriminate fairly between the candidates on the basis of the construct,
the many shapes so notes on constructing relevant stadiums.
So it's an unfair advantage of a test provision only helps people to do that.
And the tests are not really interested that the test scores for their own sake, not interested just in those performances.
We're actually interested in whether the ticket was and general skills.
So this test preparation gets in the way of our capability, and that's the truth of this.
And as I've just been arguing,
the consensus in the literature is that it's best to level the playing field in terms of test preparation by making our associates more transparent,
by giving people and past papers and showing them what they think the scoring scheme is,
and so that a broad range of people can actually prepare themselves well for the test.
Now we don't want either, as long as a matter for the test scores and just due to lack of knowledge of the test, that also helps with that.
So he had an off switch as well over the past few years.
We don't use the word not to choose. And we've actually read more of these materials available for people to practise.
So in looking at this study, we did hear and looking at things from skills assessment from the Beamer folks,
which are run by Cambridge Assessment and it's my computer.
So I'm going to show you some of the findings that we've got. But this is the kind of assessment that we're looking at for the thinking skills test.
Now the data that we looked at didn't include the rating task because it wasn't large scale, which of course, is to use the technical skills test.
They don't always use that data and which has some implications for having to trust
the data to a biomedical admissions test and as has a rating of three sections here.
So you can see that it's a problem-solving and data based on scientific knowledge.
OK, so what we did was we looked at, first of all, we've got data for the years 2010 to 2015,
and we did some of those congregations just looking at how do candidates do and whether they have any relationships with factors,
those that seem to be affecting test scores.
And once we've controlled for the kind of sequence that GCSE and actually did a 10, they might postpone adaptation, then add some quality data.
We got some questionnaire data and did interviews with students in terms of how they came to the test,
how they experience the admissions process and process as well. And then we looked at these test completion effects on our scores.
And we also looked at a range of ways in which students prepare for the tests in terms of the strategies they can use.
So here's the regression analysis.
That's what makes them an answer.
And so what we found all of this stuff at the bottom of it and then with all the different levels, including systemic ables.
But what we found for both of the tests was that there's some effects by applicant type.
So having control over how people do at GCSE and A-level people with the same scores that they were female and got lower scores with TSC Urbina.
And that's actually, you know, it's what we found in Durham. So that was debated.
And this is why the study and it could be that its population characteristics that cause this.
So we need to be a bit cautious about that. We also have a big effect and a significant effect on both of the tests, too.
And if the school has had previous applications to switch students that better having controlled for training attainment and as you would expect,
folks for widening participation that acorn,
the caller folks and the average school performance actually had significant effects
and factors that acorn and find that show it and lower outcomes in all tests.
And I believe the TSA, did you see a school? So there's some interesting effects here about an enemy applications.
And what we find is that the earlier students have fired the higher their scores
that will come back to what we know from making the military and the state schools.
This state school students depend on the TSA and independent schools.
And that's kind of constituency. But if you remember, what we've seen is that we have controls for child attainment there.
So this test seems to be showing that for the same GCSE and A-level results,
students from either state independent schools actually do better the TSC.
So it seems that it's identifying something it can.
We can imagine greatest generation things for GCSE and A-level as curriculum related tests,
so it might well be an extra some 50 state school kids and similar GCSE and A-level results.
OK, so looking at those Christian Eriksen as well, it's the best I've just picked out and the most relevant stuff from test prediction.
We asked people if the and I just wonderful.
And one of the issues about it and the general effect as well is that we we
asked people the kinds of strategies that they used and prepared for the test.
And it took the female candidates in this group that we were looking to use the memorisation strategies to to large them.
So that's actually not particularly constructive use.
It might be that gendered effects is not an instance of attempts,
but it's actually a very amateurish strategy that's being used to large extent and seems.
All right. So we asked you, did you prepare for the test students these days?
Prepared enormous and incredibly strategic. So and the vast majority of them, as you see here, but actually and prepared very well for these tests.
And they had looked at a whole range of materials.
They had support some things and certainly from their skills, but they were looking online in support for their families.
And we talked to people who take the test before there's a whole load of strategies that they were using.
So what we've got at the bottom here is a comparison between students from state schools and students from independent schools
looking at and the extent to which they prepared in terms of how long the terms of the test had they started their transition.
And what you see is a difference there, such as independent school students had started preparing for longer and.
But the state school students. And in fact, when we do the regression analysis,
what we find is that this variable amount of time with their parents, the students start the preparation.
It's actually got a significant effect on your scores for it to be beamer.
So if there's an advantage here, it's that these students have been prepared well in advance and know what's about these tests.
Hank Jones is going to talk about a student voice.
Very interested to hear from students themselves in terms of their experience of the test that they have undertaken so as possible.
So we conducted some interviews with first year undergraduates and these undergraduates have taken the view about suicide and other tests as well,
which is the range of tests, basically because you couldn't find the undergraduate students at that time to post the study.
So we took it broader than just the maths and also the TSA and the findings.
We have some great questions and I've just got some examples of their responses in the next couple of slides.
So we were interested in asking questions of independently educated students from independent sector,
and we were also interested in the views from students who had attended state schools.
So in terms of the students from the independent schools, their responses were fairly sort of.
Yes, I received plenty of support from my school in various different ways in terms of the time I had for oppression,
for the test and also access to materials, access to support from teachers who go through various amounts of feedback and also
the different strategies that they use in order to prepare and still their responses.
Where we would like more information, we would like the process in terms of the test to be demystified.
We would like to past papers. We would like to see the mark schemes as well.
And interestingly, they were very much aware of the lack of a level playing field because once they had arrived
at Oxford and met other students have come from different types of schools to theirs.
They were very much aware of the advantage that their preparation had.
Isn't that compared to their state school peers? So here is.
Of a very Typekit side. But if I can just summarise again, the request from the state school applicants was that we would like access to information.
In particular, we would like access to past papers and would like to know what it is that's being tested, said Mark Schemes ET.
And again,
here there was a strong awareness of the advantage that certain students in their peer groups on their courses that the support that they have had.
And even if that support had just given extra confidence, they felt it had provided extra advantage.
So this is in a way that summarises quite nicely from students themselves had experience.
We also did ask them questions around the whole admissions process and getting the responses were around access to better information,
demystifying what it is that they what was expected of them and a greater awareness
of the high degree of support that some students have had versus others.
This was again replicated with the response from the state schools as well.
It really boils down to privilege providing certain advantages as well.
So if I can just summarise the main findings of this research and I like to stress it's still a work in progress.
So the main findings really boil down to three main points.
And in the in the work we've done so far. So it wasn't really a surprise, perhaps to some of us,
that test preparation does have an effect on the peaks with the previous work done in the literature.
And it wasn't perhaps a surprise that students from different schools would have different amounts of test preparation,
time and resources and access to materials.
We also need to look at how test preparation undermines possibility of test scores for admissions processes.
And we need to take this into account with this trying to reduce the effect that it creates and really live up to what we liked these tests to do,
which is to produce a better level playing field across different because that applied to Oxford.
So those are the main findings to date from the report.
And I would like to end on that gloomy note.
I really would like to say that since we've been doing this work,
enormous amount of work has been going on to try and address some of this test preparation effects.
So we have been looking at the information,
advice and guidance that we do provide for our tests and made enormous amounts of inroad into providing better information,
better access to previous papers. Also mark schemes and the different types of skills the tests are looking for and that information,
which information is now, is not available on the university's website, as well as Cambridge assessments websites.
So hopefully that does responds to some degree to the request to demystify the tests and access to better information.
But again, there is more work to be done in this area.
Also, this report has fed into the review of access that was undertaken by the previous head of a real college.
On that report,
the Access Target Management Group has picked up a lot of the finds in some of the recommendations from that report was to provide more information.
So in a way, we have looked to address the information, advice and guidance points.
But again, as I've said, more work to be done in that area. We're also keen to look at targeted outreach.
We do lots of outreach right across the collegiate university and we do our best to demystify our
admissions process to provide information on what students to do when and to make sure they do it on time.
But we're also looking at providing much more information on the tests themselves,
particularly for those students who do not have access to resources, do not have access to support from school or home.
And I particularly like to mention the work undertaken by the maths departments and also the physics department,
the work that they've undertaken to hold and masterclasses on the admissions test and particularly making it targeted to schools that students needed.
Another area of average, which we're looking at,
is working with teachers because if we can support teachers and give them chapter and verse on our text, that obviously would help.
And not just a single student, but hopefully a whole generation of students.
Other outreach programmes like the unique summer school are also looking at what additional support we can offer in terms of admissions tests as well.
And also the interviews, as well as a piece of work which again was referenced in the Access Typekit Working Group,
was to to take a deep dive into our admissions process right across the different courses and not just to look at tests in isolation,
but to look at all the different stages of all the information we use.
Those videos are thought addition to the admissions process. Reviews undertaken the fear of those in some subject areas,
and the findings of those reviews has meant that we've changed some of the admissions
test to make sure that they are doing what they thought is central to effectively.
So these admissions reviews are proving to be very effective in addressing some of the findings that we are finding with some of our processes.
So again, this is work in progress and a lot more to do, but we hope we.
Continue this journey, but meanwhile, I hope you've enjoyed some talk stimulating and it's really over to you for questions.
Thank you very much, Senator John, of the really excellent presentation.
At a very important discussion. We're not going to let professor thrown on ahead of humanity's is going to force us just doesn't symptoms.
It makes those comments in response. Thank you.
Thank you very much and thank you very much, Senator John and Alison.
It's terrific to see this crossover between our Department of Education,
the university admissions office and those colleges get into some really interesting findings.
I've just been asked to make a few words by way of response. Obviously, as head of humanities, I have a great deal of interest in this issue.
The TSA test is a test taken by a number of humanities Americans, including those who do keep the history of economics.
And we have a range of tests of humanities that are specifically designed to look at questions like language aptitude or the the rather Iceland hat,
the history aptitude test. There are lots of acts maths, science elapse, that kind of thing,
and we're very conscious of the role that these tests play in our business processes.
I had the advantage of the audience of having read the whole of the draught report,
and I'm aware that it's only a draught to this, according to the book In Progress.
But I I found it really compellingly interesting.
In some ways, it's a very intuitive conclusion that preparation in tests makes you better at the test.
But then I think we've all found in this area of their emissions that a lot of what appears to be intuitive isn't so.
The finding in John's and others report that actually state schools,
students when you control for GCSE qualifications, do all the better in the TSA tests.
And the independent school counterpart is, on the one hand, counterintuitive. But actually, Lamont, 64, can understand the reasons for that.
And I think that the finding that preparation does have an effect on performance that, you know,
there's no such thing as a kind of neutral test that can somehow get under the surface of one's educational background and experience.
Get to a kind of true measure of potential is is a really important thing to evidence.
I think intuitively, we all understand that the evidence in that is important.
And then we saw that the Vienna study shows that performance in these kinds of tests does not necessarily equal general intelligence,
actually deep learning, good teaching. These are the things that will improve performance, general understanding and intelligence subjects.
I think it's very interesting to contextualise this in an international context.
Some of you may have seen the article this week in the Times Higher Education Supplement about aptitude
and skills testing nationally in the debates around using Chinese girl power schools and some of the UK.
First, he's an Australian national how to test in SATs in the US.
And I think that, you know, there's a great there's a great deal of international thinking and research about
how one assesses the aptitude and skills that students have for university study.
I think it's important to consider this in a national context. The Bhima Test is a test used by only some of the most selective schools in this
country the majority of medical school to something else called the UK cut.
I would be very interested to know how these tests compare. When you look at them, they look on the surface very different.
Obviously, the FEMA to a lunatic like myself, it's absolutely terrifying.
If you've ever seen anything that looks like you will be in complete awe of our biology and medical students for ever getting through this thing.
But they're all, as the data show, some some effects in relation to ethnicity and gender in relation to the impact on the TSA.
And that's obviously a serious issue that we need to consider very deeply.
I think there's a general question is there for us at Oxford and for universities about what are these?
What are the what are the tests telling us? What are they for? They are, as the report shows, quite predictive of a level of success.
And there is a relationship between these aptitude tests and A-level performance, and they are predictive of degree attainment.
So this relationship between these tests and degree attainment, so one might say, you know,
there is a role for the test for these kinds of tests in our evaluation of candidates.
But we also saw in one of those slides, there seem to be some effects by ethnicity and gender in terms of outcomes for those tests.
And I think that, to me, is the single most important thing that we need to understand a great deal more about in the report.
I think is very encouragingly in the case of the Thinking Skills Assessment Test, there is a suggestion that those effects are those inequality.
Effects of socioeconomic backgrounds, ethnicity and gender can be ironed out by early,
and good preparation for the TSA test is less clear in the case of the BMA, but it is clear in the case of the TSA.
So that tells us exactly.
What Samina has told us that there is a role for us working with schools, with teachers, with unique summer school teachers through mentoring,
through our interaction with sixth formers to help guide young people in their preparation for tests,
the more we can do to be transparent up front to provide a great deal of information and necessary active mentoring,
whether that's online or through ambassador schemes or however, we do this and we do this in a variety of ways.
It is really important that if we use this as we also proactively support young
people in preparing for those tests so that they don't go into the cold,
they don't go to them in a way that makes them take them by surprise.
And we can also proactively engage in those kind of equalising strategies as we know those tests are used in all universities as part of a broader,
contextual consideration of the whole person.
So a contextual consideration that includes, in the case of the interview, prior attainment in service of GCSE.
So other exams, personal statements,
and we need to think and continue to think about how that broader whole person consideration allowing for different kinds
of backgrounds and preparedness guides us in making fair judgements about who to offer a place to at our universities.
So a lot to think about some very unsurprising findings, but so important that those findings,
findings of evidence and also some very surprising findings.
I think to come, and I think it's probably the basis for what I hope will be an interesting discussion to follow now.
Thank you.