1
00:00:00,380 --> 00:00:04,190
Trained food is wonderful.

2
00:00:04,190 --> 00:00:12,800
We gave it a few minutes because obviously people are travelling from other places to get here through the grounds of the college,

3
00:00:12,800 --> 00:00:23,030
so but starting a few minutes late? Good evening, everyone, and welcome to the first of a series of seminars on student access to university.

4
00:00:23,030 --> 00:00:28,520
I'm Rebecca Surrender, University of the search for a vice chancellor for equality of Diversity,

5
00:00:28,520 --> 00:00:35,870
and I'm delighted to have been invited by the organisers of this series to the chair tonight.

6
00:00:35,870 --> 00:00:44,120
This series really couldn't be more important or timely as issues of access to higher education about

7
00:00:44,120 --> 00:00:52,310
the policy agenda not just in the UK but internationally also and debates around cost quality,

8
00:00:52,310 --> 00:01:01,520
but also equality of access to equality of outcomes of higher education become more prominent.

9
00:01:01,520 --> 00:01:11,840
And of course, nowhere is that more true than for this institution when it was pulled from media headlines and public and government scrutiny.

10
00:01:11,840 --> 00:01:17,930
So I'm really delighted that the Department of Education has put on this series.

11
00:01:17,930 --> 00:01:24,350
I think it will be of enormous value, not just for the damage analysis of the issues,

12
00:01:24,350 --> 00:01:34,700
but also the practical utility for the University for colleges, tutors and, of course, for the applicant and future students.

13
00:01:34,700 --> 00:01:39,890
Now, the organisers have asked me to say a few words about the series, so I'm just going to do that.

14
00:01:39,890 --> 00:01:47,060
It's a series of five seminars. The series was developed has been developed by Professor Jonathan and Professor Simon

15
00:01:47,060 --> 00:01:53,330
Marginson from the Higher Education Research Group College of Education here in Oxford.

16
00:01:53,330 --> 00:02:00,050
The starting point of the series is the premise that in order for the public discourse,

17
00:02:00,050 --> 00:02:07,130
sorry for the public discourse around access to social and educational inequalities and widening participation.

18
00:02:07,130 --> 00:02:14,180
In order for that debate to move forward, we need to bring to it a research based treatment of these issues.

19
00:02:14,180 --> 00:02:22,310
Reasoned and data driven approaches will help us reflect on the scope for development and reform at Oxford and in the country as a whole.

20
00:02:22,310 --> 00:02:27,350
We kick off tonight with a discussion on admissions, testing, preparation effects.

21
00:02:27,350 --> 00:02:31,310
Obviously a very important issue for this university in particular.

22
00:02:31,310 --> 00:02:37,010
Tonight's presentation views on the findings from a really exciting and actually somewhat unusual research

23
00:02:37,010 --> 00:02:43,670
collaboration involving expertise from the Central University offices and academic department and college.

24
00:02:43,670 --> 00:02:48,230
The Department, of course, the Department of Education and the college's brain science.

25
00:02:48,230 --> 00:02:54,830
The project was facilitated by Fresno's college's access officer to Organ and Race Most senior tutors,

26
00:02:54,830 --> 00:03:00,410
Simon Smith, and was part funded by Cambridge International Examinations.

27
00:03:00,410 --> 00:03:07,730
I need to tell you the proceedings are public and will be recorded as speakers tonight,

28
00:03:07,730 --> 00:03:12,380
Dr Samina Khan, director of Undergraduate Admissions and Outreach Professor,

29
00:03:12,380 --> 00:03:16,370
Joanne Fed, director of the Department of Education and sadly,

30
00:03:16,370 --> 00:03:21,350
we were to be joined also by Alison Matthews, deputy director of undergraduate admissions and outreach.

31
00:03:21,350 --> 00:03:27,620
But she can't be here this evening. They're going to do a joint presentation lasting for about 45 minutes,

32
00:03:27,620 --> 00:03:40,730
after which the official response from Professor Karen, Karen, Karen O'Brien, a very close colleague and friend of mine had.

33
00:03:40,730 --> 00:03:49,080
She really is. And of the Carolyn's response to people that have I had plenty of time for questions and discussion comments.

34
00:03:49,080 --> 00:04:08,430
Thank you all very much for. Thank you, construction.

35
00:04:08,430 --> 00:04:16,020
Welcome everybody to the first of this very exciting series of meetings that were to kick off with something that I am interested in.

36
00:04:16,020 --> 00:04:22,260
Assessment testing a scenario two for many years and when I joined Oxford,

37
00:04:22,260 --> 00:04:29,160
I was very interested to know about the various admissions tests that we do to them since I've been here.

38
00:04:29,160 --> 00:04:34,500
It's really interesting how many more applicants are taking other questions.

39
00:04:34,500 --> 00:04:41,670
So we're at a point where 88 percent of our undergraduates who apply take one of the more tests.

40
00:04:41,670 --> 00:04:50,520
We have 11 tests across different courses that we offer, but this study is mainly about being at the TSA.

41
00:04:50,520 --> 00:04:59,250
You may ask, why do we do the tests? We have so much information on only because that's apply to us right from the

42
00:04:59,250 --> 00:05:07,110
new cast forms also in to take part a level two and also do interviews as well.

43
00:05:07,110 --> 00:05:11,100
Well, if I could just give you a small insight into why we do test.

44
00:05:11,100 --> 00:05:16,380
Well, our application numbers are increasing as year and increasingly they're coming

45
00:05:16,380 --> 00:05:20,940
from parts of the world where the education system is a very different task.

46
00:05:20,940 --> 00:05:29,210
And we don't really have any segments over the interviews that looks right across the different types of performance and that potential.

47
00:05:29,210 --> 00:05:32,550
So that's one of the reasons why we do do admissions tests.

48
00:05:32,550 --> 00:05:42,990
We also have an issue now in the UK higher education system, where the reliability of predictive grades for A-levels is this questionable.

49
00:05:42,990 --> 00:05:49,370
And we I just heard today a report that said that this is something that needs to be looked at and those

50
00:05:49,370 --> 00:05:56,850
that are used to doing is asking for that and also moving to post qualifications admissions as well.

51
00:05:56,850 --> 00:06:02,370
Well, before we do that, I think we need to look at what we've already got some life into a race as an issue,

52
00:06:02,370 --> 00:06:07,400
particularly with those conditional offers being made by a vast number of universities.

53
00:06:07,400 --> 00:06:10,290
And the other thing that we often Mercia is the U.K. folks,

54
00:06:10,290 --> 00:06:16,560
especially because we have ministers who come in and tell us what is best for students in schools.

55
00:06:16,560 --> 00:06:26,200
And then they move on to talking about recycling plastic bottles so we can see of the challenges of how business.

56
00:06:26,200 --> 00:06:29,540
But this is not unique to Oxford.

57
00:06:29,540 --> 00:06:39,380
Most competitive courses, I suppose, have some form of admissions test, and I'm not the only punch, but those emissions tests for further education.

58
00:06:39,380 --> 00:06:43,540
There are other countries that do admissions tests as well.

59
00:06:43,540 --> 00:06:50,540
Emissions tests that we have in the case have been around since the introduction in two thousand and three,

60
00:06:50,540 --> 00:07:01,280
and those who did initially set pulses to be able to look at able to differentiate between spending and what comes next year to pay grades,

61
00:07:01,280 --> 00:07:11,930
not much before it takes place. So we have had for a long time before the excitement for the science to test, which was retired in 1985.

62
00:07:11,930 --> 00:07:23,000
It was concerns time because the country since independence took a seventh term in school, where it's like the state schools, the testing system.

63
00:07:23,000 --> 00:07:34,550
So you could say you could question a different journey. So students from under the test to the the working party that was put together by context

64
00:07:34,550 --> 00:07:39,240
looked at the tests and looked at the type of tests that the children are introducing.

65
00:07:39,240 --> 00:07:46,520
And they described them as aptitude test and they were tests that really could not be coached for that.

66
00:07:46,520 --> 00:07:53,720
There was no real reason for the Postal Service to help us to better understand that we did do this committee's

67
00:07:53,720 --> 00:08:00,320
study and we've got some very interesting findings that we must share with you today in the final stages.

68
00:08:00,320 --> 00:08:20,120
This one is still a work in progress. So there is more work for us yet to do in time to a couple of companies to hand over to each other.

69
00:08:20,120 --> 00:08:31,430
So I think a ridiculous generation before you help for just the thing for me is a range of tests in which you get these application effects.

70
00:08:31,430 --> 00:08:38,750
There's a whole range of studies showing this just takes a few years and we spent our life and two of them together.

71
00:08:38,750 --> 00:08:46,940
So one of these is the 11 plus exam, which is a kind of selection test for grammar schools and independent schools as well.

72
00:08:46,940 --> 00:08:57,740
And you see this preparation for the risk, even though they're sort of marketed as not being something that you can prepare for.

73
00:08:57,740 --> 00:09:07,580
So what is interesting is that these fears by and the sense of educational measurement at Durham University two presumptive years has been automated,

74
00:09:07,580 --> 00:09:15,680
with a plus test saying that this was something that you couldn't prepare for and that you could improve your scores by test generation.

75
00:09:15,680 --> 00:09:20,900
And they were drawn into investigating whether this was really true,

76
00:09:20,900 --> 00:09:24,620
and they actually claimed that they were preservation effects for their test, too.

77
00:09:24,620 --> 00:09:29,960
That was really interesting for me to observe from the sidelines could be watching the marketing and thinking,

78
00:09:29,960 --> 00:09:37,310
Really, what kind of test is it that you can prepare for, especially when you go to lunch to learn this?

79
00:09:37,310 --> 00:09:42,950
But we have an international test like Pisa. They showed in Germany,

80
00:09:42,950 --> 00:09:51,890
and this is a test run by the Organisation for Economic Co-operation and Development and looks at how real nations are doing educationally.

81
00:09:51,890 --> 00:10:01,310
And in Germany, the obvious issues there that you can prepare students for those types of things and do better little ones in this very test of

82
00:10:01,310 --> 00:10:11,510
interesting today and examples vehemence is the research study by Durham University again actually found test inflation effects.

83
00:10:11,510 --> 00:10:17,930
But in terms of some of the inequality issues that are going to talk to you later.

84
00:10:17,930 --> 00:10:21,560
So there's a vindication of these tests that I've got on this slide.

85
00:10:21,560 --> 00:10:26,240
Some of them are curriculum related, actually publishing curricula.

86
00:10:26,240 --> 00:10:32,570
You expected to prepare for the tests, and that's intended to help you do better than others.

87
00:10:32,570 --> 00:10:36,470
Different kind of logic to them altogether, which are sort of getting up.

88
00:10:36,470 --> 00:10:43,650
And I was talking about the first exams. But what we know, of course, is that teaching to the test works.

89
00:10:43,650 --> 00:10:53,510
Of course, immigration and teaching is going to help people to better, and knowledge of the test itself is going to happen here.

90
00:10:53,510 --> 00:11:03,980
And I've got an idea from the Khan Academy where they're promoting a course to help people do better on educational testing services.

91
00:11:03,980 --> 00:11:10,970
Discounting scholastic aptitude test is actually no longer quote unquote imminent.

92
00:11:10,970 --> 00:11:16,180
And so here's the counter kind of we say if you do a course, you're going to do better this test.

93
00:11:16,180 --> 00:11:25,850
No, I'm just explaining this logic because the whole idea about teacher testing and how people used to think about.

94
00:11:25,850 --> 00:11:33,170
And we have these tests. Is that what you want is a test that level the playing field a lot of table support

95
00:11:33,170 --> 00:11:37,790
students have had to do well in the first two related exams that A-levels and so on.

96
00:11:37,790 --> 00:11:43,220
What you want is a test that can identify talent aptitude for the future.

97
00:11:43,220 --> 00:11:51,710
So there was a whole discussion around that was the set of values in which these tests were developed to show the teacher in the past.

98
00:11:51,710 --> 00:11:55,490
So this kind of goes along with notions of intelligence, too,

99
00:11:55,490 --> 00:12:04,970
that you can identify people's ability through intelligence, through tests and then to assess the potential.

100
00:12:04,970 --> 00:12:09,900
So it goes on as well with ideas about accredits ability. You know, that's not necessary.

101
00:12:09,900 --> 00:12:16,490
And part of this logic, but as well is that it goes along with ideas, is permitted to proceed.

102
00:12:16,490 --> 00:12:19,610
So no matter the advantages,

103
00:12:19,610 --> 00:12:29,780
people have got to do that you cannot use these aptitude tests to talk to the really clever students who might not have all the advantages.

104
00:12:29,780 --> 00:12:40,130
So right back in the midst of the time, it was obvious from the data that even IQ tests would be tied to education privilege in the world.

105
00:12:40,130 --> 00:12:46,040
Education drove the higher IQ scores. Where to do these things?

106
00:12:46,040 --> 00:12:58,880
And so what we have here is this notion about aptitude testing this to try and promote a meritocratic approach to the selection.

107
00:12:58,880 --> 00:13:07,850
So in fact, we used to call them aptitude tests heat index, which is but there's a whole literature on this that undermines that kind of logic.

108
00:13:07,850 --> 00:13:19,220
And in the 1980s, it's in the states where were challenged on this to be discredited as the true test and issues for university entrance in the U.S.

109
00:13:19,220 --> 00:13:30,240
And what was happening was that loads of companies were setting themselves up to serve, cramming schools and helping kids and didn't get the sense.

110
00:13:30,240 --> 00:13:32,390
Given the kind of logic that there was.

111
00:13:32,390 --> 00:13:41,060
This was a problem because either you could teach to the test preparation and improve your scores, or you couldn't.

112
00:13:41,060 --> 00:13:49,250
But you couldn't have is two sets of commercial practise at the time to think about this in different ways.

113
00:13:49,250 --> 00:13:54,740
So we're not saying I can help you develop and test the other, but she can't do that with our test.

114
00:13:54,740 --> 00:13:58,580
So it spawned a huge amount of research. It's looking at this.

115
00:13:58,580 --> 00:14:07,340
So he took the 1990 synthesis of four and meta analysis boards at receptionist's.

116
00:14:07,340 --> 00:14:09,360
What they found, they call it coaching.

117
00:14:09,360 --> 00:14:18,050
And what they found is that it's virtually to test results that you get better defence score and examples of activities like maths,

118
00:14:18,050 --> 00:14:23,330
which are going to take for the variable sections of the test.

119
00:14:23,330 --> 00:14:35,720
Obviously, we're coaching and programmes that are actually larger effect size and as well as that particular very complex questions.

120
00:14:35,720 --> 00:14:47,750
Coaching helps more with that, too. And and it's just very gratifying that you actually had a second test preparation effects.

121
00:14:47,750 --> 00:14:54,650
So I mean, this is a company that have really been at the forefront of educational assessment and testing,

122
00:14:54,650 --> 00:14:59,150
but I think it's the really thought leaders in the field.

123
00:14:59,150 --> 00:15:00,230
So here they are,

124
00:15:00,230 --> 00:15:09,500
evaluating their own products and came to the conclusion right back in the 80s that actually aptitude testing was a viable way of thinking about this.

125
00:15:09,500 --> 00:15:14,720
And so they changed the name of it to score scholastic assessment test.

126
00:15:14,720 --> 00:15:19,430
But it also combined the dirty groceries. Keep forgetting to change it.

127
00:15:19,430 --> 00:15:27,260
So essentially the argument then from all other states is that what you're measuring is performance is not cheap.

128
00:15:27,260 --> 00:15:36,980
You don't have a special breed of actually assessing potential. All you can measure is how will people do on the day of these tests?

129
00:15:36,980 --> 00:15:41,750
So there are different ways of preparing for tests that we might be interested in.

130
00:15:41,750 --> 00:15:49,620
They're human and you could think about just test familiarity, actually know what the test is going to contain.

131
00:15:49,620 --> 00:15:55,190
But when you get the kinds of questions are going to be asked the content of the test,

132
00:15:55,190 --> 00:16:00,080
whether it's related to the curriculum that you've been studying and so on.

133
00:16:00,080 --> 00:16:07,880
This is quite a nice model that there was a previous work from the University of Vienna,

134
00:16:07,880 --> 00:16:19,160
so they looked at the test preparation things for a particular programme and also for their medical entrance test.

135
00:16:19,160 --> 00:16:25,750
So the model was the same, which is kind of the obvious reason, but you can think about this and the destruction equation model to look at.

136
00:16:25,750 --> 00:16:30,790
Which which would to protect the sort of things that are at the heart.

137
00:16:30,790 --> 00:16:40,230
So another approach that you can take is to just teach them tips and tricks, but actually perform it in the test.

138
00:16:40,230 --> 00:16:49,030
So they're going to be doing stuff like telling people and time management.

139
00:16:49,030 --> 00:16:58,690
I know how it turns into fights, and I've never been an item on multiple choice tests and so on.

140
00:16:58,690 --> 00:17:05,800
Eliminate the wrong answers and so on. Or you can actually teach people some solutions strategies.

141
00:17:05,800 --> 00:17:11,560
So sure, the previous items talk to them about the kind of techniques that you would use to solve problem solving.

142
00:17:11,560 --> 00:17:20,960
Items are not safe whomsoever, and that would be helpful to them, or, as you might be familiar with, the actual teaching.

143
00:17:20,960 --> 00:17:25,960
OK, doing so would be a good, viable method for getting people to test.

144
00:17:25,960 --> 00:17:32,440
So what differentiates between this or the these approaches and perhaps and test form is whether

145
00:17:32,440 --> 00:17:39,490
the impact more broadly reducing actually introduce a skill that generalises David is here.

146
00:17:39,490 --> 00:17:46,510
Another to the specific test or whether it generalise, is in fact intelligence and data.

147
00:17:46,510 --> 00:17:55,420
So they found in this particular programme the repairs and test that and the to model here, and it's in the database.

148
00:17:55,420 --> 00:18:01,090
And this is also keeping with other studies that we don't typically get from

149
00:18:01,090 --> 00:18:06,340
use test preparation programmes as generalisability to identify intelligence.

150
00:18:06,340 --> 00:18:16,240
Excuse me, General Intelligence and G for really very much to bring specific knowledge on other similar tests.

151
00:18:16,240 --> 00:18:27,540
So. So what we're getting in is tax preparation introduced people for a particular taste,

152
00:18:27,540 --> 00:18:36,750
and the question is how much of interest is on us when we've using selection tests in universities?

153
00:18:36,750 --> 00:18:44,900
Basically, what we're going to do is try and discriminate fairly between the candidates on the basis of the construct,

154
00:18:44,900 --> 00:18:51,000
the many shapes so notes on constructing relevant stadiums.

155
00:18:51,000 --> 00:18:56,760
So it's an unfair advantage of a test provision only helps people to do that.

156
00:18:56,760 --> 00:19:02,430
And the tests are not really interested that the test scores for their own sake, not interested just in those performances.

157
00:19:02,430 --> 00:19:07,170
We're actually interested in whether the ticket was and general skills.

158
00:19:07,170 --> 00:19:13,530
So this test preparation gets in the way of our capability, and that's the truth of this.

159
00:19:13,530 --> 00:19:16,140
And as I've just been arguing,

160
00:19:16,140 --> 00:19:25,560
the consensus in the literature is that it's best to level the playing field in terms of test preparation by making our associates more transparent,

161
00:19:25,560 --> 00:19:31,530
by giving people and past papers and showing them what they think the scoring scheme is,

162
00:19:31,530 --> 00:19:39,150
and so that a broad range of people can actually prepare themselves well for the test.

163
00:19:39,150 --> 00:19:49,560
Now we don't want either, as long as a matter for the test scores and just due to lack of knowledge of the test, that also helps with that.

164
00:19:49,560 --> 00:19:53,400
So he had an off switch as well over the past few years.

165
00:19:53,400 --> 00:20:03,150
We don't use the word not to choose. And we've actually read more of these materials available for people to practise.

166
00:20:03,150 --> 00:20:10,590
So in looking at this study, we did hear and looking at things from skills assessment from the Beamer folks,

167
00:20:10,590 --> 00:20:15,540
which are run by Cambridge Assessment and it's my computer.

168
00:20:15,540 --> 00:20:24,550
So I'm going to show you some of the findings that we've got. But this is the kind of assessment that we're looking at for the thinking skills test.

169
00:20:24,550 --> 00:20:33,360
Now the data that we looked at didn't include the rating task because it wasn't large scale, which of course, is to use the technical skills test.

170
00:20:33,360 --> 00:20:39,840
They don't always use that data and which has some implications for having to trust

171
00:20:39,840 --> 00:20:49,770
the data to a biomedical admissions test and as has a rating of three sections here.

172
00:20:49,770 --> 00:20:58,650
So you can see that it's a problem-solving and data based on scientific knowledge.

173
00:20:58,650 --> 00:21:08,610
OK, so what we did was we looked at, first of all, we've got data for the years 2010 to 2015,

174
00:21:08,610 --> 00:21:19,740
and we did some of those congregations just looking at how do candidates do and whether they have any relationships with factors,

175
00:21:19,740 --> 00:21:22,260
those that seem to be affecting test scores.

176
00:21:22,260 --> 00:21:32,910
And once we've controlled for the kind of sequence that GCSE and actually did a 10, they might postpone adaptation, then add some quality data.

177
00:21:32,910 --> 00:21:39,360
We got some questionnaire data and did interviews with students in terms of how they came to the test,

178
00:21:39,360 --> 00:21:50,760
how they experience the admissions process and process as well. And then we looked at these test completion effects on our scores.

179
00:21:50,760 --> 00:21:59,760
And we also looked at a range of ways in which students prepare for the tests in terms of the strategies they can use.

180
00:21:59,760 --> 00:22:03,420
So here's the regression analysis.

181
00:22:03,420 --> 00:22:05,640
That's what makes them an answer.

182
00:22:05,640 --> 00:22:14,250
And so what we found all of this stuff at the bottom of it and then with all the different levels, including systemic ables.

183
00:22:14,250 --> 00:22:21,010
But what we found for both of the tests was that there's some effects by applicant type.

184
00:22:21,010 --> 00:22:32,550
So having control over how people do at GCSE and A-level people with the same scores that they were female and got lower scores with TSC Urbina.

185
00:22:32,550 --> 00:22:36,870
And that's actually, you know, it's what we found in Durham. So that was debated.

186
00:22:36,870 --> 00:22:43,050
And this is why the study and it could be that its population characteristics that cause this.

187
00:22:43,050 --> 00:22:52,110
So we need to be a bit cautious about that. We also have a big effect and a significant effect on both of the tests, too.

188
00:22:52,110 --> 00:23:04,000
And if the school has had previous applications to switch students that better having controlled for training attainment and as you would expect,

189
00:23:04,000 --> 00:23:08,250
folks for widening participation that acorn,

190
00:23:08,250 --> 00:23:13,320
the caller folks and the average school performance actually had significant effects

191
00:23:13,320 --> 00:23:19,610
and factors that acorn and find that show it and lower outcomes in all tests.

192
00:23:19,610 --> 00:23:32,060
And I believe the TSA, did you see a school? So there's some interesting effects here about an enemy applications.

193
00:23:32,060 --> 00:23:36,410
And what we find is that the earlier students have fired the higher their scores

194
00:23:36,410 --> 00:23:42,860
that will come back to what we know from making the military and the state schools.

195
00:23:42,860 --> 00:23:49,000
This state school students depend on the TSA and independent schools.

196
00:23:49,000 --> 00:23:57,380
And that's kind of constituency. But if you remember, what we've seen is that we have controls for child attainment there.

197
00:23:57,380 --> 00:24:03,290
So this test seems to be showing that for the same GCSE and A-level results,

198
00:24:03,290 --> 00:24:08,030
students from either state independent schools actually do better the TSC.

199
00:24:08,030 --> 00:24:11,780
So it seems that it's identifying something it can.

200
00:24:11,780 --> 00:24:17,270
We can imagine greatest generation things for GCSE and A-level as curriculum related tests,

201
00:24:17,270 --> 00:24:26,420
so it might well be an extra some 50 state school kids and similar GCSE and A-level results.

202
00:24:26,420 --> 00:24:39,500
OK, so looking at those Christian Eriksen as well, it's the best I've just picked out and the most relevant stuff from test prediction.

203
00:24:39,500 --> 00:24:45,200
We asked people if the and I just wonderful.

204
00:24:45,200 --> 00:24:51,620
And one of the issues about it and the general effect as well is that we we

205
00:24:51,620 --> 00:24:56,150
asked people the kinds of strategies that they used and prepared for the test.

206
00:24:56,150 --> 00:25:04,670
And it took the female candidates in this group that we were looking to use the memorisation strategies to to large them.

207
00:25:04,670 --> 00:25:09,150
So that's actually not particularly constructive use.

208
00:25:09,150 --> 00:25:13,520
It might be that gendered effects is not an instance of attempts,

209
00:25:13,520 --> 00:25:22,610
but it's actually a very amateurish strategy that's being used to large extent and seems.

210
00:25:22,610 --> 00:25:28,330
All right. So we asked you, did you prepare for the test students these days?

211
00:25:28,330 --> 00:25:39,570
Prepared enormous and incredibly strategic. So and the vast majority of them, as you see here, but actually and prepared very well for these tests.

212
00:25:39,570 --> 00:25:42,610
And they had looked at a whole range of materials.

213
00:25:42,610 --> 00:25:49,030
They had support some things and certainly from their skills, but they were looking online in support for their families.

214
00:25:49,030 --> 00:25:55,900
And we talked to people who take the test before there's a whole load of strategies that they were using.

215
00:25:55,900 --> 00:26:04,840
So what we've got at the bottom here is a comparison between students from state schools and students from independent schools

216
00:26:04,840 --> 00:26:13,930
looking at and the extent to which they prepared in terms of how long the terms of the test had they started their transition.

217
00:26:13,930 --> 00:26:22,120
And what you see is a difference there, such as independent school students had started preparing for longer and.

218
00:26:22,120 --> 00:26:28,240
But the state school students. And in fact, when we do the regression analysis,

219
00:26:28,240 --> 00:26:34,570
what we find is that this variable amount of time with their parents, the students start the preparation.

220
00:26:34,570 --> 00:26:40,210
It's actually got a significant effect on your scores for it to be beamer.

221
00:26:40,210 --> 00:26:57,490
So if there's an advantage here, it's that these students have been prepared well in advance and know what's about these tests.

222
00:26:57,490 --> 00:27:17,150
Hank Jones is going to talk about a student voice.

223
00:27:17,150 --> 00:27:25,720
Very interested to hear from students themselves in terms of their experience of the test that they have undertaken so as possible.

224
00:27:25,720 --> 00:27:36,070
So we conducted some interviews with first year undergraduates and these undergraduates have taken the view about suicide and other tests as well,

225
00:27:36,070 --> 00:27:41,560
which is the range of tests, basically because you couldn't find the undergraduate students at that time to post the study.

226
00:27:41,560 --> 00:27:49,600
So we took it broader than just the maths and also the TSA and the findings.

227
00:27:49,600 --> 00:27:55,540
We have some great questions and I've just got some examples of their responses in the next couple of slides.

228
00:27:55,540 --> 00:28:03,460
So we were interested in asking questions of independently educated students from independent sector,

229
00:28:03,460 --> 00:28:10,060
and we were also interested in the views from students who had attended state schools.

230
00:28:10,060 --> 00:28:19,630
So in terms of the students from the independent schools, their responses were fairly sort of.

231
00:28:19,630 --> 00:28:30,430
Yes, I received plenty of support from my school in various different ways in terms of the time I had for oppression,

232
00:28:30,430 --> 00:28:40,540
for the test and also access to materials, access to support from teachers who go through various amounts of feedback and also

233
00:28:40,540 --> 00:28:45,970
the different strategies that they use in order to prepare and still their responses.

234
00:28:45,970 --> 00:28:51,910
Where we would like more information, we would like the process in terms of the test to be demystified.

235
00:28:51,910 --> 00:28:56,440
We would like to past papers. We would like to see the mark schemes as well.

236
00:28:56,440 --> 00:29:03,070
And interestingly, they were very much aware of the lack of a level playing field because once they had arrived

237
00:29:03,070 --> 00:29:07,390
at Oxford and met other students have come from different types of schools to theirs.

238
00:29:07,390 --> 00:29:11,470
They were very much aware of the advantage that their preparation had.

239
00:29:11,470 --> 00:29:21,300
Isn't that compared to their state school peers? So here is.

240
00:29:21,300 --> 00:29:33,190
Of a very Typekit side. But if I can just summarise again, the request from the state school applicants was that we would like access to information.

241
00:29:33,190 --> 00:29:41,340
In particular, we would like access to past papers and would like to know what it is that's being tested, said Mark Schemes ET.

242
00:29:41,340 --> 00:29:42,300
And again,

243
00:29:42,300 --> 00:29:53,120
here there was a strong awareness of the advantage that certain students in their peer groups on their courses that the support that they have had.

244
00:29:53,120 --> 00:29:59,640
And even if that support had just given extra confidence, they felt it had provided extra advantage.

245
00:29:59,640 --> 00:30:08,490
So this is in a way that summarises quite nicely from students themselves had experience.

246
00:30:08,490 --> 00:30:20,130
We also did ask them questions around the whole admissions process and getting the responses were around access to better information,

247
00:30:20,130 --> 00:30:25,410
demystifying what it is that they what was expected of them and a greater awareness

248
00:30:25,410 --> 00:30:32,690
of the high degree of support that some students have had versus others.

249
00:30:32,690 --> 00:30:45,830
This was again replicated with the response from the state schools as well.

250
00:30:45,830 --> 00:30:52,010
It really boils down to privilege providing certain advantages as well.

251
00:30:52,010 --> 00:31:01,340
So if I can just summarise the main findings of this research and I like to stress it's still a work in progress.

252
00:31:01,340 --> 00:31:05,360
So the main findings really boil down to three main points.

253
00:31:05,360 --> 00:31:11,150
And in the in the work we've done so far. So it wasn't really a surprise, perhaps to some of us,

254
00:31:11,150 --> 00:31:19,760
that test preparation does have an effect on the peaks with the previous work done in the literature.

255
00:31:19,760 --> 00:31:27,560
And it wasn't perhaps a surprise that students from different schools would have different amounts of test preparation,

256
00:31:27,560 --> 00:31:32,240
time and resources and access to materials.

257
00:31:32,240 --> 00:31:41,060
We also need to look at how test preparation undermines possibility of test scores for admissions processes.

258
00:31:41,060 --> 00:31:51,530
And we need to take this into account with this trying to reduce the effect that it creates and really live up to what we liked these tests to do,

259
00:31:51,530 --> 00:31:59,270
which is to produce a better level playing field across different because that applied to Oxford.

260
00:31:59,270 --> 00:32:03,590
So those are the main findings to date from the report.

261
00:32:03,590 --> 00:32:07,460
And I would like to end on that gloomy note.

262
00:32:07,460 --> 00:32:10,490
I really would like to say that since we've been doing this work,

263
00:32:10,490 --> 00:32:19,400
enormous amount of work has been going on to try and address some of this test preparation effects.

264
00:32:19,400 --> 00:32:21,890
So we have been looking at the information,

265
00:32:21,890 --> 00:32:31,320
advice and guidance that we do provide for our tests and made enormous amounts of inroad into providing better information,

266
00:32:31,320 --> 00:32:41,720
better access to previous papers. Also mark schemes and the different types of skills the tests are looking for and that information,

267
00:32:41,720 --> 00:32:49,280
which information is now, is not available on the university's website, as well as Cambridge assessments websites.

268
00:32:49,280 --> 00:32:57,560
So hopefully that does responds to some degree to the request to demystify the tests and access to better information.

269
00:32:57,560 --> 00:33:01,010
But again, there is more work to be done in this area.

270
00:33:01,010 --> 00:33:09,200
Also, this report has fed into the review of access that was undertaken by the previous head of a real college.

271
00:33:09,200 --> 00:33:10,430
On that report,

272
00:33:10,430 --> 00:33:18,180
the Access Target Management Group has picked up a lot of the finds in some of the recommendations from that report was to provide more information.

273
00:33:18,180 --> 00:33:24,860
So in a way, we have looked to address the information, advice and guidance points.

274
00:33:24,860 --> 00:33:33,860
But again, as I've said, more work to be done in that area. We're also keen to look at targeted outreach.

275
00:33:33,860 --> 00:33:40,250
We do lots of outreach right across the collegiate university and we do our best to demystify our

276
00:33:40,250 --> 00:33:47,840
admissions process to provide information on what students to do when and to make sure they do it on time.

277
00:33:47,840 --> 00:33:53,570
But we're also looking at providing much more information on the tests themselves,

278
00:33:53,570 --> 00:34:00,890
particularly for those students who do not have access to resources, do not have access to support from school or home.

279
00:34:00,890 --> 00:34:08,270
And I particularly like to mention the work undertaken by the maths departments and also the physics department,

280
00:34:08,270 --> 00:34:21,750
the work that they've undertaken to hold and masterclasses on the admissions test and particularly making it targeted to schools that students needed.

281
00:34:21,750 --> 00:34:24,560
Another area of average, which we're looking at,

282
00:34:24,560 --> 00:34:33,680
is working with teachers because if we can support teachers and give them chapter and verse on our text, that obviously would help.

283
00:34:33,680 --> 00:34:39,140
And not just a single student, but hopefully a whole generation of students.

284
00:34:39,140 --> 00:34:47,660
Other outreach programmes like the unique summer school are also looking at what additional support we can offer in terms of admissions tests as well.

285
00:34:47,660 --> 00:34:56,030
And also the interviews, as well as a piece of work which again was referenced in the Access Typekit Working Group,

286
00:34:56,030 --> 00:35:04,820
was to to take a deep dive into our admissions process right across the different courses and not just to look at tests in isolation,

287
00:35:04,820 --> 00:35:09,650
but to look at all the different stages of all the information we use.

288
00:35:09,650 --> 00:35:17,360
Those videos are thought addition to the admissions process. Reviews undertaken the fear of those in some subject areas,

289
00:35:17,360 --> 00:35:23,030
and the findings of those reviews has meant that we've changed some of the admissions

290
00:35:23,030 --> 00:35:28,280
test to make sure that they are doing what they thought is central to effectively.

291
00:35:28,280 --> 00:35:38,720
So these admissions reviews are proving to be very effective in addressing some of the findings that we are finding with some of our processes.

292
00:35:38,720 --> 00:35:44,940
So again, this is work in progress and a lot more to do, but we hope we.

293
00:35:44,940 --> 00:36:03,700
Continue this journey, but meanwhile, I hope you've enjoyed some talk stimulating and it's really over to you for questions.

294
00:36:03,700 --> 00:36:10,690
Thank you very much, Senator John, of the really excellent presentation.

295
00:36:10,690 --> 00:36:18,970
At a very important discussion. We're not going to let professor thrown on ahead of humanity's is going to force us just doesn't symptoms.

296
00:36:18,970 --> 00:36:23,320
It makes those comments in response. Thank you.

297
00:36:23,320 --> 00:36:27,150
Thank you very much and thank you very much, Senator John and Alison.

298
00:36:27,150 --> 00:36:32,530
It's terrific to see this crossover between our Department of Education,

299
00:36:32,530 --> 00:36:38,950
the university admissions office and those colleges get into some really interesting findings.

300
00:36:38,950 --> 00:36:45,430
I've just been asked to make a few words by way of response. Obviously, as head of humanities, I have a great deal of interest in this issue.

301
00:36:45,430 --> 00:36:52,620
The TSA test is a test taken by a number of humanities Americans, including those who do keep the history of economics.

302
00:36:52,620 --> 00:37:00,100
And we have a range of tests of humanities that are specifically designed to look at questions like language aptitude or the the rather Iceland hat,

303
00:37:00,100 --> 00:37:05,320
the history aptitude test. There are lots of acts maths, science elapse, that kind of thing,

304
00:37:05,320 --> 00:37:11,600
and we're very conscious of the role that these tests play in our business processes.

305
00:37:11,600 --> 00:37:15,460
I had the advantage of the audience of having read the whole of the draught report,

306
00:37:15,460 --> 00:37:19,090
and I'm aware that it's only a draught to this, according to the book In Progress.

307
00:37:19,090 --> 00:37:22,690
But I I found it really compellingly interesting.

308
00:37:22,690 --> 00:37:28,840
In some ways, it's a very intuitive conclusion that preparation in tests makes you better at the test.

309
00:37:28,840 --> 00:37:35,110
But then I think we've all found in this area of their emissions that a lot of what appears to be intuitive isn't so.

310
00:37:35,110 --> 00:37:39,370
The finding in John's and others report that actually state schools,

311
00:37:39,370 --> 00:37:45,700
students when you control for GCSE qualifications, do all the better in the TSA tests.

312
00:37:45,700 --> 00:37:53,770
And the independent school counterpart is, on the one hand, counterintuitive. But actually, Lamont, 64, can understand the reasons for that.

313
00:37:53,770 --> 00:38:00,190
And I think that the finding that preparation does have an effect on performance that, you know,

314
00:38:00,190 --> 00:38:07,500
there's no such thing as a kind of neutral test that can somehow get under the surface of one's educational background and experience.

315
00:38:07,500 --> 00:38:12,400
Get to a kind of true measure of potential is is a really important thing to evidence.

316
00:38:12,400 --> 00:38:16,510
I think intuitively, we all understand that the evidence in that is important.

317
00:38:16,510 --> 00:38:24,550
And then we saw that the Vienna study shows that performance in these kinds of tests does not necessarily equal general intelligence,

318
00:38:24,550 --> 00:38:33,430
actually deep learning, good teaching. These are the things that will improve performance, general understanding and intelligence subjects.

319
00:38:33,430 --> 00:38:37,270
I think it's very interesting to contextualise this in an international context.

320
00:38:37,270 --> 00:38:42,790
Some of you may have seen the article this week in the Times Higher Education Supplement about aptitude

321
00:38:42,790 --> 00:38:48,610
and skills testing nationally in the debates around using Chinese girl power schools and some of the UK.

322
00:38:48,610 --> 00:38:53,080
First, he's an Australian national how to test in SATs in the US.

323
00:38:53,080 --> 00:38:57,940
And I think that, you know, there's a great there's a great deal of international thinking and research about

324
00:38:57,940 --> 00:39:05,080
how one assesses the aptitude and skills that students have for university study.

325
00:39:05,080 --> 00:39:13,990
I think it's important to consider this in a national context. The Bhima Test is a test used by only some of the most selective schools in this

326
00:39:13,990 --> 00:39:18,820
country the majority of medical school to something else called the UK cut.

327
00:39:18,820 --> 00:39:24,550
I would be very interested to know how these tests compare. When you look at them, they look on the surface very different.

328
00:39:24,550 --> 00:39:29,350
Obviously, the FEMA to a lunatic like myself, it's absolutely terrifying.

329
00:39:29,350 --> 00:39:37,780
If you've ever seen anything that looks like you will be in complete awe of our biology and medical students for ever getting through this thing.

330
00:39:37,780 --> 00:39:46,990
But they're all, as the data show, some some effects in relation to ethnicity and gender in relation to the impact on the TSA.

331
00:39:46,990 --> 00:39:52,180
And that's obviously a serious issue that we need to consider very deeply.

332
00:39:52,180 --> 00:39:57,700
I think there's a general question is there for us at Oxford and for universities about what are these?

333
00:39:57,700 --> 00:40:04,930
What are the what are the tests telling us? What are they for? They are, as the report shows, quite predictive of a level of success.

334
00:40:04,930 --> 00:40:11,110
And there is a relationship between these aptitude tests and A-level performance, and they are predictive of degree attainment.

335
00:40:11,110 --> 00:40:16,540
So this relationship between these tests and degree attainment, so one might say, you know,

336
00:40:16,540 --> 00:40:22,240
there is a role for the test for these kinds of tests in our evaluation of candidates.

337
00:40:22,240 --> 00:40:30,320
But we also saw in one of those slides, there seem to be some effects by ethnicity and gender in terms of outcomes for those tests.

338
00:40:30,320 --> 00:40:37,600
And I think that, to me, is the single most important thing that we need to understand a great deal more about in the report.

339
00:40:37,600 --> 00:40:47,120
I think is very encouragingly in the case of the Thinking Skills Assessment Test, there is a suggestion that those effects are those inequality.

340
00:40:47,120 --> 00:40:53,920
Effects of socioeconomic backgrounds, ethnicity and gender can be ironed out by early,

341
00:40:53,920 --> 00:41:01,570
and good preparation for the TSA test is less clear in the case of the BMA, but it is clear in the case of the TSA.

342
00:41:01,570 --> 00:41:02,920
So that tells us exactly.

343
00:41:02,920 --> 00:41:12,010
What Samina has told us that there is a role for us working with schools, with teachers, with unique summer school teachers through mentoring,

344
00:41:12,010 --> 00:41:19,750
through our interaction with sixth formers to help guide young people in their preparation for tests,

345
00:41:19,750 --> 00:41:27,400
the more we can do to be transparent up front to provide a great deal of information and necessary active mentoring,

346
00:41:27,400 --> 00:41:33,460
whether that's online or through ambassador schemes or however, we do this and we do this in a variety of ways.

347
00:41:33,460 --> 00:41:39,700
It is really important that if we use this as we also proactively support young

348
00:41:39,700 --> 00:41:44,050
people in preparing for those tests so that they don't go into the cold,

349
00:41:44,050 --> 00:41:47,470
they don't go to them in a way that makes them take them by surprise.

350
00:41:47,470 --> 00:41:59,110
And we can also proactively engage in those kind of equalising strategies as we know those tests are used in all universities as part of a broader,

351
00:41:59,110 --> 00:42:01,690
contextual consideration of the whole person.

352
00:42:01,690 --> 00:42:09,040
So a contextual consideration that includes, in the case of the interview, prior attainment in service of GCSE.

353
00:42:09,040 --> 00:42:11,770
So other exams, personal statements,

354
00:42:11,770 --> 00:42:18,370
and we need to think and continue to think about how that broader whole person consideration allowing for different kinds

355
00:42:18,370 --> 00:42:25,750
of backgrounds and preparedness guides us in making fair judgements about who to offer a place to at our universities.

356
00:42:25,750 --> 00:42:31,660
So a lot to think about some very unsurprising findings, but so important that those findings,

357
00:42:31,660 --> 00:42:34,930
findings of evidence and also some very surprising findings.

358
00:42:34,930 --> 00:42:42,400
I think to come, and I think it's probably the basis for what I hope will be an interesting discussion to follow now.

359
00:42:42,400 --> 00:42:54,005
Thank you.