1
00:00:00,440 --> 00:00:06,200
[Auto-generated transcript. Edits may have been applied for clarity.]
Welcome to our students in the room and only two guests in the room, but also to those joining online.

2
00:00:07,040 --> 00:00:12,600
This is the guest lecture for healthcare evaluation and research impact.

3
00:00:13,480 --> 00:00:23,240
And Tom, who's going to introduce himself fully, I know making part of a team from Rand Europe, which is a kind of policy facing research consultancy.

4
00:00:24,320 --> 00:00:34,520
Yeah. Um, who actually collaborate with us because we in the university are good at certain things like ivory tower academic stuff.

5
00:00:35,160 --> 00:00:41,320
But actually the interfacing with policy is something that Rand really majors on.

6
00:00:41,520 --> 00:00:48,080
And so so we like collaborating with them. And I can't remember how we ended up inviting you.

7
00:00:48,680 --> 00:00:55,320
I think I just put a throwaway line in an email, throwaway line about evaluation and AI.

8
00:00:56,520 --> 00:01:00,940
And I said, well, that's it. I think it was actually we on a team's call and you put it in the chat?

9
00:01:01,740 --> 00:01:05,220
Wait a minute. I'm looking for someone to give a talk on it anyway. That's exactly what it was.

10
00:01:05,860 --> 00:01:10,100
So welcome. Thank you very much for coming all the way from Cambridge. You've been, said Morton.

11
00:01:11,020 --> 00:01:14,980
Do introduce yourself. You've got 90 minutes and you're going to be interactive.

12
00:01:15,940 --> 00:01:22,420
Apparently, when you ask a question, it will be edited out of the video.

13
00:01:24,260 --> 00:01:28,380
It will be edited out of the video for data protection reasons.

14
00:01:29,140 --> 00:01:34,980
So you may find Tom repeating your question. Uh, and that's kind of for the video capture.

15
00:01:35,700 --> 00:01:39,580
So. So I mean, this is a funny guy who's just repeated my question back to me, that's why.

16
00:01:40,140 --> 00:01:43,900
Anyway. Welcome, Tom. Great. Thank you. Thank you so much and lovely to be here.

17
00:01:44,780 --> 00:01:51,820
Um, and this has been great. Kind of preparing my thoughts for talking to you, because it's always a great way to consolidate your own thinking.

18
00:01:52,380 --> 00:01:57,260
And that thinking in relation to AI is going to be rapidly evolving.

19
00:01:58,000 --> 00:02:05,840
so it's great. It's a kind of stock taking moment for me as well. Um, but the that all of that is changing and in flux and and evolving.

20
00:02:06,920 --> 00:02:09,940
Uh, so whatever we finish up agreeing in 90 minutes,

21
00:02:10,560 --> 00:02:15,080
the world will continue to change and we'll need to be adaptive and learning as we as we go forward.

22
00:02:16,000 --> 00:02:21,920
Um, so we're going to make this quite interactive. There'll be lots of opportunities to, to pause and to chip in.

23
00:02:22,680 --> 00:02:26,840
Um, and then we aim to finish by, by by three, 3:00.

24
00:02:27,960 --> 00:02:32,120
Um, right. So AI and the ethics of evaluation.

25
00:02:32,680 --> 00:02:43,480
So it's not the evaluation of AI, right. This is going to be about how we use, um, uh, AI in, in evaluation and in particular how we do that.

26
00:02:44,240 --> 00:02:50,440
Um, and how we, we do that in the world of evaluation around Europe.

27
00:02:50,960 --> 00:02:54,400
We already mentioned is a not for profit policy research institute.

28
00:02:55,520 --> 00:03:02,020
Um, We we work in the in the public interest and we aim to publish everything we do, just like an academic institution would do.

29
00:03:02,620 --> 00:03:07,220
But we have the kind of internal structures of the management sciences, um, and,

30
00:03:08,340 --> 00:03:12,820
and we focus particularly on policy related issues, covering the whole range of public policy.

31
00:03:13,420 --> 00:03:20,580
But my work is particularly in the area of health care. Um, but I've also got an ongoing interest in international development.

32
00:03:21,660 --> 00:03:24,740
Um, I work with Save the Children for a while way before that.

33
00:03:24,900 --> 00:03:30,780
It was also at random. Before that, I was at the National Audit Office as their research fellow, working on kind of value for money evaluations.

34
00:03:32,180 --> 00:03:36,660
Um, in the long distant past, I had a few academic appointments along the way.

35
00:03:37,660 --> 00:03:41,900
Um, so I am if I would, I would claim to be an evaluation expert.

36
00:03:42,380 --> 00:03:49,420
I absolutely would not claim to be an AI expert. So, um, you've probably got more expertise in AI than than I do,

37
00:03:49,700 --> 00:03:56,400
but we can bring that our insights together and learn together in the coming 90, 90 minutes.

38
00:03:56,960 --> 00:04:00,160
Um, as well as, um, around Europe.

39
00:04:00,520 --> 00:04:04,400
I'm the, the, uh, recently the president of the European Evaluation Society.

40
00:04:05,320 --> 00:04:09,120
I'm still an advisor to their board. Um, I ran out.

41
00:04:09,360 --> 00:04:13,200
I can't engage with the UK Evaluation Society as well, but they do great work.

42
00:04:13,760 --> 00:04:17,040
They produce good guidance for evaluators. If you go on their website.

43
00:04:18,000 --> 00:04:25,360
Um, on on the use of AI for evaluation, um, and the European Evaluation Society, we've got our conference in Lille.

44
00:04:25,680 --> 00:04:30,880
You're very welcome to come in October. Um, etc., to pay for it, but you're very welcome.

45
00:04:31,560 --> 00:04:39,640
Um, and, uh, an AI will certainly be a kind of ongoing topic, and I think I'm giving a session on that while we're there.

46
00:04:40,600 --> 00:04:48,160
What I'm about to, to outline comes from a chapter that was recently finished, which will be published early next year with Routledge.

47
00:04:49,240 --> 00:04:54,180
Um, and it's from algorithms to evidence using generative AI in evaluation practice.

48
00:04:55,220 --> 00:04:58,380
And our piece, our chapter is on the ethics and practice of evaluation.

49
00:04:59,660 --> 00:05:08,460
Um, and I should acknowledge that I'm doing that with Ananda Millard, um, who is is a European colleague of mine.

50
00:05:09,260 --> 00:05:13,300
Um, so a lot of this stuff draws on on that.

51
00:05:14,180 --> 00:05:20,340
Any questions? We're kind of okay on on what we're what we're about to do and who who I am.

52
00:05:20,660 --> 00:05:26,100
I suppose the other thing is that I, um, spend as much time as I can writing poetry and music.

53
00:05:26,460 --> 00:05:32,420
So you can find more about that on Tomlin. musicOMH should you, should you wish to do so?

54
00:05:33,060 --> 00:05:40,500
Um, I mean more slightly more seriously that maintaining I recommend in your careers that you maintain a creative

55
00:05:41,700 --> 00:05:46,980
element to your life where you're you're doing things that are kind of metaphorical and interesting and creative.

56
00:05:47,820 --> 00:05:58,320
And I believe it sparks a lot of useful thinking in the harder work that you do in from the evaluation and and in report writing and so forth.

57
00:05:58,560 --> 00:06:03,600
Getting, getting some sense of that. So let us get underway now.

58
00:06:03,680 --> 00:06:10,320
Are you not to be looking up to music as I as I speak? So what is the what is the issue that we're going to be exploring?

59
00:06:11,960 --> 00:06:22,400
Um, evaluators are exploring are embracing AI, but actually we're just saying earlier it's quite recent really the last 2 or 3 years.

60
00:06:23,360 --> 00:06:27,080
Um, and a lot of people were either just pretending it was going to go away or were,

61
00:06:28,040 --> 00:06:32,720
um, uh, we're saying, well, it's the it's the, it's the it's the spawn of Satan.

62
00:06:33,120 --> 00:06:40,320
It's going to ruin everything. Um, and that whole that attitude, I think has shifted a lot, even if it doesn't,

63
00:06:42,520 --> 00:06:46,800
because it's evaluation works more much more than academia, which is competitive.

64
00:06:47,440 --> 00:06:52,980
And of course, backbiting and mutually supportive evaluation is very competitive or is competitive interdependence?

65
00:06:54,100 --> 00:06:57,860
We work together a lot. We collaborate a lot, but we're competing for the same business.

66
00:06:59,100 --> 00:07:02,300
So Rand Europe is not for profit, but it's also not for loss.

67
00:07:02,740 --> 00:07:06,820
And we've got 200 star researchers. And that's a lot of mouths to feed.

68
00:07:07,020 --> 00:07:13,860
And we've got to win a lot of work. So if we don't engage with AI then our competitors will.

69
00:07:14,500 --> 00:07:17,580
They will be more efficient and more effective than we can be. And we will.

70
00:07:17,740 --> 00:07:18,580
We'll just not win work.

71
00:07:18,860 --> 00:07:26,660
So, uh, until very recently, if we were going to do a reasonably good literature review as part of the first station evaluation,

72
00:07:27,700 --> 00:07:32,540
we'd probably cost that up at ten, ten grand or something, or 20 grand of that order.

73
00:07:33,660 --> 00:07:39,380
And now the first draft will be done by AI, and we will then check all of that by hand.

74
00:07:40,820 --> 00:07:45,060
There are a few loose nations, and there used to be. There are a few. I mean, it's all a good bit better than it used to be.

75
00:07:45,300 --> 00:07:52,360
We'll check it all by hand and redo it. Um, but but we'll probably that that budget for that bit will probably be five grand.

76
00:07:52,840 --> 00:07:56,200
And if we try to charge 20 grand, then somebody else is going to go.

77
00:07:56,480 --> 00:07:58,920
So it is the thing about evaluation,

78
00:07:59,680 --> 00:08:08,320
if those of you that finish up working in that area is that even if you're working within an organisation and you're there evaluator right,

79
00:08:08,600 --> 00:08:15,320
you will still be, uh, the question will be asked, why should we do it in-house if you're far more expensive than getting someone else to do it?

80
00:08:15,440 --> 00:08:22,200
So that's the reality we're going to we'll definitely be moving into AI, and that makes this discussion even more important,

81
00:08:22,760 --> 00:08:26,800
because we've got to think about how we're going to do that and what the challenges are,

82
00:08:28,000 --> 00:08:31,840
not focusing on the specific techniques, although they will crop up from time to time.

83
00:08:32,160 --> 00:08:39,240
It's about the broader question about the relationship between reflective human inquiry and AI led reasoning.

84
00:08:39,720 --> 00:08:49,500
What's how do we what's the hybridity that works within ethical guidelines And within and produces

85
00:08:50,060 --> 00:08:56,260
results that we can defend in terms of their rigour and the quality of those data that come out of that.

86
00:08:56,620 --> 00:09:01,380
That's going to be where we hope to get to. Um, but what's peculiar about evaluation,

87
00:09:03,020 --> 00:09:10,860
it what's particular about evaluation is that it requires you to come to a judgement about the value or worth of something, right?

88
00:09:11,540 --> 00:09:16,860
Um, academic research is very often an inquiry based.

89
00:09:17,140 --> 00:09:20,220
It's just like, let's find out more about let's what is going on here.

90
00:09:20,660 --> 00:09:23,860
Let's let's understand the world more deeply than we did.

91
00:09:24,180 --> 00:09:27,660
Very important. Evaluation uses the same methods,

92
00:09:28,260 --> 00:09:35,140
but those methods are oriented towards arriving at a judgement about the value or worth or and or whether it can be improved.

93
00:09:35,900 --> 00:09:40,140
And you know, so those those are the, the things that you do as an evaluator.

94
00:09:40,780 --> 00:09:45,260
You have to come to that. And typically there would be kind of more practical recommendations, um,

95
00:09:45,560 --> 00:09:50,680
obviously some academic research does that as well, but, um, not all would do that.

96
00:09:50,960 --> 00:09:54,800
So what's often what's often called the Industrial revolution.

97
00:09:55,600 --> 00:10:00,640
So you've got the classic 19th century industrial revolution. You've got forward production lines.

98
00:10:01,360 --> 00:10:10,000
Next you've got the information explosion. Um, in the 60s and 70s, and now you've got kind of AI led as the fourth Industrial revolution.

99
00:10:11,040 --> 00:10:16,480
Um, and that that is marked by the fusion of digital intelligence and human systems.

100
00:10:17,600 --> 00:10:25,040
And I want to emphasise it's about the fusion of that rather than just the, the replacement of human judgement by AI.

101
00:10:25,920 --> 00:10:29,240
Um, but it is reshaping how knowledge is produced and how we apply it.

102
00:10:29,880 --> 00:10:33,560
Um, uh, of course, AI is not a single thing.

103
00:10:34,000 --> 00:10:35,960
It's a family of computational approaches.

104
00:10:37,280 --> 00:10:48,260
Um, and we will touch on, but we're not going to, um, over focus on, artificial superintelligence, which some argue is going to come.

105
00:10:48,700 --> 00:10:56,420
Some argue it won't happen, but that's where, um, AI agents spend a lot of their time learning.

106
00:10:57,260 --> 00:11:04,340
From other AI agents. And those AI agents then share the fruits of that thinking, that knowledge,

107
00:11:04,940 --> 00:11:12,060
and you are quickly moving into something which is beyond human comprehension at all as to how we got there.

108
00:11:12,500 --> 00:11:13,620
Right? So that's the kind of superintelligence.

109
00:11:14,780 --> 00:11:21,220
Whereas a lot of the things that we're looking at will be more specific AI agents, which we set to work on a particular task.

110
00:11:21,660 --> 00:11:31,100
So, for example, I did the evaluation, I did the review of UNICEF's, uh, evaluation policy a couple of years ago.

111
00:11:31,980 --> 00:11:40,500
Um, they've got, as you can imagine, 30 years worth of evaluations, um, which just sit on shelves and absolutely never, ever get used.

112
00:11:41,220 --> 00:11:45,710
They've digitised all of that, and it's now entirely searchable using AI.

113
00:11:46,150 --> 00:11:48,190
And you can just ask, for example,

114
00:11:48,670 --> 00:11:56,510
what is the best way to support girls education in a humanitarian crisis or in the first six months of a humanitarian crisis?

115
00:11:57,110 --> 00:11:59,950
And it will just give you everything that is available on that.

116
00:12:00,350 --> 00:12:09,350
And that is such a step forward in terms of the utilisation of vast amounts of knowledge that previously really were just just sitting, sitting there.

117
00:12:10,030 --> 00:12:14,630
Um, um, and yeah, so it's about, uh, judgement.

118
00:12:15,150 --> 00:12:19,670
Let's, uh, moving on. Uh, very, very briefly,

119
00:12:20,070 --> 00:12:23,670
I mentioned the kind of the people talk about the four industrial revolutions in terms

120
00:12:23,870 --> 00:12:28,590
of another way of thinking about it is to think about the four steps of digitisation,

121
00:12:29,910 --> 00:12:38,310
uh, in the 1990s, um, and digital data collection and performance systems and data application with large um,

122
00:12:38,750 --> 00:12:43,530
geospatial analytics and big data, cloud based infrastructures, all of that.

123
00:12:44,370 --> 00:12:50,130
Um, and then, uh, algorithmic transformations, the integration of AI in machine learning.

124
00:12:50,690 --> 00:12:54,970
And then now where we're at is, I believe, kind of hybrid intelligence.

125
00:12:55,610 --> 00:13:02,930
It's about this kind of interaction between human judgement and human, uh, leadership and, uh, AI leadership.

126
00:13:04,010 --> 00:13:11,090
Um, and so what we need to do is to try and think through how we, how we balance that, um, in knowledge generation.

127
00:13:12,250 --> 00:13:17,170
So coming back to focusing on, on, um, evaluation and AI, um,

128
00:13:17,770 --> 00:13:24,090
probably the best way to think about it is that in any evaluation, we just go through a series of steps.

129
00:13:25,210 --> 00:13:29,370
Um, and the rule of AI is slightly different in each, each of those steps.

130
00:13:30,490 --> 00:13:35,730
Um, if you can imagine going that's kind of chronological, isn't it?

131
00:13:36,370 --> 00:13:38,370
Uh, from designing through to, to delivering.

132
00:13:39,430 --> 00:13:47,470
Um, but it's the little the bits between these, the step back opportunities for reviewing what you've done, planning the next stage,

133
00:13:47,990 --> 00:13:55,310
making sure that it is reflecting your interests and the needs of your clients, but also your ethical requirements as an evaluator.

134
00:13:56,710 --> 00:14:04,270
Um, and on the right hand side, then we we commission, we design work, um, but also we commission evaluations.

135
00:14:05,030 --> 00:14:09,310
So AI is already you can read it in the in the invitations to tender.

136
00:14:09,950 --> 00:14:14,750
AI is already being used in the commissioning process, for better and for worse.

137
00:14:15,030 --> 00:14:17,950
Actually, sometimes it is very difficult to work out what's going on.

138
00:14:18,110 --> 00:14:24,070
But, um, but there again, it was always difficult to work out, um, rather than in addition to the tender from some sources.

139
00:14:24,910 --> 00:14:32,550
Um, so designing and commissioning, um, then the kind of inception meeting where you're really digging down,

140
00:14:32,790 --> 00:14:37,850
how are we going to manage the we've got 250,000. We've got 18 months to deliver.

141
00:14:38,570 --> 00:14:42,170
We've got these questions to answer. We've got these databases that are available.

142
00:14:42,850 --> 00:14:46,410
We've got this resource that we can collect new data. How are we going to do that?

143
00:14:46,650 --> 00:14:56,370
That that actually, um, has always been something that the experienced old hands like myself would just kind of go, that feels like 25 grand.

144
00:14:57,210 --> 00:14:58,050
I think that will probably cost.

145
00:14:58,690 --> 00:15:04,490
And you kind of have a get a feel for it will probably be much more accurate and specific about how we go about doing that now.

146
00:15:05,210 --> 00:15:07,890
Um, and I think what they'll do is they'll learn.

147
00:15:08,210 --> 00:15:16,050
We'll, we will look at previous examples and train the AI on what we said we would do previously, and then what actually happened.

148
00:15:17,090 --> 00:15:25,530
Um, and hopefully that will not reproduce the errors of what we, what we did for what was the one there was um, recently guys in Sir Thomas's,

149
00:15:25,850 --> 00:15:32,930
I think there were it was on their recruitment processes and the questions that they were asking and how they went about recruiting.

150
00:15:33,730 --> 00:15:38,390
And the AI agent was instructed to replicate the existing processes.

151
00:15:39,070 --> 00:15:40,950
Right. And then they implemented that.

152
00:15:41,270 --> 00:15:47,190
And they were shocked to discover that it was significantly discriminating against women, for example, amongst other things.

153
00:15:47,750 --> 00:15:53,150
And then so then we went back into it and what they realised was they were already discriminating against women.

154
00:15:53,670 --> 00:15:58,270
It just became visible and apparent when you when you put that into the AI agent.

155
00:15:59,030 --> 00:16:07,790
So that's yeah, that's kind of interesting examples. The um, so the that kind of inception meeting getting you know, you've won the contract.

156
00:16:08,230 --> 00:16:14,590
So you've, you've got the NIH grant, you've then got to say, right, specifically, what are we going to do about this?

157
00:16:15,310 --> 00:16:17,790
Um, and then you're a data collection.

158
00:16:18,910 --> 00:16:27,750
Uh, of course, there are all sorts of opportunities for, for searching social media for, uh, for doing, um, AI based interviews.

159
00:16:28,790 --> 00:16:36,490
Uh, and so the AI based interviews don't just take the interviewee through a preset set of questions.

160
00:16:37,730 --> 00:16:47,010
They respond to your response. Um, and, um, uh, there's a bunch in Bristol that I've developed all the software for that,

161
00:16:47,730 --> 00:16:52,130
um, for example, on quite intimate topics of sexual health or whatever.

162
00:16:53,450 --> 00:17:01,930
Um, the most the majority of people respondents said that they preferred being interviewed by an AI agent, which I hadn't really thought about.

163
00:17:02,530 --> 00:17:04,570
I thought, well, it's probably cheaper and whatever,

164
00:17:05,090 --> 00:17:11,370
but actually people are saying that in many circumstances they quite like being interviewed by an AI agent,

165
00:17:12,010 --> 00:17:20,250
even though they know perfectly well it's an AI agent. But the so the the agent will then respond to your reply and change the next question.

166
00:17:20,810 --> 00:17:26,570
But of course it will also, um, then do all the analysis as you're going through.

167
00:17:27,050 --> 00:17:32,010
So one of the particular uses of that would be um causal mapping.

168
00:17:33,010 --> 00:17:38,270
Um, so don't. This is not it doesn't prove that this is the real causal mapping in the world.

169
00:17:38,750 --> 00:17:43,190
This proves what people think are the causal maps, the causal causality.

170
00:17:43,950 --> 00:17:51,830
So you ask people across a whole. Population group what what what would happen if.

171
00:17:52,470 --> 00:17:57,950
Why do what. What causes that to happen if that had happened, what must have been the antecedent for that?

172
00:17:58,350 --> 00:18:02,510
And then use AI to build up the user a questionnaire.

173
00:18:03,230 --> 00:18:08,270
And then that will give you a map of the causal maps that people have in their heads.

174
00:18:08,670 --> 00:18:10,870
And of course, that's not necessarily true,

175
00:18:12,030 --> 00:18:17,710
because people have in their heads all sorts of misconceptions about causality, but it's a fantastic starting point.

176
00:18:18,550 --> 00:18:26,110
Um, I mean, just for example, on misconceptions, we did, um, the evaluation of the integrated care pilots,

177
00:18:26,710 --> 00:18:32,610
where ten years ago now, um, 16 integrated care pilots, really quite interesting.

178
00:18:33,610 --> 00:18:38,930
And many of them. Over half of them had as an outcome that you would reduce unplanned admissions to hospital.

179
00:18:39,770 --> 00:18:41,970
So a pretty obvious target to have.

180
00:18:42,570 --> 00:18:51,650
Um, and um, uh, or every single GP that we interviewed agreed that what they were doing was reducing unplanned admissions.

181
00:18:52,810 --> 00:18:57,290
We looked at the data and actually there was no evidence that if anything was increasing our population.

182
00:18:58,570 --> 00:19:02,930
Um, and, and we didn't have enough time to dig back in, and we were shocked by that.

183
00:19:03,090 --> 00:19:13,930
But probably what was happening is that, um, better integrated care meant that, um, Mr. Smith on a Friday afternoon is visited by his health visitor.

184
00:19:14,890 --> 00:19:20,890
Um, and he's a bit, you know, a bit unwell, a bit iffy bit, and normally wouldn't get picked up.

185
00:19:21,490 --> 00:19:28,930
And normally I'll come back. I'll come back, uh, Tony on Tuesday and we'll have a and we'll see how you're doing with this improved system.

186
00:19:30,010 --> 00:19:33,270
Mr. Smith was whipped into hospital. Spent the whole weekend in hospital.

187
00:19:34,350 --> 00:19:36,750
Then by Monday or Tuesday, it was probably better anyway.

188
00:19:37,950 --> 00:19:45,470
Um, so there's nothing appears in in improving morbidity rates or mortality rates, but you've got an increase in unplanned admissions.

189
00:19:46,470 --> 00:19:49,910
So it's better care. Almost certainly better care.

190
00:19:50,870 --> 00:19:57,830
Um, and desirable. But in terms of the outcomes intended by the integrated care, it wasn't achieving that.

191
00:19:58,110 --> 00:20:05,070
So. So it's a little footnote to the point that asking people about causal maps and what happens

192
00:20:05,430 --> 00:20:09,870
if I do this is only a starting point to then try to understand what really happens.

193
00:20:10,310 --> 00:20:13,510
And you triangulate that with other data, but really, really useful.

194
00:20:14,070 --> 00:20:22,590
And then, of course, data analysis. Um, and one of the first things in data analysis that we would normally do would be cleaning the data sets,

195
00:20:22,950 --> 00:20:26,390
which how many of you have been involved in cleaning data so desperately?

196
00:20:26,910 --> 00:20:30,450
Yeah. Um, I mean, the big uh,

197
00:20:31,010 --> 00:20:38,890
the really big advantage of at some point early in your career of cleaning data is you really understand how the data is constructed.

198
00:20:39,850 --> 00:20:46,930
You get a real feel for what works. If you just ask A.I. to clean the data, it will remove immediately.

199
00:20:47,450 --> 00:20:52,410
Remove all the outliers. But some of those outliers are the very things that you want to find out more about.

200
00:20:52,850 --> 00:20:56,370
Of course, we'll talk about prompts later. You can prompt it to to pay more attention.

201
00:20:56,890 --> 00:21:03,730
You can do other things, but the tendency would be to say and very confidently to say, we've cleaned the data and this is what it looks like.

202
00:21:04,610 --> 00:21:07,410
Um, but you as a researcher have no idea how that was put together.

203
00:21:07,930 --> 00:21:15,010
So I would always, you know, encourage people working with data early on, at least in their career,

204
00:21:15,290 --> 00:21:20,490
to spend a bit of time actually cleaning the data sets and, and seeing how their, how they constructed and how they might be used.

205
00:21:21,130 --> 00:21:25,490
Um, and not just ask AI to do it because then you've got no understanding of that.

206
00:21:26,370 --> 00:21:29,870
Um. Uh, another kind of quite useful thing.

207
00:21:30,430 --> 00:21:33,950
Very often, uh, we benchmark or compare.

208
00:21:34,590 --> 00:21:38,630
Compare against standards would be an obvious thing. You can ask AI.

209
00:21:38,990 --> 00:21:42,390
Say these are the standard procedures for this illness.

210
00:21:43,510 --> 00:21:46,670
For this surgical intervention, whatever it might be.

211
00:21:47,590 --> 00:21:49,910
Um, you can you can establish what those are.

212
00:21:50,390 --> 00:21:57,470
And then you can use AI to check how people are talking about it, what they're doing, and what practices are actually being used.

213
00:21:57,990 --> 00:22:02,270
Um, and that, again, will be a starting point for doing something else,

214
00:22:02,550 --> 00:22:07,070
which you're going back to the surgeons and saying, why are you not doing the standard procedure?

215
00:22:07,790 --> 00:22:14,830
And you need to try and understand more because some of them will be doing it because they simply inept and they should be doing it,

216
00:22:14,950 --> 00:22:15,990
but they don't know what they should be doing.

217
00:22:16,470 --> 00:22:21,990
Others are doing other things because they're innovating and trying out new approaches, and they're really very good at what they do.

218
00:22:22,670 --> 00:22:26,810
Um, we did some work with on quality improvement with surgeons.

219
00:22:28,210 --> 00:22:32,570
This is quite a long time ago. Um, and so you would say to surgeons.

220
00:22:33,770 --> 00:22:38,330
Um, these are the guidelines. Um, why you why don't you follow them?

221
00:22:38,570 --> 00:22:40,410
As a significant group, we're not following them.

222
00:22:40,690 --> 00:22:51,690
And they would almost everyone said I, um, I know that this produces better results for the average surgeon, but, uh, Tom, I'm not an average surgeon.

223
00:22:52,650 --> 00:22:58,890
I'm actually a lot better than that. The techniques that I've evolved over time are my skill that I bring to it.

224
00:22:59,090 --> 00:23:05,690
I mean, that I get better results, not following the techniques. But what came out of that was every every surgeon is above average.

225
00:23:07,690 --> 00:23:16,050
It would seem. Uh, so, um, yeah, comparing against uh, against guidelines and standards can be quite, quite, quite effective.

226
00:23:16,970 --> 00:23:20,330
Um, but also um, benchmarking against other countries, for example.

227
00:23:20,650 --> 00:23:30,070
So OECD, a lot of their learning that they try and do is to just say to governments without any, any other, without a lot of additional comment.

228
00:23:30,910 --> 00:23:36,870
You're doing it this way. You're getting these results in Sweden or Denmark's always better.

229
00:23:37,510 --> 00:23:40,630
Um, they're doing it that way. And they get and they get other results.

230
00:23:41,590 --> 00:23:47,590
But actually, to be honest with it's quite difficult to translate a lot of that into your own thing.

231
00:23:48,110 --> 00:23:53,790
And then, um, then there's this whole thing about kind of reporting and sensemaking.

232
00:23:55,190 --> 00:24:06,190
One interesting thing is that expert judgement, human expert judgement, um, is swayed, is influenced by the quality of the writing.

233
00:24:06,950 --> 00:24:11,070
So very grammatical, very beautifully constructed arguments.

234
00:24:12,710 --> 00:24:16,990
Um, humans respond to that quite positively. AI is going there.

235
00:24:17,310 --> 00:24:20,630
Well, whatever, uh, is less less impressed by that.

236
00:24:21,510 --> 00:24:30,050
On the other hand, if you ask AI to help you write your report, it will arrive generally.

237
00:24:31,090 --> 00:24:33,690
And again, it depends a bit on prompts, but I left it's own devices.

238
00:24:34,290 --> 00:24:41,850
It will be much more overconfident about the findings and it will give you a much smoother Gulliver interpretation of your results.

239
00:24:42,610 --> 00:24:47,330
The nuances will be lost. The the outliers will get lost in all of that.

240
00:24:47,530 --> 00:24:49,210
So. So it kind of cuts both ways.

241
00:24:49,890 --> 00:24:58,090
Which leads you to the conclusion that I will come on to this, really probably a great way is to do both expert solicitation.

242
00:25:00,090 --> 00:25:03,610
So do your Delphi's or whatever you're familiar with.

243
00:25:04,170 --> 00:25:07,690
And so Delphi is a way where the evidence is unclear.

244
00:25:08,770 --> 00:25:10,730
What is the best treatment for lower back pain?

245
00:25:11,650 --> 00:25:18,970
You just ask the ten talk surgeons or clinicians in the country who know back pain, and you ask that questions,

246
00:25:19,530 --> 00:25:24,590
and then you get them to arrive at a consensus, usually with 2 or 3 rounds of questions.

247
00:25:25,510 --> 00:25:30,070
And then then that's your source of consensus seeking advice amongst the experts.

248
00:25:30,910 --> 00:25:36,350
Do that and do the AI and then ask yourself, why are we getting different results here?

249
00:25:36,630 --> 00:25:40,870
What's going on? So you you don't have to privilege one over the other.

250
00:25:41,390 --> 00:25:50,670
You can kind of engage. And so let's, let's let's look at do the expert solicitation, do the do the AI led analysis and then bring that together.

251
00:25:51,630 --> 00:25:58,430
Um, the other bit, to be honest, we haven't really we've only just got started on this is on project management.

252
00:25:59,470 --> 00:26:05,750
And um, so just, just how do you make sure that the project is running smoothly, putting all the dates in advance.

253
00:26:06,110 --> 00:26:09,110
You've got your when, when when interim reports have to be done.

254
00:26:09,430 --> 00:26:14,830
Da da da da da da da da. And it just prompts you to make sure that you're, you're ah, target with, uh, with all of that.

255
00:26:15,510 --> 00:26:20,370
Um, and that will make quite a change, um, in terms of bidding for work.

256
00:26:20,610 --> 00:26:25,330
For example, you always are asked to produce your track record.

257
00:26:26,210 --> 00:26:30,970
What have you done as an organisation? God, it's a boring bit and you always put it to the end and you haven't got enough

258
00:26:31,250 --> 00:26:34,450
time and you miss out with the most important report that somebody else did.

259
00:26:34,690 --> 00:26:36,330
And, and it's a, it's a real pain.

260
00:26:37,170 --> 00:26:45,850
Um, now we're just going to, we're putting all our reports into, into an AI agent, and it will just ask it to give us our track record every time.

261
00:26:46,010 --> 00:26:53,170
And that little one page, it's like just a little bit, but it's going to come up that's happening, you know, 3 or 4 times a week for us.

262
00:26:53,810 --> 00:27:00,690
Uh, it's an enormous saving. Uh, and we're also less likely to miss stuff out than we than we did previously, because it was just.

263
00:27:00,970 --> 00:27:04,490
Well, speaking personally, because I was extremely poor. And of course, you want us to do it.

264
00:27:05,490 --> 00:27:10,610
Uh. Um, anyhow, so that's that's the stages that we're at.

265
00:27:11,010 --> 00:27:17,650
And at each of these stages, we've got things around, um, kind of validation checks and so on.

266
00:27:17,890 --> 00:27:23,390
The one I just wanted. land on in particular is prompt design, as something is called prompt engineering.

267
00:27:24,510 --> 00:27:26,550
Um, that's the prompts that you give it.

268
00:27:27,270 --> 00:27:37,630
Um, and the more precise and carefully engineered your prompts are, the more the more likely it is that you will get to the results that you want.

269
00:27:37,830 --> 00:27:41,830
Right. Um, so so they're using the prompt so you don't just ask your first question.

270
00:27:43,110 --> 00:27:49,790
Um, why is Tomlin a great poet? Um, you then have these other prompts that you would then kind of go through.

271
00:27:50,550 --> 00:27:55,310
Uh, it actually will tell. Sorry. It will tell you it said something is a great poet.

272
00:27:56,230 --> 00:28:00,990
Uh, he writes in Cambridge. He does these things. So, um, and none of which is particularly true.

273
00:28:01,310 --> 00:28:06,350
So you have to be really careful. But if you then use prompt, is he better than so-and-so?

274
00:28:07,190 --> 00:28:11,150
And no critic so far has thought that Tomlin is better than Shakespeare.

275
00:28:12,070 --> 00:28:17,110
Um, or whatever it might be. Um, but I mentioned the comparisons with human judgement as well.

276
00:28:17,310 --> 00:28:21,130
I think that what do. What do expert humans say? What does what?

277
00:28:21,370 --> 00:28:25,010
What does AI say? What do we think about the balance between that?

278
00:28:25,610 --> 00:28:30,810
Oh, another great I talked about, um, uh, just doing doing your, uh,

279
00:28:31,050 --> 00:28:35,250
track record using citation requirements is another incredibly boring bit of our work.

280
00:28:35,970 --> 00:28:39,050
Um, and it will just do that for you, which has got to be got to be good.

281
00:28:39,650 --> 00:28:44,290
Um, so it's got a number of useful things that you can do at each stage.

282
00:28:45,250 --> 00:28:57,250
All of this is quite positive, right? Um, and it is kind of encouraging, um, a degree of optimism about AI, at least amongst my fellow evaluators.

283
00:28:58,050 --> 00:29:02,850
So even in the book I mentioned, all the previous chapters are pretty smart.

284
00:29:03,410 --> 00:29:06,250
They're not entirely gung ho about it, but they're pretty much.

285
00:29:06,810 --> 00:29:11,290
This has got to happen because it's a competitive market and it's making us more efficient.

286
00:29:11,610 --> 00:29:15,570
It's making us more effective. And I think there is insufficient step back.

287
00:29:15,870 --> 00:29:20,470
about what you know, what might go wrong and what are the what are the issues around that?

288
00:29:21,270 --> 00:29:28,230
Yeah. Chris, I want to start a campaign to drop the term prompt engineering.

289
00:29:29,790 --> 00:29:32,990
Huh? Um, because I read this book How to Talk to AI.

290
00:29:33,630 --> 00:29:37,030
So it really changed my life. I know you shouldn't have books that changed your life.

291
00:29:37,870 --> 00:29:50,030
Um, because the idea that there are these magic phrases that you can put in actually is part of the problem with the way we are thinking about AI,

292
00:29:50,510 --> 00:29:56,270
that it's this kind of oracle box, and if you put in the right magic phrase, then it will spit up.

293
00:29:57,030 --> 00:30:03,390
Uh, and what this book talks about, I can't remember the author, is you've got to have a conversation with it.

294
00:30:04,350 --> 00:30:12,550
It is just a random text generator. And so if you give it a crappy prompt, you will get something that's incredibly bland and possibly confabulation.

295
00:30:13,390 --> 00:30:20,650
But if you actually tell it something unique, like, I'm a professor at the University of Oxford,

296
00:30:21,010 --> 00:30:27,690
and I've got to give a lecture to the students later on about, um, developmental evaluation and what I really am.

297
00:30:28,010 --> 00:30:32,130
And the students are coming from these backgrounds, and I want to make just six PowerPoints,

298
00:30:32,890 --> 00:30:36,530
like, I give it all that information and then it produces something.

299
00:30:37,650 --> 00:30:43,770
And then I say, that's not what I want at all. And the reason it's no good is this, this and this.

300
00:30:44,410 --> 00:30:48,370
And then it will say, oh, sorry. And then it changes its mind and then you go back and forth.

301
00:30:49,010 --> 00:30:51,090
But it's not like there's a magic prompt.

302
00:30:51,490 --> 00:31:02,770
It's just that if it if you're going to not just generate random bits of statistics, like, you know, the statistically most probable words,

303
00:31:03,490 --> 00:31:08,330
you've got to tell it something unique and then you get great stuff out of it and then it stops confabulation.

304
00:31:09,210 --> 00:31:12,530
But I think one of the problems is the word engineering. Yeah.

305
00:31:12,770 --> 00:31:19,790
I could probably. At least you die in a particular way of thinking. I would call them prompt conversations because it's it's a bacteria in dialogue.

306
00:31:20,390 --> 00:31:25,390
Hey, they've got vaccine in there. Yeah, I like that. That's that's a really, really nice, nice thought.

307
00:31:25,710 --> 00:31:34,630
But it is the term that's very often used. Okay. So we've just heard about the reasons for exercising some care.

308
00:31:35,070 --> 00:31:39,350
And I want to now begin to move into why we need to exercise some care.

309
00:31:39,910 --> 00:31:46,630
Having the really up to now we've been emphasising some of the virtues and benefits providing you do it with care.

310
00:31:46,870 --> 00:31:55,110
But there are some other challenges. It's quite helpful to try and separate out the levels of risk.

311
00:31:55,750 --> 00:32:02,470
Or if you like, you know, what is it good for? What is it? What where is it quite straightforwardly helpful and where is it?

312
00:32:02,590 --> 00:32:06,710
A more complicated thing. And um, and this will change over time.

313
00:32:07,270 --> 00:32:11,290
By the way, I think that this is already I mean, you we mentioned the the literature review.

314
00:32:11,770 --> 00:32:16,370
I think we just get far fewer hallucinations now. I don't know how that's happening, but it is.

315
00:32:17,090 --> 00:32:20,850
Um, so, um, it's very good for text summaries.

316
00:32:21,850 --> 00:32:32,650
Um, uh, for basic qualitative coding. Translation. Um, uh, it can, uh, it can do basic code generation statistics and document formatting.

317
00:32:33,250 --> 00:32:37,330
So quite, quite a, I mean, these are, you know, not not useless things by any means.

318
00:32:38,050 --> 00:32:42,650
Um, but for more complex analysis, you need to have a bit more care, probably doing what we're talking about,

319
00:32:43,130 --> 00:32:47,250
checking it against other agents, checking it against expert, uh, expert views.

320
00:32:48,010 --> 00:32:52,530
Um, the kind of logic model, theory of change development, real time data collection.

321
00:32:53,410 --> 00:32:58,210
Uh, you're familiar with the idea of theory of change, aren't you? If you all been. Yeah, yeah, we've done a lot of downloads of that.

322
00:32:58,690 --> 00:33:03,330
That is probably the one, um, idea that is unique.

323
00:33:03,930 --> 00:33:07,490
There's a unique contribution from the evaluation world, actually.

324
00:33:08,290 --> 00:33:13,630
Uh, there's that most of the time. If you like parasitic on the other social sciences.

325
00:33:14,550 --> 00:33:18,270
But the theory of changes is quite kind of critical to the way that we think.

326
00:33:19,510 --> 00:33:24,510
Um, but we're talking over lunch about the sometimes the theory of change can be very static.

327
00:33:25,070 --> 00:33:34,070
And what you're really interested in would be a theory of, um, of, of, of evolution, of the trajectory that you're on.

328
00:33:34,550 --> 00:33:37,910
If you've got a complex intervention or you've got an intervention in a complex system.

329
00:33:39,630 --> 00:33:43,710
Um, it can be one a very small intervention in a very large system.

330
00:33:44,510 --> 00:33:51,430
And you need to understand the trajectory of that wider system before you can begin to make sense of what's happening.

331
00:33:52,110 --> 00:33:56,390
Of how the intervention of interest is, is shaping that.

332
00:33:57,110 --> 00:34:01,830
Um, so I'm just doing some work for the Government of Northern Ireland, not in health for some reason.

333
00:34:02,630 --> 00:34:06,990
Um, or because they wanted a realistic evaluation on the training system in Northern Ireland.

334
00:34:07,430 --> 00:34:12,810
So the training system in Northern Ireland has received vast amounts of money in the last 4 or 5 years.

335
00:34:13,650 --> 00:34:19,010
There are some great examples of trainers. There's some great further education outfits providing good quality training.

336
00:34:19,970 --> 00:34:25,770
The gap between the skills needed in the economy and the skills available hasn't shifted, if anything has got worse.

337
00:34:26,250 --> 00:34:30,290
So you've got what appears to be a good system, but it's not delivering.

338
00:34:31,250 --> 00:34:34,010
It's a bit like, you know, the operation was a complete success, but the patient died.

339
00:34:34,970 --> 00:34:41,050
So, um, so but but in order for us to make sense of that, we have to understand that at the UK level,

340
00:34:41,730 --> 00:34:47,610
what's happening to training, what's happened to young people in particular, NEETs not in employment, education or training.

341
00:34:48,490 --> 00:34:53,970
Um, and how is that evolving? How is that distinctive, rather different in Northern Ireland?

342
00:34:54,650 --> 00:35:02,330
What shapes that? And then why is the training system landing in the way that it is within this very complex and evolving system?

343
00:35:02,810 --> 00:35:06,610
But unless you think about that trajectory, then I think you're you're missing that.

344
00:35:06,820 --> 00:35:10,900
I think a really useful area for. I would be trying to understand trajectories more.

345
00:35:11,980 --> 00:35:15,540
Um, that where we're particularly in complex systems.

346
00:35:16,540 --> 00:35:26,180
Um, and, uh, I've spent quite a bit of time trying to understand why interventions to reduce health inequalities don't work.

347
00:35:27,500 --> 00:35:30,620
Um, and there's that's pretty much as broad as that.

348
00:35:30,780 --> 00:35:37,540
They never have. We haven't seen any significant, uh, state led interventions that have actually improved health for everyone,

349
00:35:37,980 --> 00:35:39,140
but they haven't narrowed the inequalities.

350
00:35:39,900 --> 00:35:49,860
Brown was a bit of an example where that because of a start, we saw some some just slight flexing of the gradation, um, which links,

351
00:35:51,140 --> 00:35:58,220
uh, poverty or social inequalities with health inequalities that just kind of tilted a little bit there, but it hasn't changed.

352
00:35:58,540 --> 00:36:05,560
And that's because what we need to be able to do is to understand the intersection amongst neuroscience, uh, foetal development.

353
00:36:06,600 --> 00:36:15,080
Um, uh, biology, sociology, economics, uh, uh, history, very much kind of historical circuit and.

354
00:36:15,480 --> 00:36:19,680
Bring those together and it's beyond our capacity as humans to to manage that.

355
00:36:20,040 --> 00:36:27,560
I kind of see some of these really areas of high level complexity, trying to use AI to explore those and bring in multi disciplines.

356
00:36:28,440 --> 00:36:31,240
Because the other problem is that the disciplines don't speak to each other very easily.

357
00:36:32,280 --> 00:36:39,080
Um, I mean, there's almost no sociologists will really have any familiarity with neuroscience, I think, and foetal development.

358
00:36:39,640 --> 00:36:44,760
And yet that's kind of crucial to if you're looking at health inequalities for people in their 30s,

359
00:36:45,880 --> 00:36:51,480
those are to a very large extent linked to their first thought the first thousand days after conception.

360
00:36:52,520 --> 00:36:55,840
So very, very powerful. And it's intergenerational as well.

361
00:36:56,200 --> 00:37:00,320
It floods through the generations. So understanding that it's very difficult for us.

362
00:37:01,080 --> 00:37:06,420
Um, but I would like to think that we can build models using AI that are genuinely interdisciplinary.

363
00:37:07,700 --> 00:37:14,300
So welcome. Are trying to do interdisciplinary work or fund interdisciplinary work, I should say.

364
00:37:14,900 --> 00:37:19,060
Ukri Trisha was just talking about that are doing doing work in that space.

365
00:37:19,900 --> 00:37:24,460
Um, it's very difficult for us that even when we want to talk to other disciplines to do that.

366
00:37:24,660 --> 00:37:31,500
Well, I suspect the AI can do that better than we can. Um, it is less it is less locked into its own disciplines.

367
00:37:32,420 --> 00:37:40,540
Um, so but the so this kind of causal inference, um, is at the moment it's not reliable at all.

368
00:37:40,820 --> 00:37:46,020
But I wonder whether that's the kind of area where we can begin to push forward, um,

369
00:37:46,260 --> 00:37:56,140
going into this kind of interdisciplinary, highly complex social interactions that produce outcomes that are of, of interest.

370
00:37:56,780 --> 00:38:03,000
So I distinguish between flatter things or smooth terrains and rugged terrains.

371
00:38:04,480 --> 00:38:11,120
A rugged terrain is where the same intervention has very different outcomes depending on where it lands in that terrain.

372
00:38:12,040 --> 00:38:15,680
Um, and so the same that it will.

373
00:38:16,320 --> 00:38:22,200
Yeah. And a smooth terrain is where the same intervention has very similar, uh, outcomes where,

374
00:38:22,560 --> 00:38:25,320
wherever it's implemented, when you're doing a randomised controlled trial,

375
00:38:26,040 --> 00:38:31,240
what you're trying, what you consciously do is to turn what might be a rugged terrain into a smooth terrain,

376
00:38:31,720 --> 00:38:34,840
and that you narrow down what the variables that you're looking at.

377
00:38:35,200 --> 00:38:41,880
You focus on one population and you focus on one, one particular, um, uh,

378
00:38:42,720 --> 00:38:47,960
intervention and one particular outcome or a set of outcomes, and you kind of get your measure.

379
00:38:48,360 --> 00:38:52,160
That's a very, very smooth terrain. What you're talking about is very rugged terrain.

380
00:38:53,080 --> 00:38:57,520
Uh, and that is where I think I wonder whether there's more we can do.

381
00:38:57,680 --> 00:39:00,320
What? It's beyond my limited capacity.

382
00:39:00,760 --> 00:39:08,500
If you guys are great, I would get off and try and really think about causal inference in rugged terrains and how we can make some progress on that.

383
00:39:09,020 --> 00:39:11,300
And I'm in a good place to start might be health inequalities,

384
00:39:11,860 --> 00:39:18,980
but that might actually be such a challenging thing because it is the causal mechanisms are so varied and context dependent.

385
00:39:20,220 --> 00:39:20,700
I think it would be.

386
00:39:20,940 --> 00:39:30,700
But I think, yeah, I think making some progress beyond the evaluation conclusions that say the results are patchy and context dependent,

387
00:39:31,540 --> 00:39:36,500
I think would be very helpful because I've finished up seeing that so many times, and I'm embarrassed when I have to say that again.

388
00:39:36,900 --> 00:39:40,620
Yeah, there's another dimension of complexity here, Thomas.

389
00:39:41,100 --> 00:39:47,820
No, in inequalities. Um, it's it's partly because terrains are rugged, like you say.

390
00:39:48,020 --> 00:39:52,420
It's partly because there are so many multi-level causal mechanisms all interacting.

391
00:39:53,980 --> 00:39:59,200
And that is mathematically complex. But there's also another aspect of complexity which I.

392
00:39:59,680 --> 00:40:03,640
I really value your view on how I might help with this one, because I think it can.

393
00:40:04,680 --> 00:40:13,320
Is the idea that certain findings and certain solutions are just politically unacceptable or ideologically unacceptable?

394
00:40:13,960 --> 00:40:18,440
And I'm thinking of Margaret Thatcher pulping the Black Report, for example,

395
00:40:19,520 --> 00:40:25,000
that the idea that you could say that poverty is a social determinant of help.

396
00:40:26,040 --> 00:40:32,880
Rather than, say, people's behaviour. 80% of people's health is explained by their own behaviour.

397
00:40:33,600 --> 00:40:40,920
So the second of those explanations, airbrushing out the fact that being poor might shape your behaviour.

398
00:40:41,880 --> 00:40:46,760
And that's what we call value complexity.

399
00:40:47,960 --> 00:40:54,680
Now, I'm fine about the mathematical complexity. I'm playing around with AI doing that, but I just wonder, could we bring in value complexity?

400
00:40:55,280 --> 00:41:03,940
Could we get AI to start saying, rank these solutions or tell us where the political touch points are here.

401
00:41:04,940 --> 00:41:14,580
Um, yes, a lot we have, um, I mean, just in passing the black report, of course, um,

402
00:41:15,420 --> 00:41:19,820
it passed unnoticed at the time that it looked to health inequalities amongst men.

403
00:41:20,180 --> 00:41:25,900
Women didn't really appear in it. Um, but, um, you didn't have any women in it?

404
00:41:26,260 --> 00:41:29,260
No, no, I say it's about it's about inequalities amongst men. Yeah.

405
00:41:29,740 --> 00:41:42,820
Yeah, it's it's worth it. Um, so, um, the the yeah, the, the I what I wonder whether you have, um, clusters of,

406
00:41:43,860 --> 00:41:50,740
um, solutions that can be organised around differing ideological positions.

407
00:41:52,140 --> 00:41:55,380
Um, uh. Oh yes. That, that might just be probabilistic.

408
00:41:57,000 --> 00:42:02,880
but it'll be quite in going back to that point. You know that these normally are associated with these particular views.

409
00:42:03,280 --> 00:42:06,680
So you would um, so that's a really good idea. And it could do that.

410
00:42:07,120 --> 00:42:08,520
Yeah, it would definitely be able to do that.

411
00:42:09,280 --> 00:42:23,480
Um, and um, so yeah, if you, if you could understand if you want to really go down a market driven set of solutions for reducing,

412
00:42:25,800 --> 00:42:33,400
uh, harmful health behaviours, then you would be going down the road which says you want to make sure the competition

413
00:42:35,480 --> 00:42:43,160
amongst providers and the information to consumers both drive opportunities to,

414
00:42:43,720 --> 00:42:49,280
to, uh, eat healthily, for example, to gamble less, to drink less, to smokeless.

415
00:42:50,400 --> 00:42:58,060
Um, and so that's the solution that says, um, and part of that is responsible ization in the present language.

416
00:42:59,260 --> 00:43:06,460
Um, so what the, uh, tobacco companies did originally, what the gambling industry is doing today.

417
00:43:07,740 --> 00:43:17,900
Um, what the food industry and alcohol industries are also following their playbook is to use responsible ization as a first step,

418
00:43:18,780 --> 00:43:26,820
despite the fact that many of us might want to argue that the individuals making their choices over gambling addiction,

419
00:43:27,580 --> 00:43:32,580
particularly addictive behaviours, are not in a position to exercise their choice responsibly and freely.

420
00:43:33,460 --> 00:43:38,620
Um, and that that's shaped by, for example, availability of quality shops near where you live.

421
00:43:39,420 --> 00:43:45,180
Um, the way in which supermarkets, uh, press certain products on you uh, the two for one office and alcohol.

422
00:43:45,820 --> 00:43:49,300
Well, you know all this stuff, right? Um, the, the glass sizes.

423
00:43:50,220 --> 00:43:56,720
Have you come across that one? Good, good study. Theresa. Motto in Cambridge, um, uh, this is the perverse results of research.

424
00:43:57,160 --> 00:44:04,000
Right. Um, uh, the glass size when people were buying, uh, wine by the bottle in restaurants,

425
00:44:04,960 --> 00:44:11,000
there was randomised buy days of the week and across the different restaurants, and they would be given different glass sizes.

426
00:44:12,000 --> 00:44:14,920
And the ones that were given like, like neat little small glasses,

427
00:44:15,520 --> 00:44:21,720
drank less wine than bought fewer bottles than the ones that had their kind of small buckets in front of them.

428
00:44:22,280 --> 00:44:29,000
Uh, and we're kind of guzzling that down. Um, but of course, of course, that information is fantastically useful for public health policymakers,

429
00:44:29,880 --> 00:44:33,760
but it's also very useful for volunteers who want to sell more of their wine.

430
00:44:35,160 --> 00:44:38,760
So the results of that study seems to be that now, whenever you go into a Cambridge,

431
00:44:39,640 --> 00:44:46,720
uh, restaurant, uh, the glass sizes are getting bigger and bigger. Um, and it just because you drink, you know, you don't drink more.

432
00:44:47,320 --> 00:44:50,840
You say, well, just I'll just have one glass or whatever. And then anyway, you know, the point.

433
00:44:51,320 --> 00:44:58,660
Um, the point is about this obesity genetic environment. Um, we do create an obese people.

434
00:44:59,540 --> 00:45:03,220
Um, and, um, so just to go back to the clustering point,

435
00:45:03,620 --> 00:45:10,540
there's a set of solutions that would relate to what you might call public health interventions addressing inequalities,

436
00:45:11,780 --> 00:45:14,780
building on social justice and meaningful empowerment.

437
00:45:15,740 --> 00:45:17,140
That's one cluster that would go together.

438
00:45:17,860 --> 00:45:23,780
There's another cluster, I would suspect, which is about encouraging competition, encouraging responsible behaviour,

439
00:45:24,500 --> 00:45:33,340
empowering individuals so that they can can behave responsibly, um, and using market mechanisms to give you better health outcomes.

440
00:45:34,260 --> 00:45:41,140
And that's the kind of neoliberal cluster. I'm sure there's probably an authoritarian cluster in all of that as well.

441
00:45:41,740 --> 00:45:45,220
Uh, well, actually the aids, the first response and oh, sorry, Covid,

442
00:45:45,700 --> 00:45:51,320
they allegedly one of the junior ministers, um, uh, when asked about how they were going to deal with Covid.

443
00:45:51,800 --> 00:45:59,720
The response was just a scare. The pants off them was the was the idea to terrify them so that they start to behave themselves,

444
00:46:00,840 --> 00:46:04,080
which is, of course, as we know, are not a very good way of changing behaviour in the long term.

445
00:46:05,560 --> 00:46:15,840
So how do we get on to this? We got into this because there are areas where it now seems we're pretty comfortable with how we are using AI.

446
00:46:16,880 --> 00:46:23,160
Um, on, on these, you know, just improving the drafts that we've got and reviewing documents and a little bit of that.

447
00:46:23,720 --> 00:46:30,560
We're a little bit more complicated on logic models, theory of change development, but actually it's pretty useful for first drafts on that.

448
00:46:31,040 --> 00:46:38,720
And then this uncertain area where, um, the causal inference, normative assessment,

449
00:46:39,640 --> 00:46:45,280
it's still really kind of emerging as to how comfortable we're going to be using it in that, in that space.

450
00:46:45,680 --> 00:46:50,620
But, um, now that we've been thinking about normative assessment, You've got your new idea to take that.

451
00:46:51,580 --> 00:46:55,300
Take that forward. Just looking at our time, we're doing pretty well.

452
00:46:56,020 --> 00:47:01,820
Um, but taking a bit more deeply into what might be problematic, uh, in, uh, in this then.

453
00:47:02,540 --> 00:47:12,100
Um, so, um, the transition from AI is a two under the supervision of humans to AI tools or functions that are more autonomous.

454
00:47:13,220 --> 00:47:23,380
Um, and this is the superintelligence, uh, challenge, where, uh, AI agents are no longer given their parameters.

455
00:47:24,340 --> 00:47:33,420
No longer take their prompts, engineered or not from from humans, but are taking it from other agents and are responding in that,

456
00:47:33,820 --> 00:47:42,100
in that world and that that kind of creates, um, significant issues about evaluative knowledge you might have.

457
00:47:42,420 --> 00:47:47,760
So your evaluation comes to certain conclusions. But when you're asked, how did you arrive at that?

458
00:47:48,200 --> 00:47:57,160
You can't actually say how we got to that conclusion. Um, your answer is, is going to really be about probabilistic logic and about algorithms.

459
00:47:57,720 --> 00:48:01,600
We use these algorithms to the extent that we know what algorithms were used.

460
00:48:02,280 --> 00:48:03,320
These are the ones that were used.

461
00:48:03,800 --> 00:48:11,720
And this is why we're getting the kind of results that we got that you would call algorithmic truth, right for evaluators.

462
00:48:13,120 --> 00:48:18,680
We're also interested in surfacing differences of experience amongst service users.

463
00:48:19,480 --> 00:48:30,120
We're interested in different notions of justice. We're interested in, um, differing accounts of what something means or different different groups.

464
00:48:30,880 --> 00:48:33,880
Um, and that we would call deliberative truth.

465
00:48:34,720 --> 00:48:37,720
I don't want to argue that one is necessarily better than the other.

466
00:48:38,480 --> 00:48:41,880
I think the algorithmic truth is interesting, but it's not. It's probabilistic.

467
00:48:43,120 --> 00:48:49,700
It is not the same as deliberative truth, which is what we normally talk about when we're talking certainly an evaluation.

468
00:48:50,740 --> 00:48:52,100
What is the what? What are your findings?

469
00:48:52,780 --> 00:49:02,340
Our findings are and you would talk about that kind of deliberative process in the hybrid world towards which we are heading.

470
00:49:03,340 --> 00:49:12,740
Then those two, I think you want to be able to talk clearly about those two ideas without necessarily abandoning one over the other.

471
00:49:13,500 --> 00:49:19,580
Um, so the um, yeah, the current discourse, I think, in sorry,

472
00:49:20,140 --> 00:49:26,260
I should say for evaluators, this is um, it really is about optimising AI implementation,

473
00:49:27,340 --> 00:49:35,700
and it's not really getting to grips with those deeper problems about the difference between deliberative truth and, and algorithmic truth,

474
00:49:36,580 --> 00:49:41,860
um, large language models, which is the main there's not the only bit of AI, but it's the main thing that we are often using.

475
00:49:42,780 --> 00:49:48,240
Um, the other thing that they do is that they present their findings with exactly

476
00:49:48,840 --> 00:49:53,440
the same language as we as humans use when they present deliberative findings,

477
00:49:54,040 --> 00:49:59,120
write the truth in their own way. So we are seduced into this false.

478
00:49:59,640 --> 00:50:04,040
This is the the other bit that Trish was talking about, about using the word engineering.

479
00:50:05,000 --> 00:50:10,600
We're seduced into a way of thinking which is not merited by the technology that we're using.

480
00:50:11,760 --> 00:50:15,240
Uh, and, um, yeah.

481
00:50:15,600 --> 00:50:22,680
So this kind of the key thing words for me, inclusive deliberation, interpreting meaning from different groups,

482
00:50:23,320 --> 00:50:31,680
communicating to diverse audiences, um, and addressing what you think is the value or worth of the thing.

483
00:50:32,320 --> 00:50:41,600
Those that sentence for me is kind of where, where I think the important, uh, pushback from an evaluation point of view needs to, needs to set.

484
00:50:42,480 --> 00:50:49,620
Um, and it's not that we don't use the other stuff, but the the algorithmic truth doesn't get to those, those points.

485
00:50:50,660 --> 00:51:01,900
Um, and, um, that's just this has, um, I think six I'm going to talk about, um, uh, particular challenges that it creates.

486
00:51:03,740 --> 00:51:09,740
Okay. So there are kind of six little points are what there is now as we head towards our conclusions.

487
00:51:10,780 --> 00:51:21,260
Um, the there is a problem if we arrive at evaluative conclusions and recommendations based on moral reasoning that we cannot understand.

488
00:51:23,220 --> 00:51:26,820
Um, and we simply have to be very overt about that.

489
00:51:27,020 --> 00:51:32,220
We can't, um, simply bring that into a conclusion, uh,

490
00:51:32,380 --> 00:51:41,320
and treat it in the same way that we would treat a conclusion that we've arrived at through the collecting our own data, analysing our own data.

491
00:51:41,920 --> 00:51:50,200
Interviewing people. Doing our ethnographic studies, whatever it is that we've done and formed a judgement, however imperfect that judgement is,

492
00:51:50,680 --> 00:51:57,760
it would be our judgement and we are entitled to say this is our judgement and we are like and we put it out for peer review.

493
00:51:58,280 --> 00:52:05,000
We're welcome. We welcome your thoughts on that. That is very different from saying this is the we've done this, this work.

494
00:52:05,880 --> 00:52:10,360
We're going to be transparent and tell you that we've used these different AI agents in these ways.

495
00:52:10,720 --> 00:52:16,440
So we're not going to lie about it. But then we we can't then say these are our conclusions.

496
00:52:17,440 --> 00:52:21,840
You can only say these are the conclusions that that emerge driven by this technology.

497
00:52:23,640 --> 00:52:31,800
Um, and the integration of AI into evaluative practice, um, will disrupt the traditional model of the responsible evaluator.

498
00:52:33,280 --> 00:52:38,280
Um, and we need to be able to talk overtly about distributed agency.

499
00:52:39,320 --> 00:52:42,500
What have we done? How do we do that? What is the AI done?

500
00:52:42,940 --> 00:52:49,740
Which bits do we know about from AI? What bits do we not know? And then we can we can talk about distributed agency across that.

501
00:52:49,980 --> 00:52:52,660
And I think that's fine in my in my view.

502
00:52:53,460 --> 00:53:03,900
Um, the EU 2024 act, um, the EU, I mean obviously a particular problem that the EU and wouldn't be wonderful to be part of it,

503
00:53:04,060 --> 00:53:10,740
but um, the particular problem of the EU is that you've got to get all the member states to agree before you get.

504
00:53:11,060 --> 00:53:18,380
So it tends to be quite behind the times, the recommendation that that AI can really only be used if it's got human oversight.

505
00:53:19,180 --> 00:53:23,580
That ship has sailed and it already had sail by 24, I think.

506
00:53:23,900 --> 00:53:31,500
But um, then so there is a problem about how you use AI within and comply with EU requirements on that.

507
00:53:31,740 --> 00:53:37,500
I think, um, and it's much more in the format that I'm talking about that we need to go forward.

508
00:53:38,320 --> 00:53:40,320
Um, and and. Yeah. So they've got.

509
00:53:40,600 --> 00:53:46,720
I think the EU has got themselves in a little bit of a pickle on that, as I understand it, which we might be able to use.

510
00:53:47,320 --> 00:53:54,240
Um, and their knowledge, concentration and representation, this is I mean, one of the big issues is that, uh, and of course,

511
00:53:54,480 --> 00:54:02,640
you can use prompts to try and address this, but AI agents are generally trained on dominant discourse in the global north.

512
00:54:03,600 --> 00:54:12,600
Um, and, um, when you're looking at, um, minority views in the global North that you won't find those very,

513
00:54:13,320 --> 00:54:18,120
uh, apparent, but also indigenous views are almost entirely absent from this.

514
00:54:18,320 --> 00:54:29,200
So if you're if you're trying to understand the uptake of a health intervention in Hawaii and you're using only an AI,

515
00:54:29,600 --> 00:54:36,760
a kind of standard AI agents, it will tell you that there's a lack of concordance, there's a lack of understanding.

516
00:54:38,060 --> 00:54:39,380
There's a refusal to participate.

517
00:54:40,140 --> 00:54:46,620
There's a kind of irrationality on the part of the local population, and they're not engaging with this perfectly sensible thing.

518
00:54:47,140 --> 00:54:50,300
On whatever, whatever the health intervention is ignoring.

519
00:54:50,940 --> 00:54:55,540
I mean, ignoring, um, indigenous understanding of health care.

520
00:54:55,780 --> 00:55:04,140
So an example of that would be after one tsunami, um, the, the, the Western intervention was encouraged,

521
00:55:04,700 --> 00:55:11,140
was actually pushing the idea of counselling, uh, step apart from the people, your family and friends.

522
00:55:11,740 --> 00:55:15,980
You talk with your counsellor, uh, you go through the trauma. Um, and actually,

523
00:55:16,380 --> 00:55:20,020
what was very clear is what they were saying is what I need to do when I'm

524
00:55:20,260 --> 00:55:24,060
after a traumatic event like this is to meet with my family and my community,

525
00:55:25,380 --> 00:55:28,540
dance and sing. Um, and that's how I heal.

526
00:55:29,220 --> 00:55:34,580
And it's kind of like, oh, yeah. So there's, there's this, this kind of what counts as knowledge.

527
00:55:35,140 --> 00:55:41,000
And how is that represented in in the way that we are working with, with AI?

528
00:55:41,840 --> 00:55:48,920
Um, so as it says it can. It reinforces, um, profound asymmetries that are already there,

529
00:55:49,400 --> 00:55:57,440
but it actually amplifies those unless you deliberately go about addressing those and undercutting that.

530
00:55:58,120 --> 00:56:07,360
Um, however, what I don't want to argue this has been the kind of theme that somehow human knowledge is, um, absolutely clear.

531
00:56:07,840 --> 00:56:15,200
And there's no there's no inequalities. There's no, uh, you know, that we don't we experts don't exclude indigenous knowledge and all of that as well.

532
00:56:15,680 --> 00:56:20,040
That all knowledge is constructed and all social knowledge is constructed.

533
00:56:20,600 --> 00:56:26,000
All human knowledge is constructed. What we need to understand are the different ways in which that knowledge is being constructed.

534
00:56:27,200 --> 00:56:32,040
Um, and so in saying the first point that it's reproducing asymmetries,

535
00:56:32,880 --> 00:56:38,780
I don't want to contrast that with this perfect realm of unbiased human knowledge.

536
00:56:39,900 --> 00:56:44,860
What I want to do is to say, we've got, you know, that we need to think about these different forms of construction and,

537
00:56:45,140 --> 00:56:49,820
and biases and how they and how they may, may work together.

538
00:56:50,380 --> 00:56:58,540
But I do think it's very, very important. The shin point, um, which is that algorithmic truth is computed.

539
00:56:59,260 --> 00:57:07,780
It's a probabilistic, you know, it's a probabilistic truth and it's not and it is rooted in sociotechnical systems.

540
00:57:09,340 --> 00:57:13,020
And you have to understand that if you want to be able to know how you want to,

541
00:57:13,580 --> 00:57:17,060
uh, to, to use, use it, which isn't to say that you shouldn't be using it.

542
00:57:17,220 --> 00:57:26,420
You just need to know that that's what's going on. Um, and some people, I think, quite rightly raised the spectre of AI colonialism.

543
00:57:27,540 --> 00:57:34,360
It's just like, um, without necessarily intending it becomes another form of colonialism or neo colonialism.

544
00:57:35,480 --> 00:57:41,120
So where are we going next? Um, I mean, what I think.

545
00:57:41,360 --> 00:57:47,720
I mean, you guys will be you guys that drive it. I mean, um, I think that, uh,

546
00:57:47,920 --> 00:57:54,760
the main point is that we we can't meaningfully review conclusions when we've no understanding about those we arrived at,

547
00:57:55,160 --> 00:57:57,800
and we have to be very open and overt about that.

548
00:57:58,600 --> 00:58:05,080
And we can't take responsibility for the outcomes derived through that process when it's beyond our cognitive ability to do so.

549
00:58:05,840 --> 00:58:14,440
Um, so the response in many quarters EU, uh, UK evaluation Society, um, to a limited degree.

550
00:58:16,640 --> 00:58:23,320
Um, so this is kind of like the uh, is um, is, by the way, um,

551
00:58:24,200 --> 00:58:30,890
introduced guidelines on using AI for reviewers when they're reviewing, uh, Nature research.

552
00:58:32,010 --> 00:58:41,770
Um, and the kind of thing we have to be told. So, like reviewing proposals, it goes out for external review and it would seem some reviewers,

553
00:58:42,410 --> 00:58:46,210
instead of actually sitting down and reading the damn thing or pulling it through AI anyway.

554
00:58:46,770 --> 00:58:53,010
So the NIH is saying, don't do that anyway. Um, but, um, so it's a procedural solution.

555
00:58:53,530 --> 00:59:02,050
It's one and it's not a bad start. You know, where you're just saying, um, that you, you, you, you have to be transparent and overt about,

556
00:59:03,090 --> 00:59:08,090
um, how you adhere to specific ethical constraints and you produce and do all of that.

557
00:59:08,410 --> 00:59:13,570
The difficulty is that that's I mean, it is very tempting, and that is important to aim for that.

558
00:59:14,050 --> 00:59:16,210
It's very difficult now to deliver that in practice.

559
00:59:17,370 --> 00:59:25,890
Um, because the procedure, the human control, human led procedures don't have sufficient grip to be able to control what the AI is doing.

560
00:59:26,850 --> 00:59:36,030
Um, and so I think a better approach is going to be to be much more clear about the hybridity of what we're doing,

561
00:59:36,550 --> 00:59:44,790
what is human, what is not, and how we combine those, um, in, in saying these are judgements arrived at one way.

562
00:59:45,150 --> 00:59:51,110
These are judgements arrived at in other ways. Isn't it interesting that they align or they don't align or whatever?

563
00:59:51,990 --> 00:59:56,710
Um, and then you have your deliberation using, using that uh, element.

564
00:59:57,590 --> 01:00:05,710
Um, and of course a superintelligent AI will is, is whether if it does happen will be a major challenge to that.

565
01:00:06,590 --> 01:00:14,790
Um, almost at the end now, but it'd be great to open up for questions five, five questions one is about kind of data and value inputs.

566
01:00:15,230 --> 01:00:18,710
And that's those are the questions about who who's creating the data. How is that being done?

567
01:00:19,150 --> 01:00:22,470
What cultural, geographic and social values dominate training application.

568
01:00:23,710 --> 01:00:32,770
Um, and what analytic frameworks are going into that. And that's how do we not just by default allow AI colonialism to to operate?

569
01:00:33,730 --> 01:00:38,010
Um, the interpretive authority filter is who participates in making sense of the data.

570
01:00:38,890 --> 01:00:45,330
Um, it's very tempting for, you know, people like us, the experts sitting in the room just to make make sense of the data.

571
01:00:46,210 --> 01:00:51,010
There's a really strong case for saying we just have to take the data back to the service users,

572
01:00:51,650 --> 01:00:57,570
back to professionals in the field and say, this is what we're finding. Well, how does that cohere or not with your experience?

573
01:00:58,210 --> 01:01:03,290
What what does that mean for you? And if once you find that out, would you would you change your behaviour?

574
01:01:04,370 --> 01:01:08,130
Is that something that you think, oh, right. Actually actually I do need to change that.

575
01:01:08,330 --> 01:01:14,050
So going back to the GPS and saying we've done the stats and I can tell you your behaviour is not reducing unplanned admission.

576
01:01:15,730 --> 01:01:20,450
What what are you going to do about that. So um, then agency and accountability checkpoints.

577
01:01:21,490 --> 01:01:28,750
Uh, and that's the thing about who did what when and just having and um, documenting that as you go through your, your your work.

578
01:01:29,510 --> 01:01:32,030
Um, and then that kind of final alignment and transparency.

579
01:01:32,670 --> 01:01:39,630
How does the final evaluative recommendations reflect kind of ethical standards, local values and clear accountability.

580
01:01:40,630 --> 01:01:46,470
That kind of local values is really important. Um, how will the how do these findings resonate?

581
01:01:47,230 --> 01:01:53,350
And in in localities up and down the country, the public health team in nice.

582
01:01:54,390 --> 01:02:00,190
Years ago used to have the five WS, which is will it work on a Wednesday afternoon in Wigan.

583
01:02:00,990 --> 01:02:05,470
Um, um, which is at IHI. You know, this is a great idea thought up by experts.

584
01:02:06,470 --> 01:02:06,990
Uh, but actually,

585
01:02:07,390 --> 01:02:14,430
will it be remotely meaningful on the ground when we try and make it work and that that five WS question maybe doesn't have to be wicked?

586
01:02:15,110 --> 01:02:18,950
Um, that, um, is kind of another way of thinking.

587
01:02:19,350 --> 01:02:23,190
Well, who needs to. Who do we need to consult about? Uh, about the meaning.

588
01:02:23,750 --> 01:02:32,090
What meaning we can get to this. The final bit is that if you're going to be an evaluator in the future, you've just got to acquire these skills.

589
01:02:33,210 --> 01:02:41,650
You know, it's not a it's not a matter of choice. I'm afraid you're going to be using this need to be able to talk to people that

590
01:02:41,770 --> 01:02:46,770
are experts in using AI and and understand what it is they're telling you.

591
01:02:47,410 --> 01:02:53,770
Um, it's no longer something that you can say, well, I hope it just passes us by or I hope that somebody else.

592
01:02:54,170 --> 01:02:57,530
I hand that over to the AI team, the techie experts.

593
01:02:58,530 --> 01:03:04,130
So at Rand, we've created a kind of AI expert team. And it's, you know, I think it will work.

594
01:03:04,330 --> 01:03:12,450
Okay. But the worrying bit is that somehow in people's heads, they think, oh, those guys do AI and I do evaluation and research.

595
01:03:13,330 --> 01:03:16,050
Um, and it it can't that isn't going to work.

596
01:03:16,410 --> 01:03:25,790
So, um, the skills and capabilities and you're going to be good at this, you're going to need to have those skills as you develop your careers.

597
01:03:26,670 --> 01:03:31,950
Um, is this a transformative moment? Um, a word transformation is overused.

598
01:03:32,390 --> 01:03:33,190
But putting that to one side.

599
01:03:33,790 --> 01:03:41,630
Um, I mean, for sure, the hybridity of human and AI intelligence, um, has changed the evaluation workflow where we started.

600
01:03:42,470 --> 01:03:44,590
Without a doubt, that is now looking very different.

601
01:03:45,550 --> 01:03:51,630
Um, it's quicker, it's more responsive, and it's got biases written into it because of the AI that we're using.

602
01:03:51,990 --> 01:03:58,630
So but it's changed. Um, the skill sets that we require last change, we're not going it's not going to go back.

603
01:03:59,230 --> 01:04:04,230
Um, and as Trish was saying, if you haven't done it, you've just got to go out and do a few hours training.

604
01:04:04,670 --> 01:04:10,030
It's not, you know, it's not rocket science. Um, it's digital science.

605
01:04:10,750 --> 01:04:19,750
Um, so it's like in 1971, the economist Simon, um, talked about, uh, the info glut, then just the glut of information.

606
01:04:20,510 --> 01:04:24,570
And you said that there's the the problem today is not a lack of information.

607
01:04:25,810 --> 01:04:29,770
Um, it's that we've got more information than we've got the ability to attend to.

608
01:04:30,210 --> 01:04:36,730
The value is an attention. And back. Back in 71, he was saying people will pay for your attention.

609
01:04:37,210 --> 01:04:47,050
Which of course is exactly what's happening now. Um, and, um, but it's a tension which is then very confined to what they want you to attend to.

610
01:04:47,930 --> 01:04:53,050
Um, and that does pose problems for how we think of ourselves as a community and as a society.

611
01:04:54,290 --> 01:05:02,450
Um, and, uh, yeah. And your ability to, to really, um, interpret, uh, a vast amount of data.

612
01:05:02,930 --> 01:05:09,490
I mean, certainly I've been working in with EU policymakers a little bit.

613
01:05:09,650 --> 01:05:15,410
Not a huge man in the last year or so. Um, what they constantly say is what they don't, what the last thing they want is more information.

614
01:05:16,650 --> 01:05:23,790
They want to be able to make sense of it. Firstly, they want to when they need it because of the timescales, but they also want to be inspired.

615
01:05:24,350 --> 01:05:25,430
That was the bit that surprised me.

616
01:05:26,030 --> 01:05:32,830
They said we want we want research that makes us think of new ideas that we haven't thought of and that you won't get them from AI.

617
01:05:33,270 --> 01:05:45,790
Interestingly, um, I agree with Tom, but I would I would challenge the idea that AI is good at the kind of routine stuff and creative stuff.

618
01:05:46,590 --> 01:05:57,070
I want to get a t shirt with move 37 on it. You know that, um, example where the AI made a move and it was move 37 in a game of go,

619
01:05:57,790 --> 01:06:03,190
and everyone who was watching the computer playing the human at go thought, oh, look, the computer's gone completely crazy.

620
01:06:03,430 --> 01:06:04,710
It's just done this really stupid move.

621
01:06:05,630 --> 01:06:13,590
Um, and all the go experts were saying the computer will lose, but actually it was totally transforming how the game was played.

622
01:06:14,790 --> 01:06:22,330
And so the AI nerds will have t shirts saying route 37 and what they mean by that is actually really good.

623
01:06:22,570 --> 01:06:27,450
AI can be incredibly creative only if you have the right kind of prompt conversations with it.

624
01:06:28,170 --> 01:06:37,650
Um, and you said earlier on the bit about making connections across the disciplines because the AI isn't in a tribe,

625
01:06:38,210 --> 01:06:48,530
whereas I only read the stuff that's produced by my own tribe. Um, and I mean, I'm just looking into discipline through AI.

626
01:06:48,810 --> 01:06:54,290
I'm plugging systems I didn't even know existed, but also it will take stuff from there,

627
01:06:54,690 --> 01:06:58,250
repackage them, and then come up with very novel, interdisciplinary ideas.

628
01:06:58,530 --> 01:07:04,690
So I think one of the great advantages of using AI is in generating creative ideas.

629
01:07:05,730 --> 01:07:13,570
So I was talking last night to someone from welcome, uh, on this very topic about interdisciplinarity, that their funding work in that space.

630
01:07:14,290 --> 01:07:18,550
And I was suggesting one of the ways into that would be to take in any one discipline.

631
01:07:19,470 --> 01:07:24,470
What is at the boundary of your discipline, where we're really coming up against problems that we can't solve.

632
01:07:25,110 --> 01:07:34,430
And to open that question up to interdisciplinary engagement, uh, as, as as just as a way of structuring that process.

633
01:07:35,590 --> 01:07:39,230
And I don't know the answer to that, but I think that strikes me as quite an interesting possibility.

634
01:07:40,230 --> 01:07:45,870
And that would include bringing creative, uh, as well as other social research disciplines.

635
01:07:46,350 --> 01:07:52,550
I'll leave you with two thoughts. One is that human the the radical incompleteness of human knowledge.

636
01:07:53,590 --> 01:07:59,830
Maybe it's just like there is radically that any discipline will only give you a partial account of the world.

637
01:08:00,310 --> 01:08:03,830
Any AI agent will give you a partial account of the world.

638
01:08:04,230 --> 01:08:08,590
Any. Um, however, we we go about trying to understand the world.

639
01:08:08,830 --> 01:08:12,550
There's a radical incompleteness, which is why we should listen to each other,

640
01:08:12,950 --> 01:08:20,370
which is why we should listen to indigenous communities and the wisdom that they will have generated, which is so all of that becomes.

641
01:08:21,170 --> 01:08:29,810
So once you adopt a slightly more humble position, um, and say, actually knowledge, there is a radical incompleteness to what we what we can know.

642
01:08:30,530 --> 01:08:35,410
Um, but that kind of I think that opens you up to, uh, partly you become a bit more humble,

643
01:08:35,770 --> 01:08:43,650
but also opens you up to being genuinely curious about what other people, other things might other approaches might tell you.

644
01:08:44,090 --> 01:08:46,810
And the final thought I'll just leave you with, oh, no, it's not there, is it?

645
01:08:47,490 --> 01:08:51,330
I've got there's some of the reading I've used is just there at the end of the. We'll share this presentation.

646
01:08:52,210 --> 01:08:57,290
I was in Copenhagen last week and um, at an evaluation conference, of course.

647
01:08:57,930 --> 01:09:08,890
Um, and, uh, so I'll just leave you with a kind of thought of Heidegger, which, to paraphrase him, he would say, use AI and you will regret it.

648
01:09:09,970 --> 01:09:14,010
Don't use AI and you will regret it, too. Very good.