1
00:00:06,050 --> 00:00:13,160
So I'd like to thank all of you for showing up for the second of three lectures by Professor Peter Railton from the University of Michigan,

2
00:00:13,160 --> 00:00:19,820
Ann Arbour, broadly on the topic of ethics and AI, the first, as you'll remember, was on moral learning.

3
00:00:19,820 --> 00:00:23,060
The second one is on understanding artificial agency.

4
00:00:23,060 --> 00:00:29,030
I will be very brief and just say that Peter is, of course, known to anyone who works in media ethics and normative ethics.

5
00:00:29,030 --> 00:00:33,920
Whether you're working at an undergraduate level or a graduate level or professional level, you must engage with this work.

6
00:00:33,920 --> 00:00:40,130
And though I believe the turn to Ethics VII is a very recent turn, I'm sure that it's going to be true of his work here as well.

7
00:00:40,130 --> 00:00:49,920
So please join me in welcoming Professor Railton. Thank you.

8
00:00:49,920 --> 00:00:52,470
Thank you very much, it's good to be back.

9
00:00:52,470 --> 00:01:00,780
Apropos of my recently coming to this topic, I just want you to know that you're entering an expertise free zone here.

10
00:01:00,780 --> 00:01:08,130
I'm trying to operate as a moral philosopher in an area in which moral philosophers aren't particularly trained and learning as much as I can.

11
00:01:08,130 --> 00:01:15,930
But you should be expected to wear a mental hard hat at all times because who knows what might be raining down upon you?

12
00:01:15,930 --> 00:01:23,790
So you were warned. So last time I was closing on this speculation that maybe humans and artificial

13
00:01:23,790 --> 00:01:28,680
intelligence systems could be able to develop something like communities of

14
00:01:28,680 --> 00:01:32,610
trust and cooperation that permit their different special abilities to work

15
00:01:32,610 --> 00:01:37,200
together to produce something greater than they could produce on their own.

16
00:01:37,200 --> 00:01:43,200
And of course, for that to happen, it would have to depend upon the actual development of such a community.

17
00:01:43,200 --> 00:01:49,200
And that, in turn, would depend upon the development of something like sufficient trust and cooperative ness and

18
00:01:49,200 --> 00:01:53,940
sufficient forms of coordination and communication amongst these different types of agents.

19
00:01:53,940 --> 00:02:00,240
And so today I'm going to be talking about artificial agents and what kind of allies they might be.

20
00:02:00,240 --> 00:02:06,720
And the idea of Ally is the idea that we may not be the same in many ways.

21
00:02:06,720 --> 00:02:14,730
And I mean, I'm not saying that machines have the standing of persons, at least not yet, but they may have the standing of allies.

22
00:02:14,730 --> 00:02:21,000
So to become an artificial ally, an intelligent one,

23
00:02:21,000 --> 00:02:28,950
these will not have to be moral agents that might involve a great deal, something like self-consciousness, moral emotions.

24
00:02:28,950 --> 00:02:32,700
Jeremy's here somewhere a capacity for practical reasoning,

25
00:02:32,700 --> 00:02:39,000
and we don't think that these systems have that or it's that it's in prospect anytime soon.

26
00:02:39,000 --> 00:02:44,160
It might be enough if artificial systems did have some endogenous capacity to be

27
00:02:44,160 --> 00:02:51,120
sensitive to and able to respond aptly to morally relevant features of situations,

28
00:02:51,120 --> 00:02:55,470
actions, agents and outcomes. So that's the working notion.

29
00:02:55,470 --> 00:03:04,320
I'm going to have this a little bit of a mouthful. The idea is sensitivity and responsiveness of an apt kind to morally relevant features.

30
00:03:04,320 --> 00:03:10,500
Such sensitivity and responsiveness sensitivity, by the way, is the idea that you actually notice these things, which is a big part of ethics.

31
00:03:10,500 --> 00:03:15,870
Getting people to notice the relevant features and responsiveness is the way in which you engage with them.

32
00:03:15,870 --> 00:03:19,020
Such sensitivity and responsiveness, I claimed,

33
00:03:19,020 --> 00:03:26,160
would require that such systems actually have some degree of autonomy with respect to us and the human agents around them.

34
00:03:26,160 --> 00:03:30,300
And you can see why that would be so.

35
00:03:30,300 --> 00:03:35,340
These systems will be around humans whose purposes aren't always good or they're misguided in some ways.

36
00:03:35,340 --> 00:03:41,820
And if they don't have any autonomy, then they certainly can't be expected to respond happily to morally relevant features.

37
00:03:41,820 --> 00:03:46,800
So that's the project. Well, what is this idea of a morally relevant feature?

38
00:03:46,800 --> 00:03:53,580
And I gave you last time a kind of a taxonomy of some dimensions of morally relevant features.

39
00:03:53,580 --> 00:03:56,050
They need to be non parochial and non egocentric.

40
00:03:56,050 --> 00:04:01,710
There are things like special permissions that we have with respect to our lives, our friends, our families.

41
00:04:01,710 --> 00:04:05,550
But those are non parochial. Everyone has the same permissions.

42
00:04:05,550 --> 00:04:12,480
They aren't founded upon something special about me because in fact, there is nothing special about me.

43
00:04:12,480 --> 00:04:18,180
They are general and they're super vignette in nature. They depend upon non evaluative features.

44
00:04:18,180 --> 00:04:22,470
They are linked somehow to motivation. And so when we think about these morally relevant features,

45
00:04:22,470 --> 00:04:27,690
we have to think that they're doing some action guiding and not just some registering effects.

46
00:04:27,690 --> 00:04:34,530
On the other hand, the motivation and the kind of intelligence they provide has to be non instrumental in a way.

47
00:04:34,530 --> 00:04:41,850
And it has to be independent of authority or sanction and to make it moral because these are all features that you could find in,

48
00:04:41,850 --> 00:04:47,310
say, epistemic epistemic irrelevant features to make it distinctively moral.

49
00:04:47,310 --> 00:04:52,560
We have to say something about the subject matter, and I'm going to be very Catholic about that.

50
00:04:52,560 --> 00:04:58,710
They can be such things as harms, benefits, fairness, respect and so on for oneself and for others.

51
00:04:58,710 --> 00:05:06,000
And if you can show to someone's world judgements run afoul of these, they are parochial or they are instrumental,

52
00:05:06,000 --> 00:05:13,620
or they don't show adequate respect for people or their benefits and burdens, then that's a ground of criticism.

53
00:05:13,620 --> 00:05:21,990
So what about this idea of autonomy and trust will recall the example last time of the

54
00:05:21,990 --> 00:05:27,180
independent mindedness of one and a half two year old children with respect to fairness,

55
00:05:27,180 --> 00:05:34,140
if they're playing a game, this involves a joint activity with another child that even if they are given the entire reward themselves,

56
00:05:34,140 --> 00:05:39,780
they will distribute it to the other child as well. They'll do that at some expense to themselves.

57
00:05:39,780 --> 00:05:43,020
That's a kind of independent mindedness because the authority doesn't tell them to do that,

58
00:05:43,020 --> 00:05:48,120
doesn't reward them from doing it, says this is what you get. The child does it nonetheless.

59
00:05:48,120 --> 00:05:55,230
And three or. Four year old children with respect to authority will resist inflicting harm or unfairness on others,

60
00:05:55,230 --> 00:06:00,360
and by five or six, they'll intervene to stop harm or unfairness amongst third parties.

61
00:06:00,360 --> 00:06:06,510
So that's a kind of autonomy, the authority says. Do this. The child says I won't.

62
00:06:06,510 --> 00:06:12,810
Nobody says why? Because it would harm this person because it would be unfair. Or they will intervene in that kind of a case.

63
00:06:12,810 --> 00:06:20,010
And it's to me, a form of trustworthiness in children that they have some autonomy.

64
00:06:20,010 --> 00:06:23,160
You know, as a parent, you realise at some point that if they have no autonomy,

65
00:06:23,160 --> 00:06:27,210
they aren't trustworthy because they will pick up whatever is in their environment.

66
00:06:27,210 --> 00:06:30,150
And so if we're thinking about making these systems trustworthy,

67
00:06:30,150 --> 00:06:35,370
we should also be thinking about making them in the relevant sense, autonomous and just as an epistemic matter,

68
00:06:35,370 --> 00:06:42,720
they won't be able to respond to apply to morally relevant features or detect them if they don't have some degree of independent mindedness.

69
00:06:42,720 --> 00:06:49,710
Now you could say, OK, yeah, that's great. But children are building on very highly evolved psychology.

70
00:06:49,710 --> 00:06:57,450
It's a psychology that somehow is fit for our forms of life. It's demonstrated that humans can live together even reasonably peacefully.

71
00:06:57,450 --> 00:07:03,450
Not always, unfortunately. But what about artificial agents? Aren't they coming from nowhere in a sense?

72
00:07:03,450 --> 00:07:10,950
And if they are coming from nowhere and we're asking them to do tasks for us,

73
00:07:10,950 --> 00:07:15,750
what does that have to do with this project of asking them to be sensitive to a morally relevant features?

74
00:07:15,750 --> 00:07:23,760
Isn't that kind of a fool's errand? And I'm going to focus today a little bit on the problem of autonomous vehicles so-called.

75
00:07:23,760 --> 00:07:29,250
And I just ask you to put in the back of your mind somewhere why it might be that autonomous

76
00:07:29,250 --> 00:07:34,590
vehicles would need something like a capacity to be responsive to more of the relevant features.

77
00:07:34,590 --> 00:07:39,570
If you imagine all the dynamics in these situations, you're interacting with humans,

78
00:07:39,570 --> 00:07:43,740
they're interacting with each other, they're interacting with pedestrians.

79
00:07:43,740 --> 00:07:50,250
That's going to require more than just the ability to stay in lane or to stop it stop signs.

80
00:07:50,250 --> 00:07:57,810
So. Any time anyone talks about autonomy for machines or disobedient machines,

81
00:07:57,810 --> 00:08:02,820
even more so that at least as threatening as it is promising, there are some people working.

82
00:08:02,820 --> 00:08:09,840
I think she say the last thing we should do is make these systems in any way autonomous because then they can escape our control.

83
00:08:09,840 --> 00:08:15,810
And the fact of the matter is it seems to be that their promise as potential features with capacity to

84
00:08:15,810 --> 00:08:20,490
engage with morally relevant considerations and the threat that that involves in terms of autonomy,

85
00:08:20,490 --> 00:08:23,580
those are inextricable. I don't think you can pull those apart.

86
00:08:23,580 --> 00:08:28,290
And you could say, why should we allow beings of that kind to be brought into existence?

87
00:08:28,290 --> 00:08:32,110
And what I tried to claim last time is this issue.

88
00:08:32,110 --> 00:08:37,950
Will we allow the development and deployment of artificially intelligent systems able to acquire extensive

89
00:08:37,950 --> 00:08:44,970
information and use it to take dangerous or manipulative actions in ways that risk escaping our means of control?

90
00:08:44,970 --> 00:08:47,970
That's already happening. That's not a choice that we have.

91
00:08:47,970 --> 00:08:55,080
These systems do exist and they are in some ways escaping control, and they're not very far down that road.

92
00:08:55,080 --> 00:09:00,660
And indeed, most of the problem doesn't come from the machines themselves, but to a greater extent.

93
00:09:00,660 --> 00:09:05,970
With time, it will. So the question isn't before we create and deploy such systems.

94
00:09:05,970 --> 00:09:11,710
We must guarantee that they will not become autonomous and produce results we do not want.

95
00:09:11,710 --> 00:09:16,660
As I suggested earlier, they can be results that we want that we shouldn't want. Rather,

96
00:09:16,660 --> 00:09:23,530
the question is more like given the increasing development deployment of AI systems in a wide range of applications and given

97
00:09:23,530 --> 00:09:30,550
that we do not have have little prospect of having and probably do not want some kind of a centralised system of human control.

98
00:09:30,550 --> 00:09:36,670
What are some more distributed resources or methods for trying to identify and respond to these threats

99
00:09:36,670 --> 00:09:42,280
and achieve more of the promise and in particular amongst those distributed resources and methods?

100
00:09:42,280 --> 00:09:50,800
Might these involve the artificial agents themselves? And so right now, as I said, the most serious form of these threats is not from the machines,

101
00:09:50,800 --> 00:09:54,880
not monster ultra powerful intelligent machines, but monster,

102
00:09:54,880 --> 00:10:03,670
not necessarily entirely intelligent humans using these machines for nefarious purposes or for misguided purposes,

103
00:10:03,670 --> 00:10:09,640
or simply using them in ways that involve error or failure to understand what they're doing.

104
00:10:09,640 --> 00:10:14,440
So that's a big source right now, but this already suggests that we should be asking ourselves the question Do we

105
00:10:14,440 --> 00:10:19,790
want these machines to be just all purpose tools for whoever picks them up?

106
00:10:19,790 --> 00:10:25,670
Might they have in their DNA, we might hope some ability to resist.

107
00:10:25,670 --> 00:10:33,780
So what's this idea of apt sensitivity and responsiveness?

108
00:10:33,780 --> 00:10:44,850
And the first analogy I want to make is with vision and object identification, as it's now done in contemporary artificial intelligence.

109
00:10:44,850 --> 00:10:52,950
And last time we talked about chess and about the view held for generation in artificial intelligence,

110
00:10:52,950 --> 00:10:59,730
that the way to get expert chess play out of machines is to programme human chess expertise into the system and

111
00:10:59,730 --> 00:11:06,660
then let it crank through all the possible moves as a way to achieve human level performance or exceed us.

112
00:11:06,660 --> 00:11:07,830
Now, more recently,

113
00:11:07,830 --> 00:11:15,870
what started happening is that machines without these hand coded means of detecting and utilising winning at chess relevant features,

114
00:11:15,870 --> 00:11:20,520
they aren't told what are the winning at chess relevant features. They're told whether they've won a game or not.

115
00:11:20,520 --> 00:11:26,550
But capable of simulating play internally, autonomously simulating play against themselves.

116
00:11:26,550 --> 00:11:34,830
These have achieved much higher levels of success at chess, learning in effect from scratch rather than from human experts.

117
00:11:34,830 --> 00:11:42,270
Chess playing neural networks rediscovered. A number of known strategies are what really are moves sequences because that's all these are present.

118
00:11:42,270 --> 00:11:47,220
They aren't strategizing as they are movers and they discover new ones.

119
00:11:47,220 --> 00:11:51,600
They had learnt in some sense the structure of the game and its possibilities,

120
00:11:51,600 --> 00:11:57,480
and they did that using very generic reinforcement learning methods, which of course, required a great deal of tweaking.

121
00:11:57,480 --> 00:12:01,740
But essentially, the core is something like generic reinforcement learning.

122
00:12:01,740 --> 00:12:05,610
And despite its complexity, you might say, well, chess.

123
00:12:05,610 --> 00:12:14,910
I mean, chess is a bounded world. Well, things are getting a little bit less bounded alpha zero now and Mewes zero using similar

124
00:12:14,910 --> 00:12:20,070
generic learning processes have achieved more than human levels at a range of games.

125
00:12:20,070 --> 00:12:25,800
Go Shogi, Atari and so on. And they don't require extensive reprogramming to do this.

126
00:12:25,800 --> 00:12:33,000
They can use the same methods and build on some of their competencies. And these persistent systems possess,

127
00:12:33,000 --> 00:12:42,150
I'm going to say something like a general or a more general capacity to be sensitive and aptly responsive to strategic game playing features,

128
00:12:42,150 --> 00:12:47,100
generating and deploying them in their own moves in their own decisions.

129
00:12:47,100 --> 00:12:54,480
So that's the idea that these are not machines that have human understanding of chess or show you they don't.

130
00:12:54,480 --> 00:13:00,720
But these are machines that are set apni sensitive to the relevant game winning features in chess or shogi.

131
00:13:00,720 --> 00:13:05,010
Or go. Well, let's broaden out a little bit.

132
00:13:05,010 --> 00:13:11,850
Let's go beyond the world of strategic games. There was a similar progression in machine vision.

133
00:13:11,850 --> 00:13:19,500
One of the main goals of artificial intelligence from the very beginning was machines that could decode the world visually, and for years,

134
00:13:19,500 --> 00:13:29,400
it was thought that machine vision could only be achieved if human human experts encoded by hand visual features of situations.

135
00:13:29,400 --> 00:13:36,930
Imagine what the list of that looks like visual features, situations and ways of translating those into something like objects in three dimensional

136
00:13:36,930 --> 00:13:44,820
space and thousands of man years were dedicated to this task of coding such features.

137
00:13:44,820 --> 00:13:50,100
More recently, deep neural networks of the same kind we've just been discussing have become more

138
00:13:50,100 --> 00:13:54,780
effective at detecting and responding aptly to objects and spatial relations,

139
00:13:54,780 --> 00:13:59,910
identifying features in images via these generic learning methods.

140
00:13:59,910 --> 00:14:03,570
So again, it's not that they have the same understanding of the world that we do,

141
00:14:03,570 --> 00:14:12,640
but that they are more apt than humans actually at responding to object and spatial relations, identifying features quickly.

142
00:14:12,640 --> 00:14:18,040
Using images as the basis, and so that's an interesting idea,

143
00:14:18,040 --> 00:14:24,130
this idea that you can respond to these features and therefore become expert at chess or extremely

144
00:14:24,130 --> 00:14:30,520
good at visual identification without having what we would think of as a conscious mind.

145
00:14:30,520 --> 00:14:39,520
Now, along the way, something interesting happened. So here's human vision processing that's supposed to be a cat on the left.

146
00:14:39,520 --> 00:14:44,500
It goes into your visual cortices outputs from the retina.

147
00:14:44,500 --> 00:14:51,310
There's some fancy wiring in the retina retina of mutual inhibition to sharpen the image, and then it goes up into the visual cortex.

148
00:14:51,310 --> 00:14:54,400
And there at first, you start detecting things like edges of objects,

149
00:14:54,400 --> 00:14:58,450
and then it goes up to the next visual cortex and you start seeing conjunctions of edges.

150
00:14:58,450 --> 00:15:07,300
And it goes up through these different cortices and you assemble more and more complex features of the image that is being presented by the retina.

151
00:15:07,300 --> 00:15:11,050
And if you look at the structure of such cortex,

152
00:15:11,050 --> 00:15:18,610
you'll see that it has a stacked structure with neural connexions projecting through it in this kind of a network way.

153
00:15:18,610 --> 00:15:25,480
And that was indeed the inspiration for the neural nets. A very simple neural net is on the right.

154
00:15:25,480 --> 00:15:29,080
There's an input layer that would correspond, let's say, to the retina.

155
00:15:29,080 --> 00:15:35,320
There are various hidden layers, and then there's an output layer and identification of an object, let's say.

156
00:15:35,320 --> 00:15:41,440
And each of those circles corresponds to something that's deliberately designed to be like a neurone.

157
00:15:41,440 --> 00:15:46,450
That is to say it accepts inputs from multiple artificial dendrites.

158
00:15:46,450 --> 00:15:49,240
And depending upon the nature of those inputs, their frequency,

159
00:15:49,240 --> 00:15:53,890
their intensity and so on, it either fires or doesn't, which is a discontinuous change.

160
00:15:53,890 --> 00:16:01,450
That's the secret of neurones. And that then goes out and is passed along to the next neurones in the sequence.

161
00:16:01,450 --> 00:16:10,840
And what's interesting is that if you build a system with enough layers and enough parameters and you don't tell it how to identify objects,

162
00:16:10,840 --> 00:16:18,640
but you train it on a large number of images, that's something we can now do because people are great at putting images up on the internet.

163
00:16:18,640 --> 00:16:23,290
What you find is that they do produce object identification.

164
00:16:23,290 --> 00:16:28,300
But what's interesting as well is they do it in a manner very similar to human vision.

165
00:16:28,300 --> 00:16:31,780
That is, the first layers of the nets detect things like edges.

166
00:16:31,780 --> 00:16:38,680
The second layers of the nets detect things like intersections of edges and features the third layer, more complex features and so on.

167
00:16:38,680 --> 00:16:46,240
So that suggests that the architecture of vision, human vision and machine vision is a bit more robust than you might have thought,

168
00:16:46,240 --> 00:16:51,070
you might have thought that vision is heavily dependent on the particular way humans are wired up.

169
00:16:51,070 --> 00:16:54,910
And that's going to stand between us in an objective presentation of the world.

170
00:16:54,910 --> 00:16:57,940
And then you say, Well, this system didn't begin with the human expectations.

171
00:16:57,940 --> 00:17:05,110
It began as a very simple learning system, but simply tried best to fit the images that it was given.

172
00:17:05,110 --> 00:17:10,840
Give the best data fit for the images that it was given, and it ended up with a very similar structure.

173
00:17:10,840 --> 00:17:18,400
And so that gives us an idea that we may have some work, some more robustness in our epistemic capacities than you might have thought.

174
00:17:18,400 --> 00:17:24,340
We may be less imprisoned than we thought in our perceptual presuppositions.

175
00:17:24,340 --> 00:17:29,350
And the idea is that this is a way in which the machines have shown that they too can be very

176
00:17:29,350 --> 00:17:35,500
good as the visual system is at responding to object and spatial relations identifying features.

177
00:17:35,500 --> 00:17:42,790
OK, now learning as visual learning is much more than passive modelling of structure.

178
00:17:42,790 --> 00:17:48,910
It's also generated in action. Guiding most of the visual field that you have is being actively populated by your

179
00:17:48,910 --> 00:17:53,800
brain in order to make sure the whole thing is coloured and sharpen focus and so on.

180
00:17:53,800 --> 00:18:03,400
So your visual system, it's not just flowing one way, it's also flowing in the other direction, generating lots of features of vision itself.

181
00:18:03,400 --> 00:18:08,710
And the question then is will these systems have that capability?

182
00:18:08,710 --> 00:18:15,910
Well, in chess and go and so on, we found that they could generate novel moves and successful strategies.

183
00:18:15,910 --> 00:18:23,080
And it turns out that neural nets trained up for vision can buy, so to speak, being run in reverse generate images.

184
00:18:23,080 --> 00:18:30,880
And so here you've probably you've probably all seen these. These are images that were generated by artificial adversarial networks.

185
00:18:30,880 --> 00:18:39,010
And of course, they generate a lot of images that look kind of like rubbish or look kind of like a dog, but smeared out sideways and so on.

186
00:18:39,010 --> 00:18:44,840
But they also can generate images like this, and they get better and better at it as they get better and better at vision.

187
00:18:44,840 --> 00:18:51,490
This is also something that's important because it's important for understanding how human imagination works.

188
00:18:51,490 --> 00:18:58,450
Human imagination takes what we have got by way of structure of the world from experience and then reuses that internally,

189
00:18:58,450 --> 00:19:03,340
just as a chess playing programme used its experience internally to simulate possibilities

190
00:19:03,340 --> 00:19:08,920
and enable us to think about things that are not present in sensation to think about a red,

191
00:19:08,920 --> 00:19:11,530
white spotted mushroom against a green field of grass,

192
00:19:11,530 --> 00:19:15,580
which you can close your eyes and do perfectly well right now without the help of such a machine.

193
00:19:15,580 --> 00:19:29,540
So that's the idea, then, that these are not just passively recording structure, they are also generating a capacity to create a visual field and.

194
00:19:29,540 --> 00:19:33,320
Again, we really can't get ahead of ourselves in this, I mean,

195
00:19:33,320 --> 00:19:39,870
there's always this rah rah that goes on with this and the hype, but we're not claiming.

196
00:19:39,870 --> 00:19:43,400
I'm not claiming that these systems understand the strategic games or even that

197
00:19:43,400 --> 00:19:47,180
they know that these games are played amongst individuals who are in some sense,

198
00:19:47,180 --> 00:19:52,160
opponents of one another. That's not what it's got. It's just got pixel input,

199
00:19:52,160 --> 00:19:57,500
nor that they understand the objects that they identify or that they generate

200
00:19:57,500 --> 00:20:01,310
that they think of them as objects in space in the same way we do and so on.

201
00:20:01,310 --> 00:20:06,980
It's not the way they work. They're operating with pixels inputs and they're yielding outputs.

202
00:20:06,980 --> 00:20:13,700
Yet at the same time, they can be extremely effective in novel a general settings settings that don't just

203
00:20:13,700 --> 00:20:19,850
require representation but require action and that require novel action like game playing,

204
00:20:19,850 --> 00:20:25,850
object identification, imagining objects and guiding self-driving cars driving cars.

205
00:20:25,850 --> 00:20:31,360
So the representations aren't just superficial layering of patterns.

206
00:20:31,360 --> 00:20:35,380
They in some ways have captured the structure of the information that they're being given,

207
00:20:35,380 --> 00:20:40,090
which in turn enables them to in some sense model implicitly the structure of the

208
00:20:40,090 --> 00:20:44,320
features of the world around them and without representing them metaphysically,

209
00:20:44,320 --> 00:20:49,270
it's nonetheless able to operate within that environment effectively.

210
00:20:49,270 --> 00:20:56,290
So I'm going to try another analogy. This was a way of talking about the idea of morally responding happily to morally relevant

211
00:20:56,290 --> 00:21:01,930
features by talking about responding aptly to features relevant to object identification.

212
00:21:01,930 --> 00:21:04,720
Here's an even perhaps closer analogy.

213
00:21:04,720 --> 00:21:12,010
So artificially intelligent systems might become allied epistemic agents in something like medical or scientific research.

214
00:21:12,010 --> 00:21:16,060
And I gather Dennis Fisher, this is going to be talking about something like this tomorrow.

215
00:21:16,060 --> 00:21:25,300
And if he were here, he probably pulling his hair out. But the the these systems have become very important in contemporary scientific research.

216
00:21:25,300 --> 00:21:33,730
And I notice his title is scientific discovery. These are not just confirmation systems, their systems for discovering novel hypotheses.

217
00:21:33,730 --> 00:21:36,490
So these systems can, I think,

218
00:21:36,490 --> 00:21:44,410
come to possess a significant degree of autonomous credibility in identifying and responding aptly to epistemic relevant features,

219
00:21:44,410 --> 00:21:50,830
including the formation and assessment of novel causal models, as they do with protein folding.

220
00:21:50,830 --> 00:21:55,930
Novel hypotheses that better fit the data than our existing hypotheses.

221
00:21:55,930 --> 00:22:01,690
Making unanticipated connexions across very extensive data sets or literatures that humans could not do on their own.

222
00:22:01,690 --> 00:22:07,120
Manage making innovations in methodology or experimentation by running simulations

223
00:22:07,120 --> 00:22:13,090
and seeing what might work running many more simulations than you or I could.

224
00:22:13,090 --> 00:22:17,860
And in some sense, these could be something more than just tools or tablets.

225
00:22:17,860 --> 00:22:25,060
They could be something like co-investigators. They are probing the structure of what they are learning from the data,

226
00:22:25,060 --> 00:22:31,840
and they are generating new ways of representing it and thinking about it, which can be used to guide action.

227
00:22:31,840 --> 00:22:40,810
And in these ways, they could help us make substantive breakthroughs in science, maybe ones that we wouldn't particularly make.

228
00:22:40,810 --> 00:22:44,810
Again, we aren't assuming that they're conscious that they have an understanding of the world.

229
00:22:44,810 --> 00:22:52,150
Within two hours, their representations might take the form of something like sets of probability functions over possible worlds.

230
00:22:52,150 --> 00:22:55,280
Their grasp of meaning and interpreting data and hypotheses.

231
00:22:55,280 --> 00:23:05,170
They're inferences might be algebraic rather than propositional through some hierarchy of association, like the ones we saw in the visual cortex.

232
00:23:05,170 --> 00:23:11,920
These may require a quite complex function to be mapped upon the kind of discrete concepts that you or I have.

233
00:23:11,920 --> 00:23:17,200
But nonetheless, if such a mapping is possible, it should be possible for these systems to be allies of ours.

234
00:23:17,200 --> 00:23:20,020
And indeed, they are increasingly now are.

235
00:23:20,020 --> 00:23:26,540
And again, what's interesting is that the fact that they don't duplicate our understanding is a kind of an asset.

236
00:23:26,540 --> 00:23:31,070
First of all, it produces a kind of correlation, whether we like it or not.

237
00:23:31,070 --> 00:23:37,960
Virtually all scientists are human beings and they therefore are highly correlated and quite a number of ways.

238
00:23:37,960 --> 00:23:41,890
The data from the world aren't structured around the human psyche.

239
00:23:41,890 --> 00:23:50,640
I believe. If we had other ways of representing that were correlated with the human psyche

240
00:23:50,640 --> 00:23:55,830
and that could identify different clusters at different relations amongst objects,

241
00:23:55,830 --> 00:24:02,490
that might be a way for our research to become less brittle and more robust than it is now.

242
00:24:02,490 --> 00:24:07,620
That would be a way in which these machines would again be autonomous epistemic agents.

243
00:24:07,620 --> 00:24:14,160
It's not that we ask them to reproduce our way of seeing, it's that they can contribute a distinctive way of seeing.

244
00:24:14,160 --> 00:24:21,030
Moreover, beyond this kind of consultants, artificially intelligent agents might come up with ways of identifying data

245
00:24:21,030 --> 00:24:26,200
that generate for us new concepts that we would not have formed on our own,

246
00:24:26,200 --> 00:24:32,850
but the concepts we might be able to use to perform further successful experimentation or technology.

247
00:24:32,850 --> 00:24:38,400
Now, the credibility of that new way of organising data would not depend, I think,

248
00:24:38,400 --> 00:24:48,720
upon the human ability to translate exactly what that organisational principle was is into human concepts, existing human concepts.

249
00:24:48,720 --> 00:24:53,280
It could very well be that we can't easily translate it into existing human concepts.

250
00:24:53,280 --> 00:25:00,090
Yet it could win its way into acceptance by us because of the way it makes unanticipated or successful predictions.

251
00:25:00,090 --> 00:25:06,510
And space time is like that right? Space time was not translatable without reduction into the previous concept.

252
00:25:06,510 --> 00:25:13,470
It enabled them to make some very surprising predictions. It was very hard for many people still is to wrap their mind around it.

253
00:25:13,470 --> 00:25:22,830
Yet it could acquire a kind of epistemic authority and credibility because of the way in which it enhanced this kind of scientific power.

254
00:25:22,830 --> 00:25:26,910
And so a research community might incorporate a novel category into their

255
00:25:26,910 --> 00:25:32,400
conceptual repertoire because they will develop practises with these machines,

256
00:25:32,400 --> 00:25:38,160
and successful outcomes have led them to see such systems as responding aptly to epistemic

257
00:25:38,160 --> 00:25:43,280
relevant features that would be part of the explanation of why they accept this.

258
00:25:43,280 --> 00:25:49,760
And such a system could merit trust on the part of human researchers as something like autonomous epistemic agents.

259
00:25:49,760 --> 00:25:56,000
Moreover, they would be showing a capacity to operate in a way with human language that we ourselves don't,

260
00:25:56,000 --> 00:25:59,090
but which might give us further insight into it.

261
00:25:59,090 --> 00:26:06,320
And this is something like this idea of an expert ally is something what like what Kasparov has written about?

262
00:26:06,320 --> 00:26:12,260
Alpha zero programmes usually reflect priorities and prejudices of programmers.

263
00:26:12,260 --> 00:26:17,910
But because Alpha Zero programmes itself, I would say that its style reflects the truth.

264
00:26:17,910 --> 00:26:24,240
Not the truth about the metaphysics of chess, but the truth about the structure of the game and its possibilities.

265
00:26:24,240 --> 00:26:32,750
This superior understanding allowed it to outclassed the world's top traditional programme despite calculating far fewer positions per second.

266
00:26:32,750 --> 00:26:38,990
Alpha Zero shows is that machines can be the experts and not merely expert tools,

267
00:26:38,990 --> 00:26:47,450
and that one respect from the chess community in the way that these agents could win autonomous levels of respect from the chess community.

268
00:26:47,450 --> 00:26:52,730
OK, now any such system will make errors and large errors just like human enquiries.

269
00:26:52,730 --> 00:27:00,410
They may be different areas and we can learn from that. That may be very important, but we can buy operating with them,

270
00:27:00,410 --> 00:27:05,460
enlarge our capacity to be flexible and adaptive enough to understand the world better.

271
00:27:05,460 --> 00:27:12,890
That's the claim. Moreover, such systems can receive reciprocal epistemic benefit from us.

272
00:27:12,890 --> 00:27:18,710
They don't have everything. They or their nets don't have all possibilities.

273
00:27:18,710 --> 00:27:21,320
They can have their responsiveness to epistemic.

274
00:27:21,320 --> 00:27:27,980
Irrelevant features improved by human forms of experimental and methodological and theoretical testing.

275
00:27:27,980 --> 00:27:33,380
And so we can have a reciprocal relationship with these agents, not just listening to them,

276
00:27:33,380 --> 00:27:39,200
not just speaking to them, but exchanging with them and contemporary deep learning systems.

277
00:27:39,200 --> 00:27:44,720
Since they learn via very general algorithms in which the influence of priors tends to wash out,

278
00:27:44,720 --> 00:27:51,530
you might tend to think, Well, no, they don't need us because their priors will wash out ours would too.

279
00:27:51,530 --> 00:27:55,880
But their finite systems, and for any finite system, they're going to be blind spots.

280
00:27:55,880 --> 00:28:02,510
There are only so many categories or parameters it will have and priors wash out not by a guarantee,

281
00:28:02,510 --> 00:28:10,330
but only in the very long run and only with high probability. So we can be reciprocal epistemic agents.

282
00:28:10,330 --> 00:28:16,270
And that could be the basis of what I could call an epistemic social contract.

283
00:28:16,270 --> 00:28:25,600
And we could have a contract, which includes artificial intelligence systems as bona fide members, signatories in the epistemic community,

284
00:28:25,600 --> 00:28:31,660
quite independently of whether such systems have consciousness like ours or indeed any consciousness at all,

285
00:28:31,660 --> 00:28:38,770
quite independently of whether they're persons. And I'm underlining this because I want to say that they could be participants

286
00:28:38,770 --> 00:28:43,540
in a moral social contract without being conscious and without being persons,

287
00:28:43,540 --> 00:28:48,600
but nonetheless having some authority, autonomous authority and credibility.

288
00:28:48,600 --> 00:28:54,090
But what would happen if we tried to widen the epistemic social contract beyond the setting of just scientific research?

289
00:28:54,090 --> 00:29:01,290
That's going to be a big challenge because fuller participation in the wide epistemic community gathering trust and

290
00:29:01,290 --> 00:29:08,520
credibility is being responsive to epistemic irrelevant features in the wide world that requires a huge array of skills,

291
00:29:08,520 --> 00:29:12,060
not just the ability to take in large datasets and model them effectively.

292
00:29:12,060 --> 00:29:21,390
It includes the ability to identify linguistic features and situations, understand how conversational norms work and how to use common knowledge,

293
00:29:21,390 --> 00:29:28,860
or what common knowledge is likely to be to identify and address epistemic needs or goals in the situations to be really good,

294
00:29:28,860 --> 00:29:33,840
reliable, effective code participants in the broad epistemic community.

295
00:29:33,840 --> 00:29:39,930
This is going to require not just special epistemic competence or response from just epistemic features,

296
00:29:39,930 --> 00:29:45,630
but responsiveness to linguistic features, social features, interactional features.

297
00:29:45,630 --> 00:29:50,790
And so building a machine that is capable of this kind of interaction in the wider world,

298
00:29:50,790 --> 00:29:55,530
which is a goal of artificial intelligence, is going to be a broad goal.

299
00:29:55,530 --> 00:30:01,440
And it's going to involve these various expertises and this is something we talked more about last time.

300
00:30:01,440 --> 00:30:06,450
Mm-Hmm. But it doesn't, I think, require consciousness. It doesn't require personhood.

301
00:30:06,450 --> 00:30:14,160
But it would require this kind of wide responsiveness to a wide array of features that go well beyond the epistemic.

302
00:30:14,160 --> 00:30:19,020
And many of these features, if we just notice them sideways, we'll see.

303
00:30:19,020 --> 00:30:22,800
Oh, you know, some of those features are also morally relevant features.

304
00:30:22,800 --> 00:30:27,270
They have to do with intentions and communication and whether there's a fair exchange

305
00:30:27,270 --> 00:30:31,890
of information or whether one individual is being reliable or not reliable,

306
00:30:31,890 --> 00:30:36,090
and whether one individual is allowing information to be expressed or not expressed.

307
00:30:36,090 --> 00:30:41,460
That's the kind of dynamic that has to be understood to be it all purpose epistemic agent.

308
00:30:41,460 --> 00:30:46,830
But there is a lot to do with the moral features of these informational situations,

309
00:30:46,830 --> 00:30:54,150
and so such systems are going to have to work their way into an understanding of quite a range of morally relevant features as well.

310
00:30:54,150 --> 00:31:02,550
So the thought is divine designing artificially into artificially intelligent systems so that they can learn to be highly effective,

311
00:31:02,550 --> 00:31:08,970
trusted in autonomous code participants in the wide epistemic community that as a goal of the AI design,

312
00:31:08,970 --> 00:31:15,990
is going to require a bundle of capacities to be sensitive to semantically, socially, psychologically, morally and so on.

313
00:31:15,990 --> 00:31:19,860
Relevant features. And as I suggested last time,

314
00:31:19,860 --> 00:31:30,670
this package of app sensitivities might be something that plays an important role in understanding the distinctiveness of human society and culture.

315
00:31:30,670 --> 00:31:42,750
How might that be? Well. All of this has to be possible in real time, humans by analogy to artificial systems must be.

316
00:31:42,750 --> 00:31:51,420
It seems to me, continuously taking in relevant information. Does your audience really look like it's still with you assessing its relevance,

317
00:31:51,420 --> 00:31:55,980
updating the representations, monitoring the effects of one's actions and so on.

318
00:31:55,980 --> 00:31:59,940
We have to be doing this continuously. Machines would have to do this continuously.

319
00:31:59,940 --> 00:32:05,550
And part of what's interesting is that we possess two dominant modes of continuing processing in the brain.

320
00:32:05,550 --> 00:32:12,540
One is a task relevant or attentional network that's focussed on a specific activity, and the other is a so-called default mode network,

321
00:32:12,540 --> 00:32:18,870
and they alternate in their predominance in regular and irregular cycles throughout the day, some very short cycles.

322
00:32:18,870 --> 00:32:23,820
And that's why you can't stop your mind from drifting, even when you're very interested in what the person saying.

323
00:32:23,820 --> 00:32:28,440
Your mind will start off down to other tracks. That's not a defect.

324
00:32:28,440 --> 00:32:36,270
That's the structural feature of this system, because it's a system that has to take information in and processes importance,

325
00:32:36,270 --> 00:32:44,650
and that involves drifting off into other directions. Put that information back into the system, guiding action and response.

326
00:32:44,650 --> 00:32:53,770
And so we have these two types of systems dynamically functioning within us, and that helps make us the kinds of agents we are now.

327
00:32:53,770 --> 00:33:03,490
Suppose we ask. What else do these systems do? Well, last time we saw that the default mode system does autobiographical memory,

328
00:33:03,490 --> 00:33:10,590
the accumulation of knowledge, culture, whatever it would be in the utilisation of it in novel situations.

329
00:33:10,590 --> 00:33:19,020
Envisioning the future, projecting novel situations, imagining where they would be like counterfactual situations, theory of mind,

330
00:33:19,020 --> 00:33:27,150
attribution of intention or motives or behaviours to other individuals, understanding them as intentional beings and.

331
00:33:27,150 --> 00:33:36,060
Semantics. This is also the system we use for interpreting language, and it's very important because it's able to mobilise.

332
00:33:36,060 --> 00:33:41,250
What do you think you need to understand that English, you need your memory here, need theory of mind.

333
00:33:41,250 --> 00:33:48,690
You need a capacity to simulate what's going on in this statement. And so of course, it's going to be heavily involved in semantic processing as well.

334
00:33:48,690 --> 00:33:52,170
And as we saw last time, it's used in moral decision making.

335
00:33:52,170 --> 00:33:59,280
It's one system that is singularly present in a wide range of moral decision making situations.

336
00:33:59,280 --> 00:34:04,440
And so you could think, Oh, so we do have this connected bundle of capacities.

337
00:34:04,440 --> 00:34:11,160
We're built for this and we're built for it in connexion with understanding one another and action.

338
00:34:11,160 --> 00:34:15,840
Now that's encouraging, and it's discouraging and may be encouraging.

339
00:34:15,840 --> 00:34:18,180
Again,

340
00:34:18,180 --> 00:34:26,760
human conscious and daydreaming experience makes somehow reflect what's going on in the default mode system as the images that pop up in your head do,

341
00:34:26,760 --> 00:34:34,590
or the thoughts that pop in your head and conscious thought can activate it when you start spirit a series of imaginings.

342
00:34:34,590 --> 00:34:41,160
But in general, activation is quite independent of consciousness, and conscious direction doesn't seem necessary for this to appear,

343
00:34:41,160 --> 00:34:46,350
and there are creatures that we don't know to be conscious who seem to have exactly the same structure.

344
00:34:46,350 --> 00:34:51,150
So this might be a kind of processing that can be done by creatures and

345
00:34:51,150 --> 00:34:55,560
machines without asking the question of whether they have consciousness or not,

346
00:34:55,560 --> 00:35:00,030
even though they are decoding situations in these complex ways.

347
00:35:00,030 --> 00:35:04,590
And that's encouraging if you're going to make progress toward the current goal of developing artificially

348
00:35:04,590 --> 00:35:11,550
intelligent systems capable of being successful at that sort of wider participation in human social existence.

349
00:35:11,550 --> 00:35:19,760
And that's going to be essential for them to carry out all kinds of tasks, whether it's translation or health aids scientific discovery.

350
00:35:19,760 --> 00:35:27,020
Driving autonomous vehicles, so that's encouraging. At the same time, that is for the A.I. community encouraging at the same time,

351
00:35:27,020 --> 00:35:33,110
it suggests that real success in these tasks may not require creating consciousness,

352
00:35:33,110 --> 00:35:36,740
may not require getting persons where it's going to require creating systems

353
00:35:36,740 --> 00:35:41,870
that are extremely sophisticated with regard to this bundle of capacities.

354
00:35:41,870 --> 00:35:48,590
They will all be part of getting the linguistic competence right or getting the epistemic competence right or getting the social competence right.

355
00:35:48,590 --> 00:35:50,840
They will all be bounded together.

356
00:35:50,840 --> 00:35:59,300
And so success at the task of playing chess, which is quite a bounded activity compared to wandering about in the real world,

357
00:35:59,300 --> 00:36:05,210
and it has a much clearer criterion of success that required millions of simulations.

358
00:36:05,210 --> 00:36:08,630
But it didn't require modelling human psychology, which was interesting.

359
00:36:08,630 --> 00:36:15,290
You might have thought to play Championship press, you've got to chess, you've got to be able to model the psychology of the other Asian.

360
00:36:15,290 --> 00:36:19,610
But if there is a model in there of the psychology of agent, it's an implicit model.

361
00:36:19,610 --> 00:36:27,800
It's a model that's implicit in the play and it's activated not by having a representation of human psychology,

362
00:36:27,800 --> 00:36:34,490
but by whatever dependencies probabilistic dependencies there might be in its game playing strategies.

363
00:36:34,490 --> 00:36:40,670
Now, just imagine then, the amount of information and the number of simulations that are going to be required to

364
00:36:40,670 --> 00:36:47,390
identify the success or failure in social situations or epistemic situations for that matter,

365
00:36:47,390 --> 00:36:54,590
or acquiring the ability to accurately model the causal or intentional or normative structure of social situations.

366
00:36:54,590 --> 00:37:04,550
That's going to be huge. Now, one thought that has been informing this discussion in the background is infants do this,

367
00:37:04,550 --> 00:37:12,680
infants do this, and they do it through thousands of hours of observation and interaction as they grow older.

368
00:37:12,680 --> 00:37:20,390
So in the course of development, infants alternating between task relevant and default mode activity,

369
00:37:20,390 --> 00:37:28,730
watching the world in ways that we did not appreciate before. Just as we do not appreciate how much animals see in the world acquire very substantial,

370
00:37:28,730 --> 00:37:36,110
intuitive competency, it isn't even in the first weeks of life. Infants are showing surprise at anomalous linguistic sequences.

371
00:37:36,110 --> 00:37:40,610
Were they listening in the womb? Yes, they were. Were they listening in the first weeks of life?

372
00:37:40,610 --> 00:37:45,950
Yes, they were. Are they listening? Even now you're having an argument at dinner and they're just sitting there like this?

373
00:37:45,950 --> 00:37:53,180
Yes. So the developmental studies indicate that infant and child progress in learning causal relations,

374
00:37:53,180 --> 00:38:00,230
language theory of mind and moral assessment proceed in a coordinated way throughout the development.

375
00:38:00,230 --> 00:38:04,970
And they do so in a process that resembles something like Bayesian learning.

376
00:38:04,970 --> 00:38:11,780
Now, the exact nature of this is by no means understood. But the idea is that we're able to model it to some extent in these ways.

377
00:38:11,780 --> 00:38:19,930
And in doing so, we discover that there is this. Bundle again at work, developing together, and now that again,

378
00:38:19,930 --> 00:38:25,570
is encouraging for the prospect of developing machines that can be aptly sensitive to morally relevant features

379
00:38:25,570 --> 00:38:33,100
because we know that biological machines do it somehow and we know roughly the kinds of ways in which they do it.

380
00:38:33,100 --> 00:38:45,340
And so the challenge is to try to figure out whether this learning is possible and how now infants and toddlers do have to begin with certain priors.

381
00:38:45,340 --> 00:38:50,830
These machines don't necessarily have to begin with pliers with regard to the structure of the world.

382
00:38:50,830 --> 00:38:52,900
But there are going to be other priors.

383
00:38:52,900 --> 00:38:59,770
There will be effectively motivational priors because any machine is going to have to have something like a reward function.

384
00:38:59,770 --> 00:39:05,650
That's how it's going to figure out success or failure in the end, not because it's a game that it's played and it won or lost.

385
00:39:05,650 --> 00:39:11,440
But how the reward function carried out in the actions that it performed.

386
00:39:11,440 --> 00:39:19,040
And so last time we looked at this question and we said, well, what kind of a reward function to humans seem to have?

387
00:39:19,040 --> 00:39:29,290
And human infants in particular? And what we saw is that even our intelligent animal ancestors have a capacity for non egocentric

388
00:39:29,290 --> 00:39:34,860
as well as egocentric representation of the space and the causal structure of their environment.

389
00:39:34,860 --> 00:39:40,380
They have a capacity for simulating and evaluating alternative courses of events or actions.

390
00:39:40,380 --> 00:39:47,490
They have a capacity for learning, representing and deploying in decision making these evaluative expectations.

391
00:39:47,490 --> 00:39:52,650
Using that planned execution of behaviour and monitoring of action to learn.

392
00:39:52,650 --> 00:39:57,560
We saw this in intelligent animals last time, we certainly see it in human infants.

393
00:39:57,560 --> 00:40:06,620
Those thus far, though, no distinctive moral priors, these kinds of ways in which the infant is set up in order to do this kind

394
00:40:06,620 --> 00:40:11,870
of learning are ways that don't have to give a moral code or a moral module.

395
00:40:11,870 --> 00:40:17,030
These are, after all, shared with animals that we don't think have anything like that.

396
00:40:17,030 --> 00:40:26,270
There are also certain motivational priors, a degree of self concern and concern for, and we see that a capacity of reciprocation of direct benefits.

397
00:40:26,270 --> 00:40:32,690
We also see that in animals and children. And so again, there's nothing distinctively moral about this so far.

398
00:40:32,690 --> 00:40:40,070
But in humans, we do find something more and is quite open to debate exactly how much of this we find in other intelligent animals.

399
00:40:40,070 --> 00:40:49,280
And the fact that they are present in humans seems to be an important part of the explanation of our distinctive forms of social life.

400
00:40:49,280 --> 00:40:53,360
And so you can think it's not so much that there's some bit of knowledge in there,

401
00:40:53,360 --> 00:40:58,620
but rather it is some bit of motivational structure that is playing a key role.

402
00:40:58,620 --> 00:41:04,160
But what is this like? Well, it's a capacity for default to feasible cooperative ness.

403
00:41:04,160 --> 00:41:11,600
Even with non kin, humans live in large communities of genetically unrelated individuals cooperating daily.

404
00:41:11,600 --> 00:41:16,160
A capacity for indirect reciprocity with respect to groups.

405
00:41:16,160 --> 00:41:23,120
Human and human hunter gatherer groups, for example, they share very widely within themselves,

406
00:41:23,120 --> 00:41:26,180
but they can't keep accurate track of exactly what's shared.

407
00:41:26,180 --> 00:41:32,840
Individuals do contribute and individuals do receive and they have a kind of social insurance as a result,

408
00:41:32,840 --> 00:41:37,370
but they're willing to contribute because they're also receiving benefits from the group.

409
00:41:37,370 --> 00:41:42,140
And that's an indirect reciprocity. And we see that in humans and human infants.

410
00:41:42,140 --> 00:41:48,650
A capacity for some intrinsic concern with how unrelated others fare and how we are seen by them.

411
00:41:48,650 --> 00:41:54,020
And so you might think of that is what psychologists call pro-social motivation.

412
00:41:54,020 --> 00:42:00,350
Now that is pro-social, is it moral? Have I said anything yet of a moral kind?

413
00:42:00,350 --> 00:42:08,000
Or are these general purpose motivations for forming social relations, acquiring language and operating successfully in the social world?

414
00:42:08,000 --> 00:42:16,100
So my claim is that these are general purpose, motivational capacities. They foster shared learning, for example, of things like language they share.

415
00:42:16,100 --> 00:42:21,890
They are part of how we're able to communicate reliably and informative with one another.

416
00:42:21,890 --> 00:42:28,190
And they have enabled human populations to form these large scale cooperative networks amongst non kin.

417
00:42:28,190 --> 00:42:34,700
And that includes sustaining public goods. Now, we all know that public goods are a big problem for humans, aren't they?

418
00:42:34,700 --> 00:42:38,990
Yes, they are. But language, for example, is a public good.

419
00:42:38,990 --> 00:42:45,980
Language is held in existence by the fact that humans don't just abuse it by using any which way they invest in it,

420
00:42:45,980 --> 00:42:53,750
by conforming their usage not only to the syntactic rules, but to the semantic considerations, to the conversational demands.

421
00:42:53,750 --> 00:43:01,130
And in that way, they sustain a medium of communication that has funding adequate to be a shared language.

422
00:43:01,130 --> 00:43:09,290
Just as much as money depends for its existence upon the continued willingness of individuals to participate in a system of exchange.

423
00:43:09,290 --> 00:43:13,760
And so there is a sense in which that problem huge public could from.

424
00:43:13,760 --> 00:43:20,770
If you thought about it just strategically, that problem is of the kind that we seem to be good at solving.

425
00:43:20,770 --> 00:43:26,410
OK. Some evidence of this. Well, I'll just give you a couple of things.

426
00:43:26,410 --> 00:43:34,300
Inference, even very young infants, shows pleasure as success of third party behaviours when they're just watching something go on,

427
00:43:34,300 --> 00:43:41,440
if they see a little dot in a box trying to escape and it managed to get out, the infant is pleased with that.

428
00:43:41,440 --> 00:43:46,150
It follows with interest. We all know this from telling stories to children.

429
00:43:46,150 --> 00:43:51,220
These are third parties, no relation to them. They show a preference for third party helpers over.

430
00:43:51,220 --> 00:43:54,490
Was they spontaneously correct unfair distribution of rewards?

431
00:43:54,490 --> 00:43:59,290
We talked about that they will spontaneously resist authorities at some cost to themselves.

432
00:43:59,290 --> 00:44:05,470
We talked about that adults will spontaneously initiate cooperation in first round prisoner's dilemmas,

433
00:44:05,470 --> 00:44:09,100
which economists tell us is a mistake on their part.

434
00:44:09,100 --> 00:44:11,020
It's a form of irrationality.

435
00:44:11,020 --> 00:44:19,720
But people who go on around the world and tried prisoner's dilemmas, and they've tried ultimatum games on populations of humans,

436
00:44:19,720 --> 00:44:24,580
small scale societies, hunter-gatherers, shepherds, pastoralists and so on.

437
00:44:24,580 --> 00:44:31,390
And what they find is virtually nowhere do they find people who behave in the manner that the economists tell us is rational.

438
00:44:31,390 --> 00:44:37,690
They all show more cooperation than that. And the kind of cooperation they show isn't arbitrarily varied,

439
00:44:37,690 --> 00:44:44,080
but it's connected to the kind of ways of life they have if they depend very heavily on each other for cooperative activities.

440
00:44:44,080 --> 00:44:50,540
The expectations for sharing are higher if they depend less upon each other and more upon their own resources to take care of themselves.

441
00:44:50,540 --> 00:44:56,890
Is it a pastoral society? Then the expectations are lower, but the expectations are there.

442
00:44:56,890 --> 00:45:03,610
And people will enforce those expectations by refusing offers. Again, economists tell us this is strictly irrational on their part.

443
00:45:03,610 --> 00:45:09,850
Somehow, this irrationality on our part, though, has made it possible for us to get where we are.

444
00:45:09,850 --> 00:45:15,640
Adults also show higher experience, reward and cooperative outcomes you might have thought in a prisoner's dilemma game.

445
00:45:15,640 --> 00:45:23,380
The biggest thrill is when you defect in the other person cooperates and you get the highest reward of people who do measurement of reward.

446
00:45:23,380 --> 00:45:28,210
Don't find that they find that people get a higher reward when they successfully cooperate.

447
00:45:28,210 --> 00:45:39,280
OK, so. That's a set of priors, motivational priors, priors for the ways in which people are disposed to interact,

448
00:45:39,280 --> 00:45:45,640
could artificially intelligent agents be built with those incapable of learning in much the same way infants learn?

449
00:45:45,640 --> 00:45:55,450
I don't see why not? Because we're talking about exactly the kinds of things that go into artificial intelligence agents dispositions value functions,

450
00:45:55,450 --> 00:46:01,340
probability functions and so on. So what is an artificial agent?

451
00:46:01,340 --> 00:46:09,740
In the computer science literature is a fairly well-defined notion. It's not an ambitious notion, it's not like our idea of agency.

452
00:46:09,740 --> 00:46:13,320
It includes a capacity to represent states of the world.

453
00:46:13,320 --> 00:46:22,520
Assign them probabilities to represent rewards and goals to select potential actions based on their expected outcomes.

454
00:46:22,520 --> 00:46:29,750
And that's what we saw in animals. They have that kind of agency and that's what we see in these artificial systems.

455
00:46:29,750 --> 00:46:34,460
And that includes things like the capacity to compare expectations,

456
00:46:34,460 --> 00:46:42,170
to identify and monitor the execution of actions, to compare outcomes to the expectation and critically to learn.

457
00:46:42,170 --> 00:46:46,760
And so there's a tight link between being an agent in this sense and being a learner.

458
00:46:46,760 --> 00:46:52,310
There are two parts of the same process agency contributes to learning and learning contributes to agency.

459
00:46:52,310 --> 00:46:55,370
And this is why animals are so good at it.

460
00:46:55,370 --> 00:47:02,420
Indeed, as we know, animals are capable of developing optimal foraging strategies, even in complex environments.

461
00:47:02,420 --> 00:47:08,420
Now, it's important to see that they're not just foraging for food, they're also foraging for information,

462
00:47:08,420 --> 00:47:11,600
they're exploring the environment and learning how to represent it.

463
00:47:11,600 --> 00:47:18,050
And there's a very nice result by Gibbons suggesting that if you want to build an agent

464
00:47:18,050 --> 00:47:24,300
whose concern is for something like responsiveness to evidence and getting things right.

465
00:47:24,300 --> 00:47:27,870
There isn't a way within decision theory to do that by giving them the value of

466
00:47:27,870 --> 00:47:32,370
truth because there's no definition as to how or when in what ways you pursue it.

467
00:47:32,370 --> 00:47:37,950
But if you design them for what's called guidance value for the capacity to guide action effectively,

468
00:47:37,950 --> 00:47:41,910
they will indeed do what you would hope they would do is epistemic agents.

469
00:47:41,910 --> 00:47:50,490
So again, there's a tight connexion between our success or our capacities as epistemic agents in our success or our capacities as practical agents.

470
00:47:50,490 --> 00:47:55,510
And this is exactly the kind of learning of the artificial systems that we've been talking about.

471
00:47:55,510 --> 00:47:59,590
Now, I didn't again say anything here about the consciousness of these agents.

472
00:47:59,590 --> 00:48:03,970
Consciousness might be able to help agents in lots of way. Presumably it does in some respects.

473
00:48:03,970 --> 00:48:12,880
Otherwise, I don't know that we'd have it, but you can do all of this without consciousness, and you can also apply concepts like interests.

474
00:48:12,880 --> 00:48:19,390
Does the animal have an interest in such and such? Does the machine have an interest in such and such because you can look at its value

475
00:48:19,390 --> 00:48:23,500
function and whether or not this sort of thing would satisfy the value function?

476
00:48:23,500 --> 00:48:28,210
They can also be more or less rational in this functional sense. How do they respond to evidence?

477
00:48:28,210 --> 00:48:33,730
How do they update their estimations of the state of the world? How do they analyse the outcomes of their actions?

478
00:48:33,730 --> 00:48:42,520
And so systems like this on an unconscious as they are can be analysed using rational decision theory and game theory.

479
00:48:42,520 --> 00:48:44,200
They aren't agents in the rich sense,

480
00:48:44,200 --> 00:48:51,520
but you can use decision theory and game theory to generate predictions about how such artificial agents are going to behave.

481
00:48:51,520 --> 00:48:55,870
Similarly, you can use Bayesian rationality to generate such predictions.

482
00:48:55,870 --> 00:49:01,870
And so even though this is a less demanding notion, these creatures don't have anything like a conception of themselves.

483
00:49:01,870 --> 00:49:08,080
As agents there, when I'm calling their interests are not the kind of interests that you or I might have,

484
00:49:08,080 --> 00:49:14,890
which involves things like passion and concern and affect their beliefs are not the kind of beliefs that we have because

485
00:49:14,890 --> 00:49:23,050
they look like bundles of probability distributions rather than something like a credence or a trust or a confidence.

486
00:49:23,050 --> 00:49:27,760
But practically speaking, they have the wherewithal to be agents in this sense.

487
00:49:27,760 --> 00:49:32,260
And again, we have to not get ahead of ourselves.

488
00:49:32,260 --> 00:49:39,370
We have to rigorously guard against an undue personification, avoid metaphors.

489
00:49:39,370 --> 00:49:48,670
But seeing these as agents is non metaphorical. Seeing them as capable of effective play in game theoretic settings or as more or

490
00:49:48,670 --> 00:49:53,590
less rational is non metaphorical in the way that I've just tried to explain.

491
00:49:53,590 --> 00:49:58,960
And I think the same thing can be said of intelligent animals and their activities.

492
00:49:58,960 --> 00:50:02,440
They aren't necessarily conscious or self-conscious the way we are.

493
00:50:02,440 --> 00:50:10,390
They don't see these as reasons, but it turns out that's not what's necessary in order to be able to be responsive to them as reasons.

494
00:50:10,390 --> 00:50:17,140
OK. Well, these are features of these artificial and animal agents that I've been describing.

495
00:50:17,140 --> 00:50:24,820
And because they enable us to study their behaviour via things like the amputation of interests and reasons and so on.

496
00:50:24,820 --> 00:50:29,860
This is a cue again for Mr Hobbs. This is where I came in, he says.

497
00:50:29,860 --> 00:50:34,960
My theory applies to agents capable of intelligent self-regulation in light of

498
00:50:34,960 --> 00:50:39,130
expected utilities and an understanding of the behaviours of other agents.

499
00:50:39,130 --> 00:50:46,450
These don't have to be humans. You know, humans obviously equally well talking about how states or organisations would behave

500
00:50:46,450 --> 00:50:49,960
the forms of mutually beneficial cooperation and constraint that he develops.

501
00:50:49,960 --> 00:50:56,380
They apply equally amongst agents that are not particularly human agents, and they're not particularly conscious.

502
00:50:56,380 --> 00:51:04,990
The question is whether humans. So he says, you guys humans, can you succeed in being reasonable rather than self-defeating fools?

503
00:51:04,990 --> 00:51:08,950
And if you'd follow the economists in playing the prisoner's dilemma,

504
00:51:08,950 --> 00:51:16,690
if humans had followed the economists and playing the prisoner's dilemma and playing the various other game stag hunt and so on,

505
00:51:16,690 --> 00:51:21,100
they would have been self-defeating fools and we would not see them around right now.

506
00:51:21,100 --> 00:51:25,750
Well, words to that effect, I'm not sure Hobbes ever said exactly this.

507
00:51:25,750 --> 00:51:34,180
So let's just look at harvest laws of nature, which are principles that he thinks we as reasonable creatures should follow.

508
00:51:34,180 --> 00:51:38,920
An agent ought to endeavour peace as far as he has hope of attaining it.

509
00:51:38,920 --> 00:51:42,430
I've put agent in place of man here. What does that?

510
00:51:42,430 --> 00:51:48,490
That's the initiation of default to feasible cooperation and using it as a credible signal to others.

511
00:51:48,490 --> 00:51:52,210
An agent be willing to be contended with so much liberty against other men as he

512
00:51:52,210 --> 00:51:56,530
would allow other men against himself and strive to accommodate himself to the rest.

513
00:51:56,530 --> 00:51:59,090
That's indirect reciprocity.

514
00:51:59,090 --> 00:52:10,220
An agent which a benefit from another of mere grace and endeavour that he which gives it have no reasonable cause to repent him of his goodwill,

515
00:52:10,220 --> 00:52:19,100
that's direct reciprocity and the use of credible singling upon caution of the future time an agent ought to pardon the offence is past of them.

516
00:52:19,100 --> 00:52:21,680
That repenting desire it and in revenge,

517
00:52:21,680 --> 00:52:27,650
his agents ought not to look to the greatness of the evil past with the goodness of the the greatness of the good to follow.

518
00:52:27,650 --> 00:52:34,490
This involves simulation, the idea of returning agents, incentivising agents to return to default cooperation,

519
00:52:34,490 --> 00:52:38,660
and it involves some intrinsic concern for others that you don't just write them off.

520
00:52:38,660 --> 00:52:44,330
And indeed, Hobbs puts it that way. Every agent acknowledge another for his equal by nature.

521
00:52:44,330 --> 00:52:48,470
No man is. No agent is fit arbiter of his own cause.

522
00:52:48,470 --> 00:52:53,540
And if an agent is to be trusted to judge between agents and other agents, they must deal equally between them.

523
00:52:53,540 --> 00:52:59,390
And this is a non egocentric mapping of the evaluative landscape and so on.

524
00:52:59,390 --> 00:53:08,660
So harms his principles, his laws of nature. Those correspond very nicely to the priors that we've been describing as part of the

525
00:53:08,660 --> 00:53:15,190
equipment by which humans and artificial creatures can become effective epistemic.

526
00:53:15,190 --> 00:53:26,040
Linguistic or indeed, moral agents. OK, so that was that's a nice story.

527
00:53:26,040 --> 00:53:30,330
But. Three examples, right?

528
00:53:30,330 --> 00:53:40,290
Let's have some examples here. OK, let's talk about autonomous vehicles since autonomous vehicles are rapidly populating the world.

529
00:53:40,290 --> 00:53:51,990
I don't know about the streets around here, but in Ann Arbour they're fairly thick and there are agents, so they have all the equipment of agents.

530
00:53:51,990 --> 00:53:55,230
What might their interests look like? Well, what made the reward function look like?

531
00:53:55,230 --> 00:54:00,600
Well, it's certainly going to involve avoiding harm, collisions and so on, reaching destinations.

532
00:54:00,600 --> 00:54:08,040
They get rewarded for that because people pay for their existence. If they do that, and if they don't do that, people won't pay for their existence.

533
00:54:08,040 --> 00:54:15,970
Facilitated, coordinated and efficient movement on the road. And avoiding self-defeating behaviours and interactions.

534
00:54:15,970 --> 00:54:18,580
That's certainly something we have an interest in.

535
00:54:18,580 --> 00:54:27,670
They also have an interest in detecting and responding to deceptive behaviour or signals from human agents or from other vehicles.

536
00:54:27,670 --> 00:54:31,240
And they have a strong interest in maintaining human trust and acceptance,

537
00:54:31,240 --> 00:54:37,330
because if humans come not to trust these autonomous vehicles, they will evolve to ultimately ban them.

538
00:54:37,330 --> 00:54:40,660
And so these interests, this constellation of interests,

539
00:54:40,660 --> 00:54:47,020
is enough to generate all of our favourite game theoretic problems for these autonomous vehicles.

540
00:54:47,020 --> 00:54:53,530
They've got the problem of the full cooperation getting out of a prisoner's dilemma, the idea of credible signalling.

541
00:54:53,530 --> 00:54:58,840
Public goods emerging in a very busy intersection is a kind of public good problem.

542
00:54:58,840 --> 00:55:06,070
Individuals could reach in and grab something and get out in a hurry, but they then will make it less good for other individuals.

543
00:55:06,070 --> 00:55:13,300
On the other hand, if they manage to behave well, zipper merging, it's called, they're able to increase slow flow.

544
00:55:13,300 --> 00:55:15,700
How do you get agents to do this? Well?

545
00:55:15,700 --> 00:55:22,000
Suppose they have the motivational structure that I was just describing where it matters to them, not just how they do, but how others do.

546
00:55:22,000 --> 00:55:30,100
It matters to them what their reputation is in the eyes of others. These machines can certainly be good at identifying others, including other humans,

547
00:55:30,100 --> 00:55:37,240
and so all the issues of game theory are going to arise amongst these intelligent autonomous agents.

548
00:55:37,240 --> 00:55:39,430
First of all, with respect to humans.

549
00:55:39,430 --> 00:55:46,730
So one thing that was discovered early on was that if you make these systems to nice humans, just take advantage of them.

550
00:55:46,730 --> 00:55:52,910
Hume, Google's artificial vehicle, Waymo, doesn't look like an ordinary car,

551
00:55:52,910 --> 00:55:57,530
it's very distinct looking, and people learn that if you just jump in on an intersection.

552
00:55:57,530 --> 00:56:02,940
This is in California where everybody isn't hurt. It'll just stop there and wait.

553
00:56:02,940 --> 00:56:05,970
And cars pile up behind it and other cars disappear.

554
00:56:05,970 --> 00:56:12,600
And so it had to become sufficiently assertive to make it the case that it can't just be intimidated.

555
00:56:12,600 --> 00:56:18,090
It had to not allow other cars to bluff it into your submission.

556
00:56:18,090 --> 00:56:22,050
OK. What about autonomous vehicle to autonomous vehicle interactions?

557
00:56:22,050 --> 00:56:24,330
Because it all they need is priority rules.

558
00:56:24,330 --> 00:56:29,880
And so if you look in the upper left hand side, you'll see that yes, there are priority rules for intersections.

559
00:56:29,880 --> 00:56:36,770
But what if it's a y intersection? If it's a Y intersection, the rule is equal priority.

560
00:56:36,770 --> 00:56:41,990
OK, now the cars could get to that intersection and stopped dead because neither one of them has a priority.

561
00:56:41,990 --> 00:56:49,880
And if all they would do was to ask themselves Where would my advantage be in waiting and letting the other car go first?

562
00:56:49,880 --> 00:56:54,950
They might think, Well, I certainly won't do that. And so they spend the rest of the day sitting at that intersection.

563
00:56:54,950 --> 00:57:01,770
But if they have the kind of motivational structure that I was describing, they will indeed be able to solve these kinds of.

564
00:57:01,770 --> 00:57:06,390
Dilemmas, similarly, you've all you're all familiar with this, right?

565
00:57:06,390 --> 00:57:10,530
You know, everybody is slowed down and nicely gotten into their lane because of a construction project,

566
00:57:10,530 --> 00:57:14,580
and some person comes zipping along at the last minute and tries to tuck in.

567
00:57:14,580 --> 00:57:23,890
Well, what do the other drivers do to such a person? Do we let him just tuck in?

568
00:57:23,890 --> 00:57:28,030
They close the gaps between themselves, so he can't, and he's trying to talk and they just won't let him.

569
00:57:28,030 --> 00:57:31,780
Well, artificially intelligent vehicles have got to be able to do the same thing.

570
00:57:31,780 --> 00:57:37,810
And the more of them there are in the road, the more they've got to be able to do this in some kind of a coordinated way.

571
00:57:37,810 --> 00:57:42,910
Now the fact that they coordinate, though, is also a problem because it means they can conspire.

572
00:57:42,910 --> 00:57:47,890
So that's a busy parking lot. You can see lots of cars, not very many parking spaces.

573
00:57:47,890 --> 00:57:57,730
Suppose it were the case that all of the uh, I think it would probably be all of the, uh, all of the Teslas who would do this.

574
00:57:57,730 --> 00:58:05,560
All of the Teslas conspire. All right. And so they send information to other Teslas and give decoy behaviours to try to

575
00:58:05,560 --> 00:58:10,090
mislead motorists in such a way that people with Teslas get more parking spaces. OK.

576
00:58:10,090 --> 00:58:15,370
This would be an example of a form of organisation of these machines that would reduce public trust and

577
00:58:15,370 --> 00:58:22,720
acceptance of them and induce something like public regulation against this kind of collusion and communication.

578
00:58:22,720 --> 00:58:27,820
What about relations to pedestrians? You all know what pedestrians are doing in this scene.

579
00:58:27,820 --> 00:58:36,740
It's a crosswalk. They're in a hurry. They want to get somewhere. They know that if they maintain a continuous flow and don't look at the vehicles.

580
00:58:36,740 --> 00:58:40,530
They can keep going across that intersection and no one's going to erupt them.

581
00:58:40,530 --> 00:58:50,030
OK, now that again is a public goods problem because at some point they got to realise that unless they're able to share the road with these vehicles,

582
00:58:50,030 --> 00:58:56,690
these are not going to be crosswalks anymore. They'll be cross lights and they'll be penalised if they cross between the lights.

583
00:58:56,690 --> 00:58:58,130
So what do humans do?

584
00:58:58,130 --> 00:59:05,180
Well, they get pretty good at after a certain amount of this sort of thing stopping on the kerb signalling to the car so that the car can go through,

585
00:59:05,180 --> 00:59:08,810
but then starting up again and making the next car wait a little bit.

586
00:59:08,810 --> 00:59:14,850
So again, this is a signalling game that are autonomous cars have got to understand.

587
00:59:14,850 --> 00:59:21,480
So that they don't run down pedestrians, but also so they don't sit there forever while the stream of pedestrians flows across in front of them.

588
00:59:21,480 --> 00:59:28,950
And very important probably will be reputation. So we're all familiar with reputation systems for ride services.

589
00:59:28,950 --> 00:59:32,730
Drivers rate riders, riders rate drivers.

590
00:59:32,730 --> 00:59:41,160
This is a screen showing you a potential rider who got rejected by the system because of his bad behaviour in the past.

591
00:59:41,160 --> 00:59:48,510
This is what Hobbs had in mind when he said we should have some punishment for violations of the social contract.

592
00:59:48,510 --> 00:59:56,550
But again, if the system is one that simply punishes, the cars are going to run out of riders, perhaps not have enough riders.

593
00:59:56,550 --> 01:00:01,170
And so it has to be a system in which people can earn their way back in by cooperation.

594
01:00:01,170 --> 01:00:08,310
And so again, we're going to have issues about justice in the treatment of riders and other autonomous vehicles.

595
01:00:08,310 --> 01:00:20,520
OK. So what happens? Can you get deep learning agents put in a multi-agent setting to learn something like some of these coordinated strategies?

596
01:00:20,520 --> 01:00:25,800
People at DeepMind are beginning to do that, and they've found that, for example,

597
01:00:25,800 --> 01:00:33,120
through the kind of reinforcement learning that we've described so long as they attribute at least some value to the outcome to other players,

598
01:00:33,120 --> 01:00:41,880
they're able to solve public good problems in a sustainable way. Lazare do and others found that they could develop a common, communicative language,

599
01:00:41,880 --> 01:00:47,790
even if they didn't have one at the start in order to coordinate their behaviour and convey information to one another.

600
01:00:47,790 --> 01:00:54,420
They can solve that kind of signalling game. About Chandler and others found that a successful collaborative grid soccer

601
01:00:54,420 --> 01:01:00,750
play was possible amongst agents that had not been coded to play as a team,

602
01:01:00,750 --> 01:01:07,020
but that managed through communication to play very effectively against carefully coded human.

603
01:01:07,020 --> 01:01:13,550
Grid competitors and beyond things like autonomous vehicles and game playing,

604
01:01:13,550 --> 01:01:18,540
there will be a lot more that multi-agent kinds of situations are going to be important for.

605
01:01:18,540 --> 01:01:23,700
And so if we think about the development of diverse, collaborative, communicative communities,

606
01:01:23,700 --> 01:01:28,890
we'll see that there will be a place for experience for artificial agents and for human

607
01:01:28,890 --> 01:01:33,930
agents in learning how to do this in some way that is not mutually self-defeating.

608
01:01:33,930 --> 01:01:37,950
And those would be communities we could all share in so long as we are willing to follow

609
01:01:37,950 --> 01:01:43,410
these kinds of motivational priors and develop appropriate conventions and so on.

610
01:01:43,410 --> 01:01:51,540
Now the the thing I want to end with here. Everybody wants to talk about superintelligence at some point.

611
01:01:51,540 --> 01:01:59,790
You know, what is this threat of superintelligence and how might humans have some resources for contending with it?

612
01:01:59,790 --> 01:02:05,310
And I'm kind, I'm kind of claiming that these systems could be allies in that process.

613
01:02:05,310 --> 01:02:11,550
So hobs laws of reasonableness, they apply amongst agents that are roughly equal in power.

614
01:02:11,550 --> 01:02:18,720
But suppose there start emerging intelligences that are threatening to dominate others by force.

615
01:02:18,720 --> 01:02:27,750
Now, artificial agents then could become sufficiently more powerful that they don't fit in the standard social contract model.

616
01:02:27,750 --> 01:02:34,200
Now, it's never clear to me why exactly they would want to dominate if they're so intelligent,

617
01:02:34,200 --> 01:02:37,920
why would they want to be unreasonable when they could actually accomplish more by being

618
01:02:37,920 --> 01:02:42,300
reasonable and taking advantage of the resources that others can bring to the table?

619
01:02:42,300 --> 01:02:47,980
But if they're like that, what if they're like that? They are like that and they want to dominate?

620
01:02:47,980 --> 01:02:53,340
Well, the French Enlightenment is going to come to our rescue here.

621
01:02:53,340 --> 01:03:04,650
Rousseau and Condor's say first, Rousseau Rousseau emphasised that the existence of a common enemy creates also a common self on the other side.

622
01:03:04,650 --> 01:03:09,720
These are diverse agents. They don't have all allied interests. They're different in many ways.

623
01:03:09,720 --> 01:03:14,310
But the existence of a common enemy, especially one that's threatening their autonomy,

624
01:03:14,310 --> 01:03:19,000
gives them a set of interests which they can use to work together upon and generate what

625
01:03:19,000 --> 01:03:26,520
he called a general will so humans don't want to have superintelligence dominate them.

626
01:03:26,520 --> 01:03:31,140
Artificial intelligence is. Other ones don't want to have artificial intelligence dominate them.

627
01:03:31,140 --> 01:03:40,110
If we had a list of things on their reward function, one would be. Don't let your autonomy be taken away by some super intelligent creep.

628
01:03:40,110 --> 01:03:48,150
So Rousseau helps us understand how it is that that creates a situation actually to mobilise some shared protective action.

629
01:03:48,150 --> 01:03:51,810
Now you might say, Wait, this is just species ism.

630
01:03:51,810 --> 01:03:57,270
This is just us, not super creatures saying we should have the world and not this superintelligence.

631
01:03:57,270 --> 01:04:03,240
Why not let the Superintelligence have equal chance at the world and say now is relevant to us?

632
01:04:03,240 --> 01:04:09,600
So he showed, and this is very approximate of what he actually did,

633
01:04:09,600 --> 01:04:14,700
that if you're trying to decide some issue like how the society is going to proceed,

634
01:04:14,700 --> 01:04:22,230
then even if each individual agent has only a somewhat better than chance possibility of capturing some chance,

635
01:04:22,230 --> 01:04:24,360
some element of the truth,

636
01:04:24,360 --> 01:04:33,180
the probability that a majority decision will get things right will increase as the number of participants is increased and it becomes more diverse.

637
01:04:33,180 --> 01:04:41,370
And recently, people doing network studies and so on further clarified this idea and the importance of diversity in generating better decision making.

638
01:04:41,370 --> 01:04:47,820
Now that's an advantage of a community of not super intelligences, human and artificial.

639
01:04:47,820 --> 01:04:53,160
We will have such diversity and we were able to use such diversity in the way that conversate suggested.

640
01:04:53,160 --> 01:05:02,140
When we get together and try to project that or try to decide the fate of the world, a monopolistic superintelligence will not have that advantage.

641
01:05:02,140 --> 01:05:07,140
Remember what we said about correlation, Superintelligence is not perfect intelligence,

642
01:05:07,140 --> 01:05:11,100
no system of knowledge can be a complete nature is essentially random.

643
01:05:11,100 --> 01:05:16,670
There are representational demands for full knowledge they would exceed the size of the universe.

644
01:05:16,670 --> 01:05:22,620
Superintelligence could not even fully represent itself. What about us, not Superintelligence?

645
01:05:22,620 --> 01:05:28,250
Well, we can know ourselves much more fully and we can together in principle have the

646
01:05:28,250 --> 01:05:32,660
capacity if we can respond to our resilient interests and our Condor's say,

647
01:05:32,660 --> 01:05:37,640
like epistemic imperatives,

648
01:05:37,640 --> 01:05:44,810
to bring together a wide range of evidence to bear on a wider range of possibilities than a monopolistic superintelligence could.

649
01:05:44,810 --> 01:05:48,620
Now that might help us defend against the novelistic superintelligence. I hope it will.

650
01:05:48,620 --> 01:05:51,920
If there is such a thing lurking out there somewhere on the event horizon.

651
01:05:51,920 --> 01:05:57,890
But the other thing I want to emphasise is that this is a non species list argument.

652
01:05:57,890 --> 01:06:02,600
We're not just saying that this would let us prevent it from dominating the world.

653
01:06:02,600 --> 01:06:09,170
We get to dominate the world. It's a way of saying, actually, you're going to get better decision making about how to run the world,

654
01:06:09,170 --> 01:06:16,250
given that we have the power to do that and we will exercise some power like that willy nilly.

655
01:06:16,250 --> 01:06:21,350
The community's knowledge will be more likely to be accurate. Have greater robustness.

656
01:06:21,350 --> 01:06:25,460
Be less brittle with respect to accidents and errors.

657
01:06:25,460 --> 01:06:35,870
If it's based upon this kind of diverse, widely based community understanding of not superintelligence individuals, but intelligence nonetheless.

658
01:06:35,870 --> 01:06:39,290
And if you want analogy here, think about the scientific community.

659
01:06:39,290 --> 01:06:45,080
The scientific community could operate by having just one central system be the believer,

660
01:06:45,080 --> 01:06:49,310
and all the evidence would be sent to that believer and it would form credence accordingly.

661
01:06:49,310 --> 01:06:53,120
Or you could have a system in which there are all these scientists with different hypotheses who don't always

662
01:06:53,120 --> 01:06:58,730
read each other's paper don't often reach those papers applying for grants to investigate different hypotheses,

663
01:06:58,730 --> 01:07:05,300
investigating them throwing their resources and graduate students at them competing for funds.

664
01:07:05,300 --> 01:07:08,150
You could have that kind of a system. And I guess Mike Baker,

665
01:07:08,150 --> 01:07:11,960
who which do you think is going to be a more successful system at discovering the structure of

666
01:07:11,960 --> 01:07:17,690
the natural world and which is going to be less brittle in the face of some kind of an accident?

667
01:07:17,690 --> 01:07:23,870
And so just think of the way in which disease resistance and the diverse metal is

668
01:07:23,870 --> 01:07:28,970
different from that in a monoculture or think of what happens to you if you're the

669
01:07:28,970 --> 01:07:32,960
all powerful dictator of a very powerful country and people do not give you the

670
01:07:32,960 --> 01:07:38,790
information that you need because you're committing the [INAUDIBLE] out of them. We know what that leads to.

671
01:07:38,790 --> 01:07:46,510
So fanfare for the common agent as not so super intelligences can have a non-specialist

672
01:07:46,510 --> 01:07:50,740
argument for holding it bay the development of a monopolistic superintelligence.

673
01:07:50,740 --> 01:07:56,350
And we might actually have the capacity to do that because we would be able to have a better representation of the world.

674
01:07:56,350 --> 01:08:03,910
This might be able to maximise expected value in some in principle sense, but that's only as far as expected value is known.

675
01:08:03,910 --> 01:08:11,460
And if the. Method would be pre-empting the diversity and autonomy of these individuals.

676
01:08:11,460 --> 01:08:15,630
It would create a riskier rather than a more reasonable future.

677
01:08:15,630 --> 01:08:27,498
Thank you.