So I'd like to thank all of you for showing up for the second of three lectures by Professor Peter Railton from the University of Michigan,
Ann Arbour, broadly on the topic of ethics and AI, the first, as you'll remember, was on moral learning.
The second one is on understanding artificial agency.
I will be very brief and just say that Peter is, of course, known to anyone who works in media ethics and normative ethics.
Whether you're working at an undergraduate level or a graduate level or professional level, you must engage with this work.
And though I believe the turn to Ethics VII is a very recent turn, I'm sure that it's going to be true of his work here as well.
So please join me in welcoming Professor Railton. Thank you.
Thank you very much, it's good to be back.
Apropos of my recently coming to this topic, I just want you to know that you're entering an expertise free zone here.
I'm trying to operate as a moral philosopher in an area in which moral philosophers aren't particularly trained and learning as much as I can.
But you should be expected to wear a mental hard hat at all times because who knows what might be raining down upon you?
So you were warned. So last time I was closing on this speculation that maybe humans and artificial
intelligence systems could be able to develop something like communities of
trust and cooperation that permit their different special abilities to work
together to produce something greater than they could produce on their own.
And of course, for that to happen, it would have to depend upon the actual development of such a community.
And that, in turn, would depend upon the development of something like sufficient trust and cooperative ness and
sufficient forms of coordination and communication amongst these different types of agents.
And so today I'm going to be talking about artificial agents and what kind of allies they might be.
And the idea of Ally is the idea that we may not be the same in many ways.
And I mean, I'm not saying that machines have the standing of persons, at least not yet, but they may have the standing of allies.
So to become an artificial ally, an intelligent one,
these will not have to be moral agents that might involve a great deal, something like self-consciousness, moral emotions.
Jeremy's here somewhere a capacity for practical reasoning,
and we don't think that these systems have that or it's that it's in prospect anytime soon.
It might be enough if artificial systems did have some endogenous capacity to be
sensitive to and able to respond aptly to morally relevant features of situations,
actions, agents and outcomes. So that's the working notion.
I'm going to have this a little bit of a mouthful. The idea is sensitivity and responsiveness of an apt kind to morally relevant features.
Such sensitivity and responsiveness sensitivity, by the way, is the idea that you actually notice these things, which is a big part of ethics.
Getting people to notice the relevant features and responsiveness is the way in which you engage with them.
Such sensitivity and responsiveness, I claimed,
would require that such systems actually have some degree of autonomy with respect to us and the human agents around them.
And you can see why that would be so.
These systems will be around humans whose purposes aren't always good or they're misguided in some ways.
And if they don't have any autonomy, then they certainly can't be expected to respond happily to morally relevant features.
So that's the project. Well, what is this idea of a morally relevant feature?
And I gave you last time a kind of a taxonomy of some dimensions of morally relevant features.
They need to be non parochial and non egocentric.
There are things like special permissions that we have with respect to our lives, our friends, our families.
But those are non parochial. Everyone has the same permissions.
They aren't founded upon something special about me because in fact, there is nothing special about me.
They are general and they're super vignette in nature. They depend upon non evaluative features.
They are linked somehow to motivation. And so when we think about these morally relevant features,
we have to think that they're doing some action guiding and not just some registering effects.
On the other hand, the motivation and the kind of intelligence they provide has to be non instrumental in a way.
And it has to be independent of authority or sanction and to make it moral because these are all features that you could find in,
say, epistemic epistemic irrelevant features to make it distinctively moral.
We have to say something about the subject matter, and I'm going to be very Catholic about that.
They can be such things as harms, benefits, fairness, respect and so on for oneself and for others.
And if you can show to someone's world judgements run afoul of these, they are parochial or they are instrumental,
or they don't show adequate respect for people or their benefits and burdens, then that's a ground of criticism.
So what about this idea of autonomy and trust will recall the example last time of the
independent mindedness of one and a half two year old children with respect to fairness,
if they're playing a game, this involves a joint activity with another child that even if they are given the entire reward themselves,
they will distribute it to the other child as well. They'll do that at some expense to themselves.
That's a kind of independent mindedness because the authority doesn't tell them to do that,
doesn't reward them from doing it, says this is what you get. The child does it nonetheless.
And three or. Four year old children with respect to authority will resist inflicting harm or unfairness on others,
and by five or six, they'll intervene to stop harm or unfairness amongst third parties.
So that's a kind of autonomy, the authority says. Do this. The child says I won't.
Nobody says why? Because it would harm this person because it would be unfair. Or they will intervene in that kind of a case.
And it's to me, a form of trustworthiness in children that they have some autonomy.
You know, as a parent, you realise at some point that if they have no autonomy,
they aren't trustworthy because they will pick up whatever is in their environment.
And so if we're thinking about making these systems trustworthy,
we should also be thinking about making them in the relevant sense, autonomous and just as an epistemic matter,
they won't be able to respond to apply to morally relevant features or detect them if they don't have some degree of independent mindedness.
Now you could say, OK, yeah, that's great. But children are building on very highly evolved psychology.
It's a psychology that somehow is fit for our forms of life. It's demonstrated that humans can live together even reasonably peacefully.
Not always, unfortunately. But what about artificial agents? Aren't they coming from nowhere in a sense?
And if they are coming from nowhere and we're asking them to do tasks for us,
what does that have to do with this project of asking them to be sensitive to a morally relevant features?
Isn't that kind of a fool's errand? And I'm going to focus today a little bit on the problem of autonomous vehicles so-called.
And I just ask you to put in the back of your mind somewhere why it might be that autonomous
vehicles would need something like a capacity to be responsive to more of the relevant features.
If you imagine all the dynamics in these situations, you're interacting with humans,
they're interacting with each other, they're interacting with pedestrians.
That's going to require more than just the ability to stay in lane or to stop it stop signs.
So. Any time anyone talks about autonomy for machines or disobedient machines,
even more so that at least as threatening as it is promising, there are some people working.
I think she say the last thing we should do is make these systems in any way autonomous because then they can escape our control.
And the fact of the matter is it seems to be that their promise as potential features with capacity to
engage with morally relevant considerations and the threat that that involves in terms of autonomy,
those are inextricable. I don't think you can pull those apart.
And you could say, why should we allow beings of that kind to be brought into existence?
And what I tried to claim last time is this issue.
Will we allow the development and deployment of artificially intelligent systems able to acquire extensive
information and use it to take dangerous or manipulative actions in ways that risk escaping our means of control?
That's already happening. That's not a choice that we have.
These systems do exist and they are in some ways escaping control, and they're not very far down that road.
And indeed, most of the problem doesn't come from the machines themselves, but to a greater extent.
With time, it will. So the question isn't before we create and deploy such systems.
We must guarantee that they will not become autonomous and produce results we do not want.
As I suggested earlier, they can be results that we want that we shouldn't want. Rather,
the question is more like given the increasing development deployment of AI systems in a wide range of applications and given
that we do not have have little prospect of having and probably do not want some kind of a centralised system of human control.
What are some more distributed resources or methods for trying to identify and respond to these threats
and achieve more of the promise and in particular amongst those distributed resources and methods?
Might these involve the artificial agents themselves? And so right now, as I said, the most serious form of these threats is not from the machines,
not monster ultra powerful intelligent machines, but monster,
not necessarily entirely intelligent humans using these machines for nefarious purposes or for misguided purposes,
or simply using them in ways that involve error or failure to understand what they're doing.
So that's a big source right now, but this already suggests that we should be asking ourselves the question Do we
want these machines to be just all purpose tools for whoever picks them up?
Might they have in their DNA, we might hope some ability to resist.
So what's this idea of apt sensitivity and responsiveness?
And the first analogy I want to make is with vision and object identification, as it's now done in contemporary artificial intelligence.
And last time we talked about chess and about the view held for generation in artificial intelligence,
that the way to get expert chess play out of machines is to programme human chess expertise into the system and
then let it crank through all the possible moves as a way to achieve human level performance or exceed us.
Now, more recently,
what started happening is that machines without these hand coded means of detecting and utilising winning at chess relevant features,
they aren't told what are the winning at chess relevant features. They're told whether they've won a game or not.
But capable of simulating play internally, autonomously simulating play against themselves.
These have achieved much higher levels of success at chess, learning in effect from scratch rather than from human experts.
Chess playing neural networks rediscovered. A number of known strategies are what really are moves sequences because that's all these are present.
They aren't strategizing as they are movers and they discover new ones.
They had learnt in some sense the structure of the game and its possibilities,
and they did that using very generic reinforcement learning methods, which of course, required a great deal of tweaking.
But essentially, the core is something like generic reinforcement learning.
And despite its complexity, you might say, well, chess.
I mean, chess is a bounded world. Well, things are getting a little bit less bounded alpha zero now and Mewes zero using similar
generic learning processes have achieved more than human levels at a range of games.
Go Shogi, Atari and so on. And they don't require extensive reprogramming to do this.
They can use the same methods and build on some of their competencies. And these persistent systems possess,
I'm going to say something like a general or a more general capacity to be sensitive and aptly responsive to strategic game playing features,
generating and deploying them in their own moves in their own decisions.
So that's the idea that these are not machines that have human understanding of chess or show you they don't.
But these are machines that are set apni sensitive to the relevant game winning features in chess or shogi.
Or go. Well, let's broaden out a little bit.
Let's go beyond the world of strategic games. There was a similar progression in machine vision.
One of the main goals of artificial intelligence from the very beginning was machines that could decode the world visually, and for years,
it was thought that machine vision could only be achieved if human human experts encoded by hand visual features of situations.
Imagine what the list of that looks like visual features, situations and ways of translating those into something like objects in three dimensional
space and thousands of man years were dedicated to this task of coding such features.
More recently, deep neural networks of the same kind we've just been discussing have become more
effective at detecting and responding aptly to objects and spatial relations,
identifying features in images via these generic learning methods.
So again, it's not that they have the same understanding of the world that we do,
but that they are more apt than humans actually at responding to object and spatial relations, identifying features quickly.
Using images as the basis, and so that's an interesting idea,
this idea that you can respond to these features and therefore become expert at chess or extremely
good at visual identification without having what we would think of as a conscious mind.
Now, along the way, something interesting happened. So here's human vision processing that's supposed to be a cat on the left.
It goes into your visual cortices outputs from the retina.
There's some fancy wiring in the retina retina of mutual inhibition to sharpen the image, and then it goes up into the visual cortex.
And there at first, you start detecting things like edges of objects,
and then it goes up to the next visual cortex and you start seeing conjunctions of edges.
And it goes up through these different cortices and you assemble more and more complex features of the image that is being presented by the retina.
And if you look at the structure of such cortex,
you'll see that it has a stacked structure with neural connexions projecting through it in this kind of a network way.
And that was indeed the inspiration for the neural nets. A very simple neural net is on the right.
There's an input layer that would correspond, let's say, to the retina.
There are various hidden layers, and then there's an output layer and identification of an object, let's say.
And each of those circles corresponds to something that's deliberately designed to be like a neurone.
That is to say it accepts inputs from multiple artificial dendrites.
And depending upon the nature of those inputs, their frequency,
their intensity and so on, it either fires or doesn't, which is a discontinuous change.
That's the secret of neurones. And that then goes out and is passed along to the next neurones in the sequence.
And what's interesting is that if you build a system with enough layers and enough parameters and you don't tell it how to identify objects,
but you train it on a large number of images, that's something we can now do because people are great at putting images up on the internet.
What you find is that they do produce object identification.
But what's interesting as well is they do it in a manner very similar to human vision.
That is, the first layers of the nets detect things like edges.
The second layers of the nets detect things like intersections of edges and features the third layer, more complex features and so on.
So that suggests that the architecture of vision, human vision and machine vision is a bit more robust than you might have thought,
you might have thought that vision is heavily dependent on the particular way humans are wired up.
And that's going to stand between us in an objective presentation of the world.
And then you say, Well, this system didn't begin with the human expectations.
It began as a very simple learning system, but simply tried best to fit the images that it was given.
Give the best data fit for the images that it was given, and it ended up with a very similar structure.
And so that gives us an idea that we may have some work, some more robustness in our epistemic capacities than you might have thought.
We may be less imprisoned than we thought in our perceptual presuppositions.
And the idea is that this is a way in which the machines have shown that they too can be very
good as the visual system is at responding to object and spatial relations identifying features.
OK, now learning as visual learning is much more than passive modelling of structure.
It's also generated in action. Guiding most of the visual field that you have is being actively populated by your
brain in order to make sure the whole thing is coloured and sharpen focus and so on.
So your visual system, it's not just flowing one way, it's also flowing in the other direction, generating lots of features of vision itself.
And the question then is will these systems have that capability?
Well, in chess and go and so on, we found that they could generate novel moves and successful strategies.
And it turns out that neural nets trained up for vision can buy, so to speak, being run in reverse generate images.
And so here you've probably you've probably all seen these. These are images that were generated by artificial adversarial networks.
And of course, they generate a lot of images that look kind of like rubbish or look kind of like a dog, but smeared out sideways and so on.
But they also can generate images like this, and they get better and better at it as they get better and better at vision.
This is also something that's important because it's important for understanding how human imagination works.
Human imagination takes what we have got by way of structure of the world from experience and then reuses that internally,
just as a chess playing programme used its experience internally to simulate possibilities
and enable us to think about things that are not present in sensation to think about a red,
white spotted mushroom against a green field of grass,
which you can close your eyes and do perfectly well right now without the help of such a machine.
So that's the idea, then, that these are not just passively recording structure, they are also generating a capacity to create a visual field and.
Again, we really can't get ahead of ourselves in this, I mean,
there's always this rah rah that goes on with this and the hype, but we're not claiming.
I'm not claiming that these systems understand the strategic games or even that
they know that these games are played amongst individuals who are in some sense,
opponents of one another. That's not what it's got. It's just got pixel input,
nor that they understand the objects that they identify or that they generate
that they think of them as objects in space in the same way we do and so on.
It's not the way they work. They're operating with pixels inputs and they're yielding outputs.
Yet at the same time, they can be extremely effective in novel a general settings settings that don't just
require representation but require action and that require novel action like game playing,
object identification, imagining objects and guiding self-driving cars driving cars.
So the representations aren't just superficial layering of patterns.
They in some ways have captured the structure of the information that they're being given,
which in turn enables them to in some sense model implicitly the structure of the
features of the world around them and without representing them metaphysically,
it's nonetheless able to operate within that environment effectively.
So I'm going to try another analogy. This was a way of talking about the idea of morally responding happily to morally relevant
features by talking about responding aptly to features relevant to object identification.
Here's an even perhaps closer analogy.
So artificially intelligent systems might become allied epistemic agents in something like medical or scientific research.
And I gather Dennis Fisher, this is going to be talking about something like this tomorrow.
And if he were here, he probably pulling his hair out. But the the these systems have become very important in contemporary scientific research.
And I notice his title is scientific discovery. These are not just confirmation systems, their systems for discovering novel hypotheses.
So these systems can, I think,
come to possess a significant degree of autonomous credibility in identifying and responding aptly to epistemic relevant features,
including the formation and assessment of novel causal models, as they do with protein folding.
Novel hypotheses that better fit the data than our existing hypotheses.
Making unanticipated connexions across very extensive data sets or literatures that humans could not do on their own.
Manage making innovations in methodology or experimentation by running simulations
and seeing what might work running many more simulations than you or I could.
And in some sense, these could be something more than just tools or tablets.
They could be something like co-investigators. They are probing the structure of what they are learning from the data,
and they are generating new ways of representing it and thinking about it, which can be used to guide action.
And in these ways, they could help us make substantive breakthroughs in science, maybe ones that we wouldn't particularly make.
Again, we aren't assuming that they're conscious that they have an understanding of the world.
Within two hours, their representations might take the form of something like sets of probability functions over possible worlds.
Their grasp of meaning and interpreting data and hypotheses.
They're inferences might be algebraic rather than propositional through some hierarchy of association, like the ones we saw in the visual cortex.
These may require a quite complex function to be mapped upon the kind of discrete concepts that you or I have.
But nonetheless, if such a mapping is possible, it should be possible for these systems to be allies of ours.
And indeed, they are increasingly now are.
And again, what's interesting is that the fact that they don't duplicate our understanding is a kind of an asset.
First of all, it produces a kind of correlation, whether we like it or not.
Virtually all scientists are human beings and they therefore are highly correlated and quite a number of ways.
The data from the world aren't structured around the human psyche.
I believe. If we had other ways of representing that were correlated with the human psyche
and that could identify different clusters at different relations amongst objects,
that might be a way for our research to become less brittle and more robust than it is now.
That would be a way in which these machines would again be autonomous epistemic agents.
It's not that we ask them to reproduce our way of seeing, it's that they can contribute a distinctive way of seeing.
Moreover, beyond this kind of consultants, artificially intelligent agents might come up with ways of identifying data
that generate for us new concepts that we would not have formed on our own,
but the concepts we might be able to use to perform further successful experimentation or technology.
Now, the credibility of that new way of organising data would not depend, I think,
upon the human ability to translate exactly what that organisational principle was is into human concepts, existing human concepts.
It could very well be that we can't easily translate it into existing human concepts.
Yet it could win its way into acceptance by us because of the way it makes unanticipated or successful predictions.
And space time is like that right? Space time was not translatable without reduction into the previous concept.
It enabled them to make some very surprising predictions. It was very hard for many people still is to wrap their mind around it.
Yet it could acquire a kind of epistemic authority and credibility because of the way in which it enhanced this kind of scientific power.
And so a research community might incorporate a novel category into their
conceptual repertoire because they will develop practises with these machines,
and successful outcomes have led them to see such systems as responding aptly to epistemic
relevant features that would be part of the explanation of why they accept this.
And such a system could merit trust on the part of human researchers as something like autonomous epistemic agents.
Moreover, they would be showing a capacity to operate in a way with human language that we ourselves don't,
but which might give us further insight into it.
And this is something like this idea of an expert ally is something what like what Kasparov has written about?
Alpha zero programmes usually reflect priorities and prejudices of programmers.
But because Alpha Zero programmes itself, I would say that its style reflects the truth.
Not the truth about the metaphysics of chess, but the truth about the structure of the game and its possibilities.
This superior understanding allowed it to outclassed the world's top traditional programme despite calculating far fewer positions per second.
Alpha Zero shows is that machines can be the experts and not merely expert tools,
and that one respect from the chess community in the way that these agents could win autonomous levels of respect from the chess community.
OK, now any such system will make errors and large errors just like human enquiries.
They may be different areas and we can learn from that. That may be very important, but we can buy operating with them,
enlarge our capacity to be flexible and adaptive enough to understand the world better.
That's the claim. Moreover, such systems can receive reciprocal epistemic benefit from us.
They don't have everything. They or their nets don't have all possibilities.
They can have their responsiveness to epistemic.
Irrelevant features improved by human forms of experimental and methodological and theoretical testing.
And so we can have a reciprocal relationship with these agents, not just listening to them,
not just speaking to them, but exchanging with them and contemporary deep learning systems.
Since they learn via very general algorithms in which the influence of priors tends to wash out,
you might tend to think, Well, no, they don't need us because their priors will wash out ours would too.
But their finite systems, and for any finite system, they're going to be blind spots.
There are only so many categories or parameters it will have and priors wash out not by a guarantee,
but only in the very long run and only with high probability. So we can be reciprocal epistemic agents.
And that could be the basis of what I could call an epistemic social contract.
And we could have a contract, which includes artificial intelligence systems as bona fide members, signatories in the epistemic community,
quite independently of whether such systems have consciousness like ours or indeed any consciousness at all,
quite independently of whether they're persons. And I'm underlining this because I want to say that they could be participants
in a moral social contract without being conscious and without being persons,
but nonetheless having some authority, autonomous authority and credibility.
But what would happen if we tried to widen the epistemic social contract beyond the setting of just scientific research?
That's going to be a big challenge because fuller participation in the wide epistemic community gathering trust and
credibility is being responsive to epistemic irrelevant features in the wide world that requires a huge array of skills,
not just the ability to take in large datasets and model them effectively.
It includes the ability to identify linguistic features and situations, understand how conversational norms work and how to use common knowledge,
or what common knowledge is likely to be to identify and address epistemic needs or goals in the situations to be really good,
reliable, effective code participants in the broad epistemic community.
This is going to require not just special epistemic competence or response from just epistemic features,
but responsiveness to linguistic features, social features, interactional features.
And so building a machine that is capable of this kind of interaction in the wider world,
which is a goal of artificial intelligence, is going to be a broad goal.
And it's going to involve these various expertises and this is something we talked more about last time.
Mm-Hmm. But it doesn't, I think, require consciousness. It doesn't require personhood.
But it would require this kind of wide responsiveness to a wide array of features that go well beyond the epistemic.
And many of these features, if we just notice them sideways, we'll see.
Oh, you know, some of those features are also morally relevant features.
They have to do with intentions and communication and whether there's a fair exchange
of information or whether one individual is being reliable or not reliable,
and whether one individual is allowing information to be expressed or not expressed.
That's the kind of dynamic that has to be understood to be it all purpose epistemic agent.
But there is a lot to do with the moral features of these informational situations,
and so such systems are going to have to work their way into an understanding of quite a range of morally relevant features as well.
So the thought is divine designing artificially into artificially intelligent systems so that they can learn to be highly effective,
trusted in autonomous code participants in the wide epistemic community that as a goal of the AI design,
is going to require a bundle of capacities to be sensitive to semantically, socially, psychologically, morally and so on.
Relevant features. And as I suggested last time,
this package of app sensitivities might be something that plays an important role in understanding the distinctiveness of human society and culture.
How might that be? Well. All of this has to be possible in real time, humans by analogy to artificial systems must be.
It seems to me, continuously taking in relevant information. Does your audience really look like it's still with you assessing its relevance,
updating the representations, monitoring the effects of one's actions and so on.
We have to be doing this continuously. Machines would have to do this continuously.
And part of what's interesting is that we possess two dominant modes of continuing processing in the brain.
One is a task relevant or attentional network that's focussed on a specific activity, and the other is a so-called default mode network,
and they alternate in their predominance in regular and irregular cycles throughout the day, some very short cycles.
And that's why you can't stop your mind from drifting, even when you're very interested in what the person saying.
Your mind will start off down to other tracks. That's not a defect.
That's the structural feature of this system, because it's a system that has to take information in and processes importance,
and that involves drifting off into other directions. Put that information back into the system, guiding action and response.
And so we have these two types of systems dynamically functioning within us, and that helps make us the kinds of agents we are now.
Suppose we ask. What else do these systems do? Well, last time we saw that the default mode system does autobiographical memory,
the accumulation of knowledge, culture, whatever it would be in the utilisation of it in novel situations.
Envisioning the future, projecting novel situations, imagining where they would be like counterfactual situations, theory of mind,
attribution of intention or motives or behaviours to other individuals, understanding them as intentional beings and.
Semantics. This is also the system we use for interpreting language, and it's very important because it's able to mobilise.
What do you think you need to understand that English, you need your memory here, need theory of mind.
You need a capacity to simulate what's going on in this statement. And so of course, it's going to be heavily involved in semantic processing as well.
And as we saw last time, it's used in moral decision making.
It's one system that is singularly present in a wide range of moral decision making situations.
And so you could think, Oh, so we do have this connected bundle of capacities.
We're built for this and we're built for it in connexion with understanding one another and action.
Now that's encouraging, and it's discouraging and may be encouraging.
Again,
human conscious and daydreaming experience makes somehow reflect what's going on in the default mode system as the images that pop up in your head do,
or the thoughts that pop in your head and conscious thought can activate it when you start spirit a series of imaginings.
But in general, activation is quite independent of consciousness, and conscious direction doesn't seem necessary for this to appear,
and there are creatures that we don't know to be conscious who seem to have exactly the same structure.
So this might be a kind of processing that can be done by creatures and
machines without asking the question of whether they have consciousness or not,
even though they are decoding situations in these complex ways.
And that's encouraging if you're going to make progress toward the current goal of developing artificially
intelligent systems capable of being successful at that sort of wider participation in human social existence.
And that's going to be essential for them to carry out all kinds of tasks, whether it's translation or health aids scientific discovery.
Driving autonomous vehicles, so that's encouraging. At the same time, that is for the A.I. community encouraging at the same time,
it suggests that real success in these tasks may not require creating consciousness,
may not require getting persons where it's going to require creating systems
that are extremely sophisticated with regard to this bundle of capacities.
They will all be part of getting the linguistic competence right or getting the epistemic competence right or getting the social competence right.
They will all be bounded together.
And so success at the task of playing chess, which is quite a bounded activity compared to wandering about in the real world,
and it has a much clearer criterion of success that required millions of simulations.
But it didn't require modelling human psychology, which was interesting.
You might have thought to play Championship press, you've got to chess, you've got to be able to model the psychology of the other Asian.
But if there is a model in there of the psychology of agent, it's an implicit model.
It's a model that's implicit in the play and it's activated not by having a representation of human psychology,
but by whatever dependencies probabilistic dependencies there might be in its game playing strategies.
Now, just imagine then, the amount of information and the number of simulations that are going to be required to
identify the success or failure in social situations or epistemic situations for that matter,
or acquiring the ability to accurately model the causal or intentional or normative structure of social situations.
That's going to be huge. Now, one thought that has been informing this discussion in the background is infants do this,
infants do this, and they do it through thousands of hours of observation and interaction as they grow older.
So in the course of development, infants alternating between task relevant and default mode activity,
watching the world in ways that we did not appreciate before. Just as we do not appreciate how much animals see in the world acquire very substantial,
intuitive competency, it isn't even in the first weeks of life. Infants are showing surprise at anomalous linguistic sequences.
Were they listening in the womb? Yes, they were. Were they listening in the first weeks of life?
Yes, they were. Are they listening? Even now you're having an argument at dinner and they're just sitting there like this?
Yes. So the developmental studies indicate that infant and child progress in learning causal relations,
language theory of mind and moral assessment proceed in a coordinated way throughout the development.
And they do so in a process that resembles something like Bayesian learning.
Now, the exact nature of this is by no means understood. But the idea is that we're able to model it to some extent in these ways.
And in doing so, we discover that there is this. Bundle again at work, developing together, and now that again,
is encouraging for the prospect of developing machines that can be aptly sensitive to morally relevant features
because we know that biological machines do it somehow and we know roughly the kinds of ways in which they do it.
And so the challenge is to try to figure out whether this learning is possible and how now infants and toddlers do have to begin with certain priors.
These machines don't necessarily have to begin with pliers with regard to the structure of the world.
But there are going to be other priors.
There will be effectively motivational priors because any machine is going to have to have something like a reward function.
That's how it's going to figure out success or failure in the end, not because it's a game that it's played and it won or lost.
But how the reward function carried out in the actions that it performed.
And so last time we looked at this question and we said, well, what kind of a reward function to humans seem to have?
And human infants in particular? And what we saw is that even our intelligent animal ancestors have a capacity for non egocentric
as well as egocentric representation of the space and the causal structure of their environment.
They have a capacity for simulating and evaluating alternative courses of events or actions.
They have a capacity for learning, representing and deploying in decision making these evaluative expectations.
Using that planned execution of behaviour and monitoring of action to learn.
We saw this in intelligent animals last time, we certainly see it in human infants.
Those thus far, though, no distinctive moral priors, these kinds of ways in which the infant is set up in order to do this kind
of learning are ways that don't have to give a moral code or a moral module.
These are, after all, shared with animals that we don't think have anything like that.
There are also certain motivational priors, a degree of self concern and concern for, and we see that a capacity of reciprocation of direct benefits.
We also see that in animals and children. And so again, there's nothing distinctively moral about this so far.
But in humans, we do find something more and is quite open to debate exactly how much of this we find in other intelligent animals.
And the fact that they are present in humans seems to be an important part of the explanation of our distinctive forms of social life.
And so you can think it's not so much that there's some bit of knowledge in there,
but rather it is some bit of motivational structure that is playing a key role.
But what is this like? Well, it's a capacity for default to feasible cooperative ness.
Even with non kin, humans live in large communities of genetically unrelated individuals cooperating daily.
A capacity for indirect reciprocity with respect to groups.
Human and human hunter gatherer groups, for example, they share very widely within themselves,
but they can't keep accurate track of exactly what's shared.
Individuals do contribute and individuals do receive and they have a kind of social insurance as a result,
but they're willing to contribute because they're also receiving benefits from the group.
And that's an indirect reciprocity. And we see that in humans and human infants.
A capacity for some intrinsic concern with how unrelated others fare and how we are seen by them.
And so you might think of that is what psychologists call pro-social motivation.
Now that is pro-social, is it moral? Have I said anything yet of a moral kind?
Or are these general purpose motivations for forming social relations, acquiring language and operating successfully in the social world?
So my claim is that these are general purpose, motivational capacities. They foster shared learning, for example, of things like language they share.
They are part of how we're able to communicate reliably and informative with one another.
And they have enabled human populations to form these large scale cooperative networks amongst non kin.
And that includes sustaining public goods. Now, we all know that public goods are a big problem for humans, aren't they?
Yes, they are. But language, for example, is a public good.
Language is held in existence by the fact that humans don't just abuse it by using any which way they invest in it,
by conforming their usage not only to the syntactic rules, but to the semantic considerations, to the conversational demands.
And in that way, they sustain a medium of communication that has funding adequate to be a shared language.
Just as much as money depends for its existence upon the continued willingness of individuals to participate in a system of exchange.
And so there is a sense in which that problem huge public could from.
If you thought about it just strategically, that problem is of the kind that we seem to be good at solving.
OK. Some evidence of this. Well, I'll just give you a couple of things.
Inference, even very young infants, shows pleasure as success of third party behaviours when they're just watching something go on,
if they see a little dot in a box trying to escape and it managed to get out, the infant is pleased with that.
It follows with interest. We all know this from telling stories to children.
These are third parties, no relation to them. They show a preference for third party helpers over.
Was they spontaneously correct unfair distribution of rewards?
We talked about that they will spontaneously resist authorities at some cost to themselves.
We talked about that adults will spontaneously initiate cooperation in first round prisoner's dilemmas,
which economists tell us is a mistake on their part.
It's a form of irrationality.
But people who go on around the world and tried prisoner's dilemmas, and they've tried ultimatum games on populations of humans,
small scale societies, hunter-gatherers, shepherds, pastoralists and so on.
And what they find is virtually nowhere do they find people who behave in the manner that the economists tell us is rational.
They all show more cooperation than that. And the kind of cooperation they show isn't arbitrarily varied,
but it's connected to the kind of ways of life they have if they depend very heavily on each other for cooperative activities.
The expectations for sharing are higher if they depend less upon each other and more upon their own resources to take care of themselves.
Is it a pastoral society? Then the expectations are lower, but the expectations are there.
And people will enforce those expectations by refusing offers. Again, economists tell us this is strictly irrational on their part.
Somehow, this irrationality on our part, though, has made it possible for us to get where we are.
Adults also show higher experience, reward and cooperative outcomes you might have thought in a prisoner's dilemma game.
The biggest thrill is when you defect in the other person cooperates and you get the highest reward of people who do measurement of reward.
Don't find that they find that people get a higher reward when they successfully cooperate.
OK, so. That's a set of priors, motivational priors, priors for the ways in which people are disposed to interact,
could artificially intelligent agents be built with those incapable of learning in much the same way infants learn?
I don't see why not? Because we're talking about exactly the kinds of things that go into artificial intelligence agents dispositions value functions,
probability functions and so on. So what is an artificial agent?
In the computer science literature is a fairly well-defined notion. It's not an ambitious notion, it's not like our idea of agency.
It includes a capacity to represent states of the world.
Assign them probabilities to represent rewards and goals to select potential actions based on their expected outcomes.
And that's what we saw in animals. They have that kind of agency and that's what we see in these artificial systems.
And that includes things like the capacity to compare expectations,
to identify and monitor the execution of actions, to compare outcomes to the expectation and critically to learn.
And so there's a tight link between being an agent in this sense and being a learner.
There are two parts of the same process agency contributes to learning and learning contributes to agency.
And this is why animals are so good at it.
Indeed, as we know, animals are capable of developing optimal foraging strategies, even in complex environments.
Now, it's important to see that they're not just foraging for food, they're also foraging for information,
they're exploring the environment and learning how to represent it.
And there's a very nice result by Gibbons suggesting that if you want to build an agent
whose concern is for something like responsiveness to evidence and getting things right.
There isn't a way within decision theory to do that by giving them the value of
truth because there's no definition as to how or when in what ways you pursue it.
But if you design them for what's called guidance value for the capacity to guide action effectively,
they will indeed do what you would hope they would do is epistemic agents.
So again, there's a tight connexion between our success or our capacities as epistemic agents in our success or our capacities as practical agents.
And this is exactly the kind of learning of the artificial systems that we've been talking about.
Now, I didn't again say anything here about the consciousness of these agents.
Consciousness might be able to help agents in lots of way. Presumably it does in some respects.
Otherwise, I don't know that we'd have it, but you can do all of this without consciousness, and you can also apply concepts like interests.
Does the animal have an interest in such and such? Does the machine have an interest in such and such because you can look at its value
function and whether or not this sort of thing would satisfy the value function?
They can also be more or less rational in this functional sense. How do they respond to evidence?
How do they update their estimations of the state of the world? How do they analyse the outcomes of their actions?
And so systems like this on an unconscious as they are can be analysed using rational decision theory and game theory.
They aren't agents in the rich sense,
but you can use decision theory and game theory to generate predictions about how such artificial agents are going to behave.
Similarly, you can use Bayesian rationality to generate such predictions.
And so even though this is a less demanding notion, these creatures don't have anything like a conception of themselves.
As agents there, when I'm calling their interests are not the kind of interests that you or I might have,
which involves things like passion and concern and affect their beliefs are not the kind of beliefs that we have because
they look like bundles of probability distributions rather than something like a credence or a trust or a confidence.
But practically speaking, they have the wherewithal to be agents in this sense.
And again, we have to not get ahead of ourselves.
We have to rigorously guard against an undue personification, avoid metaphors.
But seeing these as agents is non metaphorical. Seeing them as capable of effective play in game theoretic settings or as more or
less rational is non metaphorical in the way that I've just tried to explain.
And I think the same thing can be said of intelligent animals and their activities.
They aren't necessarily conscious or self-conscious the way we are.
They don't see these as reasons, but it turns out that's not what's necessary in order to be able to be responsive to them as reasons.
OK. Well, these are features of these artificial and animal agents that I've been describing.
And because they enable us to study their behaviour via things like the amputation of interests and reasons and so on.
This is a cue again for Mr Hobbs. This is where I came in, he says.
My theory applies to agents capable of intelligent self-regulation in light of
expected utilities and an understanding of the behaviours of other agents.
These don't have to be humans. You know, humans obviously equally well talking about how states or organisations would behave
the forms of mutually beneficial cooperation and constraint that he develops.
They apply equally amongst agents that are not particularly human agents, and they're not particularly conscious.
The question is whether humans. So he says, you guys humans, can you succeed in being reasonable rather than self-defeating fools?
And if you'd follow the economists in playing the prisoner's dilemma,
if humans had followed the economists and playing the prisoner's dilemma and playing the various other game stag hunt and so on,
they would have been self-defeating fools and we would not see them around right now.
Well, words to that effect, I'm not sure Hobbes ever said exactly this.
So let's just look at harvest laws of nature, which are principles that he thinks we as reasonable creatures should follow.
An agent ought to endeavour peace as far as he has hope of attaining it.
I've put agent in place of man here. What does that?
That's the initiation of default to feasible cooperation and using it as a credible signal to others.
An agent be willing to be contended with so much liberty against other men as he
would allow other men against himself and strive to accommodate himself to the rest.
That's indirect reciprocity.
An agent which a benefit from another of mere grace and endeavour that he which gives it have no reasonable cause to repent him of his goodwill,
that's direct reciprocity and the use of credible singling upon caution of the future time an agent ought to pardon the offence is past of them.
That repenting desire it and in revenge,
his agents ought not to look to the greatness of the evil past with the goodness of the the greatness of the good to follow.
This involves simulation, the idea of returning agents, incentivising agents to return to default cooperation,
and it involves some intrinsic concern for others that you don't just write them off.
And indeed, Hobbs puts it that way. Every agent acknowledge another for his equal by nature.
No man is. No agent is fit arbiter of his own cause.
And if an agent is to be trusted to judge between agents and other agents, they must deal equally between them.
And this is a non egocentric mapping of the evaluative landscape and so on.
So harms his principles, his laws of nature. Those correspond very nicely to the priors that we've been describing as part of the
equipment by which humans and artificial creatures can become effective epistemic.
Linguistic or indeed, moral agents. OK, so that was that's a nice story.
But. Three examples, right?
Let's have some examples here. OK, let's talk about autonomous vehicles since autonomous vehicles are rapidly populating the world.
I don't know about the streets around here, but in Ann Arbour they're fairly thick and there are agents, so they have all the equipment of agents.
What might their interests look like? Well, what made the reward function look like?
Well, it's certainly going to involve avoiding harm, collisions and so on, reaching destinations.
They get rewarded for that because people pay for their existence. If they do that, and if they don't do that, people won't pay for their existence.
Facilitated, coordinated and efficient movement on the road. And avoiding self-defeating behaviours and interactions.
That's certainly something we have an interest in.
They also have an interest in detecting and responding to deceptive behaviour or signals from human agents or from other vehicles.
And they have a strong interest in maintaining human trust and acceptance,
because if humans come not to trust these autonomous vehicles, they will evolve to ultimately ban them.
And so these interests, this constellation of interests,
is enough to generate all of our favourite game theoretic problems for these autonomous vehicles.
They've got the problem of the full cooperation getting out of a prisoner's dilemma, the idea of credible signalling.
Public goods emerging in a very busy intersection is a kind of public good problem.
Individuals could reach in and grab something and get out in a hurry, but they then will make it less good for other individuals.
On the other hand, if they manage to behave well, zipper merging, it's called, they're able to increase slow flow.
How do you get agents to do this? Well?
Suppose they have the motivational structure that I was just describing where it matters to them, not just how they do, but how others do.
It matters to them what their reputation is in the eyes of others. These machines can certainly be good at identifying others, including other humans,
and so all the issues of game theory are going to arise amongst these intelligent autonomous agents.
First of all, with respect to humans.
So one thing that was discovered early on was that if you make these systems to nice humans, just take advantage of them.
Hume, Google's artificial vehicle, Waymo, doesn't look like an ordinary car,
it's very distinct looking, and people learn that if you just jump in on an intersection.
This is in California where everybody isn't hurt. It'll just stop there and wait.
And cars pile up behind it and other cars disappear.
And so it had to become sufficiently assertive to make it the case that it can't just be intimidated.
It had to not allow other cars to bluff it into your submission.
OK. What about autonomous vehicle to autonomous vehicle interactions?
Because it all they need is priority rules.
And so if you look in the upper left hand side, you'll see that yes, there are priority rules for intersections.
But what if it's a y intersection? If it's a Y intersection, the rule is equal priority.
OK, now the cars could get to that intersection and stopped dead because neither one of them has a priority.
And if all they would do was to ask themselves Where would my advantage be in waiting and letting the other car go first?
They might think, Well, I certainly won't do that. And so they spend the rest of the day sitting at that intersection.
But if they have the kind of motivational structure that I was describing, they will indeed be able to solve these kinds of.
Dilemmas, similarly, you've all you're all familiar with this, right?
You know, everybody is slowed down and nicely gotten into their lane because of a construction project,
and some person comes zipping along at the last minute and tries to tuck in.
Well, what do the other drivers do to such a person? Do we let him just tuck in?
They close the gaps between themselves, so he can't, and he's trying to talk and they just won't let him.
Well, artificially intelligent vehicles have got to be able to do the same thing.
And the more of them there are in the road, the more they've got to be able to do this in some kind of a coordinated way.
Now the fact that they coordinate, though, is also a problem because it means they can conspire.
So that's a busy parking lot. You can see lots of cars, not very many parking spaces.
Suppose it were the case that all of the uh, I think it would probably be all of the, uh, all of the Teslas who would do this.
All of the Teslas conspire. All right. And so they send information to other Teslas and give decoy behaviours to try to
mislead motorists in such a way that people with Teslas get more parking spaces. OK.
This would be an example of a form of organisation of these machines that would reduce public trust and
acceptance of them and induce something like public regulation against this kind of collusion and communication.
What about relations to pedestrians? You all know what pedestrians are doing in this scene.
It's a crosswalk. They're in a hurry. They want to get somewhere. They know that if they maintain a continuous flow and don't look at the vehicles.
They can keep going across that intersection and no one's going to erupt them.
OK, now that again is a public goods problem because at some point they got to realise that unless they're able to share the road with these vehicles,
these are not going to be crosswalks anymore. They'll be cross lights and they'll be penalised if they cross between the lights.
So what do humans do?
Well, they get pretty good at after a certain amount of this sort of thing stopping on the kerb signalling to the car so that the car can go through,
but then starting up again and making the next car wait a little bit.
So again, this is a signalling game that are autonomous cars have got to understand.
So that they don't run down pedestrians, but also so they don't sit there forever while the stream of pedestrians flows across in front of them.
And very important probably will be reputation. So we're all familiar with reputation systems for ride services.
Drivers rate riders, riders rate drivers.
This is a screen showing you a potential rider who got rejected by the system because of his bad behaviour in the past.
This is what Hobbs had in mind when he said we should have some punishment for violations of the social contract.
But again, if the system is one that simply punishes, the cars are going to run out of riders, perhaps not have enough riders.
And so it has to be a system in which people can earn their way back in by cooperation.
And so again, we're going to have issues about justice in the treatment of riders and other autonomous vehicles.
OK. So what happens? Can you get deep learning agents put in a multi-agent setting to learn something like some of these coordinated strategies?
People at DeepMind are beginning to do that, and they've found that, for example,
through the kind of reinforcement learning that we've described so long as they attribute at least some value to the outcome to other players,
they're able to solve public good problems in a sustainable way. Lazare do and others found that they could develop a common, communicative language,
even if they didn't have one at the start in order to coordinate their behaviour and convey information to one another.
They can solve that kind of signalling game. About Chandler and others found that a successful collaborative grid soccer
play was possible amongst agents that had not been coded to play as a team,
but that managed through communication to play very effectively against carefully coded human.
Grid competitors and beyond things like autonomous vehicles and game playing,
there will be a lot more that multi-agent kinds of situations are going to be important for.
And so if we think about the development of diverse, collaborative, communicative communities,
we'll see that there will be a place for experience for artificial agents and for human
agents in learning how to do this in some way that is not mutually self-defeating.
And those would be communities we could all share in so long as we are willing to follow
these kinds of motivational priors and develop appropriate conventions and so on.
Now the the thing I want to end with here. Everybody wants to talk about superintelligence at some point.
You know, what is this threat of superintelligence and how might humans have some resources for contending with it?
And I'm kind, I'm kind of claiming that these systems could be allies in that process.
So hobs laws of reasonableness, they apply amongst agents that are roughly equal in power.
But suppose there start emerging intelligences that are threatening to dominate others by force.
Now, artificial agents then could become sufficiently more powerful that they don't fit in the standard social contract model.
Now, it's never clear to me why exactly they would want to dominate if they're so intelligent,
why would they want to be unreasonable when they could actually accomplish more by being
reasonable and taking advantage of the resources that others can bring to the table?
But if they're like that, what if they're like that? They are like that and they want to dominate?
Well, the French Enlightenment is going to come to our rescue here.
Rousseau and Condor's say first, Rousseau Rousseau emphasised that the existence of a common enemy creates also a common self on the other side.
These are diverse agents. They don't have all allied interests. They're different in many ways.
But the existence of a common enemy, especially one that's threatening their autonomy,
gives them a set of interests which they can use to work together upon and generate what
he called a general will so humans don't want to have superintelligence dominate them.
Artificial intelligence is. Other ones don't want to have artificial intelligence dominate them.
If we had a list of things on their reward function, one would be. Don't let your autonomy be taken away by some super intelligent creep.
So Rousseau helps us understand how it is that that creates a situation actually to mobilise some shared protective action.
Now you might say, Wait, this is just species ism.
This is just us, not super creatures saying we should have the world and not this superintelligence.
Why not let the Superintelligence have equal chance at the world and say now is relevant to us?
So he showed, and this is very approximate of what he actually did,
that if you're trying to decide some issue like how the society is going to proceed,
then even if each individual agent has only a somewhat better than chance possibility of capturing some chance,
some element of the truth,
the probability that a majority decision will get things right will increase as the number of participants is increased and it becomes more diverse.
And recently, people doing network studies and so on further clarified this idea and the importance of diversity in generating better decision making.
Now that's an advantage of a community of not super intelligences, human and artificial.
We will have such diversity and we were able to use such diversity in the way that conversate suggested.
When we get together and try to project that or try to decide the fate of the world, a monopolistic superintelligence will not have that advantage.
Remember what we said about correlation, Superintelligence is not perfect intelligence,
no system of knowledge can be a complete nature is essentially random.
There are representational demands for full knowledge they would exceed the size of the universe.
Superintelligence could not even fully represent itself. What about us, not Superintelligence?
Well, we can know ourselves much more fully and we can together in principle have the
capacity if we can respond to our resilient interests and our Condor's say,
like epistemic imperatives,
to bring together a wide range of evidence to bear on a wider range of possibilities than a monopolistic superintelligence could.
Now that might help us defend against the novelistic superintelligence. I hope it will.
If there is such a thing lurking out there somewhere on the event horizon.
But the other thing I want to emphasise is that this is a non species list argument.
We're not just saying that this would let us prevent it from dominating the world.
We get to dominate the world. It's a way of saying, actually, you're going to get better decision making about how to run the world,
given that we have the power to do that and we will exercise some power like that willy nilly.
The community's knowledge will be more likely to be accurate. Have greater robustness.
Be less brittle with respect to accidents and errors.
If it's based upon this kind of diverse, widely based community understanding of not superintelligence individuals, but intelligence nonetheless.
And if you want analogy here, think about the scientific community.
The scientific community could operate by having just one central system be the believer,
and all the evidence would be sent to that believer and it would form credence accordingly.
Or you could have a system in which there are all these scientists with different hypotheses who don't always
read each other's paper don't often reach those papers applying for grants to investigate different hypotheses,
investigating them throwing their resources and graduate students at them competing for funds.
You could have that kind of a system. And I guess Mike Baker,
who which do you think is going to be a more successful system at discovering the structure of
the natural world and which is going to be less brittle in the face of some kind of an accident?
And so just think of the way in which disease resistance and the diverse metal is
different from that in a monoculture or think of what happens to you if you're the
all powerful dictator of a very powerful country and people do not give you the
information that you need because you're committing the [INAUDIBLE] out of them. We know what that leads to.
So fanfare for the common agent as not so super intelligences can have a non-specialist
argument for holding it bay the development of a monopolistic superintelligence.
And we might actually have the capacity to do that because we would be able to have a better representation of the world.
This might be able to maximise expected value in some in principle sense, but that's only as far as expected value is known.
And if the. Method would be pre-empting the diversity and autonomy of these individuals.
It would create a riskier rather than a more reasonable future.
Thank you.