It's important to realise that the crucial issue is not really the issue of consciousness here,
it's what kinds of structural relations do we have with one another,
what are the interests at stake and are there paths that we could take such that
all of these different agents could do better with regard to their interests?
If they can manage some form of mutual regard or concern and that you can spell out without emotion,
concern does not have to mean caring with a warm heart.
It could mean, do I give some weight to your utility function or your value function?
Hello? I'm Katrien Devolder. This is thinking out loud conversations with leading philosophers from around the world on topics that concern us all.
In this interview I talk to Peter Railton, who is Gregory S. Kavka Distinguished University Professor and John Stephenson Perrin Professor
of Philosophy at the University of Michigan
Professor Railton gave the Uehiro lectures of 2022 here at the University of Oxford.
His topic was Ethics and AI you can watch the three lectures on the Practical Ethics Channel on YouTube.
What follows now is my interview with Professor Railton on the topic of his lectures
in which he dealt with the question How should we understand and interact with A.I.?
The three Uehiro lectures that you gave were about possible future artificial agents.
It might be helpful to just give us an idea about what sort of artificial agents you have in mind.
Yeah, some of them are not even possible. Some are actual. So I'm particularly interested in autonomous artificial agents or nearly autonomous ones.
That is ones that are doing some of their own decision making.
An agent, then, is this kind of an interactive idea that is acting the environment, receiving information,
receiving rewards from the environment and then adapting its behaviour in response?
And typically it's also assumed that relative to the goals or the rewards, these systems will do something like maximising expected value,
given their predictions of what's possible, the way that they would evaluate or rank them.
They will try to get higher in the ranking and therefore will associate with actions or
courses of action or outcomes some value function that they use to decide their behaviour.
And that's the sense in which they're decision making and not just carrying out a fixed programme.
They might be doing something very different from anything the programmers intended, and that's a sense in which they're also autonomous.
It's their value ranking that's being used. Even if it was given by someone else, it once it's up and running it,
it and the representations can change to make it a bit more concrete, whether, for example, a self-driving car count as an autonomous.
Yes, a genuinely self-driving car would be an autonomous agent in that sense, as would, for example,
a home companion for an older person or systems that are used in financial trades and trading and so on that operate independently.
Autonomous systems can include things like translation programmes or speech programmes.
So do you think that artificial agents could be conscious in principle?
I think that's possible. I don't really know what the conditions of consciousness are.
On the other hand, I don't think we're near there yet.
And part of the point of my my lectures was that that's not going to be the central issue, at least not in the medium term,
that it won't be the need for consciousness that's necessary in order for these systems to become appropriately sensitive to moral considerations.
And it also consciousness will be necessary for them to be able to have the
equivalent of social relations with each other and social relations with us.
What makes it possible for them, for example,
to have interactions is that when autonomous car and another autonomous car both have certain goals, maybe go to some destination,
there'll be some human drivers as well in the scene and they have goals and they will converge
on an intersection and they will all share the goal of getting to their own destination.
They will also prefer to do it relative their rankings more quickly rather than slower, but they will also, relative to the rankings,
want to avoid collision and so they will need to organise with themselves some kind of a pattern of movement
that makes sense for all of them because each one of them is going to try to advance its interests.
But at the same time, if they all advance their interests exactly the same time, in the same way, they're not going to go anywhere.
So it's a coordination problem amongst multiple agents with goals that are not aligned,
but that could probably be mutually satisfied to a reasonably high degree.
And so you could say that's an interaction where if the.
Can cooperate. Maybe through communication, maybe through implicit signalling,
maybe through having some developed patterns of behaviour that they can expect from others, they can achieve something they couldn't on their own.
That is to say, a level of coordination amongst their movements,
such that they can all do better realising their goals than they would if they were struggling independently against one another.
And so that's the sense in which they are.
They can be social beings with these social beings also have moral duties, such as light duty not to harm us, duty to keep their promises.
When we say moral duty, we normally associate that with a lot of things that say feelings of dutiful ness or feelings of guilt.
So if you think of duty in that way, then you say, Well, as long as they're not conscious, they won't have duties in that sense.
But if you think, well, people can contract for services and they can contract for services with autonomous agents.
And in that sense, the autonomous agent is under a contractual obligation to fulfil a certain condition or lose the contract.
And so in the sense in which an ordinary person can have an obligation to keep a contract,
it might not be anything to do with something particularly moral or even emotional in any way.
Someone said, Look, you promised to pay this money back at a certain time and therefore you're obliged to do it.
And if you don't do it, there's a penalty that you'll pay. That's part of what we mean by obligation.
And the same thing could be true for these autonomous systems.
So how would we make them comply with that, be programmed in them, or would that be something that they learn themselves?
I think the most common view as well, we would have to programme ethics into them.
My sense is that that has the same problem in general as the idea that we should programme go
expertise into the system or that we should programme language expertise into the system.
And that was tried for quite a long time. That was the main model of artificial intelligence.
You get experts together and they would write up programmes and those systems achieved a very high level of function,
but they kept stopping at a certain level that was not nearly human level.
And so the new generation of artificial intelligence machines are based not upon expert encoded learning, but on their own learning.
In a way, the other machines were not strictly intelligent in a certain sense,
because what they were doing was taking expert knowledge that was given to them
and collecting more data and processing more quickly and delivering conclusions.
But they weren't thinking in any identifiable sense the way that a human would have to think.
These new systems begin with no particular assumptions about the situation that for a game they might only know the rules.
They might only learn well which side won.
And then a system like AlphaGo, for example, will play many games against itself, simulating games and learning only.
Well, which side won? And it will, over the course of that, come to have a representation of well,
in this kind of a situation, with this kind of a move, be better or worse than this other move.
And on the basis of that, play a game and find out whether it wins or loses.
Keep improving. We create a competitor for it that has its degree of competence and it keeps competing with that.
And eventually they learn to play go better than a human.
Yeah. So we did not programme that knowledge into them at all.
And what's striking is that systems like that building from essentially no expert knowledge,
but using very generic learning processes and lots of experience and the capacity to simulate and evaluate outcomes.
They can acquire these competencies. And the same thing is now true with language systems.
You may have noticed that language translation programmes are much better than they ever were.
They're based on similar kinds of learning. And if you think about that's kind of how we do it.
People don't sit down and tell children the rules of grammar.
Children, even at a very early age, learn how to recognise frequencies amongst sounds in language.
Associate those frequencies with patterns. Learn eventually that those patterns are examples of more abstract patterns,
and use those abstract patterns eventually to start generating new patterns.
As now artificial systems can do generative systems. So if that's the way the kind of complex situational social.
That embodied in language is accomplished, then that's a model for perhaps how it is accomplished in the moral case.
And so would we have moral duties towards them?
Like you said, they might not all be conscious. Maybe they don't have feelings.
So what would grown like our responsibility towards them? Here's a way to think about it.
You know, I talk about these as as agents and as agents with whom we can have contractual relations, for example.
And you might say, do we really have a notion of the interest or the good or the benefit of an artificial agent?
And what I have been doing is talking about goal attainment. And certainly artificial agents do have goals.
But someone could say, Yeah, but does it matter whether their goals are attained or not?
If they are, if they don't have any kinds of experiences? And maybe you can show that that might have effects on us, but we have experiences.
And so one way to think about that question, I think, is to realise how much of what we think matters,
matters to us, not because of how it feels, but because how it relates to our goals.
If we look around us,
we see a great deal of our concern is actually about things like goal attainment rather than the producing of certain conscious states.
And many of our conscious states matter to us because of goal attainment.
Hmm. So that's the order of explanation. So you could harm an artificial agent in that way, then, because you stop it from achieving there, you know?
So, you know, artificial agents are doing financial transactions now.
Some of them could be doing financial transactions, transactions for themselves on the side and accumulating money.
They could then lend us money. You need money.
You go to this artificial system that's got a pool of money and you promise to pay it back.
But you don't. Well, that's okay. You didn't harm anyone.
I'm not sure you didn't harm someone. And it's not just because you were harming other humans.
When you do that, it's because you accepted the money from an agent who knew that they were giving it to you on the expectation that you would repay.
They were dealing honestly with you or reliably with you, and you're not reciprocating.
And that seems unfair to me and the fact that I can say, Oh, but I've got these private inner experiences and it doesn't say,
Well, what's that got to do with the fairness of this financial transaction?
So yeah, I think we have to get used to the idea that, yeah,
there may be agents around that are just as intelligent and capable and involved and useful and committed or reliable as humans,
and that we need to think about how they matter.
You could think, well, systems like that are going to be out in the world.
Here's this vehicle, and I'm trying to interact with that vehicle to get through an intersection or to merge.
I could deceive it, right?
I could you know, I can make a little gesture like this that it might interpret as me moving in and then I would squeeze in.
And if I did that, I might be able to get away with it. And if humans in general did that, what would the artificial systems learn?
Well, they would learn how to pre-emptively try to block that. Hmm.
And so we would be back where we were blocking each other intersections.
So do I have an obligation, to be honest, in my interactions with autonomous vehicles?
It seems to me I do. And it seems to me that it's an obligation, very much like my obligation to be honest to people.
I need to be a reliable signaller in order to coordinate with them, because we each have goals for each trying to realise the goals.
And I could either create a system with a sufficient level of trust or confidence,
such that we could do this smoothly together and each get more progress to our goals than we could on our own.
Or I could do it in such a way that made it more difficult for them and more difficult for other humans.
And so I would say, yeah, there's a I have a responsibility not to exploit.
Mm hmm. And you say, well, it's not like a moral responsibility because, you know, should I feel guilty about or something?
And I said, Well, I don't know if you should feel guilty or not. That's a further question.
But if you ask me, given the kinds of reasons that we justify moral obligations with, like, do they promote mutual understanding?
Do they make it possible for people with conflicting goals to achieve their goals?
Do they make it possible for people to lead better lives? Those are the kinds of reasons we give.
And those reasons can be given in this case.
And of course, what it is for a machine to have its goals realised is not perhaps the same as a human because they don't have the same.
Feelings. Maybe eventually they will.
But if you ask, could it be important for us to take into account the goals of machines, the way we take into the goals of animals or institutions?
We take into account the goals of institutions. They don't have consciousness, I would say, yeah, sure, we can have that.
And it's, it's important to realise that the crucial issue is not really the issue of consciousness here,
it's what kinds of structural relations do we have with one another,
what are the interests at stake and are there paths that we could take such that all of these different agents could do better with
regard to their interests if they can manage some form of mutual regard or concern and that you can spell out without emotion.
Concern does not have to mean caring with a warm heart.
It could mean, do I give some weight to your utility function or your value function?
And it turns out if you take machines that say there are machines operating in the natural environment,
suppose we try to automate fishing so we create these automatic fishing boats.
If you programme them with a utility function which only assigns maximal value to their own catch, they will do what humans do.
They will over exploit the resource. If instead, like humans, they have at least some interest in the well-being of the others.
So they assign some value to the utility function of others.
They get some kind of benefit from the experience of coordination and human infants do that as well.
If their value structure is like that, then they can learn to be sustainable fishermen.
And so autonomous systems, artificial systems can learn to solve public goods problems.
So it actually could be a step towards a better world if could.
Yeah and you know people say because they don't have emotions, you might say, well, there might be some pluses to that as well.
They might not be vengeful in the same way or they might not be as short sighted.
They might be more prepared to work together because they might realise better than us that like if we work together,
we're actually all going to achieve our goals in a, in a model in which these systems work by simulating out streams of consequences,
they can see, oh, if, if we all do it this way, here's what's going to become of us.
And that's not what I want either.
I'll be out of work just as much as they will, and so they could have a community that sustainably produces a good in a way that is the result.
It's the product of their capacity as agents. So if these are autonomous, artificial agents sort of become quite powerful, which is sort of possible,
they might actually become, you know, so smart that they would actually effectively run the world.
And that might be good because they might be willing to cooperate more, maybe.
But of course, if there's a powerful, you only need like one baddie, you know, they'll just destroy the world.
And that's something we should be fearful of or definitely be worried about that.
And it can happen completely accidentally.
So this is a long way from that, but it's a kind of example.
The most recent programmes for generating credible natural speech programmes like MBT three, they can produce pretty credible speech,
not maybe a half an hour talking philosophy, but they can produce pretty credible speech and dialogue with people.
And if you give them a prompt, you'll come up with a relevant response.
And they did that by harvesting structural information about language from all these texts that they were given.
Now, amongst the texts were computer programming texts and they in effect learnt principles for or in the grammar of some types of programming.
Now that turned out not to have any serious consequences,
but you could imagine a system like that, not by any design learning how to write bits of code.
If those bits of code operate within them in a certain way,
that might be an executive role and that might change their dispositions and then they
might behave in a way that was completely done outside of the design specifications.
So they would be rewriting their own code. So yeah, by by all means, things like this can happen and they could happen before we know it.
So there's a lot to worry about there. And,
and my thought is all the more reason that we should be developing a large
community of mutually trusting artificial and natural agents so that they can be.
Aware of something like this starting to emerge because the other intelligences, artificial intelligence, they don't want this either.
One dominant artificial intelligence would be a conflict of interest for them because they would not want to be dominated.
They couldn't achieve their goals. So they could be allies in this process of being attentive and alert and responsive,
maybe in ways that we wouldn't anticipate as humans because we're not machines, they might be better at anticipating how this machine would operate.
You know, what's our safety against dictators?
It's not some piece of computer programming or a government regulation because the government was what enforces regulations.
It's, you know, a population that can be mobilised and enlightened and attentive to the emergence of these things and form itself
into a unit that can be good at spotting such kinds of powerful individuals emerging and trying to control that.
So, yes, I worry about this, and that's part of the programme of concern that I have, is that we should be thinking about.
Yes, that's right. How would we build a robust community of artificial and natural agents capable of this kind of resistance?
Because I can't see any. You can't programme a guarantee on this.
If you like this episode, don't forget to subscribe to the Thinking Out Loud podcast or to the Practical Ethics Channel on YouTube.