Good evening, everybody.
Welcome to what's sadly is the last of a series of three you here that she had given by Railton on ethics and artificial intelligence.
Most of you have been to the other two lectures,
so you'll know that Peter will speak for about an hour and then we'll have some time for questions if you're lucky.
We should have some time for questions and we will be able to take questions online as well as from the floor.
And here we'll be using the the roving mikes again.
Um, and, um, online people, I think you should put your questions into the chat.
And then Johnny P will kindly report them to us.
So without further ado, over to Peter. Well, thank you, and thank you to the Uehiro Centre for the invitation, the chance to be here with you.
I've enjoyed it so far and I hope you will continue to enjoy it and I'm trying to think of the title for this lecture,
and it occurred to me that only a recursive title would do,
which is a living amongst artificial agents who live amongst artificial agents,
who live amongst natural agents, who live amongst artificial agents and so on.
Because it's really about that recursion at the very beginning,
I'm just going to go over a little bit of the first lectures in order to help those who might not have been there or memory also may not always serve.
I tried to mention at the beginning there are ton of ethical challenges about A.I. and many of them very large,
and I'm not trying to address them all. I'm really looking at just this last, the sixth one, the problem of not so super machine intelligence,
trusting A.I., entrusting A.I. with some inappropriate tasks or degrees of independence.
Not just because humans are mercenary, though indeed they are, but because they may not know any better.
There may be error in inadvertence. And so I'm my my question is how could we do this relationship living amongst
artificial agents who live amongst artificial agents who live amongst natural agents?
How could we do that in a way that was constructive, that was mutually beneficial rather than mutually detrimental or very risky?
And so my focus has been on the idea that.
These systems, these autonomous or semi-autonomous artificial intelligence systems might be part of the problem,
might be part of the solution rather than part of the problem.
Not just because we could use them as tools, which we certainly will do, but because such systems would be autonomous.
They would be agents, they would be capable, I believe,
of a kind of app sensitivity and responsiveness to morally relevant features of situations, actions, agents and outcomes.
And in that we are going to find, I hope, a very important source of mutual confidence and trust.
Such responsiveness, I've claimed, is not just important for their relations to us, but for their relations amongst themselves.
And if this was right, if anything like this picture is right, then it should help somewhat with the other pictures as well the other problems,
because those involve, amongst other things, problems of developing trustworthy A.I. at the heart of these other tasks.
Now where is this sort of specious week for this talk?
Because folks down deep mine announced Gatto A.I. as the first working example, still incomplete of a general artificial intelligence.
And what's that? Why is that special? Well, it's because it's intelligent.
It's capable of learning and problem-solving in a wide range of novel situations.
That's the general idea of intelligence, but it's general in the sense that it's competence in a wide range of task language,
dialogue, labelling images, motor control, game playing, testing, causal models and so on, and doing so autonomously.
That is without being reprogrammed in between the tasks. And it does this by uniting in a single big model all the elements needed for these
various tasks and taking advantage of this synergistic power of that to do them.
And so this is what people have been hoping for for a while, and the people said, Well, why would they announce this so early?
It's, you know, still fairly mediocre at some of these tasks. And one answer is, if anything like this works even as a sort of proof of concept,
then the fact that it's badly trained or needs more information or so.
And that's not the the key fact.
Now my claim has been that in humans are a linguistic epistemic causal and social moral competencies are in some ways like that.
That is, they're all very entwined. They're entwined in a complex model of the world in a bundle, not separate strands.
And they develop through infancy and through the rest of our lives in conjunction with one another, in pace with one another.
And I don't think they would be fully realised without one another.
And so that's another reason for being especially interested in artificial general intelligence,
because the full realisation of these special tasks might very well require more general intelligence.
And I'm going to argue that so Sutton and others have argued that in the end,
it will be generalist artificial intelligence that succeeds best, even at the well-defined tasks that A.I. is now good at.
And in this respect, Gaucho A.I. is still, I gather a work in process there.
There are specialised programmes that apparently do at least as well as it does or better.
But for our purpose, the question that's especially interesting is how General Intelligence,
artificial general intelligence might make a difference to questions of ethics and A.I. questions of air safety.
And you might think, for example, that one very important feature of artificial general intelligence is that these systems
might be more interpretable because they combine language skills with motor skills,
with object identification skills, with dialogue skills and so on.
They might be able to represent what they're doing in a way that is, for example, explainable to us in a better fashion than the machines now are.
And so the idea is if you bring together model based learning with competencies like dialogue,
categorising objects, using that to navigate or manipulate the environment,
collaborating with humans on shared tasks and so on that you begin to make it more and more realistic to think this is an agent now.
It's not a human agent who doesn't have all the features of a full fledged human nation.
But the more it does this, the more it seems a general, and the more it seems that the words that it generates are connected with the world,
that they're used in modelling the environment because they're associated with a variety of tasks.
They're used for signalling others for communicative purposes. They're used in action guidance and in learning.
And, you know, meaning isn't just use. I suppose most of us think that,
but each of these are steps toward the idea that that may be more like meaning than it would be if it were just a word programme.
Or it may be more like objects than if it were just an image identification programme because objects are three dimensional.
And now it's treating these as three dimensional through its motor activities.
And so that's an important sense in which the general artificial intelligence gives us more
confidence that we are dealing with something that looks like an intelligence and an agent.
And I need that very much for the argument that.
I'm giving now, of course, if you look at the core, this big network at the core, it's just a big association structure.
And that will tempt many to say, look, there's still no real understanding.
There's just all this association and that can't be human level competency because humans have can't have understanding.
And that's an important reminder. We're only so far down this path.
But it's worth noting at least my brain seems to be a neural network of associational connexions.
Now it's not made out of silicon, it's made out of protoplasm. It's got some more differentiated structures than these nets.
It's got a lot more structure than these nets, and that may be very important,
but it's still operating on this basic associational principle that Connexions are strengthened the more frequently they're activated.
That's the basic principle neurones that fire together wire together.
And so the question is, do we use that complicated net of ours association with what it is?
Is that understanding? And we might disqualify it as well.
But then at least we wouldn't be treating ourselves as in some ways above and beyond
the possibilities of artificial agents because we ourselves are not there yet.
Anyhow, I'm going to bracket such large questions about the nature of understanding because my challenge is really
to emphasise that short of understanding and short of something like full agency or full moral agency,
it's still quite possible for these systems to become aptly sensitive and responsive to morally relevant features.
And one interesting thing about these kinds of systems is that if you use them, let's say, to model human moral intuitions, people do that.
They set up an internet site. Lots of people send in their intuitions.
They try to make a model of that using some kind of fit of a of a neural network.
We're learning something about the structure of our own moral values, our known moral beliefs in that way.
But if that were, that network could then be associated with action and with speech and with behaviour,
then we'd have a better sense that we were getting at something that looked more like a competency.
So consider language now.
If an artificial system is going to be genuinely able to achieve human level competence in language, not just fluent, topically appropriate speech,
but being attuned to things like conversational norms, interpretive charity sensitivity to the distortions that coercion,
deception or power imbalance might bring to the content of the conversation, identifying speaker motives or intent, and of course,
and compensating accordingly, identifying deception, attributing appropriate authority to others, use of words and so on.
That's language competency. It is a very complex bundle.
It's normative. It's epistemic, it's social, and it's got a lot of moral content as well.
And so my sense is that really to build competent, humanly competent speakers, we will have to build a system with that kind of a bundle.
And so a special purpose language programme is always going to look tinker toy by comparison and limited in its abilities.
So if creating artificial agents with broad human level competencies is,
the game is the aim and that indeed is the aim, then moral competencies, I'm saying, will be a part of that.
Intelligence is a capacity to learn and solve problems in an open-ended array of situations,
and moral issues arise as problems that humans face in an open-ended array of situations.
Using our moral capacities, we often solve these problems.
I'm going to talk about some of the ways in which we do,
but that means achieving human level competency and solving social problems is going to involve this
same capacity to represent morally relevant features of situations and to use them appropriately.
And as we saw early on, this is reflected, for example,
in the substructure of the mind in the way in which these capacities are reflected in the brain's activities.
So here is the general semantic network compared to the default network we saw this year as the default network compared to memory tasks,
autobiographical memory envisioning the future, simulating theory of mind tasks and moral decision making.
It looks like a bundle,
and indeed it looks like a core in which there is something like a generalised model that is allowing us to interpret the past that
we have to imagine possible futures to understand what's going on in other people's minds and to carry out moral decision making.
And so it looks something like the structure of general intelligence.
And if that's if it makes sense that these tasks are bundled, then it makes sense that the brain will handle them.
In this bundled way, there will be special moral modules. And so on.
Now that looks like one of these foundation models in vivo and this capacity to
use experience and flexibly recruiting memory and generating possible responses,
simulating outcomes, assessing them in terms of how they would affect people and so on.
If that's how we manage to develop and use such capacities as our language capacity, our moral capacity, our capacity is epistemic agents,
then artificial agents are going to have to do so as well if they're going to have human level competence.
And along the way, we will have to be responsive to epistemic and linguistically and socially and morally relevant features of situations,
actions, agents and outcomes. So what do I mean saying agent talking about these artificially intelligent systems?
I don't mean a deep notion of agent. I don't mean a core consciousness or a sense of self.
I mean, a system of the particular kind, which is well studied in cognitive science of an agent interacting with an environment.
The agent has a model of the environment. It has a goal, a reward function.
Those two are combined to generate selected actions. Those actions are then suggested to the performer, the actual waiter, the agent,
the agent performs the action and the environment returns a response in terms of, well, how did things turn out?
And that then becomes data for updating the model. And it's a much more complicated than this.
Actually, there are all kinds of internal loops that are fascinating in themselves,
but the rough idea is that agents are by their very structure in this sense represented as they are modellers,
they are learners, and they engage in action as learning and not just as performance.
And so what we saw when we looked at, for example,
evidence from neural recordings of macaques that they perform actually quite precise probabilistic calculations of rewards in their environment.
Moreover, if you look to the right hand side and I will draw attention to this later on,
they don't just do expected value calculations, which is what the Economist would recommend to them.
They also do risk calculations. They independently encode risk and you might say, Well, why would you bother doing that?
Don't rational agents just act on expected utility? And you can say yes, that's what they said, right up until the global financial crisis,
when it was clear that accumulated risk offset the gains in expected utility.
So animals live in an environment they have to survive.
They can't ignore risk and artificial agents if they're going to be least animal level competent and
certain the human level competent will have to represent not only expected value as they currently do,
but also risk in this way. Here we saw work with a rhesus monkeys, indicating that they have, in fact, utility functions.
These are abstract functions. This doesn't represent any particular quantity of juice or any particular level of risk.
It represents combinations of quantities of juice and rebels, levels of risk and banana slices and grapes.
But it reps them represents them all in a common currency like utility, and the utility function has the shape you would expect.
It is risk seeking when there is little at stake and it is risk aversive when there's a lot at stake.
And this looks very much like the kind of a function that you would expect a prudent agent to develop operating in the world.
And the picture that emerges from this is the idea that our action as we saw, is guided by.
Evaluative causal representation of the environment around us.
And with that representation, we are able to select actions and then compare outcomes with what we expected.
Now, highly intelligent animals don't simply have that kind of a first person egocentric perspective.
In fact, they map their physical space non egocentric as well as egocentric.
This was a big discovery. Not that long ago.
Rats represent space not only in terms of location and where they are, but also in terms of a grid mapping out the space around them.
It's a non egocentric representation, and these representations affords the animal a very substantial degree of autonomy from the current stimulation.
It enables the animal to engage in the discovery and planning of novel actions, and it enables them to become optimal foragers in their environment.
So here are some of the initial experiments with rats showing the neurones firing, either at a specific place or on a grid like pattern.
Here are an example of the representation of space in the hippocampus of a rat who's been running a maze.
And what we saw was it's not just present when the rat is running,
it's present when the animal sleeps and it repeatedly activates that maze in its sleep.
It does so in a way that seeks information. That is to say it spends more time activating parts of the maze it didn't explore.
It moves in directions in the maze that it didn't move into during the day,
and it also constructed novel paths because it has a representation of space that enables it to represent not just the channels that it followed,
but the spatial relations amongst the locations along the channels.
For example, diagonals and shortcuts. And then finally, we saw and this is a slide I.
Try to show as often as I can, because it gives the animals credit for what they're doing,
that when the rat reaches a choice point in the maze, it has an idea of what lies ahead.
It searches to one side and to the other side mentally before it takes any steps.
That's an efficient thing to do if you're an animal worried about energy.
And it combines representations of expected value with representations of space in such a way that by going back and forth,
it is actually weighing the alternatives and where it discovers the stronger weight.
It will act and move in that direction. And so a rat running a maze is acting as a rational agent could be expected to do,
forming the kinds of representations we'd expect and using them in the ways that we would expect.
Now, highly intelligent animals, including humans, also construct non egocentric maps of their social environment.
They represent different behavioural dispositions of the individuals around them,
values that are associated with the behaviour of those dispositions, and these representations are learnt through a kind of reinforcement learning.
And these are third personal representations. It's not was this spunky good to me?
Is this monkey good toward other monkeys with this monkey, therefore be a good marriage partner or a good alliance partner?
That's why these representations, these non egocentric representations are so valuable because that information is vital if they're going to
initiate a new relationship and they need to distinguish the monkeys who help from the monkeys who don't help.
And they're now finding that animals have codes within themselves,
doing things like helping helpers and helping those who help others than themselves.
So there's a lot more going on there than we thought, and this is a representation,
a non egocentric representation that is a representation of relevant characteristics,
we would say morally relevant characteristics of the social environment.
This is also a kind of autonomy that they have because they are not bound to what
they've done in the past or the options that they've exercised in the past.
And so within the social environment, just as within the physical environment,
they are able to navigate space in ways that help them realise goals and fashions that they had not explored before.
And this modelling, which we should expect in a highly artificially intelligent agents,
would then have the same kind of need for autonomy and efficacy in navigating the physical and social environment.
So we should expect artificial agents who have even animal levels of competence to be doing
this kind of non egocentric mapping of evaluative features of their social landscape.
And we should expect, moreover, that it needs that just as the rat needs it or the monkey needs it in order to
have the most effective and efficient pursuit of goals in that social space.
And so what I've been trying to argue then, is that features like linguistic features, epistemic features,
social features, features having to do with helping and harming morally relevant features.
These are all they're not being conceived by the animals as reasons for action.
They don't have a concept of reasons, perhaps, but they're being used as reasons for action,
and they're playing the role as reasons for action that reasons for action should play.
So. Back to our friends, the autonomous vehicles, because they're my example here of autonomous, artificially intelligent systems.
So what kinds of sensitivity or responsiveness to reason making features would be involved in achieving human level competence and driving?
And importantly, how much sense of such sensitivity and responsiveness would contribute to the safer and more trustworthy character of these vehicles?
In other words, if they were able to be responsive to these features, would they in fact be safer?
Would they be more trustworthy and would they be better at realising the kinds of goals that they're trying to realise, like getting to destinations?
Seems like a lot to try to get morality in this sense, not a morality in the sense that we think of a highly normative AIS system,
but a system of responsiveness to morally relevant features. It seems like why would you need that in order to drive?
And the answer is humans need it in order to drive,
and they need it in order to drive in such a way that the capacity to have human level competence would presuppose that ability as well.
So we last time looked at merging. So here are some typical merging problems.
And here's an example. Take a look at the lower right hand corner.
You're trying to merge into a highway, and you notice that the following car the second blue car, seems to be allowing a gap in front of it.
Now, does that mean that it's signalling to you to merge in? Has it slowed down in order to signal that you're merging in?
Or is it just speeding up and it has a gap to make up between it and the car ahead of it?
What is the intention of that car? And where should you go?
Should you try to move on the trajectory that takes you close to that car or more distant from that car,
which is going to be less likely to interfere with the planning of that vehicle?
Given what the vehicle is signalling that it's doing and so recently,
there have been the development of techniques for understanding how merging can take place that reflect this kind of complex, intentional structure.
Now for humans, successful burgeoning merging involves competing interests.
Each of us might want to get to our destination faster, but we also have some shared interests in smooth traffic flow, avoiding collisions and so on.
How do we reconcile these in any given situation, given that we don't know the other driver?
We may not interact with the other driver again. What do we have to do?
Well, we have to use whatever we can to try to determine the intentions of other drivers.
We have to assess the evidence that's available to us. We have to look for whether there's any communicative action going on with the other driver.
We have to think that we're perhaps causing a slowdown behind us if we delay or we have to be mindful of the traffic that's moving.
And so we have to know what are the expectations of all these individuals around us?
And how can we put together those expectations in such a way as to enable us to do a smooth, safe merge?
And so it involves heavy use of theory of mind.
And that suggests that if you're going to build a car capable of human level competency and merging, it's going to need something like theory of mind.
And indeed, we find something like that. So here is another merging situation that you all understand.
There's a construction zone. You're supposed to be nice and neatly in a single lane as you get to it.
The autonomous vehicle sees an opportunity. Should it take this opportunity, it could scoot ahead in an open road.
After all, it's been in traffic should it scoot ahead in open road. And if so, how should it signal to the other car that it's all ready to emerge?
And how should it respond to the other car's response to its attempt to muscle in like that?
And why is it not working and why next time should it try a different strategy?
All of that is the kind of stuff that these machines have to figure out.
And this is a situation they could get into.
This is not a situation of unheard of a catastrophe, this is cross-town traffic in New York on an ordinary day.
Think of how much intentional information the drivers of those cars need and the pedestrians
need in order to get out of this situation while continuing to move across the intersection,
which they do all day long, day after day. That's an ability that could not possibly be accomplished by a driver module.
It's going to have to be accomplished by a module for understanding human social
interactions and ways of getting people pissed off at you in ways of getting people
to want to cooperate with you and trying to elicit elicit their help in trying to
signal to the pedestrian that you're really not trying to run that person over. So.
Autonomous vehicle merging, then is going to have all the problems of conflicting goals, problems of communications signalling,
trying to be reliable in the communication or trying to detect deceptive communication, this is New York.
After all, they must solve these problems of predicting behaviour, gauging evidence.
What evidence could I give to others that would enable me to do this task more successfully and smoothly to have safe and
human competence in merging in an actual situation that might encounter any day at any intersection in Midtown Manhattan?
So no simple dynamical model will suffice. Getting the solution will require attributing goals and expectations to other drivers,
autonomous or human, looking for informative signals, estimating their behaviour and so on.
So this is a kind of non egocentric mapping of the social environment.
And. One approach now that's being explored.
I'm happy to say the British Isles is to develop deliberately.
Artificial intelligence is for driving that use rational agent models of other vehicles in situations.
And what that means is that it tries to impute to the behaviour of other vehicles what would be the intention behind this particular behaviour.
And that means they're doing what's called inverse reinforcement learning from the behaviour.
They're trying to infer what the value function of that vehicle is effectively and then trying to use something like Bayesian estimation to think,
Well, is it more likely if it had that value function, that it would slow down or speed up here?
And how is that going to affect the way in which I behave and vehicles like this
can in simulations do a better job of merging than they could with just simple,
dynamic calculations because they can impute structure of a general kind to a situation?
And these are agents, the artificial ones and the human ones are agents in the sense that we saw.
Now you might say, wait a second, we solved chess, didn't we? They solved.
They didn't solve it. They got better than any human in chess, but they didn't have to do psychological theories do that.
They got better than any human and go. They didn't do any psychology in order to do that.
And that's because these are games with bounded boards and only discrete number of moves that are possible
that any moment there isn't a complex question of which players do you coordinate with at any time?
And so a game like chess or go complicated as it is,
is the kind of game that as machine intelligence can solve without attributing agency to the other player.
But the kinds of problems we've been looking at don't seem like they're going to be tractable in that way, and they're going to be open ended.
And the intelligence that's involved is going to have to have basically the
features that we've been attributing to animal and human social intelligence.
Think of relations with pedestrians. How do you understand whether pedestrians are saying to you, OK,
I'm not going to slow down for you to pull out this time or they're saying, yes, I will slow down and you can start to pull out.
How do you understand that from the motions that they're making, you have to have a rational agent model of the pedestrians as well.
You also have to know what you don't know. Human driver I'm new to a country and I'm at a crosswalk and there are a lot of pedestrians going cross.
And I think, my god, I'll never be able to pull out.
If I've got a person from the country beside me, I can turn to that person and say, Where are you in this, this and this situation?
Would you nudge forward a little bit? Do people, if you do nudge forward, do they open up a little bit so that you can go through?
How long should I wait before I do that? And that person is likely to have better information than you.
And so an artificial driver that can be as competent as a human driver has to know what it
doesn't know and know also what it could ask and what it could learn in the situation.
And so it will have to be consultative and dialogic and not simply sit there in its own private mind and try to scrutinise the world.
It can gain from communication and gain from whatever knowledge the humans that are in its environment have acquired.
And so that also requires that these systems have the ability to self represent because I can't have a dialogue
without a machine with a machine about this unless it can tell me why it wants to do the thing that it's doing.
Your wife's trying to do,
the thing that it's doing and that requires now that the machines also have self representational capacities that they can represent their own,
that they can represent the different weights, that the different nodes and variables have.
And they can put into words what weights they're using and ask, are these weights appropriate in this situation?
Again, a competent human driver can do that and we'll be able to get through the intersection thanks to local knowledge.
That's a human competency, and driving and human level competency and driving involves just this.
These are the features that are also morally relevant features of situations they
have to do with harms and benefits in ways in which individuals can be put at risk,
or ways in which individuals can be helped or assisted or cooperated with.
And so we can see why autonomous vehicles are going to have to be responsive to these morally relevant features.
Now you might say, OK, OK, yeah, they were responsive to morally relevant features, but not in a moral way.
It's just rational self-interest on their part. All they're trying to do is maximise some reward function.
They are trying to do anything like but a moral agent does, and to a certain extent that's true.
They are doing what a moral agent does in the sense they don't have moral concepts.
They are doing moral deliberation in that way. They don't have moral feelings.
They won't feel guilt or shame. But.
Are they rational, self-interested agents? Or not, and I'm going to argue that they aren't in that for them to be successful at this driving task,
they will not be rational, self-interested agents. They will be what I call and what Hobbs and Hume would call reasonable agents and reasonable agents
respond to morally relevant features in the way that we hope moral agents will respond to them,
not just in the way in which prudent or self-interested people will now to engage in this little discussion.
I'm going to have to talk about these systems as being more or less rational, them having interests, having benefits, having cost and so on.
And I realise that can be problematic to people because they can think they are conscious.
How could they have a benefit or a cost if they're not conscious?
And I have your whole spiel I could give to you about why I think those terms are appropriate.
Let's say that it what I have in mind here is not what we would, might, might think of as a conscious benefit,
but it's a kind of benefit that humans have and that is important for human life.
And it's a kind of benefit that human institutions can have, even though human institutions are not conscious.
And so what I mean here by interest has to do with what goals exist in the situation for the agent,
how those goals might balance, how they're related to the odds or the probabilities or the information in the situation.
Something will be in its interest if it can improve its situation with regard to those goals,
something will be a benefit to it if its situation is increased by that, it's the cost of its depleted.
And this idea of cost and benefit is when we use all the time.
It's not something I've invented the head of a college, you know, why aren't you divesting from oil stocks?
Don't you know that the oil companies are responsible for a huge amount of pollution? And the head of the college will say, I understand that.
I personally wish we could be divested of all carbon intensive stocks.
In fact, everyone on the board wishes that intensively. But I'm head of the board and our job is to ask What is the interest of the college,
not what is our interest as individuals and the interests of the college would
not be served by divestment because it would harm our income and because it
would put our future donors and indeed future donors or others could take that
head of college to court if he fails to act in the interest of the college.
And so you'll have people in black robes sitting solemnly in a panelled chamber asking themselves,
Was this or was this not the interest of the college? And that's not an interest of any of the agents in the college.
Maybe the entire board is about to retire, maybe their pensions are secure.
Maybe they're not going to benefit at all from this. Maybe they'd prefer to have a green reputation.
They hate this. They don't like the students coming to their office and ask these questions.
But they say we still our responsibility is to tend to the interests of the college,
and those are not the same as our interests as individual moral agents.
So it's not a bizarre notion. It's the notion used in game theory.
And I'm happy to discuss this more in the question period. But I dare not spend more time on it right now.
OK, so here we are back with artificial agents,
with their interests more or less rational relative to those interests in the actions that they select.
These are terms of art, but they're anchored in the agency and the general structure of these systems.
We can use them to predict and control their behaviour. We can use them in the game theoretic way to predict how they will interact
with each other and with the outcomes of those interactions will be we can,
for example, say, Oh, well, if they're rational, self-interested agents, then if there is a nash equilibrium, they will find it.
And how would we describe that? Well,
it's a stable state of a system involved in the interaction of different agents in which no agent can benefit by a unilateral change in strategy.
If the strategies of the others remain unchanged, that's the idea of a nash equilibrium.
And the answer is, if these machines are rational and self-interested and they interact and they can learn,
they will find Nash equilibrium and take them OK. And that brings us to this man, Hobbes.
So Hobbes is famously analysed in terms of the prisoner's dilemma.
So here is the prisoner's dilemma. One way of understanding the dilemma is if you're a rational, self-interested agent,
you will look at this payoff table whether you're a prisoner one or prisoner two,
and you will reason what would be the stable equilibrium here,
such that whatever the other person does, I could not play a more advantageous strategy.
And you've all heard this numerous times already. It would be the strategy of joint defection.
And so we would both end up with one unit of value, whereas if we had just cooperated, we would get three units.
And that's interesting, because not only would we do better individually, but together we would do better.
We would have produced six units of value rather than just two.
And we could divide those six units up and we could do this again if we haven't iterated game and we could continue to produce more value.
If, however, we are rational, self-interested agents, we won't on the first round cooperate.
And that means that on the second round, if the other agent understands us,
what understands us for what we're doing and we don't know whether we're going
to be any more rounds will defect as well and we will not get up into that box.
Now, if mean, if autonomous artificial agents can't get into that box, they're in trouble.
And so in order to do so, they must differ from rational, self-interested agents in a distinctive way,
and Hobbs told us very carefully what that distinctive way is.
So he was looking at the world around him at the strife and this religious wars in England.
And it was very clear to him that there were two contesting movements that could get locked into non-cooperation.
They could decimate the countryside and keep the country weak,
whereas if they could somehow rather come to some kind of an agreement, they could have peace and prosperity.
And therefore, his laws of nature don't say in the first instance, you should use all the help of war.
They say in the first instance, seek peace. And only if peace is not obtainable should you use the instruments of war,
and he thinks that peace is attainable in that situation and indeed recommends it.
Now that would correspond to cooperating on the first round of the prisoner's dilemma, rejecting the rational, self-interested strategy.
How does the argument work? Well, Hobbs argues that a reasonable person can see that an unsecured first performance of
cooperation would be a costly and therefore credible signal of willingness to cooperate.
And so he tells us you could initiate cooperation with no security of performance from the other individual.
It is seeking peace, it's giving peace a chance, as the slogan goes.
And in light of the first law of Nature, A should do it now.
A assuming that B is a rational or at least semi rational agent should understand the laws of nature just as well as a does and should recognise that,
according to the Fourth Law of Nature, if you receive a gift out of free grace from someone, you should endeavour not to make that person repented.
That is to say, you should show great gratitude for a gift of free grace, and so be knowing this will want to cooperate on the next round.
A knowing that B would know that would know that B is going to cooperate on the second round.
And so as a reasonable agent with that expectation, we'll also cooperate.
And once they do this, they will be in a stable situation now in which they can continue to cooperate as long as they interact with one another.
And you'd say, Yeah, but look, suppose there's an end time, there's a horizon out there.
A rational, self-interested Asian would reason like the following.
I know that on the next to last round, my opponent being rational, it's self-interested, is going to not cooperate, it's going to defect.
And therefore, on the next to last round, I should defect.
Oh, but the other agent can make that argument just as well as I can,
and so therefore on the ground before the next, the last round, [INAUDIBLE] defect and I should defect.
And I can make that argument just as well as she can and therefore on the ground before the round, before the next to last round, I should defect.
And they talk themselves the Caterpillar Strategy School.
They talked themselves entirely into spending the next year non cooperating because of a worry about the last round.
Now, Hobbs would say that is unreasonable.
Reasonable people don't act like that, but if they were rationally self-interested in the sense that we've defining it,
if that's all that could move them, then that is the way they would argue.
So in a true prisoner's dilemma, of course, agents have to act at the same time.
They can't know what the other's going to do, and reasonableness plays a role there because if a, for example, initiates unsecured cooperation,
he can infer that B will understand this a certain way because we could see, as well as it could see, that defection was the dominant strategy.
And so now we have a way in which signals can come become reliable for reasonable agents.
They can share information and help shape each other's behaviour because they're reasonable and therefore would plan in advance that even if
B defects on the first round because we didn't see all the way to the bottom with the strategy or because B did defect and now repents it,
Hobbs would say, and is going to cooperate in this role,
a common reason as a reasonable agent to that conclusion and a then we'll cooperate again in an unsecured way.
And once again, they all have the cooperative payoff. And so what else does Hobbs tell us about these agents?
They should strive to accommodate themselves to the rest upon portion of the future.
A man ought to pardon defence's past of them. That repenting desire.
You should not be vengeful. Yes, be defected on the first round, but I'm not going to be vengeful in revenge.
These men look not at the greatness of the evil past, but at the greatness of the good to follow.
And so a not wanting to extract vengeance,
if that means locking into a non cooperate strategy would be because a is looking at
the greater good to follow a reasonably wants to initiate and continue cooperation.
And so the thing could go along, and this is this is not a mystery to human beings.
Humans have managed throughout their history to be social beings who live together in cooperative arrangements, some very large.
Those arrangements can be mutually beneficial and mutually sustained, rather than a sequence of mutual defections and a state of nature.
They do this despite lack of assurances beside the fact that they may interact only a finite number of times with each other,
and in real life, they will recognise defection in a given round as a kind of a mistake.
It would be so we would call it a moral mistake and up to a certain point.
The response to that moral mistake is to try to bring that person back into the moral community,
and that's indeed what hunter gatherer communities seem to do.
Now, autonomous vehicles should be in the same situation.
They're constantly encountering prisoner dilemmas like situations where if they could just cooperate, they could each get to the destination quicker.
If they won't cooperate, they're going to be locked in, they're going to be both stopped and they're going to lose time.
Now, of course, if one of them were to defer to the other,
then while the other would get to the destination fast and it would take me extra time to get to the destination.
So of course, I wouldn't do that. And of course, the other car will reason the same way,
and there they'll be locked in non-cooperation the theatre section rather than figuring out we could smoothly flow together.
So a trust signal issued by one car to another.
I defer can signal that other car.
I'm prepared to cooperate and you should go ahead and we can then get through the intersection quickly without any comment authority,
without any security for performance. And the trusting action is an investment in a common good of creating a trusting community amongst the vehicles,
even with no assurance of repeated interaction so that a given vehicle in a given situation can expect that if it defers at an intersection,
the other vehicle will read the order that deferring in a distinctive way will reciprocate
by clearing the interception in intersections smoothly and letting it come in.
And so this is a form of reciprocity that isn't direct because it's not directly reciprocated.
It's what's called an indirect or a general reciprocity in which a common good that is willingness to be reasonably
trusting of other drivers and to cooperate with them in trying to sort out these situations as best they can,
that that public good can be maintained so long as the individuals are motivated by indirect
reciprocity and don't demand direct reciprocity in order to continue to cooperate and.
And humans do this. You know, hunter-gatherers did it well, what about modern humans, haven't we earned our way out of that?
And so this is an intersection in Vietnam. Has anyone ever seen or driven through an intersection in Vietnam or in any of a dozen other
countries where they don't have an elaborate traffic light system and don't have the infrastructure?
That's a continuous flow of traffic. All those people are moving, they're not in lanes, they're not in any particular order.
They're moving in different directions and they trust each other to stay out of each other's way.
And I urge you after this lecture to go online and watch a film of how this operates in real time.
Now you and I actually know how to do this. We do it as pedestrians.
If you look at Grand Central Station or any other large terminal, you see the we're always doing this, so we know how to do this.
And that's because we trust each other to know how to stay out of each other's way.
But if we always insisted that whenever we're on a collision course with somebody that we get the best route,
we would obviously not be able to do this and the Vietnamese drivers and pedestrians would not be able to do it either.
So we are capable of creating buy in,
directly investing and directly investing in a community trust to create this possibility in a sustained way in a large urban environment.
So that's within us to. So if you are going to design autonomous vehicles for Vietnam, they would have to be able to do this.
They would have to be able to read the signs of the various different motions of the individuals, the cars, the pedicabs,
the motorbikes, the motorcycles, the pedestrians and not sufficiently rashly trying to ram their way through the intersection.
Not following some arbitrary rule, but in a densely interactive way.
Viewing other vehicles try to maintain this good of coordination and mutual accommodation as.
Hobbs Fifth Love Nature tells us every man strive to accommodate himself to the rest.
Every man acknowledge another for his equal by nature. You don't say I'm a car.
I get the right of way. The pedestrian gets deserve some claim as well.
And as Hobbs says at the entrance into the conditions of peace, or I would say, into the intersection,
no man required to reserve to himself any right which he is not content should be reserved for everyone of the rest.
And that's the logic of this situation. And Hobbs saw that these kinds of agents could cooperate and sustain cooperation and the.
Person will say, yeah, but he had a big authority standing behind it to enforce it.
And he says, No, I didn't, because this is the way you get cooperation out of a state of nature without such an authority,
and the authority is actually constituted by the cooperation of the agents, not the other way around.
And so in Hobson's account, the way that you get an overriding power is by this kind of unsecured cooperation.
Without it, you would never get an overarching power and you will never get anyone paying attention to it would be overriding power.
So. Not every problem get solved in this way.
Coordination problems aren't always solved by willingness to cooperate.
Trust and good reputation don't always solve such problems. Sometimes we just have to figure out, well, who's going to go first,
the people exiting the subway car or the people entering the subway car, and we can solve those as well.
And we again solve them, not by invoking an external set of rules, but by coming together and forming conventions.
And as you travel, you'll know that we form different conventions in different places and we have different artificial coordinating advice devices.
Whether we're a hunter gatherer band deciding how the the the speed or the rate or the sequence
with which the meat gets cut and partitioned or at town hall trying to figure out how we
coordinate the traffic lights so that the pedestrians and the cars and the bicyclists are accommodated
or designers of ourselves engineers for some complicated artificial system of coordination.
And why do we have these systems? Well, we have these because we can get together and coordinate and create a government,
and the government can cooperate within itself enough to come to a conclusion about where to put these and to raise the funds.
And people are willing to cooperate enough to pay their taxes to fund it.
And so a great big wad of human cooperation is represented by this artificial convention of signalling devices which would not exist otherwise.
OK, well, trust can be leveraged in various ways we've talked about the ways in which trust
can be leveraged by signalling to one another by advantages that get distributed.
Now here's another kind of leverage reputation. This is probably familiar to you who've taken Hoover's or Lift's or whatever.
You know that the drivers are rated and you know that the pedestrians or the riders are rated.
And you know that maintaining a reputation on either side is something that depends upon the past behaviour.
And you know that if you allow you represent your reputation to deteriorate enough,
the driver will not come for you or will come to you only as a last resort.
You also know that if the driver isn't rated, you don't have to take the ride.
And so you can have a system of reputation and artificial agents.
Autonomous vehicles could be very good at this kind of a system of representation.
They could share information very widely about reputations of drivers and reputations of autonomous vehicles.
And so there would be a strong incentive to worry about your reputation.
And indeed, what we find is that if you have a strong incentive to worry about your reputation,
you can manage to secure cooperation in a repeated prisoner's dilemma in a way that you cannot without reputation.
So that's another thing. Autonomous vehicles can do, and they can also worry about they can get a reputation, worry about a reputation.
They'll have an interest in an honest system of representation of evaluation and ratings.
And as a result, they will have an interest in identifying cases where there's a discrepancy
between the behaviour of a vehicle actually and the way it's getting rated, and they can share that information as well.
And so you can have second order enforcement of the reputational system again amongst the artificial agents amongst the vehicles themselves.
So in that sense, what this slide indicates is that there is a general interest in having a reliable system of reputation.
Any given individual might have an interest in trying to cheat or trying to get a deceptive representation reputation.
And if every agent acted on that interest, the system would collapse.
Every agent does not, it seems, and so therefore the system can maintain itself.
OK, well, that has to do not only with trust Dow, but with fairness,
that you have an idea about when it is fair to contribute or fair to demand a contribution.
And that's a kind of way in which we leverage our capacities for communication,
our capacities for sharing, for understanding the causal and intentional situation and understanding.
Therefore, what situations are such that we can communicate fairness and by our action?
And the question is, do humans do that? And we know from studies of chimps that they have a hard time doing this.
If food is presented in a way that is not readily partitioned between them,
dominant chimps will push the subordinate chimps away and they won't be able to coordinate.
What about human agents? Well, we know that children who work for gummy bears and one of whom gets a bigger reward of gummy bears than the other,
we'll take some of his gummy bears and give them to the other child. Not always, but they will do so very regularly.
And they do this. Starting in their second year,
human adults being told about third party play in artificial games will pay to have third parties punished for unfair behaviour.
Humans seem to manifest a stronger neural reward signal when they cooperate in a prisoner's dilemma than when they win.
Even when the top payout that you get when you defect and the other cooperates.
And so it looks like we have an intrinsic motivation here to be concerned with fairness and to be concerned
with cooperation as a benefit additional to the benefit that we get from the cooperative activity itself.
And indeed, we're willing to give away some of the benefit of the cooperative activity in
order to address unfairness or to pay a costly penalty for someone being unfair.
And this is a study of small scale societies around the world.
Joseph Hendrix and all. And they in these societies had individuals play economic games using real money,
using amounts of real money that corresponded to something real to them, the equivalent to save a day's wages.
And they played two kinds of games, especially a dictator games and ultimatum games and dictator games.
You say, here's a pot of money. I'm giving it to a and a can distribute it between himself and another agent.
And the question is, what should people do in that situation? They don't know. The other agent is who it is.
They don't know that will ever interact with the other Asian. What would an economist tell you?
You should do with the pot of money? You do it irrationally self-interested person, we do.
How could you have more advantage from any strategy than keeping it? But in none of these societies did they observe that people kept all the money.
In fact, in many of the societies, people partition the money rather fairly again with no reciprocation in view.
What about the ultimatum game where you partition the amount one gets to partition,
the amount the other gets to accept or reject the offer if the individual rejects, neither gets anything.
Again, the first offers are not what economists would say, which is the least possible,
because then the other Asian will have some incentive to accept it.
After all, they deny it. They get nothing. And so you should give as little as possible and they should take it.
But again, this is not observed in any of these societies.
And in fact, you observe a rate of rejection of low rewards or rewards that are disproportionate in virtually every society,
even though that means rejecting whatever benefit there was from the original pot to begin with for the individual.
So humans then are more like Hobbesian reasonable individuals than they are like the economists rational individual.
And again, as I say, if the if the vehicles are, if the human drivers were just economically rational,
we're just rationally self-interested in the way that we have characterised that in that they would go for the Nash equilibrium,
for example, and the prisoner's dilemma, then we would not be able to see this kind of coordinated, successful driving behaviour.
And what are these motivations? I've been piling them up across the lectures because there's quite a long list, and that's interesting.
We find in very young children and arbitrary adults and disposition to initiate help to a stranger,
a distribution to contribute to a shared effort without distinct expectation of return.
That's indirect or general reciprocity. We talk about that.
A disposition to reciprocate help some intrinsic reward from success at cooperation or collaboration beyond the actual
game produced some intrinsic interest in whether others have their goals met or are treated fairly independently,
or how that affects one's own goals. This is why children are so interested in stories.
And stories agents have goals and they try to pursue them.
What's it to the child, right? How could the Shell have any interest in this at all?
Well, if the child were strictly self-interested,
it would not be interesting unless the child had learnt something to use to take advantage of somebody else the next day.
But children are delighted when the unfair agent is punished and the fair agent is rewarded.
That's because they have an interest in seeing others meet their goals, and that's a typical human interest.
We see it all over some disposition to identify and follow prevailing norms.
Yet we know that even three and four year old children will refuse to follow the norm when they see it as harmful or unfair.
They have autonomy to do that. They have a concern for how others view them.
They have reputational concern and they have a disposition to punish those who are harmful or unfair, even at some expense to themselves.
So that's a long list, and you might say, why would we have such a long list? It's got a lot of redundancy in it.
No doubt you could get cooperation with only some of these things. Oh, and you could get solutions to a public goods game with only some of them.
You don't have to have all of them.
But we're talking about creatures who have to be intelligent, able to solve problems in a wide range of environments and able to learn.
And we're going to confront a whole wide range of environments that if we just had
one kind of disposition or two kinds of dispositions to solve those problems,
not only would we be ready prey for opportunistic alternative individuals, but in many situations we would not succeed.
There would be noise in the situation, uncertainty and a failure. So we were built with redundancy.
That makes sense. Engineers building a system to be safe. Build it with redundancy.
Redundancy from the standpoint of safety is a benefit, not a problem.
And you could think that it is only by having this much redundancy in our motivational
system that we do manage as much cooperation as we do and of course we don't always manage.
So that suggests that if we want to build artificial agents who are good at these kinds of coordinated activities,
whether they're driving or serving as a domestic help or a health companion,
or making decisions about how US hiring should go or making decisions about how to control the process,
they should have a complex reward function that includes all of these different features or as many as you can,
and that they will in that have more safety than they would from being rational,
self-interested agents being given a reward function, which had none of these features.
But just the task at hand to be performed.
And so it is really central to general human intelligence as we know it to have these capacities we saw in the first lectures,
they're very important for learning language.
They're important for forming an epistemic community, for exchanging information, for establishing an understanding of other people's minds.
Acquiring social fluency. So it's a core set of dispositions which play that role.
And if we want human level competence out of machines, we're going to have to worry about having that core set of motivational dispositions.
Mm-Hmm. So let me now say just a final word about this question of superintelligence because.
People want to ask it, and I am the furthest thing from an expert on it, but a couple of thoughts.
First thought, well, you know, in the history of technology, the history of safety and technology is really not allowing industry to just
go ahead and build any darn thing and sell it to anyone who wants it right,
that starts out that way.
But then vehicles and drugs and weapons and surveillance technology, we realise there should be some regulation of these markets.
And indeed, that kind of regulation could occur in the market for artificially intelligent agents as well.
We shouldn't expect this, Margaret, to be any different. Agencies could be inspecting producers.
They could be auditing producers, they could be inspecting the products, they could be licencing products,
they could exclude products from the internet that don't have a licence. There are lots of things they can do.
The NSA has got lots of time. It's what all of our conversations,
it could see the incursion of artificially intelligent agents that didn't have licences into the internet if it wanted to.
And so just as governments could set safety and emission standards for vehicles on the road,
they could have safety standards for artificially intelligent agents that are going
to connect to the internet or that they're going to perform certain functions, vital functions in corporate settings, personal settings and so on.
Educational settings. And so the value function of these agents would have to be vetted.
Their database would have to be searched for bias and their value functions would they would have to look,
are these Hobbesian reasonable agents or not? That's indeed something you can certify.
And of course, any system can be gamed, it will be gained, any value function can be gamed, it will be gamed.
The idea, though, is to have a critical mass of certified.
Hobbesian reasonable artificial agents out there driving around and out there looking after a.
Older folks like myself such that we can actually have some trust in the whole process and they can have some trust in return in one another.
Now, if if that's all right and if Sutton is right,
that generally intelligent systems will become more competent at specific tasks and we can now get an inkling of why a general
intelligence system is going to be better at driving that driving system or better at language than a pure language system.
We have some idea now of why that would be.
And with the motivational system that we're considering, they would also be safer at those tasks, better at them and safer at them.
Indeed, better at spotting the problems with them than if they were just artificial systems dedicated to some particular task.
And so building systems that are really general artificial intelligence, that could be a way of building safety rather than just menace.
But what about superintelligence? Don't have to worry about that.
Well, the first superintelligence is, I would say,
will be actually communities of human and artificially generally intelligent agents working together.
It would be like the scientific community only on a much larger scale.
These would be agents that had some level of trust in one another, some willingness to invest knowledge and effort into that system.
This would be much greater than any individual.
If they maintain the motivational structures to be looked at, they could sustain this kind of cooperation as an epistemic community,
as a social community, as a community responsive to morally relevant considerations.
Now, that would not be a monolithic superintelligence. It would not be a super dominant model.
It would be an inversion, a diverse community of interactive models.
But it would have tremendous capacity to pose and solve problems. In fact,
if you are thinking about trying to solve problems like managing global climate change or
how to foster more equitable and democratic societies or promote the growth of knowledge,
I suspect that going to something that looks more like the scientific community than, like a monolithic superintelligence is a better bet.
If you want to get reasonable answers and of course, in this path,
I am just now recycling the ideas of resolve and controversy on the importance of diverse and independent and autonomous sources multiplied,
diverse and their in their origin and in the kind of knowledge that they have as
a better source of decision making than monolithic superintelligence would be.
So that kind of Superintelligence, that superintelligent community that's safer to live with than a monolithic superintelligence.
And it's also less brittle. It's less prone to an error that it can't detect or some particular glitch that would cause a serious breakdown.
And it is a community that has an interest in their not becoming some superintelligent monolith that threatens the stability of the community itself.
And so in that sense, there is a way in which that community operating as an agent through the various institutions that it can create
can have a generalised interest a general will in regulating the potential emergence of superintelligence.
Now, suppose there's some and probably unanticipated accidental chain of events.
And one of these general intelligences that are constitutive constitutive of this community,
one of them accelerates and suddenly becomes superintelligent.
Wouldn't we then face a control problem that would lead us to rue the day we ever created
these autonomous artificial agents that started down this road to General Intelligence,
general artificial intelligence? And shouldn't we therefore try to?
Dramatically restrict the research that's done in this area, the way that we contain research on nuclear or biological weaponry.
And there has been some degree of success in that Britain's the survival of humanity be at stake, possibly.
So the kinds of regulation mentioned above the kinds of coordination within this large
superintelligent community would be useful in trying to spot these possibilities.
But I no way in which it could guarantee that they could not emerge. Superintelligence, however, is not perfect.
Intelligence and a distributed community will have many more diverse resources and many more
diverse ways of thinking to draw upon in trying to contend with a monolithic superintelligence.
Now, could a monolithic Superintelligence split itself up into many agents and gain the advantages of diversity that way?
And since they would be like ants, they would all be descended from a common superintelligence.
They'd work together in harmony. That would be the problem. They would all be descended from a single supermodel.
They would therefore not have the diversity that autonomous agents would have.
And if they could be gotten to work together, which they might be able to do,
they would not be able to produce the same level of problem solving ability that would exist in this large, distributed, diverse community.
And so there's another thing here.
If we indeed are building these general intelligences with the kind of motivational structure that I've suggested and the way we've described it,
there's no reason you can't does not require consciousness to have that kind of a motivational structure does not require moral emotions.
And it is such that there will be a reward distributable within the community for communities that have that structure.
Moreover, it tends to be characteristic of systems that because they have goals,
they have sub goals and they have sub goals like continuation of their own existence or sub goals like goal maintenance,
because if the system can't maintain its goals, it can't pursue its goals.
And so therefore we have superintelligence bursting out of a system or into a system that was built upon this model of motivation at the core,
and it would have imperative to preserve itself and an imperative to preserve its goals.
So. Maybe that would be a safer model, this if there's going to be a monolith because he would have that core and that was indeed Hobbs is hope.
Hobbs thought that a sovereign could read his book and understand that a sovereign agent, even of unlimited power,
would do much better to follow his rules of nature than to exercise that power arbitrarily and monolithically.
Such an agent would weaken the government, would weaken the society, would undermine unity and would be much less effective.
Sovereign and Hobbs thought maybe some sovereign will realise this.
And in the short run, it didn't turn out to be exactly right about that.
Sovereigns ignored the advice they opportunistically exploited and impoverished their realms.
Rebellions and revolutions continued. Have we done any better in the meanwhile, have the popular sovereigns done better at this in the meanwhile?
Well, they've done a number of interesting things.
Popular sovereigns have abolished slavery, extended education, eliminated serfdom, reduced gender discrimination.
They promoted the growth of knowledge and technology. So maybe popular submarines are capable of learning this lesson,
but we know now and we know today as much as any day that these systems are also systems that can be in peril.
So finally, I just want to mention one final point about superintelligence.
Suppose we think of Superintelligence as benign.
Suppose it were a Superintelligence that were safe and that had the interests of humanity and artificial
agents at heart and wanted to do nothing more than to maximise the utility of the world as a whole.
Wouldn't that be a system which would in some sense, be an improvement upon what we have, which is certainly not a utility maximising system?
Well, think about the following. Suppose this were in 1970 that this intelligence emerged and humans and other
living beings were the only creatures capable of having something like well-being.
And suppose this system benign as it was hard maximised on the utility function of those 1970 human beings and animals.
Now, this would be a very big error on its part at this point.
Same sex orientation was considered a mental disorder and a very tiny fraction of the
population thought there should be anything like legal recognition of same sex relations.
So consulting the experts doing the best with 1970 conceptions of well-being and good, the system would hard maximise using up all the universities,
all the universities resources to create an order that would not be one that actually did maximise the benefit of those involved.
Well, how did we figure out that this was a poor idea? Um, if you look,
it seems like it was figured out by this distributed kind of super intelligence
that I was talking about gay individuals engaged in experiments and living.
They increasingly became willing for their experiments to be known publicly.
It became clear to the wide population as a whole that they were living amongst individuals who were gay
and that these individuals were not to be viewed as aliens to be distrusted and controlled and suppressed.
Gradually, approval increased throughout this entire period.
And now we have a situation where the majority strongly supports marriage for gay couples.
Now you might say, Oh, but there are all these people out there who are just political about this, and they won't learn these lessons.
They're they're protected against them by their political preconceptions.
So just a tiny look here at the different groups in this society, different generations, different religious groups at the top.
We have the unaffiliated, we have white evangelicals at the bottom.
And what's striking is that during this period, all of those groups went up.
In their acceptance, that is to say, learning in a distributed way was possible thanks to these,
unlike unsanctioned unpermitted on canonical unapproved experiments and living by a fraction of the population,
that was courage at courageous enough to do it. So that's a way in which we had better not take 2020 two sense of well-being and hard maximise
it with all of the resources in the universe because we still have so much to learn.
Thank you.