The robot has that recording in progress, there's an A.I. somewhere hidden in the machine.
Thank you so much, Gary, for the invitation and for the very long intro.
Sorry, sorry. There's a grey box again, there's a grey box of obscurity.
It is true. It's warning me that this meeting is being recorded. Yeah, thank you.
So first of all, for the for the invitation to speak to you all today and also for everyone, to everyone, for coming.
I hope that this will be a fun and potentially entertaining discussion about AI ethics.
So what I wanted to do this talk to do is to provide a discussion or provocation to think about the broad context in which,
where in which there's now a discussion around ethics.
You, if you're at all paying attention to the EU legislation as well as UK legislation,
there's a lot of focus these days on on AI related safety and harms.
And the question is, so why do we have a sudden focus on on AI related safety and harms?
Where does this come from?
Is it is it that we've reached some sort of threshold by which now AI systems are becoming so capable that that they're becoming harmful?
The truth is, is is a little bit more subtle than that now.
I think it's only come to the point where we've we're starting to come to an understanding of all the
ways that digital AI technology is shaping society and starting to understand what it's doing to society.
And as a result, these things are all sort of being lumped under this notion of air safety and harms.
And so what I'm trying to do in this talk is to give you a bit of a of a taster session.
But before I do that, I wanted to make sure that so that we do talk about some uncomfortable topics in this lecture.
So if there are if there are issues that I've listed here that are particularly triggering or that are very upsetting to you,
you may want to either tune out for those sections of the of the talk or or or review the talk after the fact.
As far as I really like interactive discussions, and so but I do have a lot of material I'd like to go through.
And so what I propose that we do is if you could use the chat box to bring in comments and I will I will incorporate some of your comments as you do.
And then at the end, let's have an open discussion and I'll try to leave at least 10 minutes at the end for us to talk about things.
If that's OK, does that is that all right with everybody? Sounds great, sounds great to me, but I'm only one of 16 other people.
Excellent. I imagine there are nodding people. OK, excellent.
I see one thumbs up, the couple thumbs up. So the format of this talk is going to be a little bit like a bento box.
So I grew up in Japan from a Japanese heritage. And so we're going to talk about bento boxes have little.
It's a collection of of different tastes, different topics and some of them.
We're going to go into a little bit more depth than others. You know, overall,
my treatment of the topic is not going to be particularly satisfying because they're not
going to be able to spend that much time getting into great depth into these discussions,
but hopefully they'll make a collective meal.
So, so we're going to talk about first for your talk about Trump's second talk about bias in everyday life and then sources of trouble.
And we're going to go into that in some considerable detail. So that's going to be a substantial part of the talk.
And then we will talk about various aspects of what people are called human interaction, which often deals with algorithm automation bias,
our tendency to believe machines and also the ways that that relates to confirmation bias.
And then we'll talk about in general, which is disparate, disparate impact of bad systems.
So the way I like to start this talk is one which is a little bit provocative,
but what I'd like to say is that the idea the term air safety has had many different connotations.
And from about three years ago,
when I proposed a new class for undergraduates in computer science on on computer A.I. ethics and responsible innovation,
there was a certain strong connotation that this had to do with one of these three problems.
And what I'd like to say now is that these are not the topics we're going to talk about,
because these intellectual what I call privileged distractions poor and nightmare fantasies for from well-known
Oxford philosophers are things that essentially are really are they're not that they're not interesting,
they're profoundly interesting, but they're sort of like mind viruses that will sort of suck all of the oxygen out of the air and
prevent us from being able to think or talk about things that are actually much more relevant problems.
And these are, you know, first of all, the superintelligence hypothesis,
which states that we will have general intelligence that will enslave humanity because they will start to become exponentially self improving.
The second one is that we have to solve the trolley problem before we can make any progress.
Self-driving cars, autonomous systems because you need to be able to weigh the value of human lives.
And the third one is the the idea that that information itself exerts a moral claim.
And therefore, when you think about whether potentially information is sentient or alive in some sense and perhaps robots should be granted rights.
These are all really interesting things to talk about at a pub,
and I'm more than happy to to to go drinking with you and discuss these in great length.
But there are many reasons why these are problematic in the context of a real discussion about air safety in legislation,
as well as in trying to encourage researchers,
especially junior researchers, to think about air safety because they are very, very hypothetical and almost provably impossible.
And so there's a lot of real fundamental laws that need to be really bent in order for the first one to become even feasible.
And the trolley problem is an interesting one because it's not that it's completely irrelevant to air ethics,
but an area that it gets applied to immediately is self-driving cars because of the of the similarity of the I'm sorry,
I'm not going to spend a lot of problem talking about a time talking about these problems.
I'm assuming that they're familiar to people again, just even referring to them.
They're starting to suck all the air out of out of this talk.
And so but what I just wanted to say is that people think of this and, you know, in relation to self-driving cars, when in fact,
the truth is that the trolley problem is much closer to the problem of medical ethics as well as disaster related ethics.
So when you have a finite number of resources like ICU beds and you have a disaster like a pandemic happening,
how do you then prioritise the limited resources this is, you know,
a grandmother's life who is a Pulitzer prise winning novelist, you know, but more important to save than, you know,
somebody who's a younger person who has, you know, I don't know so that they start to become trolley problem ask.
But in fact, medical assets deals with these in the most boring ways imaginable.
So as borrowing, but also conservative ways, which actually kind of makes sense.
And so there even still, the trolley problem is not particularly relevant.
So what are what are we going to talk about for the rest of this lecture?
So we like we in the computer science department,
in my research group and also the class that I teach first year undergraduates like to focus on harms and problems
that deliberately or accidentally arise from our sort of relentless march towards increasing automation.
And, you know, deliberate harms usually are emerging from using technology as a smokescreen to try to
exploit people and to manipulate people for for individual gain and accidental harms are
different class of things that usually result from us doing the things that seem exciting
or good and then actually accidentally end up doing something unintentionally bad.
But both of those classes of harms often disproportionately affect minorities,
disadvantaged groups, low paid workers who are already exploited and overlooked.
In some sense, this is not a coincidence. It's nothing to do with these groups or individuals that ourselves.
It's that these groups are already overlooked in the design of these systems.
They were once again overlooked, or perhaps some nefarious billionaire decided to build a social network that would really improve their own wealth.
And that was itself sort of, you know, sort of a selfish action which then ended up exploiting the vulnerabilities of many.
Now there are many books that have been written recently about particular groups,
and these are the textbooks that I that I recommend for understanding specific groups and the ways that they are being marginalised by technology.
Let's start dive in to a particular example. I wonder if people remember or experience the Twitter cropping scandal of October 2020.
So explain what this what this is. So if you if you take the image on the left is is one image.
I don't know if you can see the border around it. What it is is. So basically,
imagine taking two very long images splitting so so that you have one that has
Mitch McConnell on the top and Barack Obama on the bottom and the other one.
You have Barack Obama on top and Mitch McConnell on the bottom. So they're otherwise identical.
It's just the faces are swapped. OK, so they're both very long, skinny images.
OK. Say that you posted them separately side on Twitter.
What would happen, so what happens is you end up with the Twitter algorithm,
goes and figures out because it's a long, skinny image and needs to figure out how to crop the image.
Now, the thing that most people didn't realise is that it doesn't just do this by chopping the middle of the image or doing something deterministic.
It uses an algorithm to try to figure out what's the most interesting part of the image.
And this algorithm was found to simply select white people.
Right? So it was it would promote, you know, that it's literally erasing, you know, preferring white faces to black faces.
And the result is that you end up with two beautiful faces of Mitch McConnell instead of our great, you know, former president Barack Obama.
And this is a real problem, right? So the way that this sort of worked was as follows.
There was there lot there is the photo gets processed by model,
which produces as a predicted salience map and that is used to crop the area that's of highest salience.
So this created an immense sort of response that say, why is this doing this right?
So and this the result of was an investigation by the Twitter team, which deals with the ethics and responsible innovation team.
And they were looking at several problems in this in this cropping algorithm.
One of them was indeed unequal treatment of different groups, not just of different skin colour, but also gender.
And the second one was so-called objectification biases dealt with,
otherwise known as male gaze about biases in which photos of women were cropped not around faces, but around parts of the body.
And these kinds of issues they then evaluated with dataset and were able to discover exactly what the biases were in the algorithm.
Now this is the kind of thing that I'd like to talk about, because it's an example that's immediately upsetting, right?
And it's important to sort of try to to articulate what is upsetting about this and what might happen if it gets unchecked.
You know, we should we should try to rationalise our feelings around this kind of thing.
Another reason why it's upsetting is that it's one of these algorithms that we didn't even know existed.
It's one of these things that it's like, what? Why did you even have to add the algorithm if you had done nothing whatsoever?
Simply the image in the middle, you would have avoided literally erasing or hiding, you know, already marginalised groups of people.
And the point is that that these sorts of algorithms are now being rolled out and massive scale everywhere amongst the systems that we use every day.
And and so as a result, we are perpetually surrounded by algorithms that have potentially very harmful biases in them and harmful properties,
not just biases, but properties that we don't even realise.
And so, you know, and that leads to the real question of of really, what can we do about this?
And really, you know, is there an opportunity to fix all such problems?
Is this a problem? You know, is this something that we have to do post hoc or can we start to do this in sort of an anticipatory way?
And so that's sort of the reason why I proposed a course in in for computer science that will because we believe that that
computer scientists are amongst all the stakeholders who should really be involved in the process of computer engineers.
You know, people who are thinking about the algorithms, machine learning researchers,
statisticians that such as yourselves who are thinking about the various and rich contexts that these
things may be deployed and told to stand the kind of risks that that might arise from from their use.
So we use the term algorithmic bias,
and here I just wanted to say that we use this sort of broad definition of bias rather than the statistical definition of bias.
We refer to algorithmic bias as essentially in general, systematic or repeatable errors in a computer system that create unfair outcomes that work
differently for different groups that end up privileging one group arbitrarily over another.
And typically, you end up with this these kinds of errors and problems ending up in two camps.
One is probably the most familiar to all of you that deals with sort of allocate of harms,
which means that it's when a system provides different groups,
unequal opportunities, resources or capabilities, and the other class is one that it's less discussed in the aid fairness literature.
And because it's not directly statistics statistical property of of machine learning systems, but that is called a representational harm.
A representation of harm is something that where an algorithmically curated or created depiction of somebody or something or a group is harmful,
and we'll give some examples of that. Let's first talk about allocate of harms.
Give me a second here. So I look out of harms you've probably heard of in the context of many kind of high stakes decision making contexts like,
for example, you know, predictive policing systems, the judicial process, the criminal recidivism systems.
But allocating harms exist in all sorts of shapes and in routine systems, for example, things as simple as a speech recognition algorithm has,
you know, speech recognition algorithms, which we use very routinely, have been shown to have various kinds of addictive harms.
And in that the recognition rates are not evenly distributed across groups.
So there are many groups for which recognition rates are much lower than than other groups.
And you know, this is not this might seem like it's not terribly surprising because people speak so differently,
but we're talking about again, systematic differences between groups in particular.
You know, the highest performing speech recognition engine,
which you know this this study was a little bit obsolete from two years ago, but believe it still is.
Either Google or IBM was demonstrated to be 13 percent more accurate for men in general than for women.
But this this difference might. This difference is significant and interesting,
and it's even greater for people who have different accents or different ways of speaking and different backgrounds.
And so that is something to note. But unfortunately, it's not just space speech recognition,
it's basically every sort of recognition that kind of algorithm or system that we use today,
including those for basic face detection as well as recognition.
And of course, one of the sources of this, it's not that faces of people who have different ethnic backgrounds are harder to recognise.
It's that largely this is entirely the fault of the data sets that we are using.
The data that we're using are not even the distributed amongst amongst groups.
And as a result, the systems just by default perform worse for those groups for which it has less data.
So you might say, well, what's a few percentage difference anyway?
Does that make any difference? Well, everybody in the room, especially in the private statistics, now know that this is a major problem,
especially as you try to run an algorithm on millions of faces or run a system over and over again for something that's quite a critical thing.
For example, even a small chance of error can cause a lot of false positives if you do a detection problem enough.
So the ACLU has done a lot of great activism with with with government procured
face detection and recognition systems to demonstrate how badly they perform.
So. So just like in the U.K., in the U.S., there are many policing departments that we're starting to use commercial facial recognition systems to try
to identify and spot potential criminals and to make a point of this to make a point that the in particular,
the ways that marginalised people are disproportionately affected by false positives is that they applied this algorithm to members of Congress and
found that black members of Congress Black members of Congress were disproportionately mis recognised as criminals than by the algorithm and saying,
you know, this algorithm is clearly not fit for purpose.
So these are examples of the kind of things that even a small percentage of chances of error can really cause a very uncomfortable problem,
which amplifies an already delicate situation.
Now think about then. So these these kinds of things, so these recognition systems are embedded in complex systems and often,
you know, that have multiple such recognition systems, Inc.
So what happens when we embed these systems in the kinds of everyday tools that you need to do your work or to survive to live, right?
So if everything from the, you know, the sort of soap dispensers that we use to the face authentication systems we use have these biases.
What does it mean? It means that the systems just don't work as well.
For certain groups of people in particular, it doesn't work as well for people who are already marginalised that you know,
have have are less frequently at higher positions and companies and are paid less have higher unemployment rates and things like that.
And so you're making a bad situation worse if your fundamental digital tools in your environment disproportionately work more poorly for these groups.
And and that's something to think about. So the representative harms, let's talk about that for a moment.
So again, representative harms are not exactly the same as as allocate apartments where you have different groups for which an error rate maybe,
maybe disproportionately affecting one group rather than others. Examples of representational harms are, for example.
Professor Latanya Sweeney, who is a former chief of the Federal Trade Commission.
She discovered that she was doing an early study of how African-American names were more frequently associated on Google,
with adverts suggesting arrest than non-African-American names.
And they systematically ran a bunch of African-American baby names and sort of white baby names.
And, you know, and measured exactly to what extent this was the case.
And you know, the thing is, so the problem, the question that she asked is So, so why?
Why? Why is this? Why did this happen? Is it because the clients paid to have the ads do this specifically targeting black names?
Or is it some sort of, you know, through some complex feedback system that the system learnt a racist association between?
And the reason why this is the representation of harm is that it's one of these
things where you are a you are subliminally suggesting that for some reason,
if you have an African-American name, you're more likely to be associate with criminality.
Right? It's a represent. It's a harm that that is associated is that it's associated with that depiction.
But we see many kinds of representational harms of many sorts all the time you think about it.
For example, the gender bias and Google image search, a lot of people have studied this.
So if you type in nurse, you often get pictures of of white women.
And if you type in doctor you get, then you know, more often pictures of of white men and CEO is sort of old white guys.
And the question is, these are representations that then, you know, are already contain huge sort of biases and luggage, right?
And so the question is why where are the where are the black women, what we know,
what's going on here that sort of reinforces the stereotype that CEOs look like these things doctors and nurses look like these things.
And this is a form of representational harm. Now, it's more difficult to try to think about how you characterise that in terms of.
San Francisco property, because often it's a lot more subtle to identify what it is that you need to try to look for.
If you're going to try to find these disparities and these representational issues.
So let's talk specifically in context of algorithmic decision making before go on.
I don't see any comments that everything OK going forward.
And you know, the quick responses. All right, great. So let's let's go on to algorithmic decision making, Solomon,
decision making is is a whole sort of new world that I think has brought a lot of these issues to the forefront,
which deals with essentially the application of algorithmic methods to support humans in decision making processes.
So, so that's often that's called decision support, where essentially you still have an expert there.
But now a system is providing advice to make the person able to make perhaps more accurate decisions in
some sense or to help them go through more cases more quickly or in place of the human expert entirely.
And that's what we call automated decision making. And there are many,
many examples where algorithmic decision making systems are affecting and articulating and many invisible parts of our lives in ways that
you probably and I don't realise every day and this is where the title of my talk victims of algorithmic violence really comes from.
Right. Nearly every aspect of your life is now being judged right by by algorithms.
One of the earliest areas for automated decision-making was making. It was, you know, an applicant cutting edge application.
Machine learning was in credit risk prediction. You know, even in the 80s and 90s, there was early machine learning that was used for for credits.
And of course, that's being used today.
And, you know, more than ever before, we have things like, you know, whether you're going to get a loan or a mortgage or,
you know, or even be able to get a mobile phone plan uses a huge number of features,
some of which are derived from completely non-financial features like lifestyle features and other big data features, which is very interesting.
Differential pricing. You know, you might not realise that you know that the cost of your flight or your hotel
room really depends on how much the surveillance architecture knows about you.
You know, if it knows that you want, you might spend more money. It may differentially price you into a higher price tier.
And actually, for the exact same product, you will be advertised a higher price.
Law enforcement You've probably seen many examples of the ways that predictive policing systems,
as well as you know, have helped helped and also armed groups to try to distribute the resources of policing,
force it within areas particularly known for high higher crime, criminal justice, job hiring,
the gig economy, everything from Uber Eats to Uber, you know, to Deliveroo and Uber itself.
You know, all of these aspects, you know, their algorithms that articulate what clients you're going to get for your next job,
you know how you are going to be seen by others and whether or not you can continue to make a livelihood of these, you know, out of out of this job.
And increasingly, the kind of gig work algorithm uber driven kinds of stuff is making its way into into brick and mortar establishments.
And so now we have lots of surveillance systems that go and monitor worker productivity, especially during the pandemic,
to try to measure whether or not you are being productive and whether or whether you should be fired
and whether it should hire another associate professor of human computer interaction instead.
And so these are the sorts of things are amazing, even even any prioritisation.
So when you try to call 999 or when you walk into a hospital, whether you end up, you know,
waiting there for six hours or not is due to some algorithm that's trying to optimise resource in some sense.
And so so this leads to a lot of interesting questions about which of these algorithms have biases.
You know, why are they using features? You know, what kind of features are they using?
Exactly. You know, you would hope that the algorithm wasn't using your Twitter feed to try to determine whether or not you deserve to be operated on.
You know, although for me, it would probably prior to no, just kidding.
But you know, there's so many, so many questions that that it leads you to ask about this.
And so this is my algorithmic decision making is is so important because because these
algorithms are working opaquely behind the scenes of everything that we do and and nobody's
really discussing exactly whether we should be using these algorithm and what it's fair
for these systems to actually conclude based upon the information that they have.
So this and so let's just back up for a second say, so what are we talking about the context so previously?
Each of these contexts were were done by humans, right?
So there was so there were human experts for each of these contexts, whether it was applying for a mortgage or applying for a job or deciding whether
or not somebody was guilty or deserved parole was done by human experts who were.
And what we're doing here and in our decision making is saying, well,
we're going to replace that or augment that in some sense with some system, with some data driven system that is used to extrapolate from data.
So what we really mean by that is the way that that works is usually what happens is they start with a dataset or of real cases that were evaluated.
And then what happens is that based upon that dataset,
then that's that's sort of those cases are transformed into some machine readable representation
and those that representation is typically called a feature set or or feature representation.
And then what the system, what the human decided to do in that context, whether it was to give you a loan or not,
is then considered the label and that's associated with those features. And you do that for all of those examples, right?
And then you feed things into some complex, usually or hopefully not too complex.
Model and the result of this model is something that can then for new instances,
judge based upon its representation, what kind of label a human would have given.
So that is a very cartoon fresh,
fresher and computer science picture of what a machine learning system that is algorithmic decision making does, right?
And the question is OK. So that seems fine, right?
That's fine. It's nothing. Nothing wrong here. What could possibly go wrong?
Right. So so I could say there exactly, you know, four major areas where things could go wrong, right?
And there various different coloured blocks here going through very quickly.
You know that sample data that sample set coming in could be not representative.
There could be it could change over time. Right? You know, you could have you might have used a convenient sample.
You may not, you know, so that that's to begin with. The second thing is, so you relied on a human expert to come up with the label, right?
That label might be noisy. You know, human experts make mistakes.
Then the second thing about features, so you've transformed something that's a complex world into an end dimensional no right or something, right?
And that itself should raise alarms completely.
Because how? What? Right? Well, on what basis did you do that?
And we'll talk about that in a second. Then you trained this monster monster sort of model.
Right? And the model, you know, is is it's working in some ways,
it's that this generic statistical machine that doesn't actually know anything about the world and it doesn't actually know about the cases at all.
All it knows is the relationships between the variables that the representation that you,
the examples and the representations that you've given, right?
And then finally, you sort of contextualise this in society, you drop the whole thing in society and what happens.
So there's five areas that are worth discussing discussing here, and I'm sure that none of these things are terribly new to all of you.
But but. But but. And so. So let's talk a little bit about a couple of them.
So the problem with the future transformation? So there's a very, very old I theory theorem that says that's called a frame problem that says.
If so, the problem of choosing a representation for a problem is at least as hard as solving the problem.
OK, so so if you need to if you want to figure out what you need to represent,
you already need to have figured out a way to solve the problem before you.
So, so in other words, it's sort of a thing that says choosing the right representation is tremendously hard problem,
and in particular because it's impossible to figure out what to represent versus what to not represent.
You know, a butterfly in South America may be relevant for a particular court case, but it may be completely relevant.
Nine point nine nine nine Other other cases in the world The amazing thing about human beings is that we don't need future representations.
We are very flexible creatures that we can bring in evidence as in when we need it and reshape our representations.
But the kinds of systems that we're building now can't do that.
There's a wonderful, wonderful case called the hungry judges problem about, so we're talking about the fallibility of human experts.
This this case, which has been hotly debated, demonstrated that judges are far more likely to be harsh on a particular cases, right?
Sort of before lunch than they were and the first thing in the morning.
Now this is completely, you know, even if you, you know, so this is assuming that the cases came in randomly and that there shouldn't be.
The cases we're not ordered such that the hardest or most ambiguous cases were closest to light.
So, so you know, the hypothesis is when people get hungry, do they get more mean?
You know, and I could definitely reflect on my own behaviour and tell you that perhaps that's the case.
But there have been other kinds of things about similar decisions that were made in NHS contexts where misdiagnoses were,
you know, correlated with being tired or, you know, being hungry.
And so humans are fallible. And of course, we're using that.
You were plugging that into a machine learning algorithm,
which is then trying to reproduce hangry ness in our population, which I think is quite problematic.
And this one is one that it may be very basic, too basic for for this group,
but at one that I always talk about, which I think is really interesting on this.
This case was from an example from Microsoft Research where they were.
There was a research, very, very early experiment in neural networks where they collected what was at the in the 90s,
a very, very big dataset of of fifteen fourteen thousand one hundred ninety nine patients.
And what they were trying to predict was they're trying to the dataset consisted of
pneumonia patients and whether or not they they died within sort of 24 hours of a diagnosis.
And the reason why they did this is they were trying to model this is to understand for pneumonia patients,
it's often a very subtle interaction between sort of symptoms that might influence
whether somebody would go bad on somebody's condition or deteriorate quickly,
or whether you can just send them home with antibiotics and they'll be fine.
And if you do, if you're able to anticipate this well,
then you can be sure to keep those patients that are going to deteriorate in-hospital while you can send the ones
who are going to be likely to be fine home so that you don't end up using hospital resources unnecessarily.
And so. So the models so they decided they were going to build an early neural network
based upon this very large dataset of of pneumonia patients and a number
of features of each critical features that doctors thought were essential components of whether or not determiners of of of a patient's outcome.
And they trained neural networks that seemed to have high accuracy for the time 85 percent accuracy.
And they asked the researcher, OK, should we run this in production? Should we start deploying this in practise?
And the very interesting thing is that the researchers said absolutely not because we have no idea what the system is learning.
And people came up to him and said, What do you mean? What do you mean?
We don't know what the system is starting 20 years later? They wrote a paper so very recently about.
So I believe it was 2016 about a system that of a model where they could they could actually
take apart the individual contributions of what was going on within that neural network.
It's a slightly modified kind of network called general generalised additive linear modules models at the end.
And what's great about this model is that it is purely additive, right?
So you can look at the individual contributions that the model has learnt and take apart what why you know,
the features that it believes are contributing to your likelihood of probability of death.
And then it's far easier to interpret. And what's interesting is that so you know,
you can look at all the features that it took in and then look at each feature one at a time to see how it contributes the probability of death.
And they found very interesting things, so they found a couple of things that immediately raised red flags to the researcher,
and one is that they were sort of sharp increases in probability of death that occurred at round numbers like,
for example, that occurred at 65 years old, that occurred at 85 years old.
And then they also found something really peculiar at the very end,
which which was that the model was predicting that a person's probability death would go down significantly after 100.
Right. And so this led to some very sort of some head scratching and would this be worrisome so.
So this is great that this can lead to some great bar game where we go and say, Well, what does this mean?
You know, why are we seeing this kind of thing?
And the hypothesis was immediately that, you know, after a couple of rounds that essentially it's 65 and at least in the U.S.,
your insurance, you're sort of you go to retire retirement retirement mode.
And often this means that your you change insurance providers, right?
And so then what happens is that your quality of care changes.
So you may have had a really,
really great insurance provider and then you retire and now you're sort of switch to Medicare and Medicaid and things like that.
And so then it could be that your probability of death, then as a result of the quality of care goes up significantly.
It's not that you turn 65 and suddenly instantly you are more likely to die.
Of course, that doesn't happen. Human beings don't work that way,
but it's something about the fact that people are retiring and switching health care providers, but that might be doing that.
The 101 is very interesting to the idea there is that they said, Well,
when somebody turns 100, then doctors often are like, Oh, this person's a centenarian?
Right? You know, we're going to immediately put this person if this person presents with a fever and pneumonia,
they're not going to think about whether or not to put the person in hospital.
This person's going to be prioritised right regardless because they're 100 years old.
Right. And so what you end up seeing in the model is the reflection of what people do because of their irrational decision-making process.
And then you see the effects of people's care practises. Right?
I mean, and so all the data that we used to train this network is reflecting this.
So what would have happened if we had used this algorithm to decide whether or not to put people in hospital?
We would have made really grave mistakes, right?
We would have said, Oh, they're 100, therefore they're not going to die, and it would have made the opposite decision that we had made earlier.
Does that make sense? So people ask me why we want the critical models, and this is the very reason why,
because a lot of data reflects actions, reflects the things that we do in society.
And it's important for us to be able to untangle these things because no data that we get is ever perfect.
And it would be unethical indeed to run a trial that just said we'll just let people die so that we can collect the data.
Right? So in this case, we have to. We were following best practises to try to keep people alive, but the result is that we end up with blurry data.
So I know I'm running out of running a little bit of time, I think I've got a little bit more time left here.
So the other the last thing that we're talking about is feedback loops to the ways that the systems get dropped into a society.
So I know you are all control theory experts, maybe at this point, but they're sort of there are two kinds of feedback loops.
One is this sort of good kind and bad kind of positive feedback loop is the bad
kind where you end up with any small change ends up being magnified amplified.
So, so this is the kind of thing that results in the squeak when you are in a auditorium and the microphone
gets picked up by the speakers and these speakers emit a sound and that gets picked up by the microphone,
which then gets amplified by the speakers again and you end up deafening everybody in the room.
That's a kind of positive feedback.
Negative feedback is where essentially the output is used to, then to then make a response that compensates for that.
So this is the way that drones can fly as they believe,
because we have and we take that sort of if you have a strong wind that pushes the drone to the left,
the control system then goes and uses that feedback to then compensate for it to come back.
I don't know why it suddenly turns dark, but that's OK. So, so what?
So what can we say about the sort of systems that we are building? So systems that we are building often create positive feedback loops.
And why is that?
The reason is, of course, because they're perpetuating because we're just because our systems are designed to reproduce what we do, right?
So and warts and all. So the things that we do, it's at it's gold standard.
It's the drive to do it, to try to do it exactly the same way. And no sense, is it trying to do it the opposite way?
And so, for example, so this, you know, an example, and this can cause lots of problems.
If you then embed them in a society, which then ends up perpetuating the biases that you have,
for instance, the predictive policing systems that we alluded to earlier,
such as PredPol, what they do is they they try to predict what areas of the city are likely to have a crime.
Right now, the data set that they were trained on has a lot of instances of crimes that were done in in low income communities,
but also ethnic communities. And because in some sense, these groups are already over police threat,
there's a lot of focus and obsession over over sort of low impact crimes that are perpetuated by these groups, right?
And and so the overpoliced nature of these groups then gets fed in to the classifier,
which then learns that these groups are then perpetrators of these crimes and therefore creates a classifier that encourages the continued policing,
the output of the classifiers to predict the location, and then that output of the location.
It is used by the by the police department to then send more forces so that you can then survey and catch more crimes.
And so their gold standard of behaviour was the number of crimes.
Court, rather than something more fair, which is whether, you know, for example, whether all groups were sort of surveyed equally.
And so as a result, you end up with a perpetual feedback loop of these groups being over surveilled.
This is them then predicting that you need to survey the groups more and so on and so forth.
At no point dissolve them. Go, wait. What about these other people, right?
In the same way, you have a different kind of feedback loop happens with the sort of rating systems of of gig workers.
So, so Uber, for instance, it has the algorithm, you know, allows both the driver and the passenger to rate each other, right?
And what happens is very nefarious sort of thing where if a passenger passenger rates some driver down,
then the driver in turn gets matched with passengers who have lower ratings and who are also in turn are likely to then rate the driver down.
Right. And so in a in a perfect world where there are no biases, this is already an unstable system.
It's an unstable, positive feedback loop. Right? But in a world where there are biases, that makes it even worse.
So there's a wonderful paper called Vehicles of Discrimination, which shows that minority groups have lower Uber ratings to begin with.
And so then then that gets perpetuated and they get matched with with essentially passengers who have lower ratings,
which may or may not be the result of of just of bias.
It could also be a result of them being poorly behaving passengers, which makes their lives harder.
Right. And so that leads to an example, a positive feedback loop.
And is this is this the best we can do in society?
You know, and there's a Black Mirror episode about this, you know, and and really, I mean, we are information architects.
We're deciding how the system should support a more robust and sort of, you know, equal and an ethical society.
And yet we're creating these fragile, positive feedback loop systems that are that are really are sensitive to biases that we only know exist.
And this is why, you know, it's a problem.
Similarly, whenever we have new kinds of systems that that you know that people roll out, it's it's disproportionately low income families,
areas that that that are seen to be, you know, high in criminality, that get to be the guinea pigs to for new kinds of draconian surveillance systems.
Well, because one, it's not in my backyard, right? It's not in my see that I'm the iResearch doing this.
I don't want it in my house. So but let me go. Let me go.
Survey the, you know, the people who are the problem or the are marginalised groups.
And this makes things so much worse because they are already putting up with systems that work so badly for, you know, worse for them to begin with.
And then you have this horrible system that then provides surveillance architectures,
which which have lots of issues with control and and privacy to undermine their well-being.
Even more so, it's very so it's extremely contentious and interesting, I think, as far as the last part of the lecture.
I just want to mention that people talk about human interaction and, you know, the two terms that sound really alike that I that I want to untangle.
One we've talking about is this algorithmic bias of techno and automation.
Bias is essentially something slightly different.
It's just our general tendency to believe algorithms because they are and some seem to be either objective or
they're just too complicated that we just trust that this complicated machine must be doing something smart.
This sort of thing is, of course, automation bias is perpetuated by things like marketing campaigns like A.I. systems,
but it's also perpetuated by model inscrutability interdisciplinarity,
trying to understand the lack of understanding of what kind of complex problems might exist in the models and and and also knowing their limitations.
Unfortunately, this is the very salient example today, because just about two months ago,
there was a case where there was a drone strike in in in Afghanistan, I believe.
Or maybe it's in Iran, which resulted in a lot of civilian casualties as the result of a synthesis of a bunch of what they call bad intelligence.
But in fact, it was exactly the same intelligence. It was sort of.
The problem is it was a humans who are interpreting things as being relevant to what they
wanted was a massive combination completion of of automation bias and confirmation bias,
which resulted in the decision to blow up this vehicle, which was full of children and and civilians.
And the you know that one of the lead general, the Middle East,
came and apologised me and said we made a mistake, but blamed the intelligence rather than the system, right?
It was almost like you're blaming the data. The data were wrong. The data were bad and fundamentally it's much more.
Difficult to think about what we mean if we don't really understand what the system,
why this system is concluding, what it's doing and what the source of the data are.
How can you even make any sort of sense of what the what the output is and whether it's worth studying?
And so those everything that we've talked about turn out we're dealing with unintentional harms,
and I leave you with an idea of saying, Well, what about the deliberate side of things?
So there are many, many issues and ethics that deal with, you know,
individual gain that result from the exploitation of of various individuals and groups.
And this has led to a lot of very interesting and new stories that will be coming
out over the next two months that deal with the Facebook paper revelations,
which talk about how Facebook wilfully ignored the cases going on in various parts of the world,
including in Myanmar, where where various groups co-opted the advertising mechanism to spread lots of hate against Muslims,
as well as a very sensitive political case in Ethiopia.
Now these kinds of things that say, well, is it the algorithm or is it?
Is it the moderation?
And the point is that an algorithmic infrastructure was created and then it was co-opted by various actors to promote genocide or to create genocide.
And that sort of thing shows you the complexity of its, you know, whether it's, you know, the bias and the algorithm or it's the, you know,
if you think of things as an entire system, the ways that the system then evolve over time helps us understand what the potential harms can be.
So this this leads to a whole realm of exciting things to think about.
We are very lucky to have a new project looking at the potential to rethink some of these mechanisms can be more harm resilient.
I'm not in any position saying that we can ever build a system that will be ethical in some sense.
But can we build systems that that? Can we consider all of these sources of harms and build things that that maybe work
a little bit more robustly against the kind of biases we know are going to exist? Or perhaps we can do data collection a little bit more ethical way,
and perhaps we can be more transparent so that when ethical problems occur, we can identify them early.
Humans can say, Hey, look, this is bad, what's going on? Let's stop it.
And so that's the the task of our Oxford Martin School project that's just starting now and very happy to talk to you all about directions for that.
And we are the human centric computing group. And please, you know, talk to us and talk to me about things.
If you're if this talk, give you ideas or reactions. Yeah, let's have a little chat.
Thank you very much. Great. Thank you so much. Max, I was an absolutely trenchant talk as a round of applause, everybody.
So.