1 00:00:02,770 --> 00:00:07,540 The robot has that recording in progress, there's an A.I. somewhere hidden in the machine. 2 00:00:07,540 --> 00:00:11,150 Thank you so much, Gary, for the invitation and for the very long intro. 3 00:00:11,150 --> 00:00:17,770 Sorry, sorry. There's a grey box again, there's a grey box of obscurity. 4 00:00:17,770 --> 00:00:22,940 It is true. It's warning me that this meeting is being recorded. Yeah, thank you. 5 00:00:22,940 --> 00:00:31,060 So first of all, for the for the invitation to speak to you all today and also for everyone, to everyone, for coming. 6 00:00:31,060 --> 00:00:38,500 I hope that this will be a fun and potentially entertaining discussion about AI ethics. 7 00:00:38,500 --> 00:00:50,770 So what I wanted to do this talk to do is to provide a discussion or provocation to think about the broad context in which, 8 00:00:50,770 --> 00:00:55,370 where in which there's now a discussion around ethics. 9 00:00:55,370 --> 00:01:01,360 You, if you're at all paying attention to the EU legislation as well as UK legislation, 10 00:01:01,360 --> 00:01:07,390 there's a lot of focus these days on on AI related safety and harms. 11 00:01:07,390 --> 00:01:12,550 And the question is, so why do we have a sudden focus on on AI related safety and harms? 12 00:01:12,550 --> 00:01:13,420 Where does this come from? 13 00:01:13,420 --> 00:01:21,430 Is it is it that we've reached some sort of threshold by which now AI systems are becoming so capable that that they're becoming harmful? 14 00:01:21,430 --> 00:01:24,970 The truth is, is is a little bit more subtle than that now. 15 00:01:24,970 --> 00:01:29,830 I think it's only come to the point where we've we're starting to come to an understanding of all the 16 00:01:29,830 --> 00:01:36,670 ways that digital AI technology is shaping society and starting to understand what it's doing to society. 17 00:01:36,670 --> 00:01:42,520 And as a result, these things are all sort of being lumped under this notion of air safety and harms. 18 00:01:42,520 --> 00:01:48,250 And so what I'm trying to do in this talk is to give you a bit of a of a taster session. 19 00:01:48,250 --> 00:01:55,510 But before I do that, I wanted to make sure that so that we do talk about some uncomfortable topics in this lecture. 20 00:01:55,510 --> 00:02:01,720 So if there are if there are issues that I've listed here that are particularly triggering or that are very upsetting to you, 21 00:02:01,720 --> 00:02:10,510 you may want to either tune out for those sections of the of the talk or or or review the talk after the fact. 22 00:02:10,510 --> 00:02:17,770 As far as I really like interactive discussions, and so but I do have a lot of material I'd like to go through. 23 00:02:17,770 --> 00:02:26,080 And so what I propose that we do is if you could use the chat box to bring in comments and I will I will incorporate some of your comments as you do. 24 00:02:26,080 --> 00:02:32,560 And then at the end, let's have an open discussion and I'll try to leave at least 10 minutes at the end for us to talk about things. 25 00:02:32,560 --> 00:02:43,530 If that's OK, does that is that all right with everybody? Sounds great, sounds great to me, but I'm only one of 16 other people. 26 00:02:43,530 --> 00:02:47,190 Excellent. I imagine there are nodding people. OK, excellent. 27 00:02:47,190 --> 00:02:52,890 I see one thumbs up, the couple thumbs up. So the format of this talk is going to be a little bit like a bento box. 28 00:02:52,890 --> 00:03:00,170 So I grew up in Japan from a Japanese heritage. And so we're going to talk about bento boxes have little. 29 00:03:00,170 --> 00:03:04,560 It's a collection of of different tastes, different topics and some of them. 30 00:03:04,560 --> 00:03:07,980 We're going to go into a little bit more depth than others. You know, overall, 31 00:03:07,980 --> 00:03:11,760 my treatment of the topic is not going to be particularly satisfying because they're not 32 00:03:11,760 --> 00:03:15,840 going to be able to spend that much time getting into great depth into these discussions, 33 00:03:15,840 --> 00:03:18,780 but hopefully they'll make a collective meal. 34 00:03:18,780 --> 00:03:28,770 So, so we're going to talk about first for your talk about Trump's second talk about bias in everyday life and then sources of trouble. 35 00:03:28,770 --> 00:03:33,900 And we're going to go into that in some considerable detail. So that's going to be a substantial part of the talk. 36 00:03:33,900 --> 00:03:44,160 And then we will talk about various aspects of what people are called human interaction, which often deals with algorithm automation bias, 37 00:03:44,160 --> 00:03:51,720 our tendency to believe machines and also the ways that that relates to confirmation bias. 38 00:03:51,720 --> 00:03:57,430 And then we'll talk about in general, which is disparate, disparate impact of bad systems. 39 00:03:57,430 --> 00:04:03,250 So the way I like to start this talk is one which is a little bit provocative, 40 00:04:03,250 --> 00:04:12,400 but what I'd like to say is that the idea the term air safety has had many different connotations. 41 00:04:12,400 --> 00:04:15,490 And from about three years ago, 42 00:04:15,490 --> 00:04:23,950 when I proposed a new class for undergraduates in computer science on on computer A.I. ethics and responsible innovation, 43 00:04:23,950 --> 00:04:29,170 there was a certain strong connotation that this had to do with one of these three problems. 44 00:04:29,170 --> 00:04:32,980 And what I'd like to say now is that these are not the topics we're going to talk about, 45 00:04:32,980 --> 00:04:39,400 because these intellectual what I call privileged distractions poor and nightmare fantasies for from well-known 46 00:04:39,400 --> 00:04:45,040 Oxford philosophers are things that essentially are really are they're not that they're not interesting, 47 00:04:45,040 --> 00:04:51,820 they're profoundly interesting, but they're sort of like mind viruses that will sort of suck all of the oxygen out of the air and 48 00:04:51,820 --> 00:04:57,220 prevent us from being able to think or talk about things that are actually much more relevant problems. 49 00:04:57,220 --> 00:05:02,740 And these are, you know, first of all, the superintelligence hypothesis, 50 00:05:02,740 --> 00:05:12,010 which states that we will have general intelligence that will enslave humanity because they will start to become exponentially self improving. 51 00:05:12,010 --> 00:05:17,800 The second one is that we have to solve the trolley problem before we can make any progress. 52 00:05:17,800 --> 00:05:24,010 Self-driving cars, autonomous systems because you need to be able to weigh the value of human lives. 53 00:05:24,010 --> 00:05:30,640 And the third one is the the idea that that information itself exerts a moral claim. 54 00:05:30,640 --> 00:05:40,360 And therefore, when you think about whether potentially information is sentient or alive in some sense and perhaps robots should be granted rights. 55 00:05:40,360 --> 00:05:43,960 These are all really interesting things to talk about at a pub, 56 00:05:43,960 --> 00:05:49,510 and I'm more than happy to to to go drinking with you and discuss these in great length. 57 00:05:49,510 --> 00:05:57,370 But there are many reasons why these are problematic in the context of a real discussion about air safety in legislation, 58 00:05:57,370 --> 00:06:00,640 as well as in trying to encourage researchers, 59 00:06:00,640 --> 00:06:09,190 especially junior researchers, to think about air safety because they are very, very hypothetical and almost provably impossible. 60 00:06:09,190 --> 00:06:16,990 And so there's a lot of real fundamental laws that need to be really bent in order for the first one to become even feasible. 61 00:06:16,990 --> 00:06:23,050 And the trolley problem is an interesting one because it's not that it's completely irrelevant to air ethics, 62 00:06:23,050 --> 00:06:30,370 but an area that it gets applied to immediately is self-driving cars because of the of the similarity of the I'm sorry, 63 00:06:30,370 --> 00:06:33,910 I'm not going to spend a lot of problem talking about a time talking about these problems. 64 00:06:33,910 --> 00:06:39,070 I'm assuming that they're familiar to people again, just even referring to them. 65 00:06:39,070 --> 00:06:41,590 They're starting to suck all the air out of out of this talk. 66 00:06:41,590 --> 00:06:50,470 And so but what I just wanted to say is that people think of this and, you know, in relation to self-driving cars, when in fact, 67 00:06:50,470 --> 00:06:59,410 the truth is that the trolley problem is much closer to the problem of medical ethics as well as disaster related ethics. 68 00:06:59,410 --> 00:07:06,010 So when you have a finite number of resources like ICU beds and you have a disaster like a pandemic happening, 69 00:07:06,010 --> 00:07:09,550 how do you then prioritise the limited resources this is, you know, 70 00:07:09,550 --> 00:07:16,660 a grandmother's life who is a Pulitzer prise winning novelist, you know, but more important to save than, you know, 71 00:07:16,660 --> 00:07:22,690 somebody who's a younger person who has, you know, I don't know so that they start to become trolley problem ask. 72 00:07:22,690 --> 00:07:26,620 But in fact, medical assets deals with these in the most boring ways imaginable. 73 00:07:26,620 --> 00:07:30,460 So as borrowing, but also conservative ways, which actually kind of makes sense. 74 00:07:30,460 --> 00:07:35,230 And so there even still, the trolley problem is not particularly relevant. 75 00:07:35,230 --> 00:07:41,710 So what are what are we going to talk about for the rest of this lecture? 76 00:07:41,710 --> 00:07:44,900 So we like we in the computer science department, 77 00:07:44,900 --> 00:07:50,770 in my research group and also the class that I teach first year undergraduates like to focus on harms and problems 78 00:07:50,770 --> 00:07:58,440 that deliberately or accidentally arise from our sort of relentless march towards increasing automation. 79 00:07:58,440 --> 00:08:04,230 And, you know, deliberate harms usually are emerging from using technology as a smokescreen to try to 80 00:08:04,230 --> 00:08:11,910 exploit people and to manipulate people for for individual gain and accidental harms are 81 00:08:11,910 --> 00:08:16,380 different class of things that usually result from us doing the things that seem exciting 82 00:08:16,380 --> 00:08:21,810 or good and then actually accidentally end up doing something unintentionally bad. 83 00:08:21,810 --> 00:08:28,140 But both of those classes of harms often disproportionately affect minorities, 84 00:08:28,140 --> 00:08:33,120 disadvantaged groups, low paid workers who are already exploited and overlooked. 85 00:08:33,120 --> 00:08:40,110 In some sense, this is not a coincidence. It's nothing to do with these groups or individuals that ourselves. 86 00:08:40,110 --> 00:08:44,250 It's that these groups are already overlooked in the design of these systems. 87 00:08:44,250 --> 00:08:53,100 They were once again overlooked, or perhaps some nefarious billionaire decided to build a social network that would really improve their own wealth. 88 00:08:53,100 --> 00:09:05,130 And that was itself sort of, you know, sort of a selfish action which then ended up exploiting the vulnerabilities of many. 89 00:09:05,130 --> 00:09:08,250 Now there are many books that have been written recently about particular groups, 90 00:09:08,250 --> 00:09:18,780 and these are the textbooks that I that I recommend for understanding specific groups and the ways that they are being marginalised by technology. 91 00:09:18,780 --> 00:09:28,680 Let's start dive in to a particular example. I wonder if people remember or experience the Twitter cropping scandal of October 2020. 92 00:09:28,680 --> 00:09:35,910 So explain what this what this is. So if you if you take the image on the left is is one image. 93 00:09:35,910 --> 00:09:40,920 I don't know if you can see the border around it. What it is is. So basically, 94 00:09:40,920 --> 00:09:48,300 imagine taking two very long images splitting so so that you have one that has 95 00:09:48,300 --> 00:09:53,250 Mitch McConnell on the top and Barack Obama on the bottom and the other one. 96 00:09:53,250 --> 00:09:57,150 You have Barack Obama on top and Mitch McConnell on the bottom. So they're otherwise identical. 97 00:09:57,150 --> 00:10:03,780 It's just the faces are swapped. OK, so they're both very long, skinny images. 98 00:10:03,780 --> 00:10:11,600 OK. Say that you posted them separately side on Twitter. 99 00:10:11,600 --> 00:10:17,540 What would happen, so what happens is you end up with the Twitter algorithm, 100 00:10:17,540 --> 00:10:23,660 goes and figures out because it's a long, skinny image and needs to figure out how to crop the image. 101 00:10:23,660 --> 00:10:31,180 Now, the thing that most people didn't realise is that it doesn't just do this by chopping the middle of the image or doing something deterministic. 102 00:10:31,180 --> 00:10:36,370 It uses an algorithm to try to figure out what's the most interesting part of the image. 103 00:10:36,370 --> 00:10:41,050 And this algorithm was found to simply select white people. 104 00:10:41,050 --> 00:10:51,460 Right? So it was it would promote, you know, that it's literally erasing, you know, preferring white faces to black faces. 105 00:10:51,460 --> 00:11:00,520 And the result is that you end up with two beautiful faces of Mitch McConnell instead of our great, you know, former president Barack Obama. 106 00:11:00,520 --> 00:11:04,720 And this is a real problem, right? So the way that this sort of worked was as follows. 107 00:11:04,720 --> 00:11:08,200 There was there lot there is the photo gets processed by model, 108 00:11:08,200 --> 00:11:15,310 which produces as a predicted salience map and that is used to crop the area that's of highest salience. 109 00:11:15,310 --> 00:11:21,040 So this created an immense sort of response that say, why is this doing this right? 110 00:11:21,040 --> 00:11:30,820 So and this the result of was an investigation by the Twitter team, which deals with the ethics and responsible innovation team. 111 00:11:30,820 --> 00:11:36,070 And they were looking at several problems in this in this cropping algorithm. 112 00:11:36,070 --> 00:11:44,290 One of them was indeed unequal treatment of different groups, not just of different skin colour, but also gender. 113 00:11:44,290 --> 00:11:49,750 And the second one was so-called objectification biases dealt with, 114 00:11:49,750 --> 00:11:59,890 otherwise known as male gaze about biases in which photos of women were cropped not around faces, but around parts of the body. 115 00:11:59,890 --> 00:12:08,590 And these kinds of issues they then evaluated with dataset and were able to discover exactly what the biases were in the algorithm. 116 00:12:08,590 --> 00:12:14,710 Now this is the kind of thing that I'd like to talk about, because it's an example that's immediately upsetting, right? 117 00:12:14,710 --> 00:12:23,350 And it's important to sort of try to to articulate what is upsetting about this and what might happen if it gets unchecked. 118 00:12:23,350 --> 00:12:28,570 You know, we should we should try to rationalise our feelings around this kind of thing. 119 00:12:28,570 --> 00:12:33,670 Another reason why it's upsetting is that it's one of these algorithms that we didn't even know existed. 120 00:12:33,670 --> 00:12:40,330 It's one of these things that it's like, what? Why did you even have to add the algorithm if you had done nothing whatsoever? 121 00:12:40,330 --> 00:12:49,450 Simply the image in the middle, you would have avoided literally erasing or hiding, you know, already marginalised groups of people. 122 00:12:49,450 --> 00:12:57,910 And the point is that that these sorts of algorithms are now being rolled out and massive scale everywhere amongst the systems that we use every day. 123 00:12:57,910 --> 00:13:07,960 And and so as a result, we are perpetually surrounded by algorithms that have potentially very harmful biases in them and harmful properties, 124 00:13:07,960 --> 00:13:11,290 not just biases, but properties that we don't even realise. 125 00:13:11,290 --> 00:13:17,170 And so, you know, and that leads to the real question of of really, what can we do about this? 126 00:13:17,170 --> 00:13:22,960 And really, you know, is there an opportunity to fix all such problems? 127 00:13:22,960 --> 00:13:32,080 Is this a problem? You know, is this something that we have to do post hoc or can we start to do this in sort of an anticipatory way? 128 00:13:32,080 --> 00:13:40,720 And so that's sort of the reason why I proposed a course in in for computer science that will because we believe that that 129 00:13:40,720 --> 00:13:47,780 computer scientists are amongst all the stakeholders who should really be involved in the process of computer engineers. 130 00:13:47,780 --> 00:13:51,520 You know, people who are thinking about the algorithms, machine learning researchers, 131 00:13:51,520 --> 00:13:56,110 statisticians that such as yourselves who are thinking about the various and rich contexts that these 132 00:13:56,110 --> 00:14:03,340 things may be deployed and told to stand the kind of risks that that might arise from from their use. 133 00:14:03,340 --> 00:14:05,470 So we use the term algorithmic bias, 134 00:14:05,470 --> 00:14:14,140 and here I just wanted to say that we use this sort of broad definition of bias rather than the statistical definition of bias. 135 00:14:14,140 --> 00:14:24,910 We refer to algorithmic bias as essentially in general, systematic or repeatable errors in a computer system that create unfair outcomes that work 136 00:14:24,910 --> 00:14:32,360 differently for different groups that end up privileging one group arbitrarily over another. 137 00:14:32,360 --> 00:14:40,870 And typically, you end up with this these kinds of errors and problems ending up in two camps. 138 00:14:40,870 --> 00:14:46,660 One is probably the most familiar to all of you that deals with sort of allocate of harms, 139 00:14:46,660 --> 00:14:49,750 which means that it's when a system provides different groups, 140 00:14:49,750 --> 00:14:58,750 unequal opportunities, resources or capabilities, and the other class is one that it's less discussed in the aid fairness literature. 141 00:14:58,750 --> 00:15:05,590 And because it's not directly statistics statistical property of of machine learning systems, but that is called a representational harm. 142 00:15:05,590 --> 00:15:16,480 A representation of harm is something that where an algorithmically curated or created depiction of somebody or something or a group is harmful, 143 00:15:16,480 --> 00:15:21,250 and we'll give some examples of that. Let's first talk about allocate of harms. 144 00:15:21,250 --> 00:15:29,770 Give me a second here. So I look out of harms you've probably heard of in the context of many kind of high stakes decision making contexts like, 145 00:15:29,770 --> 00:15:38,560 for example, you know, predictive policing systems, the judicial process, the criminal recidivism systems. 146 00:15:38,560 --> 00:15:49,540 But allocating harms exist in all sorts of shapes and in routine systems, for example, things as simple as a speech recognition algorithm has, 147 00:15:49,540 --> 00:15:55,990 you know, speech recognition algorithms, which we use very routinely, have been shown to have various kinds of addictive harms. 148 00:15:55,990 --> 00:16:02,620 And in that the recognition rates are not evenly distributed across groups. 149 00:16:02,620 --> 00:16:09,340 So there are many groups for which recognition rates are much lower than than other groups. 150 00:16:09,340 --> 00:16:16,720 And you know, this is not this might seem like it's not terribly surprising because people speak so differently, 151 00:16:16,720 --> 00:16:21,640 but we're talking about again, systematic differences between groups in particular. 152 00:16:21,640 --> 00:16:24,790 You know, the highest performing speech recognition engine, 153 00:16:24,790 --> 00:16:30,580 which you know this this study was a little bit obsolete from two years ago, but believe it still is. 154 00:16:30,580 --> 00:16:38,500 Either Google or IBM was demonstrated to be 13 percent more accurate for men in general than for women. 155 00:16:38,500 --> 00:16:43,090 But this this difference might. This difference is significant and interesting, 156 00:16:43,090 --> 00:16:53,200 and it's even greater for people who have different accents or different ways of speaking and different backgrounds. 157 00:16:53,200 --> 00:16:59,530 And so that is something to note. But unfortunately, it's not just space speech recognition, 158 00:16:59,530 --> 00:17:04,120 it's basically every sort of recognition that kind of algorithm or system that we use today, 159 00:17:04,120 --> 00:17:09,430 including those for basic face detection as well as recognition. 160 00:17:09,430 --> 00:17:17,410 And of course, one of the sources of this, it's not that faces of people who have different ethnic backgrounds are harder to recognise. 161 00:17:17,410 --> 00:17:22,210 It's that largely this is entirely the fault of the data sets that we are using. 162 00:17:22,210 --> 00:17:26,980 The data that we're using are not even the distributed amongst amongst groups. 163 00:17:26,980 --> 00:17:35,980 And as a result, the systems just by default perform worse for those groups for which it has less data. 164 00:17:35,980 --> 00:17:39,670 So you might say, well, what's a few percentage difference anyway? 165 00:17:39,670 --> 00:17:46,360 Does that make any difference? Well, everybody in the room, especially in the private statistics, now know that this is a major problem, 166 00:17:46,360 --> 00:17:56,710 especially as you try to run an algorithm on millions of faces or run a system over and over again for something that's quite a critical thing. 167 00:17:56,710 --> 00:18:06,850 For example, even a small chance of error can cause a lot of false positives if you do a detection problem enough. 168 00:18:06,850 --> 00:18:13,270 So the ACLU has done a lot of great activism with with with government procured 169 00:18:13,270 --> 00:18:19,660 face detection and recognition systems to demonstrate how badly they perform. 170 00:18:19,660 --> 00:18:27,790 So. So just like in the U.K., in the U.S., there are many policing departments that we're starting to use commercial facial recognition systems to try 171 00:18:27,790 --> 00:18:34,210 to identify and spot potential criminals and to make a point of this to make a point that the in particular, 172 00:18:34,210 --> 00:18:43,750 the ways that marginalised people are disproportionately affected by false positives is that they applied this algorithm to members of Congress and 173 00:18:43,750 --> 00:18:53,410 found that black members of Congress Black members of Congress were disproportionately mis recognised as criminals than by the algorithm and saying, 174 00:18:53,410 --> 00:18:56,320 you know, this algorithm is clearly not fit for purpose. 175 00:18:56,320 --> 00:19:06,010 So these are examples of the kind of things that even a small percentage of chances of error can really cause a very uncomfortable problem, 176 00:19:06,010 --> 00:19:10,480 which amplifies an already delicate situation. 177 00:19:10,480 --> 00:19:19,990 Now think about then. So these these kinds of things, so these recognition systems are embedded in complex systems and often, 178 00:19:19,990 --> 00:19:23,950 you know, that have multiple such recognition systems, Inc. 179 00:19:23,950 --> 00:19:33,880 So what happens when we embed these systems in the kinds of everyday tools that you need to do your work or to survive to live, right? 180 00:19:33,880 --> 00:19:42,610 So if everything from the, you know, the sort of soap dispensers that we use to the face authentication systems we use have these biases. 181 00:19:42,610 --> 00:19:46,150 What does it mean? It means that the systems just don't work as well. 182 00:19:46,150 --> 00:19:53,980 For certain groups of people in particular, it doesn't work as well for people who are already marginalised that you know, 183 00:19:53,980 --> 00:20:02,080 have have are less frequently at higher positions and companies and are paid less have higher unemployment rates and things like that. 184 00:20:02,080 --> 00:20:11,590 And so you're making a bad situation worse if your fundamental digital tools in your environment disproportionately work more poorly for these groups. 185 00:20:11,590 --> 00:20:17,920 And and that's something to think about. So the representative harms, let's talk about that for a moment. 186 00:20:17,920 --> 00:20:24,250 So again, representative harms are not exactly the same as as allocate apartments where you have different groups for which an error rate maybe, 187 00:20:24,250 --> 00:20:31,420 maybe disproportionately affecting one group rather than others. Examples of representational harms are, for example. 188 00:20:31,420 --> 00:20:38,830 Professor Latanya Sweeney, who is a former chief of the Federal Trade Commission. 189 00:20:38,830 --> 00:20:47,590 She discovered that she was doing an early study of how African-American names were more frequently associated on Google, 190 00:20:47,590 --> 00:20:52,630 with adverts suggesting arrest than non-African-American names. 191 00:20:52,630 --> 00:20:59,890 And they systematically ran a bunch of African-American baby names and sort of white baby names. 192 00:20:59,890 --> 00:21:04,000 And, you know, and measured exactly to what extent this was the case. 193 00:21:04,000 --> 00:21:10,150 And you know, the thing is, so the problem, the question that she asked is So, so why? 194 00:21:10,150 --> 00:21:19,570 Why? Why is this? Why did this happen? Is it because the clients paid to have the ads do this specifically targeting black names? 195 00:21:19,570 --> 00:21:27,980 Or is it some sort of, you know, through some complex feedback system that the system learnt a racist association between? 196 00:21:27,980 --> 00:21:31,000 And the reason why this is the representation of harm is that it's one of these 197 00:21:31,000 --> 00:21:34,930 things where you are a you are subliminally suggesting that for some reason, 198 00:21:34,930 --> 00:21:38,980 if you have an African-American name, you're more likely to be associate with criminality. 199 00:21:38,980 --> 00:21:44,800 Right? It's a represent. It's a harm that that is associated is that it's associated with that depiction. 200 00:21:44,800 --> 00:21:50,140 But we see many kinds of representational harms of many sorts all the time you think about it. 201 00:21:50,140 --> 00:21:54,460 For example, the gender bias and Google image search, a lot of people have studied this. 202 00:21:54,460 --> 00:21:59,590 So if you type in nurse, you often get pictures of of white women. 203 00:21:59,590 --> 00:22:08,170 And if you type in doctor you get, then you know, more often pictures of of white men and CEO is sort of old white guys. 204 00:22:08,170 --> 00:22:16,570 And the question is, these are representations that then, you know, are already contain huge sort of biases and luggage, right? 205 00:22:16,570 --> 00:22:20,800 And so the question is why where are the where are the black women, what we know, 206 00:22:20,800 --> 00:22:27,340 what's going on here that sort of reinforces the stereotype that CEOs look like these things doctors and nurses look like these things. 207 00:22:27,340 --> 00:22:35,430 And this is a form of representational harm. Now, it's more difficult to try to think about how you characterise that in terms of. 208 00:22:35,430 --> 00:22:41,700 San Francisco property, because often it's a lot more subtle to identify what it is that you need to try to look for. 209 00:22:41,700 --> 00:22:47,590 If you're going to try to find these disparities and these representational issues. 210 00:22:47,590 --> 00:22:53,900 So let's talk specifically in context of algorithmic decision making before go on. 211 00:22:53,900 --> 00:22:58,330 I don't see any comments that everything OK going forward. 212 00:22:58,330 --> 00:23:03,610 And you know, the quick responses. All right, great. So let's let's go on to algorithmic decision making, Solomon, 213 00:23:03,610 --> 00:23:12,520 decision making is is a whole sort of new world that I think has brought a lot of these issues to the forefront, 214 00:23:12,520 --> 00:23:21,490 which deals with essentially the application of algorithmic methods to support humans in decision making processes. 215 00:23:21,490 --> 00:23:28,120 So, so that's often that's called decision support, where essentially you still have an expert there. 216 00:23:28,120 --> 00:23:34,570 But now a system is providing advice to make the person able to make perhaps more accurate decisions in 217 00:23:34,570 --> 00:23:41,620 some sense or to help them go through more cases more quickly or in place of the human expert entirely. 218 00:23:41,620 --> 00:23:47,080 And that's what we call automated decision making. And there are many, 219 00:23:47,080 --> 00:23:55,570 many examples where algorithmic decision making systems are affecting and articulating and many invisible parts of our lives in ways that 220 00:23:55,570 --> 00:24:02,980 you probably and I don't realise every day and this is where the title of my talk victims of algorithmic violence really comes from. 221 00:24:02,980 --> 00:24:08,260 Right. Nearly every aspect of your life is now being judged right by by algorithms. 222 00:24:08,260 --> 00:24:14,240 One of the earliest areas for automated decision-making was making. It was, you know, an applicant cutting edge application. 223 00:24:14,240 --> 00:24:22,030 Machine learning was in credit risk prediction. You know, even in the 80s and 90s, there was early machine learning that was used for for credits. 224 00:24:22,030 --> 00:24:24,010 And of course, that's being used today. 225 00:24:24,010 --> 00:24:29,320 And, you know, more than ever before, we have things like, you know, whether you're going to get a loan or a mortgage or, 226 00:24:29,320 --> 00:24:34,330 you know, or even be able to get a mobile phone plan uses a huge number of features, 227 00:24:34,330 --> 00:24:42,700 some of which are derived from completely non-financial features like lifestyle features and other big data features, which is very interesting. 228 00:24:42,700 --> 00:24:48,880 Differential pricing. You know, you might not realise that you know that the cost of your flight or your hotel 229 00:24:48,880 --> 00:24:53,110 room really depends on how much the surveillance architecture knows about you. 230 00:24:53,110 --> 00:24:58,810 You know, if it knows that you want, you might spend more money. It may differentially price you into a higher price tier. 231 00:24:58,810 --> 00:25:03,670 And actually, for the exact same product, you will be advertised a higher price. 232 00:25:03,670 --> 00:25:08,320 Law enforcement You've probably seen many examples of the ways that predictive policing systems, 233 00:25:08,320 --> 00:25:21,550 as well as you know, have helped helped and also armed groups to try to distribute the resources of policing, 234 00:25:21,550 --> 00:25:30,460 force it within areas particularly known for high higher crime, criminal justice, job hiring, 235 00:25:30,460 --> 00:25:36,490 the gig economy, everything from Uber Eats to Uber, you know, to Deliveroo and Uber itself. 236 00:25:36,490 --> 00:25:43,480 You know, all of these aspects, you know, their algorithms that articulate what clients you're going to get for your next job, 237 00:25:43,480 --> 00:25:52,480 you know how you are going to be seen by others and whether or not you can continue to make a livelihood of these, you know, out of out of this job. 238 00:25:52,480 --> 00:26:00,860 And increasingly, the kind of gig work algorithm uber driven kinds of stuff is making its way into into brick and mortar establishments. 239 00:26:00,860 --> 00:26:06,400 And so now we have lots of surveillance systems that go and monitor worker productivity, especially during the pandemic, 240 00:26:06,400 --> 00:26:12,370 to try to measure whether or not you are being productive and whether or whether you should be fired 241 00:26:12,370 --> 00:26:16,420 and whether it should hire another associate professor of human computer interaction instead. 242 00:26:16,420 --> 00:26:19,780 And so these are the sorts of things are amazing, even even any prioritisation. 243 00:26:19,780 --> 00:26:25,690 So when you try to call 999 or when you walk into a hospital, whether you end up, you know, 244 00:26:25,690 --> 00:26:32,830 waiting there for six hours or not is due to some algorithm that's trying to optimise resource in some sense. 245 00:26:32,830 --> 00:26:37,480 And so so this leads to a lot of interesting questions about which of these algorithms have biases. 246 00:26:37,480 --> 00:26:41,110 You know, why are they using features? You know, what kind of features are they using? 247 00:26:41,110 --> 00:26:48,910 Exactly. You know, you would hope that the algorithm wasn't using your Twitter feed to try to determine whether or not you deserve to be operated on. 248 00:26:48,910 --> 00:26:52,030 You know, although for me, it would probably prior to no, just kidding. 249 00:26:52,030 --> 00:26:58,960 But you know, there's so many, so many questions that that it leads you to ask about this. 250 00:26:58,960 --> 00:27:04,810 And so this is my algorithmic decision making is is so important because because these 251 00:27:04,810 --> 00:27:09,910 algorithms are working opaquely behind the scenes of everything that we do and and nobody's 252 00:27:09,910 --> 00:27:15,850 really discussing exactly whether we should be using these algorithm and what it's fair 253 00:27:15,850 --> 00:27:19,840 for these systems to actually conclude based upon the information that they have. 254 00:27:19,840 --> 00:27:24,910 So this and so let's just back up for a second say, so what are we talking about the context so previously? 255 00:27:24,910 --> 00:27:28,420 Each of these contexts were were done by humans, right? 256 00:27:28,420 --> 00:27:34,870 So there was so there were human experts for each of these contexts, whether it was applying for a mortgage or applying for a job or deciding whether 257 00:27:34,870 --> 00:27:40,000 or not somebody was guilty or deserved parole was done by human experts who were. 258 00:27:40,000 --> 00:27:43,840 And what we're doing here and in our decision making is saying, well, 259 00:27:43,840 --> 00:27:52,720 we're going to replace that or augment that in some sense with some system, with some data driven system that is used to extrapolate from data. 260 00:27:52,720 --> 00:28:07,060 So what we really mean by that is the way that that works is usually what happens is they start with a dataset or of real cases that were evaluated. 261 00:28:07,060 --> 00:28:11,020 And then what happens is that based upon that dataset, 262 00:28:11,020 --> 00:28:17,800 then that's that's sort of those cases are transformed into some machine readable representation 263 00:28:17,800 --> 00:28:23,230 and those that representation is typically called a feature set or or feature representation. 264 00:28:23,230 --> 00:28:30,700 And then what the system, what the human decided to do in that context, whether it was to give you a loan or not, 265 00:28:30,700 --> 00:28:37,270 is then considered the label and that's associated with those features. And you do that for all of those examples, right? 266 00:28:37,270 --> 00:28:44,950 And then you feed things into some complex, usually or hopefully not too complex. 267 00:28:44,950 --> 00:28:53,670 Model and the result of this model is something that can then for new instances, 268 00:28:53,670 --> 00:29:00,250 judge based upon its representation, what kind of label a human would have given. 269 00:29:00,250 --> 00:29:03,640 So that is a very cartoon fresh, 270 00:29:03,640 --> 00:29:09,550 fresher and computer science picture of what a machine learning system that is algorithmic decision making does, right? 271 00:29:09,550 --> 00:29:14,530 And the question is OK. So that seems fine, right? 272 00:29:14,530 --> 00:29:18,890 That's fine. It's nothing. Nothing wrong here. What could possibly go wrong? 273 00:29:18,890 --> 00:29:26,410 Right. So so I could say there exactly, you know, four major areas where things could go wrong, right? 274 00:29:26,410 --> 00:29:31,750 And there various different coloured blocks here going through very quickly. 275 00:29:31,750 --> 00:29:38,260 You know that sample data that sample set coming in could be not representative. 276 00:29:38,260 --> 00:29:45,160 There could be it could change over time. Right? You know, you could have you might have used a convenient sample. 277 00:29:45,160 --> 00:29:53,980 You may not, you know, so that that's to begin with. The second thing is, so you relied on a human expert to come up with the label, right? 278 00:29:53,980 --> 00:29:59,710 That label might be noisy. You know, human experts make mistakes. 279 00:29:59,710 --> 00:30:09,880 Then the second thing about features, so you've transformed something that's a complex world into an end dimensional no right or something, right? 280 00:30:09,880 --> 00:30:15,190 And that itself should raise alarms completely. 281 00:30:15,190 --> 00:30:20,350 Because how? What? Right? Well, on what basis did you do that? 282 00:30:20,350 --> 00:30:26,290 And we'll talk about that in a second. Then you trained this monster monster sort of model. 283 00:30:26,290 --> 00:30:30,430 Right? And the model, you know, is is it's working in some ways, 284 00:30:30,430 --> 00:30:36,790 it's that this generic statistical machine that doesn't actually know anything about the world and it doesn't actually know about the cases at all. 285 00:30:36,790 --> 00:30:42,100 All it knows is the relationships between the variables that the representation that you, 286 00:30:42,100 --> 00:30:46,210 the examples and the representations that you've given, right? 287 00:30:46,210 --> 00:30:52,780 And then finally, you sort of contextualise this in society, you drop the whole thing in society and what happens. 288 00:30:52,780 --> 00:31:01,150 So there's five areas that are worth discussing discussing here, and I'm sure that none of these things are terribly new to all of you. 289 00:31:01,150 --> 00:31:05,350 But but. But but. And so. So let's talk a little bit about a couple of them. 290 00:31:05,350 --> 00:31:17,950 So the problem with the future transformation? So there's a very, very old I theory theorem that says that's called a frame problem that says. 291 00:31:17,950 --> 00:31:27,020 If so, the problem of choosing a representation for a problem is at least as hard as solving the problem. 292 00:31:27,020 --> 00:31:31,190 OK, so so if you need to if you want to figure out what you need to represent, 293 00:31:31,190 --> 00:31:35,060 you already need to have figured out a way to solve the problem before you. 294 00:31:35,060 --> 00:31:40,580 So, so in other words, it's sort of a thing that says choosing the right representation is tremendously hard problem, 295 00:31:40,580 --> 00:31:47,210 and in particular because it's impossible to figure out what to represent versus what to not represent. 296 00:31:47,210 --> 00:31:52,820 You know, a butterfly in South America may be relevant for a particular court case, but it may be completely relevant. 297 00:31:52,820 --> 00:32:00,020 Nine point nine nine nine Other other cases in the world The amazing thing about human beings is that we don't need future representations. 298 00:32:00,020 --> 00:32:06,290 We are very flexible creatures that we can bring in evidence as in when we need it and reshape our representations. 299 00:32:06,290 --> 00:32:10,420 But the kinds of systems that we're building now can't do that. 300 00:32:10,420 --> 00:32:17,020 There's a wonderful, wonderful case called the hungry judges problem about, so we're talking about the fallibility of human experts. 301 00:32:17,020 --> 00:32:30,460 This this case, which has been hotly debated, demonstrated that judges are far more likely to be harsh on a particular cases, right? 302 00:32:30,460 --> 00:32:34,420 Sort of before lunch than they were and the first thing in the morning. 303 00:32:34,420 --> 00:32:43,120 Now this is completely, you know, even if you, you know, so this is assuming that the cases came in randomly and that there shouldn't be. 304 00:32:43,120 --> 00:32:48,210 The cases we're not ordered such that the hardest or most ambiguous cases were closest to light. 305 00:32:48,210 --> 00:32:53,900 So, so you know, the hypothesis is when people get hungry, do they get more mean? 306 00:32:53,900 --> 00:32:58,900 You know, and I could definitely reflect on my own behaviour and tell you that perhaps that's the case. 307 00:32:58,900 --> 00:33:07,240 But there have been other kinds of things about similar decisions that were made in NHS contexts where misdiagnoses were, 308 00:33:07,240 --> 00:33:11,410 you know, correlated with being tired or, you know, being hungry. 309 00:33:11,410 --> 00:33:15,610 And so humans are fallible. And of course, we're using that. 310 00:33:15,610 --> 00:33:18,040 You were plugging that into a machine learning algorithm, 311 00:33:18,040 --> 00:33:24,010 which is then trying to reproduce hangry ness in our population, which I think is quite problematic. 312 00:33:24,010 --> 00:33:28,900 And this one is one that it may be very basic, too basic for for this group, 313 00:33:28,900 --> 00:33:32,590 but at one that I always talk about, which I think is really interesting on this. 314 00:33:32,590 --> 00:33:37,660 This case was from an example from Microsoft Research where they were. 315 00:33:37,660 --> 00:33:44,140 There was a research, very, very early experiment in neural networks where they collected what was at the in the 90s, 316 00:33:44,140 --> 00:33:49,450 a very, very big dataset of of fifteen fourteen thousand one hundred ninety nine patients. 317 00:33:49,450 --> 00:33:53,860 And what they were trying to predict was they're trying to the dataset consisted of 318 00:33:53,860 --> 00:34:01,240 pneumonia patients and whether or not they they died within sort of 24 hours of a diagnosis. 319 00:34:01,240 --> 00:34:07,270 And the reason why they did this is they were trying to model this is to understand for pneumonia patients, 320 00:34:07,270 --> 00:34:12,280 it's often a very subtle interaction between sort of symptoms that might influence 321 00:34:12,280 --> 00:34:16,000 whether somebody would go bad on somebody's condition or deteriorate quickly, 322 00:34:16,000 --> 00:34:20,020 or whether you can just send them home with antibiotics and they'll be fine. 323 00:34:20,020 --> 00:34:22,960 And if you do, if you're able to anticipate this well, 324 00:34:22,960 --> 00:34:29,140 then you can be sure to keep those patients that are going to deteriorate in-hospital while you can send the ones 325 00:34:29,140 --> 00:34:35,410 who are going to be likely to be fine home so that you don't end up using hospital resources unnecessarily. 326 00:34:35,410 --> 00:34:39,970 And so. So the models so they decided they were going to build an early neural network 327 00:34:39,970 --> 00:34:43,720 based upon this very large dataset of of pneumonia patients and a number 328 00:34:43,720 --> 00:34:54,580 of features of each critical features that doctors thought were essential components of whether or not determiners of of of a patient's outcome. 329 00:34:54,580 --> 00:34:59,980 And they trained neural networks that seemed to have high accuracy for the time 85 percent accuracy. 330 00:34:59,980 --> 00:35:06,010 And they asked the researcher, OK, should we run this in production? Should we start deploying this in practise? 331 00:35:06,010 --> 00:35:13,840 And the very interesting thing is that the researchers said absolutely not because we have no idea what the system is learning. 332 00:35:13,840 --> 00:35:18,100 And people came up to him and said, What do you mean? What do you mean? 333 00:35:18,100 --> 00:35:25,360 We don't know what the system is starting 20 years later? They wrote a paper so very recently about. 334 00:35:25,360 --> 00:35:31,960 So I believe it was 2016 about a system that of a model where they could they could actually 335 00:35:31,960 --> 00:35:37,120 take apart the individual contributions of what was going on within that neural network. 336 00:35:37,120 --> 00:35:44,770 It's a slightly modified kind of network called general generalised additive linear modules models at the end. 337 00:35:44,770 --> 00:35:49,450 And what's great about this model is that it is purely additive, right? 338 00:35:49,450 --> 00:35:56,230 So you can look at the individual contributions that the model has learnt and take apart what why you know, 339 00:35:56,230 --> 00:36:01,240 the features that it believes are contributing to your likelihood of probability of death. 340 00:36:01,240 --> 00:36:06,190 And then it's far easier to interpret. And what's interesting is that so you know, 341 00:36:06,190 --> 00:36:15,220 you can look at all the features that it took in and then look at each feature one at a time to see how it contributes the probability of death. 342 00:36:15,220 --> 00:36:20,710 And they found very interesting things, so they found a couple of things that immediately raised red flags to the researcher, 343 00:36:20,710 --> 00:36:27,580 and one is that they were sort of sharp increases in probability of death that occurred at round numbers like, 344 00:36:27,580 --> 00:36:32,470 for example, that occurred at 65 years old, that occurred at 85 years old. 345 00:36:32,470 --> 00:36:34,990 And then they also found something really peculiar at the very end, 346 00:36:34,990 --> 00:36:42,130 which which was that the model was predicting that a person's probability death would go down significantly after 100. 347 00:36:42,130 --> 00:36:49,420 Right. And so this led to some very sort of some head scratching and would this be worrisome so. 348 00:36:49,420 --> 00:36:56,560 So this is great that this can lead to some great bar game where we go and say, Well, what does this mean? 349 00:36:56,560 --> 00:36:58,720 You know, why are we seeing this kind of thing? 350 00:36:58,720 --> 00:37:08,620 And the hypothesis was immediately that, you know, after a couple of rounds that essentially it's 65 and at least in the U.S., 351 00:37:08,620 --> 00:37:12,790 your insurance, you're sort of you go to retire retirement retirement mode. 352 00:37:12,790 --> 00:37:17,260 And often this means that your you change insurance providers, right? 353 00:37:17,260 --> 00:37:21,220 And so then what happens is that your quality of care changes. 354 00:37:21,220 --> 00:37:22,300 So you may have had a really, 355 00:37:22,300 --> 00:37:27,640 really great insurance provider and then you retire and now you're sort of switch to Medicare and Medicaid and things like that. 356 00:37:27,640 --> 00:37:33,130 And so then it could be that your probability of death, then as a result of the quality of care goes up significantly. 357 00:37:33,130 --> 00:37:37,960 It's not that you turn 65 and suddenly instantly you are more likely to die. 358 00:37:37,960 --> 00:37:40,870 Of course, that doesn't happen. Human beings don't work that way, 359 00:37:40,870 --> 00:37:47,920 but it's something about the fact that people are retiring and switching health care providers, but that might be doing that. 360 00:37:47,920 --> 00:37:51,970 The 101 is very interesting to the idea there is that they said, Well, 361 00:37:51,970 --> 00:37:57,100 when somebody turns 100, then doctors often are like, Oh, this person's a centenarian? 362 00:37:57,100 --> 00:38:02,860 Right? You know, we're going to immediately put this person if this person presents with a fever and pneumonia, 363 00:38:02,860 --> 00:38:07,360 they're not going to think about whether or not to put the person in hospital. 364 00:38:07,360 --> 00:38:12,160 This person's going to be prioritised right regardless because they're 100 years old. 365 00:38:12,160 --> 00:38:19,990 Right. And so what you end up seeing in the model is the reflection of what people do because of their irrational decision-making process. 366 00:38:19,990 --> 00:38:23,740 And then you see the effects of people's care practises. Right? 367 00:38:23,740 --> 00:38:28,730 I mean, and so all the data that we used to train this network is reflecting this. 368 00:38:28,730 --> 00:38:35,170 So what would have happened if we had used this algorithm to decide whether or not to put people in hospital? 369 00:38:35,170 --> 00:38:38,110 We would have made really grave mistakes, right? 370 00:38:38,110 --> 00:38:44,600 We would have said, Oh, they're 100, therefore they're not going to die, and it would have made the opposite decision that we had made earlier. 371 00:38:44,600 --> 00:38:51,550 Does that make sense? So people ask me why we want the critical models, and this is the very reason why, 372 00:38:51,550 --> 00:38:57,070 because a lot of data reflects actions, reflects the things that we do in society. 373 00:38:57,070 --> 00:39:04,510 And it's important for us to be able to untangle these things because no data that we get is ever perfect. 374 00:39:04,510 --> 00:39:10,510 And it would be unethical indeed to run a trial that just said we'll just let people die so that we can collect the data. 375 00:39:10,510 --> 00:39:20,130 Right? So in this case, we have to. We were following best practises to try to keep people alive, but the result is that we end up with blurry data. 376 00:39:20,130 --> 00:39:26,910 So I know I'm running out of running a little bit of time, I think I've got a little bit more time left here. 377 00:39:26,910 --> 00:39:34,890 So the other the last thing that we're talking about is feedback loops to the ways that the systems get dropped into a society. 378 00:39:34,890 --> 00:39:42,270 So I know you are all control theory experts, maybe at this point, but they're sort of there are two kinds of feedback loops. 379 00:39:42,270 --> 00:39:46,830 One is this sort of good kind and bad kind of positive feedback loop is the bad 380 00:39:46,830 --> 00:39:53,010 kind where you end up with any small change ends up being magnified amplified. 381 00:39:53,010 --> 00:39:59,610 So, so this is the kind of thing that results in the squeak when you are in a auditorium and the microphone 382 00:39:59,610 --> 00:40:04,860 gets picked up by the speakers and these speakers emit a sound and that gets picked up by the microphone, 383 00:40:04,860 --> 00:40:09,510 which then gets amplified by the speakers again and you end up deafening everybody in the room. 384 00:40:09,510 --> 00:40:11,460 That's a kind of positive feedback. 385 00:40:11,460 --> 00:40:21,390 Negative feedback is where essentially the output is used to, then to then make a response that compensates for that. 386 00:40:21,390 --> 00:40:24,900 So this is the way that drones can fly as they believe, 387 00:40:24,900 --> 00:40:30,120 because we have and we take that sort of if you have a strong wind that pushes the drone to the left, 388 00:40:30,120 --> 00:40:37,050 the control system then goes and uses that feedback to then compensate for it to come back. 389 00:40:37,050 --> 00:40:41,460 I don't know why it suddenly turns dark, but that's OK. So, so what? 390 00:40:41,460 --> 00:40:49,320 So what can we say about the sort of systems that we are building? So systems that we are building often create positive feedback loops. 391 00:40:49,320 --> 00:40:51,360 And why is that? 392 00:40:51,360 --> 00:41:01,710 The reason is, of course, because they're perpetuating because we're just because our systems are designed to reproduce what we do, right? 393 00:41:01,710 --> 00:41:07,770 So and warts and all. So the things that we do, it's at it's gold standard. 394 00:41:07,770 --> 00:41:15,690 It's the drive to do it, to try to do it exactly the same way. And no sense, is it trying to do it the opposite way? 395 00:41:15,690 --> 00:41:20,790 And so, for example, so this, you know, an example, and this can cause lots of problems. 396 00:41:20,790 --> 00:41:27,300 If you then embed them in a society, which then ends up perpetuating the biases that you have, 397 00:41:27,300 --> 00:41:31,590 for instance, the predictive policing systems that we alluded to earlier, 398 00:41:31,590 --> 00:41:38,220 such as PredPol, what they do is they they try to predict what areas of the city are likely to have a crime. 399 00:41:38,220 --> 00:41:45,810 Right now, the data set that they were trained on has a lot of instances of crimes that were done in in low income communities, 400 00:41:45,810 --> 00:41:51,270 but also ethnic communities. And because in some sense, these groups are already over police threat, 401 00:41:51,270 --> 00:41:58,910 there's a lot of focus and obsession over over sort of low impact crimes that are perpetuated by these groups, right? 402 00:41:58,910 --> 00:42:03,570 And and so the overpoliced nature of these groups then gets fed in to the classifier, 403 00:42:03,570 --> 00:42:12,780 which then learns that these groups are then perpetrators of these crimes and therefore creates a classifier that encourages the continued policing, 404 00:42:12,780 --> 00:42:17,880 the output of the classifiers to predict the location, and then that output of the location. 405 00:42:17,880 --> 00:42:25,350 It is used by the by the police department to then send more forces so that you can then survey and catch more crimes. 406 00:42:25,350 --> 00:42:30,250 And so their gold standard of behaviour was the number of crimes. 407 00:42:30,250 --> 00:42:39,190 Court, rather than something more fair, which is whether, you know, for example, whether all groups were sort of surveyed equally. 408 00:42:39,190 --> 00:42:43,240 And so as a result, you end up with a perpetual feedback loop of these groups being over surveilled. 409 00:42:43,240 --> 00:42:48,370 This is them then predicting that you need to survey the groups more and so on and so forth. 410 00:42:48,370 --> 00:42:52,510 At no point dissolve them. Go, wait. What about these other people, right? 411 00:42:52,510 --> 00:43:01,420 In the same way, you have a different kind of feedback loop happens with the sort of rating systems of of gig workers. 412 00:43:01,420 --> 00:43:11,800 So, so Uber, for instance, it has the algorithm, you know, allows both the driver and the passenger to rate each other, right? 413 00:43:11,800 --> 00:43:18,370 And what happens is very nefarious sort of thing where if a passenger passenger rates some driver down, 414 00:43:18,370 --> 00:43:26,320 then the driver in turn gets matched with passengers who have lower ratings and who are also in turn are likely to then rate the driver down. 415 00:43:26,320 --> 00:43:31,900 Right. And so in a in a perfect world where there are no biases, this is already an unstable system. 416 00:43:31,900 --> 00:43:37,630 It's an unstable, positive feedback loop. Right? But in a world where there are biases, that makes it even worse. 417 00:43:37,630 --> 00:43:44,800 So there's a wonderful paper called Vehicles of Discrimination, which shows that minority groups have lower Uber ratings to begin with. 418 00:43:44,800 --> 00:43:51,070 And so then then that gets perpetuated and they get matched with with essentially passengers who have lower ratings, 419 00:43:51,070 --> 00:43:54,280 which may or may not be the result of of just of bias. 420 00:43:54,280 --> 00:44:00,290 It could also be a result of them being poorly behaving passengers, which makes their lives harder. 421 00:44:00,290 --> 00:44:04,720 Right. And so that leads to an example, a positive feedback loop. 422 00:44:04,720 --> 00:44:07,870 And is this is this the best we can do in society? 423 00:44:07,870 --> 00:44:14,140 You know, and there's a Black Mirror episode about this, you know, and and really, I mean, we are information architects. 424 00:44:14,140 --> 00:44:21,760 We're deciding how the system should support a more robust and sort of, you know, equal and an ethical society. 425 00:44:21,760 --> 00:44:30,190 And yet we're creating these fragile, positive feedback loop systems that are that are really are sensitive to biases that we only know exist. 426 00:44:30,190 --> 00:44:35,050 And this is why, you know, it's a problem. 427 00:44:35,050 --> 00:44:45,250 Similarly, whenever we have new kinds of systems that that you know that people roll out, it's it's disproportionately low income families, 428 00:44:45,250 --> 00:44:57,640 areas that that that are seen to be, you know, high in criminality, that get to be the guinea pigs to for new kinds of draconian surveillance systems. 429 00:44:57,640 --> 00:45:02,680 Well, because one, it's not in my backyard, right? It's not in my see that I'm the iResearch doing this. 430 00:45:02,680 --> 00:45:06,280 I don't want it in my house. So but let me go. Let me go. 431 00:45:06,280 --> 00:45:10,840 Survey the, you know, the people who are the problem or the are marginalised groups. 432 00:45:10,840 --> 00:45:17,650 And this makes things so much worse because they are already putting up with systems that work so badly for, you know, worse for them to begin with. 433 00:45:17,650 --> 00:45:21,220 And then you have this horrible system that then provides surveillance architectures, 434 00:45:21,220 --> 00:45:26,800 which which have lots of issues with control and and privacy to undermine their well-being. 435 00:45:26,800 --> 00:45:37,060 Even more so, it's very so it's extremely contentious and interesting, I think, as far as the last part of the lecture. 436 00:45:37,060 --> 00:45:46,810 I just want to mention that people talk about human interaction and, you know, the two terms that sound really alike that I that I want to untangle. 437 00:45:46,810 --> 00:45:51,340 One we've talking about is this algorithmic bias of techno and automation. 438 00:45:51,340 --> 00:45:54,490 Bias is essentially something slightly different. 439 00:45:54,490 --> 00:46:02,620 It's just our general tendency to believe algorithms because they are and some seem to be either objective or 440 00:46:02,620 --> 00:46:08,170 they're just too complicated that we just trust that this complicated machine must be doing something smart. 441 00:46:08,170 --> 00:46:15,010 This sort of thing is, of course, automation bias is perpetuated by things like marketing campaigns like A.I. systems, 442 00:46:15,010 --> 00:46:20,350 but it's also perpetuated by model inscrutability interdisciplinarity, 443 00:46:20,350 --> 00:46:29,050 trying to understand the lack of understanding of what kind of complex problems might exist in the models and and and also knowing their limitations. 444 00:46:29,050 --> 00:46:34,420 Unfortunately, this is the very salient example today, because just about two months ago, 445 00:46:34,420 --> 00:46:43,480 there was a case where there was a drone strike in in in Afghanistan, I believe. 446 00:46:43,480 --> 00:46:51,010 Or maybe it's in Iran, which resulted in a lot of civilian casualties as the result of a synthesis of a bunch of what they call bad intelligence. 447 00:46:51,010 --> 00:46:54,460 But in fact, it was exactly the same intelligence. It was sort of. 448 00:46:54,460 --> 00:46:59,290 The problem is it was a humans who are interpreting things as being relevant to what they 449 00:46:59,290 --> 00:47:05,590 wanted was a massive combination completion of of automation bias and confirmation bias, 450 00:47:05,590 --> 00:47:12,310 which resulted in the decision to blow up this vehicle, which was full of children and and civilians. 451 00:47:12,310 --> 00:47:16,840 And the you know that one of the lead general, the Middle East, 452 00:47:16,840 --> 00:47:23,710 came and apologised me and said we made a mistake, but blamed the intelligence rather than the system, right? 453 00:47:23,710 --> 00:47:30,300 It was almost like you're blaming the data. The data were wrong. The data were bad and fundamentally it's much more. 454 00:47:30,300 --> 00:47:34,880 Difficult to think about what we mean if we don't really understand what the system, 455 00:47:34,880 --> 00:47:38,540 why this system is concluding, what it's doing and what the source of the data are. 456 00:47:38,540 --> 00:47:45,740 How can you even make any sort of sense of what the what the output is and whether it's worth studying? 457 00:47:45,740 --> 00:47:51,530 And so those everything that we've talked about turn out we're dealing with unintentional harms, 458 00:47:51,530 --> 00:47:56,720 and I leave you with an idea of saying, Well, what about the deliberate side of things? 459 00:47:56,720 --> 00:48:00,770 So there are many, many issues and ethics that deal with, you know, 460 00:48:00,770 --> 00:48:06,890 individual gain that result from the exploitation of of various individuals and groups. 461 00:48:06,890 --> 00:48:13,070 And this has led to a lot of very interesting and new stories that will be coming 462 00:48:13,070 --> 00:48:16,850 out over the next two months that deal with the Facebook paper revelations, 463 00:48:16,850 --> 00:48:24,440 which talk about how Facebook wilfully ignored the cases going on in various parts of the world, 464 00:48:24,440 --> 00:48:33,920 including in Myanmar, where where various groups co-opted the advertising mechanism to spread lots of hate against Muslims, 465 00:48:33,920 --> 00:48:39,170 as well as a very sensitive political case in Ethiopia. 466 00:48:39,170 --> 00:48:42,440 Now these kinds of things that say, well, is it the algorithm or is it? 467 00:48:42,440 --> 00:48:43,640 Is it the moderation? 468 00:48:43,640 --> 00:48:51,530 And the point is that an algorithmic infrastructure was created and then it was co-opted by various actors to promote genocide or to create genocide. 469 00:48:51,530 --> 00:48:59,210 And that sort of thing shows you the complexity of its, you know, whether it's, you know, the bias and the algorithm or it's the, you know, 470 00:48:59,210 --> 00:49:10,670 if you think of things as an entire system, the ways that the system then evolve over time helps us understand what the potential harms can be. 471 00:49:10,670 --> 00:49:16,010 So this this leads to a whole realm of exciting things to think about. 472 00:49:16,010 --> 00:49:24,410 We are very lucky to have a new project looking at the potential to rethink some of these mechanisms can be more harm resilient. 473 00:49:24,410 --> 00:49:30,170 I'm not in any position saying that we can ever build a system that will be ethical in some sense. 474 00:49:30,170 --> 00:49:35,720 But can we build systems that that? Can we consider all of these sources of harms and build things that that maybe work 475 00:49:35,720 --> 00:49:41,900 a little bit more robustly against the kind of biases we know are going to exist? Or perhaps we can do data collection a little bit more ethical way, 476 00:49:41,900 --> 00:49:48,800 and perhaps we can be more transparent so that when ethical problems occur, we can identify them early. 477 00:49:48,800 --> 00:49:53,570 Humans can say, Hey, look, this is bad, what's going on? Let's stop it. 478 00:49:53,570 --> 00:50:02,900 And so that's the the task of our Oxford Martin School project that's just starting now and very happy to talk to you all about directions for that. 479 00:50:02,900 --> 00:50:10,820 And we are the human centric computing group. And please, you know, talk to us and talk to me about things. 480 00:50:10,820 --> 00:50:17,060 If you're if this talk, give you ideas or reactions. Yeah, let's have a little chat. 481 00:50:17,060 --> 00:50:32,260 Thank you very much. Great. Thank you so much. Max, I was an absolutely trenchant talk as a round of applause, everybody. 482 00:50:32,260 --> 00:50:33,472 So.