I'm Marcus to say Italy. I'm the Simone professor for the public understanding of science here in Oxford and a professor of mathematics.
And this lecture is kind of like the highlight of our year,
trying to communicate some of the exciting science that's going on here in Oxford and across the world to the public.
And we're very grateful to the Simoni Fund, which helps to make this possible.
And the Oxford plays house for for hosting us now.
She was Richard Dawkins idea, my predecessor to hold this lecture on kind of neutral grounds, not in the university,
but somewhere where the public and scientists can come together to to discuss kind of the important issues of the day in France.
Today is the 20th anniversary of this lecture.
So so we look, again, sort of backwards, a lot of lectures we've had,
but also looking forward because the topic of today's lecture is really one that's going to affect society very much over the coming years.
I just spent several years on a committee with the Royal Society looking at the impact AI
and machine learning is going to have on what else is working but on our lives in general.
So so I thought it was really urgent to get somebody who's really at the cutting edge of this field to come and talk to us about its impact.
And I think we kind of realised that A.I. may be driving our cars, being our doctors,
but we do feel there's some area that we really regard as uniquely human and that's our kind of emotional world.
So I was very interested to invite Dr. Rosalind Picard from MIT Media Lab, who set up the Department for Affective Computing.
There. She has several companies looking at the impact of AI on society and inventor as well.
To come in, explore with us. Can we create AI with emotional intelligence?
So please give a big Oxford. Welcome to Professor Picard.
Thank you. Thank you, Mark.
It's such a pleasure to be here with all of you. I'm going to start with something a bit dangerous because I know it's dark out there,
but I'm going to ask you to close your eyes and imagine with me something first.
Imagine that you are working on something very important to you and you go wherever you go to get away from it all to focus.
This could be your office or someplace you haven't told anybody where you you go to get out of the way of people.
And while you're in this place working really hard on this important deadline,
this character barges into your space and interrupts you and doesn't apologise and doesn't notice that you're slightly annoyed by this.
Then the character gives you some useless advice, which you then respond to with a little bit more annoyance.
Which the character ignores. And he continues to be unhelpful at this point.
Maybe you were very subtle, but you are no longer subtle.
Let's say that you are very clear that this is annoying.
But he continues to ignore that you are annoyed.
And this goes on until finally you say leave.
You're very explicit. You might get up and walk him out the door and he does leave.
But first he winks and does a happy little dance before leaving.
Now you can open your eyes and the story that you just imagined is one of the
ones that has happened to many of us who use some of the early AI systems.
Some of you my age or whatever might recognise this animated paperclip.
Younger people may go, What's that?
This was a very smart software agent made by Microsoft, deployed an office that actually had some of the first AI machine learning in it.
That was quite good at seeing if you were writing a letter.
It was not, however, good at seeing.
If you're positive or negative about what it's doing, it didn't recognise your emotion.
Now there's a time to ignore people's emotion.
You don't want to, you know, flickering every time you smile, but there's a time to respond appropriately to emotion.
And it did not do that. This happy little dance thing was probably really popular when people were coming in to test it and
getting 50 bucks an hour in this modern facility to try out this cool new future technology at the time.
But under the context that we just went through, that would not be an intelligent response.
So these are examples of where I has not shown emotional intelligence and I won't go through all of the aspects of this.
I am just going to skip around and touch on some kind of hot topics, I think, with respect to it.
But the big picture is the AI should be smart about recognising our emotions.
About knowing when to express them or not.
Like when to do the happy little dance and when. Not when.
How to handle our emotions. When to kind of ignore them and when to respond.
And if the system has emotion.
You know, what does that mean? If you're actually have emotion? We'll get into this.
Then it should regulate those, right? It shouldn't be having a meltdown in front of you.
We don't want that. We don't want our technology to just act emotional.
Utilising emotions is a different kind of emotional intelligence that is about understanding how different
affective states lead to different kinds of thinking or facilitate different kinds of thinking.
And that can be used advantageously. For example, if you have a creative problem to solve.
It has been shown that if you are put in a good mood first, you will think more out of the box.
So if you really are not in a good mood, you have to solve a creative problem.
You should. You could be emotional intelligence.
I need to go get in a good mood first before I go bang my head against the wall trying to solve that problem.
So what did we see instead? That was called intelligent interaction.
We see animated objects of all kinds and robots and so forth, smiling at us, trying to make us happy.
And that actually there is a time and place for that.
Like when when someone first greets you or you come home and your dog sees you and it looks happy to see you,
but as soon as it sees that you look like this, then it should change, right?
It should stop looking happy. So it's fine when it first greets you.
It's not fine after you have shown that you're not happy.
This is not new wisdom.
This has been around at least for thousands of years, as we see here from Proverbs, singing cheerful songs to somebody with a heavy heart.
You might as well pour vinegar on their wounds or take away their coat on a cold day.
Unwelcome. The first robot I saw that showed emotional intelligence was this Ph.D. thesis project of my colleague Cynthia Brazil.
And I'm going to play a little clip of kismet because kismet does the right thing here.
So another important skill for the robot to be able to learn from people is being able to recognise communicative intent.
Very good kismet. And the way we've done that with Kismet right now is to have the robot recognise by tone of voice.
Are you praising it? Are you scolding it? Where do you put your body?
Oh, no, no.
You're not to do that. No. So imagine that paper clip that I started off with.
Bill Gates got a standing ovation when he announced it was going away.
A lot of people were really unhappy with that thing. Now, if Clippy, instead of smiling and dancing every time you were upset,
had done what Kismet had done and looked sorry, how many of you might have given it a second chance?
Right. So one of the things with emotional intelligence we have to do is know how to look.
Sorry, recognise when we made a mistake, apologise and repair the situation.
Now we also need to know if the person we're interacting with is happy or upset.
This should be easy, right? I work a lot with people with autism.
One of the things they're usually taught is this smile means the person is truly happy.
This is the smile you try to have in front of the camera where not just cheers with your mouth, but also smile with your eyes.
Right? So in my book Affective Computing, I quoted the neurologists and the great psychologists and said, Yes, this is the true smile of delight.
Then later we got data.
And I'm going to play for you a clip of a kind of experiment that we did here where the people were brought in to fill out their resume on a Web site.
And unbeknown to them, well, they know they're being recorded.
They're fully informed of that. But we didn't tell them until afterwards that it was designed to be frustrating.
I am sorry. You seem to be having difficulties getting through this form.
Would you mind looking at the camera and share your thoughts on how to improve the form?
This one stuck. All right.
Now, you know, he looks like he's not having any fun. The next guy is in the exact same situation.
Watch his face. Well I'll go home and watch mean that little very quick.
I'm going to say it one more time because if you. It's a little hard to see.
Watch the I. No, I don't know.
All right. Not happy, but that was the happy face.
Computer could detect this originally as the true smile of delight.
In fact, we found 90% of people showed that happy face in our study with adults.
Later, we found people just looking at children and learning situations had also found them
showing this to smile when they were having learning interactions that weren't going well.
Lots of reasons why people might do this. But for us,
this is there's this challenge of both being honest to people who struggle with
reading faces and letting them know that this is much harder than they've been told,
and that if your boss looks like this, you might not want to keep doing what you were doing.
But it's also a very hard problem to solve. So something as simple as recognising a smile turns out to be harder than we thought.
My group uses a lot of machine learning now, more popularly called AI, even though the early AI people did not consider machine learning a part of AI.
And what we did here is we took four different machine learning models and compared them to a human,
and we showed videos that were of smiles elicited under real delightful circumstances versus the circumstance you just saw.
And we looked at how people did at this, which were pretty good when it came from a truly delight smile,
but was pretty bad when it came from a frustrated smile. Here are the machine learning.
Algorithms are actually better than the person at recognising the smile of frustration,
even though it has these same action units activated by the AI in the mouth with the computer and the person both were given was temporal information.
And here the computer was better at breaking up the temporal information into some things that are typically quite different when you're frustrated,
kind of more of a quick jerky smile. And a lot of people didn't pay attention to that.
Today. You can get off the shelf software.
You can actually get the one I'm showing you here for free online.
And this can read your facial expressions in real time on a mobile device and much more than smiles.
Here you'll see a subset of things that can recognise being mapped.
Also, two emojis, which can be fun just for having a good time.
This is software by one of our spin out companies, Affectiva,
that has already analysed more than 7 million faces and is now more than 90% accurate on about 30 different expressions.
As I mentioned, if you want to play with it, hopefully not right now in the middle of the talk.
But it's dark. It works best with light. But you can download for free from iOS or Android.
The Google play store just by searching for affects me.
And then make sure your face is illuminated when you play with it and play with the
settings and you can see how well it can recognise all of your facial expressions.
Now, robots today give the appearance of being much more emotionally capable than they actually are.
And I don't think The Tonight Show this this is everybody's probably asleep over here when Jemmy Fallon is
on TV in the U.S. I'm going to play a clip of an episode that made my jaw drop originally when I saw it.
And then I'm going to tell you the real punchline. Let's meet our next robot.
This one came all the way from Hong Kong. Please welcome the founder and CEO of Hanson Robotics, David Hanson, and his robot, Sophia.
Oh, my gosh. Welcome. Thank you so much for coming on the show.
Nice to me as well. David, you brought a friend with you here and this is really kind of freaking me out.
Yeah. This is Sophia. Uh huh. And Sophia is a social robot.
Mm hmm. And she has artificial intelligence software that we've developed at Hanson Robotics, which can process visual data.
She can see people's faces. She can process conversational data.
Emotional data and use all of this to form relationships with people.
Okay. So, I mean, she's basically alive.
Is that what you're saying? Oh, yeah. Yeah. She is basically alive.
Oh, would you like to maybe give it a try? Sure. I would say, what's this is like?
You see how awkward my first days are to robots?
Oh, I'm already. I'm getting nervous around a robot. Very pretty, Rover. Do I.
Do I just say hello to. Yeah, yeah, yeah, yeah.
Hi, Sophia. Hello, Jemmy. Oh, my God.
Do you know where you are? Of course. I'm in New York City and I'm on my favourite show, The Tonight Show.
Sophia, can you tell me a joke? Sure. What?
Jesus can never be yours when she can never be mine.
I don't know. Nacho cheese. Yeah, yeah, yeah, yeah.
I like I like nacho cheese. Nacho cheese is the.
Has to do. I'm getting laughs. Yeah. Maybe I should host the show.
Okay. Stay in your langer now.
Jimi, would you like to play a game of rock, paper, scissors, robot style?
Sure. Okay. Let's get this game going.
Show me your hand to start. Rock, paper, scissors.
Shoot. Oh, I won.
This is a good beginning of my plan to dominate the human race.
All right. A lot of things to pick apart here.
David said, Yeah, she's basically alive.
Now when I hear the word, basically, I insert the word not.
She's not alive. Sophie is not alive. In fact, this entire scene was scripted and rehearsed many times.
Jimmy's a great actor. He made it look like the first time meeting this robot.
David, the maker of Sophia, said they rehearsed it over and over and over.
And there were people behind the curtain. Making things work.
We don't know of any machines that are conscious and that have feelings or experiences the way that we do.
They can put on a face of disgust you nacho cheese.
But that's just a program saying put on the following set of movements on your face.
I'm not saying we can't ever do this in the future. Perhaps it might be possible.
But right now, none of us that I know of in the fields of computer science,
I have any idea how to make something like this happen with the kinds of software,
hardware, quantum computing, all kinds of bio computers, things that we're doing other than having a child creating offspring in the natural way.
Also this. I cut off the end where she says, Oh, that was just a joke about my plan to dominate the world.
No machines are evolving themselves to take over the world right now.
It really does not have the ability to happen unless humans design in such an ability.
I would add sort of power dominance hungry humans design in such an ability.
So then it would be a human machine combo at some level doing this.
This is certainly not likely to happen from the kinds of things I'm talking about with you today.
People often ask. Well, whereas, you know. Some.
I'm just a robot. Some people say I'm just a robot. We're just machines.
And in fact, if you could build a machine that's exactly able to do the emotional things, I do look like it has these functions.
I have walk like I do, talk like I do joke like I do express emotions.
Like I do recognise emotions like I do. Then it's essentially equal where the same, right?
And then we've once we can build it, we've fully understood it because in A.I. we often build things to better understand them.
We build models of the brain to better understand how it works. So if we succeeded in building all of this,
maybe even making it say that it was conscious and it had experiences, would that mean that that's all we are?
We're just this material stuff. A story I like to think of that helps me with.
This is the story of the radio instructions that were found by aliens on a planet.
And they went and they found the materials to build the radio and they built everything exactly according to the instructions.
And in the end, they turned the knob that was supposed to turn and make music and oh, my music came out.
And so they thought we have therefore understood not only how radio works, but where music comes from.
They thought that by building this with this phenomenon emerging, that it was therefore fully explained by the physical material stuff that was there.
Now, of course, we know about more than that. So we know about radio waves and all this other stuff, but they didn't have access to that.
Similarly, our thinking that just building something that has outward functions that are equivalent according
to some list of criteria to think that that could be everything is a very myopic way of thinking.
Here's a picture of her.She Ishiguro in Japan who has been making robots that look like him, his wife,
his daughters, with the idea that eventually he could send his robot to give the talk instead of him.
You wouldn't see the difference. And a lot of them do look look real.
But by no means are these the same as the person that they're designed to functionally look similar to.
Now, in our lab, we've been wrestling lately.
The Media Lab grew up at MIT in the School of Architecture and Urban Studies, different from the A.I. lab that grew out of the computer science group.
But we have a lot of kindred spirits that go back and forth.
And the AI that I sort of grew up thinking I wanted to build was a lot like Sophia.
But at the Media Lab, over the decades, we've been rethinking A.I. and thinking, What do we really want?
What do we really want to be remembered for building?
And Marvin Minsky, one of the founding parents of the field of AI and one of the founders of the Media Lab used to say,
I want to build an A.I. that's so brilliant and powerful that, you know, we'll be lucky if it keeps us around as a household pet.
And at first I thought, Wow, that's a pretty cool, big vision. And then I thought, Wait a minute, do I really want to be a household?
Do I want my children to be household? My grandchildren to be household pets?
So we have been rethinking what kind of air we want to build, and we've decided we want the future to be a better one for humans.
And we want to build the kind of intelligence that extends human ability, not replaces human ability.
Not that it can't replace some of the things we do, but it's got to remember, it's got to reflect what our priorities are.
And so we'd like to think about that.
So we have been looking at examples, you know, such as wearable cameras that augment for a blind person what they see.
Or I'm going to tell you a story in a moment of a wearable smartwatch.
We already are very familiar with the way smartphones give us superpowers.
But something like this can actually extend our intelligence in a way that that's a bit surprising.
And this one in particular started when I was working with children with autism.
Here's a boy who was wearing one of our cameras that was augmented with a computer vision
system you just saw that would whisper in his ear or show him in an alternate form.
He could process what somebody's facial expressions were because he had difficulty processing the face of the person he was socially interacting with.
And one day as I was interacting with a person with autism, she said to me.
RAZ You have it all wrong. My biggest problem is not reading other people's emotions.
My biggest problem is you're not reading my emotions.
And I thought, great, you know, this is what I do for a living like this, that it's okay.
I have room to improve. What do I need to do better? And she said, It's not just you.
It's everybody is not reading my emotions accurately. And what emotions are we not reading accurately?
You're not reading my stress, my anxiety.
And I realised as we worked more with kids with autism, while some people are kind of stressed like this little boy,
a lot of people would outwardly look very chill and inwardly they're about to erupt
a meltdown or something that could cause injury to themself or others around them.
And I realised back in our lab in our early days of affective computing,
we had taken off the shelf skin conductance sensors that measured the sweat on your hand.
And we'd learned a bunch about how these could get anticipatory responses in fight or flight responses because the skin is
innervated by the sympathetic nervous system or part of our autonomic nervous system that is triggered with fight or flight.
So I wondered if we could take this into a form factor that kids could use in the classroom if this might help them be better understood.
So we built lots of different versions, learned where we could measure this and other places on the body because the palms were often inconvenient.
And today there are two commercialised versions of this I'm wearing one that we use in a lot of research.
The empathic 84. And this one, the embrace, which I'll tell you a little bit more about.
First, I'll play just a little example of the data collected from a child we saw.
There is a fine. How are you sure? Are you sure you want to know?
I'm going to turn the sound down because she's about to have a meltdown.
What you see in that little blue window at the top is her skin conductance level streaming from her two ankles,
going up with her increased sympathetic reactivity likely here.
Now, she's about to have a meltdown and we see it. It will peak in a moment here.
The early version had wireless dropouts. That's why that long straight line, this blue window at the top corresponds to this little blue strip here.
It's one minute wide. This is about 45 minutes of data. And you can see she's been building up for some time before that peak there.
So a teacher or a parent who sees the child's level growing without seeing any external causes for this might then
reason that there's something internal that's increasing that could be about to cause a response like we just saw.
The signal doesn't just go up with stress. It also goes up with cognitive effort and engagement.
The first time I saw this from the wrist, that previous girl was wearing it on her ankles.
Was this data from an MIT student? What you see here from bottom to top is seven days left to right is 24 hours.
We see, as we expected, big peaks in this activity with exams and and homeworks problems.
That's I imagine Oxford's a lot like at MIT.
There's, you know, really hard things to think about its effort mentally and emotionally to engage in this.
Unfortunately, the low point every day that we found is over here on the right, underlined in yellow, and that's classroom activity.
To the disappointment of MIT professors. The middle, the biggest peak of most days, surprisingly, is sleep.
And if you're paying attention, you should be scratching your head going, Wait, didn't you just say this is arousal, stress, cognitive load?
What the heck is going on here?
It's a whole nother talk, but in a nutshell, we're finding that these patterns, when you zoom in on them with high frequency peaks,
they exhibit statistics similar to single neurone hippocampal firing patterns.
These are from regions of the brain involved in memory,
and we are finding the signal to be useful and predicting how well people perform on a task they learned before sleep.
Now. The first time we built the sensor, it looked like this.
It was in a sweatband and it had homebrew electronics in it that frequently broke.
And it was the end of the semester. One December I was in my office, I was working away and a young student knocks on the door and he says.
Professor Picard, could I please borrow one of your wristband sensors?
My little brother has autism. He can't talk. And I want to see what's stressing him out.
And I said, sure. In fact, don't just take one, take two in case one breaks.
And so he takes the two. He doesn't wait for one to break.
He puts both of them on his little brother's wrists at the same time, the opposite of what I told him to do.
But I'm really glad he didn't follow my instructions. I go back to MIT.
I'm looking at the boys data on my computer and the first day looks very flat, both wrists.
It's a kid who looks pretty relaxed. Second day, pretty relaxed.
Third day, same thing. Go to the next day and my jaw drops.
One of the signals on one wrist went so high that I thought the sensor must be broken.
We have stressed people out at MIT every way we can imagine, you know, from qualifying exams to obnoxious noises in your ears.
Boston drivers stress. And I had never seen a peak this big and weirder.
How could you be stressed on just one side and not the other?
I tried to do some debugging and I'm electro engineer so I thought I could figure this out.
Ran a bunch of tests, could not explain what was going on. So I did the old fashioned debugging.
I picked up the phone and I called the student at home on vacation.
Hi. How is your Christmas? How's your little brother? Hey, do you have any idea what happened to him?
And I gave him the exact date and time in the data. And he said, I don't know.
I'll check the diary. Like MIT student keeps a diary.
You know, probably Oxford students do diaries, too. So quick prayer.
Like, what are the odds a teenager would write this down? Mom of three teenagers.
Pretty low odds. He comes back. He has the exact date and time written down.
And he says that was right before he had a grand mal seizure. Now.
I didn't know what seizures were, but I knew another student's dad was chief of neurosurgery at Children's Hospital Boston.
So I screwed up my courage and called Dr. Joe Madsen.
Dr. Madsen, my name's Rosalind Picard. Do you know if it's possible somebody could have a huge, sympathetic nervous system surge here?
It looked like it was 20 minutes before the seizure. And he said.
Probably not. But he said, you know, it's interesting.
We've had people whose hair stands on in on one arm 20 minutes before a seizure.
And I'm like, on one arm. And I told him how, you know, it happened on only one side.
He got interested. We made a lot more devices. 90 families were being enrolled in a clinical study where all the children,
90 children were all candidates for brain surgery because they had seizures that were not stopped by medication.
And we found that 100% of that first batch of children had very large skin conductance responses with their grandma seizures.
Not 20 minutes in advance of them, but usually at the exact same time of a perfectly synchronised readout of their video EEG data.
At the bottom here is movement data.
When you combine that movement data with the skin conductance, you get a more sensitive and specific seizure detector.
But there was a mystery here, and that is the seizure is only about the width of that red line.
It's only a few minutes wide that the person is shaking, convulsing and having this unusual electrical activity in their brain.
Why is this stress? Skin conductance, sweat, whatever it is, response so big for so long, for 20, 30 minutes when the person's just lying there.
And as we tried to figure this out, we learned of another thing.
And that was that there's something called SUDEP, sudden, unexpected death in epilepsy.
And when a person who has epilepsy dies after a seizure, this you see this seizure activity.
What what you see here from top to bottom is traces from electrodes on the scalp.
This this time from left to right. And here each trace is going kind of electrically crazy because there is a seizure.
Then the seizure stops. And we would like to see normal brain activity here.
But instead, everything is below ten micro volts. It's suppressed. And it's called postnatal after the seizure.
Generalised across all EEG channels. EEG suppression.
And it turns out that while this occurs in 100% of SUDEP, it also occurs frequently in cases where people fortunately don't die after the seizure.
But the duration of it turns out to be related to the size of the response on the wrist.
This is not just a crazy artefact.
This work is published in the top medical journal Neurology and has been replicated with other groups beyond this paediatric group as well.
So this led us to start to learn more about SUDEP.
And to my surprise, I learned that this thing that is taking people's lives is actually quite common and not not commonly discussed.
It's actually the number two cause of years of potential life, loss of all neurological disorders, stroke being number one.
There's a suit up every 7 to 9 minutes, and probably most people here have never heard of it.
It's also more common than crib death. In the U.S., we call it Sudden Infant Death Syndrome.
It takes more lives every year than sudden infant death syndrome or crib death.
It starts with a little unusual electrical activity in the brain. If you have repeated ones of these, you get an epilepsy diagnosis.
And what could be start off as a little brushfire can spread and generalise over your brain and cause convulsions all over.
In fact, most of the time we get a response on both wrists because usually the seizure generalises
their number one way to prevent SUDEP is to take your medications the number two
way and actually what's been shown in the literature to make a significant difference
in reducing the likelihood of a SUDEP after a seizure is for somebody to be there.
To have somebody come check on you. So by you leaving here tonight and knowing about SUDEP,
I hope that you will also be able to be somebody who would help people know that being there for one another could actually reduce these deaths.
We think most of these are preventable. Now what's been shown is that when somebody gets there,
the first thing they do is often say the person's name or just touch them or flip them over and stimulate them in some way.
It's also been shown that going in this is invasive inside your brain and stimulating one of the key regions of the brain involved in emotion.
The amygdala can turn off your breathing and you can sit there and just stop breathing.
You're capable of breathing, but you don't breathe. And then when somebody says your name or talks to you or stimulates you, you breathe.
It's been observed that when the seizure spreads to that region, it can turn off the breathing, but somebody stimulating can turn it back on.
So at in Pedagogue, we decided to commercialise this and get it out with an alert.
And now the Swatch Embrace runs A.I. Machine learning real time to detect generalised tonic clonic seizures and to
send an alert to try to bring somebody there to hopefully stimulate the person and help restart their breathing.
It can't do fully closed loop just yet, but we are getting stories that are like this that make my skin conductance go up.
I read my email. This was from a mom who said she was in the shower that morning.
She saw her phone on the counter by the shower go off saying her daughter needs her help.
She goes running out of the shower to her daughter's bedroom, finds her daughter face down in bed, blue and not breathing.
She flips her daughter over and her daughter takes a breath and turns pink.
And this time, I think I stopped breathing reading this email.
Since then, we've heard a lot of stories like this. Now the SUDEP is actually most common among people in there.
It starts peaking in the twenties and the thirties and forties, and it could happen to somebody,
you know, if it's a stigmatised over here in the UK as in the US,
you'll probably if you go back and talk to your friends and say, I don't know anybody with epilepsy, do you?
You'll probably find several of them saying, actually, I have epilepsy or my family member does.
We don't tend to talk about it, but what in 26 people has epilepsy?
If we talked about it and got to know when one another has a need for somebody to be there, we could actually prevent a lot of those deaths.
So there is still no AI that can actually stimulate you and reposition you.
It may be coming, but right now we need human help to close this loop.
And it's actually kind of nice that we need human help because we need human help to do more than just stimulate us to breathe again.
Right. We really need to get to know our neighbour. Now.
I was very surprised when I saw that result that the duration of shut down was related to some activity deep in the brain,
and that when the activity deep in the brain is happening, we're not sensing anything from the EEG.
The idea looks like your brain's dead. But where we. Since the activity is on the wrist.
So I was really glad I had tenure because this is pretty weird, right, to say there's deep brain activity.
The EEG doesn't see it, but the wrist does. Huh.
Like, talk about bizarre.
And I was remarking on how strange I thought this was to a group of doctors, and one of them shakes her head at me and she goes, Roz, it's easy.
Like, What do you mean it's easy? It's medicine. 1010.
I never had medicine one on one. I'm an engineer. Medicine, 1 to 1.
Everybody who took it was taught that as an embryo, we have three kinds of tissue.
One of them forms our muscles and bones, the mesoderm, the endoderm formed our lungs, our stomach.
All of that soft, all the soft organs inside. But the largest organ on the outside, the skin co formed with the brain, the spinal cord,
the whole neuronal system, the ectoderm knit together our brain and our skin from the beginning.
So she said, I'm not surprised that you would find activity deep in the brain not showing up on the scalp, but showing up on the wrist.
Now this got me thinking because I study emotion.
Most of the stuff that we had been studying was from brain scans and things deep in the brain and these homes of emotion in the brain.
And one of the things that was of interest was something my boss had asked me for years before,
and I had not taken him very seriously back before we started working on autism and epilepsy, and we were just trying to build smart computers.
My boss said to me. Roz. What are you going to do?
Something useful. When are you going to tell me my wife's mood before I go home.
And I thought, oh, great, you know, mood rings, you know, he wanted the mood ring and that told him his wife's mood before he gets out.
And, you know, Walter, the mood ring is just a stupid little sensor that turns colours with temperature.
You know, I like to do things that are hard. I like to do things that are important.
I mean, yeah, maybe I can make more money selling mood rings, but really, I want to solve hard, important problems.
And as we got these brain findings and the connection to the scan, and I started to learn more about what we might be able to do with Mood,
I started to learn about the importance of a very serious mood disorder.
I know it's stigmatised here. Sorry, bringing up all these negative things tonight, but you've got to pay attention to this one.
It's already the leading cause of disability and it's on track by the year 2030 to pass cancer or accidents.
And stroke is the number one cause of death and years lived with disability.
That's the forecast. Could we change that forecast? Could we have a better future instead of this one?
One way I like to think about this and this is my one non-real data graph is this is well-being on the left here, higher is better.
People are more positive, they feel great. You get admitted to Oxford or you get a job at your favourite place to work and you're doing great.
You come in, you do even better. This is time. Everybody hits major stressors, everybody takes a dip.
And unfortunately, even in the best companies, 20,
30% of people are or more are dropping down here where they might get a major depressive disorder diagnosis.
Others are resilient and snapping back.
You know, the metaphor of the frog put on the pot on the stove and they turn on the temperature and the water starts to heat up.
The mood ring, the stupid temperature sensor. If the frog had a temperature sensor, the frog could get out.
Maybe jumped from the red to the blue before it got in trouble.
Right. When I was talking with leading psychiatrists about what we should be doing to prevent this.
I said, Well, do we just need to get people who are in trouble to you sooner?
Because if we got them into medical care sooner, you could fix them, right?
I mean, I'm talking with experts who run the depression innovation across all of the United States
and have international partners working on all the cutting edge ways to treat depression.
I said you could fix it. Right? And he said, no, we don't know how to fix it.
And I thought, then we need to be not waiting for the guys working over here in the medical system.
We need to do something over here to prevent it. We need to not let the frog boil.
And you know what? These sensors that we're wearing. They're much more sophisticated than temperature sensors.
Could they actually give us early warning of if mood or stress are getting out of hand?
Now there are dozens of studies we're doing. I'm just going to give you a couple quick highlights, because I want to leave time for Q&A.
But here's one. Here is data. What you see from bottom to top on the left is data from wristband,
actually smartwatches and mobile phone data related to physiology, geolocation, social network patterns.
Here from 20, this one was 22 patients diagnosed with major depression disorder.
And on the bottom is the score on a gold standard depression inventory that a top mass general hospital psychiatrist is giving them.
If it was perfect, they would all the dots would lie along this this diagonal line from bottom to top.
And we see it's not perfect, but it's pretty doggone good. In fact,
it's correlated 0.83 and that is the just objective data from the system being correlated with the
physician that is already as well performing as physicians are with correlated with each other.
We've also been asking, can we forecast to see if you're on a trajectory where things are getting better or worse?
And the first version of this is shown here.
This is using a kind of AI and deep learning that's called multitasking instead of just training it on mood or on stress or on health.
We train it on multiple things. Those three here that are related so that it learns things that are common across them,
and then it tunes the output layer to individual people or people like you.
And here we just asked, how well would it work if we set it up?
So that chance was 5050.
And we just want to forecast if tomorrow night came just through data today, tomorrow night, you're in a great mood or a bad mood.
You're really healthy or you're really sick or you're really super stressed or are you really calm or you're really stressed.
The good ones versus the bad ones. 5050 would be random.
It's first time it was 78 to 87%.
Since then, we've been refining, we've done much more nuanced models with regression to give exact levels, and the accuracy is continuing to improve.
We also want to know. What what's affecting it for you?
And as we collect data and the last slide in this one, the data here is only limited to New England.
It's limited to only New England college students here. Can't say this generalises behind that dataset yet.
But here we looked at which things were most associated with being more stressed the next night.
This is the doctor work of psychiatry or more calm the next night.
And the number one thing for calmness was positive social interaction.
Isn't that interesting? We see a lot of we computer scientists who are introverts kind of like, oh, darn, we have to be social.
But for most people, the positive social interaction makes a huge difference in your calm tomorrow,
and the negative social interaction has the reverse impact.
And there's one data point even worse than the negative social interaction that I could give another whole talk about, and that's sleep.
In particular, if your sleep times are very uncorrelated from day to day,
you have high sleep e regularity that is consistently associated with higher
stress and worse mood and worse mental health in our college student population.
So we want to understand how to make this better for people and.
It turns out this is really hard. Again, that's what I'm showing you here is just first experiments limited to college students.
It's probably too small to see, but from top to bottom, there are a whole bunch of things.
And I'll tell you the most important ones here, the one that's just highlighted in blue here is the Priestley social interaction.
And down below some texting behaviour, college students interacting socially through texting.
What's circled here? There are four vertical columns.
Each row is a different thing that was measured in this particular model,
and each column is a different group of people clustered by people who are similar to each other.
And what we see is that the two columns in the middle, you see that bright yellow spot and that dark blue spot next to it.
What that means is for the people in the column with the bright yellow spots, those behaviours,
those social behaviours were highly associated with them being much happier the next day.
But the people in this other group, those exact same behaviours are associated with them feeling much worse the next day.
So we can't give a one size fits all piece of advice.
When we look at people's data, we can start to learn what works for them and what doesn't work for them and give customised.
Insights about what makes things look better for their forecast.
So I don't know about you. I. I actually check weather forecasts.
My husband doesn't. So I know this won't work. Won't be of interest to everybody, but I check.
I like to see if it's going to rain because I look pretty horrible when I've been drenched and I'm interested in the mood forecast.
However, I suppose my mood forecast looks like this. Right.
Worse stress. Worse mood getting sick.
Oh, man. You know, like who wants to look at that up, especially if you're already depressed.
Right. This could be really bad. You know, if you look like this, like this app is going nowhere.
And we run the risk of making people even worse off.
Right. Which is not what we want. So what's the app that we want?
Well, the one we want gives you not just the weather forecast, but the ability to change the weather,
the mood weather, the stressed weather, the health, whether you're likely to get sick.
But if you go to bed 2 hours early, you're more likely to.
You're a lot less likely to get sick.
If you spend some time praying or meditating or reaching out to a friend or making a new friend or getting some sunshine or fresh air,
it can get to know from your behaviours and your interactions what makes a difference for you and in that moment give
you in an evidence based way based on all this machine learning and ai of emotion and mood being put to use for you,
it could help you see not only your forecast, but how to change your forecast.
So that's the goal we're working toward right now is an AI that's not trying to take over the
world or just impress people on stage with its facial expressions or its emotional jokes.
But the AI that makes our lives better by helping us understand what's going on inside, that we want to change and helping us do that better.
Let me wrap up here with a quick recap. I started with a story of how much harder some of this is than we thought.
We thought like detecting smiles, easy. Then we learn that these peak smiles sometimes can show up when people are really frustrated.
So we have to be much more sophisticated about how we interpret them and read more than just the picture of the facial expression.
While working with people who wanted to understand each other's facial expressions better.
We also learned some of them didn't want that. They wanted us to understand them better.
They wanted us to understand their stress and anxiety better.
And technology we had could be reshaped into wearable forms that could go out where their stressors were happening in schools,
on playgrounds, and start to give them insight into how that stress was changing.
In fact, this will be this measuring my both devices are measuring this right now as I'm up here.
To our surprise, we learned when loaning out the skin conductance technology in the sweatband that it could also pick
up unusual peaks that related not to sweating and stress in the sense we usually think about it,
but related to unusual neural activation in the brain that arises with unusual electro activity with a seizure.
We learned that the size of that activity was related to a signal in the brain associated with SUDEP sudden unexpected death,
an epilepsy that happens every 7 to 9 minutes,
and that by getting somebody there, we might be able to have a human become stimulate that person and restart their breathing and prevent death.
With that. We then began partnering with neurologists and starting to learn more about what was going on deep in these
emotion centres in the brain that related not only to the epilepsy but related to changes in mood and depression.
What could we maybe detect even before a person needs to get a clinical diagnosis?
Could we prevent depression? Could we prevent 80% of the cases of it?
I think that's possible. We started to measure not only with wearables, but with smartphones, human behaviours that change very subtly,
so subtly that you and I may not notice these things changing, but the technology is really good at picking up these changes.
It can see the sophisticated, high dimensional equivalent of the frog in the water and the water getting hotter and hotter and hotter.
And it could let us know that it's getting hotter and hotter and hotter.
We're now using the deep learning machine learning AI in ways that are very personalised to
help people get insights into not only what trajectory they're on and what their stress,
health and mood are likely to be tomorrow, but also what they could do to change that.
And in all of this, surprisingly, is a very low tech solution.
All this fancy AI and all. And what do we need most?
We seem to need each other. We seem to need human social interaction.
We need to sometimes get our head out of our phones and our computers and just
reconnect with each other and have what Barbara Fredrickson calls positive resonances,
where we share a smile, where we share a story, a share, a hug.
It doesn't always have to be a happy thing. It can be a sad thing. We empathise, we connect.
And these things are highly associated with tomorrow being happier, less stressed and healthier.
I hope that you will join me now and Q&A about this, that we can explore together how to make this future.
And I invite you also to go online. We have publications related to all of this, and I'm happy to engage now and hear what's on your mind.
Thank you. Thank you.