ARI: Hello, I'm Ari. [CLAUDINE: And I'm Claudine] Welcome to Proving the Negative.
ARI: We're a podcast all about exploring the different sides of cybersecurity,
from political to computer science, international relations to mathematics.
Join us as we talk to our friends about the work they do.
MARY: My name is Mary. My doctoral project was about the security of
speech interfaces and voice control.
My website says that I am a cyber security researcher and an innovator.
Being an innovator is something that I am aspiring to at present! I am keen to
deepen my understanding and, in the future, use that to innovate.
Not quite sure what that will be, but something that will be of genuine value
to people and genuine use in contributing to the field.
ARI: What are you curious about?
MARY: More broadly, is this technical understanding of detail that I find
fascinating. The technical detail of how things become vulnerable,
how it works in practice, specific bugs in specific systems, and digging into generic
descriptions and news headlines to understand how that works in practice.
That's something that I've scratched the surface of, and continue to learn about.
I didn't start from a technical background - I've tried to gain as much technical
understanding as I can during my doctorate and on an ongoing basis.
And I found that really rewarding, especially when I feel that I have fully
understood something and I know how it works, that really motivates me.
CLAUDINE: I'm curious - you said you don't come from a technical background.
How did you become interested in attacks on speech interfaces?
Well, a varied background - I came into cybersecurity as a second career,
so I had accumulated a fairly mixed bag of knowledge, some of it useful.
I started out in humanities learning Latin, which I really enjoyed at the time.
It detailed how language and grammar works. I always wanted to follow that up.
I'd studied in my own time. When I got the opportunity to do the doctorate,
I was looking to use as much of my prior knowledge as possible.
Language technology and speech,
even though at the time I didn't know much about it, seemed to fit that bill.
As I looked into it, people started to use Alexa and Google more widely.
I feel I was quite lucky as well in that it coincided with a more general trend.
I didn't know that at the time - a stroke of luck, and [it] fitted previous interests.
ARI: Which general trend was that at the time?
MARY: when I started looking at this in 2015, it was still fairly unusual for people
to be using Alexa and Google and voice on their phone and things like that.
My impression is that that's become more widespread since, more ordinary.
Perhaps less so amongst security researchers who have concerns
about Alexa listening in on their conversations,
which was kind of why I pursued this research. Certainly, looking at my family
friends, they all seem to be using Alexa in a way that they weren't five years ago.
That's not something that I can claim to have foreseen, it's been lucky for me!
When I started the research, I wondered if it was too niche,
possibly something that would go out of fashion within a couple of years.
That doesn't seem so far to have happened, so that's encouraging.
ARI: Are we looking at voice hacking here?
That sounds quite cool, doesn't it? Very sci[ence] fi[ction]! Well yes, in a sense that
hacking is trying to get a system to behave in an unintended way.
And that's what I've tried to do with voice control - provide speech or audio
input to devices that will provoke it to do something that it shouldn't do,
according to the developer's specifications.
Yes, you could describe that as voice hacking,
it's perhaps a little bit in the realms of science fiction for the moment.
ARI: You're also a penetration tester. We talked about general, headline security
but also these technical details. You've said you're interested in these details...
What is pen[etration] testing and what interests you about it?
MARY: I did an internship with a pentesting company during my doctorate.
What it consists of, is pretending to be the person who's trying to attack a company.
But your intentions are good rather than bad. You pre-empt an attacker
attacking their potential victim yourself without intending any actual harm,
and give them a chance to prepare for someone doing that for real.
Having been in academia, considering theoretical and abstract topics, to
the real world of vulnerabilities and management of large systems,
there is quite a contrast. It was quite a valuable experience to be aware of it.
You learn that you might you might solve something as a theoretical problem.
But implementing that in reality, in society is going to be very different.
Some awareness of that as an academic is good.
CLAUDINE: How did academic experience inform professional experience (and vice versa)?
Oh, that's interesting question. They are different worlds. That is as it should be.
[In] industry you're not looking for the perfect solution that will be valid forever.
You're looking for something that will be of practical use in a given time and place.
Everything is done under time pressure. You don't have time for perfection.
Whereas, in academia there is certainly a luxury of having that time and lack of
real world consequences, to consider things. Each have their own place, really.
We need people who are exploring things on a theoretical basis and there's a need
for practical implementation where academic concerns need to take
second place to more immediate, financial, practical things.
I certainly wouldn't say that one is more important or significant than the other.
I just think they both need to be valued for what they are.
CLAUDINE: Could you talk more about [your] research - what were you looking at?
MARY: I looked quite specifically at Google and Alexa because they're the
most widespread devices in use, at least in the English speaking Western world.
The first thing I needed to do was gain an overview and understanding of the
technology involved and the potential security problems that it currently has,
and that it could possibly have, and link that to broader cybersecurity context.
So really, the best part of my first and even second year was taken up with that.
In the second phase of the project, I did more practical experiments, looking at
developer versions of Google Assistant and Alexa Skills - apps for voice devices.
For the Google Assistant, what I did specifically was input nonsense sounds,
sounds which don't actually have any meaning attached to them,
(which all languages have - they have sounds which in the current usage of a
language, don't actually have any meaning attached to them).
I'd see what would happen if I directed that kind of input to Google Assistant
and whether that would trigger any unexpected effects.
I was looking for sounds which rhymed with target commands and seeing if I
could trick the device into thinking that a nonsense sound was a target command,
which worked in a couple of instances - quite satisfying!
I also tested the human perception of those nonsense sounds
to see whether they would detect them as a target command.
As it turns out, humans couldn't hear any meaning in those sounds at all.
So it opened up a gap between what a human would hear
and how a device would interpret it. Often in security there's a gap between
how things are in a system [plan] and how they are in the real world.
That's the essence of it. That gap between how humans heard sounds and
how they were picked up by Google was a potential security vulnerability.
ARI: How have you approached the biggest challenge in your work?
MARY: There are technical challenges in trying to write code, getting things to work...
Things inevitably don't work, having having to redo them...
Then an immense challenge of information overload, knowing where to stop.
You can't realistically know everything there is to know about your subject.
On the other hand, you want to know as much as possible to inform it.
The most challenging thing, really, has been a bit more personal.
Trying to be convinced that what I'm doing will (potentially) work and be of use.
One particularly challenging aspect of that is that when you start anything,
any project, inevitably your early ideas will appear, in retrospect, very naive,
very romanticised and frankly, a little bit embarrassing. To move beyond them,
You really have to be prepared to see that, but also not just stay there and
instead move on to finding something that is more realisable and more serious.
So that process, being convinced that it's worth persevering and then not getting
too caught up in your mistakes, that's been the biggest challenge for me,
especially in research where you're not told what to do, you're expected to be
exploring something that hasn't been done before, so you really are not going
to get it right. Carrying on regardless in research is an important skill, perhaps
even more so than technical understanding or reading papers.
CLAUDINE: I have to ask - what was an early idea that you had to move past?
MARY: Of the many examples, I remember submitting an early version
of my doctoral proposal plan and it was something to do with...
As I said, I studied Latin. I had this idea that I was going to somehow link the
theories of rhetoric in ancient Rome to the security of modern voice control.
That was going to (somehow) be of practical use and solve a cyber security problem.
Within six months of learning something about cyber security,
I realised that was, very clearly, never going to work. It was the starting point.
You really do have to start somewhere [and] not get too caught up
in the less than perfect aspects of your earlier work.
ARI: We ask guests "when did you fail and what have you learned?" perhaps
it's about being open to new ideas - you got to start somewhere.
Especially in an academic environment where it can be quite critical.
How do you use that? How do you keep going? What does resilience look like?
Cyber security leveraging Latin was certainly an idea. Then you got to a
very relevant, useful, impactful idea by the end of your project.
MARY: It is difficult because, as you say, especially in academia, there's this
pressure to be convincing, to sound credible all the time and authoritative.
And of course, that's just not realistic. You're constantly learning as you're
going along. There's a balance between being open about your mistakes and
humble about what you don't know. At the same time, having a degree of
composure and valuing what you DO know... having a responsibility to try and
contribute that. It's very much a balance between those two things. You become
obsessed with your mistakes or you can not see (and continue to make) them.
So it's finding that happy medium between the two.
ARI: I took part in your experiment! I remember making up nonsense,
random words, there was some wordplay involved...
What is a target command and how does word play fit into that?
You said that people didn't understand the words, but the computer did?
MARY: Yes. There were two parts to the practical experiment. One was the
nonsense sounds targeting the audio part of the voice controlled device, the
the second part was more about understanding words and meaning.
Most words have multiple meanings depending on the context they're used in.
We do that automatically. We barely think about it. We just use words in
their correct context, with their given meaning. It turns out that we don't fully
understand how we do that and struggle to teach machines how to do it.
In my second experiment (that you took part in, Ari) I was trying to demonstrate
that humans could come up with usage of words which meant something
different to a target command in a given context and feed it to Alexa. Alexa
would pick up on it as that target command (as programmed).
One example is the word "bank". We can say: I was standing on the (river) bank,
lost my balance on the bank... we would understand we're talking about water.
But Alexa, because she doesn't understand about banks being next to
water will pick it up as a command to do online banking and act accordingly.
So you could trick a system into doing something malicious, you could
use a word in a different context that sounds totally unrelated and trigger a
victim's system to do a banking transaction... any malicious act you can
think of, you might be able to work into wordplay. A sinister contrast between
the playfulness of the word play and the maliciousness it could be used for.
CLAUDINE: How pervasive do you think this issue actually is
amongst voice or speech recognition technologies across multiple platforms?
MARY: I was looking more at control of devices via speech.
For the moment, people are more worried about their devices listening in.
Applications of voice control have been limited [maybe as] it's difficult to control.
You wouldn't want to be controlling a military robot by voice if you couldn't
be sure that whatever you said was actually going to happen,
or that you weren't going to say one thing and it would do something else.
The issue of the security of control by voice is one of limit[ed] potential.
If we could solve the security, that would open a number of applications
which are currently too risky but might actually be otherwise quite valuable.
The other issue relating to voice that's happened in practice is deep fakes.
We have images, but it's also happened with voice, where people have been able
to do more and more convincing imitations of real individual's voices.
E.g., being tricked into making a bank transfer.
That kind of use of audio technology is becoming a real security concern.
That's slightly to the side of what I was looking at in my research, but related.
ARI: Does that mean that we shouldn't let recordings of our voice go online?
You've described yourself as a researcher hoping to be an innovator,
It sounds like you are working in a[n innovative] space!
What do you see as the exciting or troublesome parts here?
MARY: It's an area with many problems and as yet few solutions.
I don't think, to answer your question about voice recordings, I really don't
think the answer (in terms of what we should/can be doing) is to just not be
recorded, or limit our communication: A) This is impractical,
and B it is restrictive and fear based. That isn't the answer.
The answer is to find a way to handle this in a secure way.
That is my area of innovation that, if you look me up in 5 years time,
I may or may not have solved.
CLAUDINE: If you had unlimited time, unlimited resources, what would you fix?
MARY: Something which is perhaps even more fanciful than the work I've
already been talking about! When Perseverance landed on Mars, one of
the things that struck me about it was that when it landed, it almost
immediately started sending Twitter updates. That was probably a human
typing at NASA and not the actual robot on Mars.
But having said that, it got me thinking about the fact that, going into space,
we're going to be sending robots ahead of us. Alexa is a robot.
And I wondered, Well, what would the place be for speech technology in that?
Could we train the robot to describe to us the things that it finds?
How would we make those communications secure?
If we ever encounter anything moving on another planet, a robot will be there
on our behalf. How would we train [a robot] to talk to [the alien]?
How would we use speech technology in that kind of context?
How would [the robot] describe a landscape that a human's never seen?
As I say, drifting off into the realms of total fantasy.
CLAUDINE: I feel like that could very well be turned into a sci-fi limited series.
ARI: I'd watch that. CLAUDINE: Yeah, I would, too. MARY: Okay, next project!
ARI: We're talking about interacting or describing the world around us.
Does accessibility come into this conversation?
The potential to voice hack is not just a general issue,
it's also very specific for people who use technology in a certain way.
Have you encountered accessibility or how this might affect interaction?
MARY: Certainly, coming back to more immediate human and social topics.
That is something I've thought about - moving away from the future,
towards people's more immediate concerns.
Voice control has a part to play in assisting people with mobility issues,
turning on the light from across a room or turning off a kitchen device.
The reliability and security of voice control becomes even more important
when it's being used by someone who's who might be entirely reliant on it,
rather than using it as a convenient option.
There are all sorts of endless applications that become
very practical and perhaps less romantic in that setting.
Much as I enjoy thinking about the bigger picture and the exciting topics,
I do think that this kind of technology also has more immediate, more human
and less glamorous applications that are just as valuable, possibly more valuable.
ARI: Is there a push for this or is it something that we don't talk about?
MARY: It probably is something that's not talked about enough.
When Alexa first came in, certainly Google Home, it was more of a toy.
Very trivial - ask the weather, play music... almost a bit of a gimmick.
Perhaps there wasn't enough thought into these more serious applications,
Perhaps because it was not immediately attractive to the wider population.
There is scope for looking at voice control in that context and improving it,
making it more reliable and more genuinely useful.
ARI: even to install your smart light bulbs and connect it all to Alexa...
The barrier to entry is actually quite high.
My friend has some light bulbs, that are smart bulbs and Alexa deals with them,
and I can't. Alexa doesn't recognise my voice but recognises his voice.
MARY: As someone who spent a day trying to install a voice controlled kettle.
Yeah, I agree with that. That's after four years of doctoral research in the area!
ARI: Right? We are the experts, people!
CLAUDINE: With respect to accessibility and voice commands.
Is it your sense that technology developed for vulnerable populations,
individuals who have specific disabilities or needs around technology,
that security by design is a priority, an afterthought or not thought of at all?
MARY: We know in cyber security that... Security has been an afterthought,
full stop for the whole of the internet, for these systems that we put together.
That problem becomes exacerbated when we have sensitive applications
or applications which are used by more vulnerable people.
In a nutshell, security tends to be an afterthought rather than something
that's baked in from the beginning and the system design.
There are new security angles to the fact that we can make things happen just by
speaking to a device that haven't been explored yet.
There are all sorts of implications and security problems to be solved.
We've only just scratched the surface.
ARI: What are your tips for keeping up to speed with cybersecurity?
I try to read blogs, e.g. Bruce Schneier's which is the go-to for a lot of people.
I subscribe to a few newsletters (one produced by one of our alumni!).
When I have time, every time there's a cyber security story in the news,
I try to look into at least some of the background to it.
So beyond the headline, try to look up some technical blogs and articles and
try to develop some understanding of what actually happened.
From a more specialist point of view, I suppose that's quite helpful.
ARI: Who do you go to for these write ups?
MARY: Well, Google tends to throw up CNET, technical blogs, academic articles.
I use Wikipedia if I really don't know anything about a specific area.
Just some kind of technical material that is beneath the media, if you like.
That's a way of continuing to develop [under realistic time constraints].
Join us next week for another fascinating conversation.
In the meantime, you can tweet at us @HelloPTNPod
and you can subscribe on Apple Podcasts or wherever you listen to podcasts.
The title there is PTNPod. See you next week. ARI: Bye!
This has been a podcast from the Centre for Doctoral Training in Cybersecurity, University of Oxford.
Funded by the Engineering and Physical Sciences Research Council.