So one of the interesting things is the way the deep learning has changed,
machine learning has changed even in this physics departments is on Mondays, we've just started an informal lecture series.
One of our former postdocs, Typekit Ruiner, give me lectures on basic, basic introductions to machine learning and packed out.
We advertised only to the physics faculty and graduate students in our final year students and we packed.
We've packed out this lecture, lecture theatre and that wouldn't have happened two or three years ago.
So there's been an enormous change in the interest inside physics, as well as a huge change in interest worldwide.
So I'm going to do a very basic introduction.
Very pedestrian, if you know a lot about machine learning, I'm not going to say anything new until the end.
I apologise for that, but I'm setting up the next two speakers.
Andrea Lucas and Elliot Benton, who'll be giving you some exciting cutting edge applications of machine learning in physics.
I want to start by giving a kind of potted and partial history of the fields and on the go.
And it starts with this great man, Alan Turing, who in this unbelievably important paper computing machinery and intelligence,
which is kind of the founding paper of the field of artificial intelligence. He.
They made many important contributions there, including the Turing test.
But also said, what if instead of trying to produce a programme to simulate the adult minds,
why not rather try to produce one which simulates the child's in this world, then subjected to an appropriate course of education?
One would obtain the adult brain. We've thus divided up into two parts the child programme and the education process.
I'm going to tell you today about the child programme and the education process.
So this paper estimated enormous amounts of work, and it's important when we talk about machine learning that we look at the history
of really machine learning and artificial intelligence on this ambiguity. Those words a little bit later.
You have a little timeline from 1950 down to here arrows wrong way around.
Let's see the Turing Test 1950.
Not that long afterwards, the first automated translators came out on and they kind of worked, but they didn't work very well.
There was enormous interest, huge amounts of money went into it, and then it kind of died out.
In 1957, the man called Rosenblat the myth of the perceptron,
which is a very basic model of a neurone, which will come across a bit further in the talk.
So in the New York Times in 1957. So that's only seven years after Turing's paper and the journalist wrote the navy
revealed the embryo of an electronic computer that expects will be able to walk,
talk, see, write, reproduce itself and be conscious of its existence.
So there's nothing new about hype or used to hype cycles in lots of fields.
But in 1957, this was the the feeling and huge amounts of money poured into this area.
Over 60, there's lots of the end of the sixties, many started to dry up because the promise was not really being kept,
and the very famous book in 1969 by Minsky and Puppets on the perceptron showed that in fact,
these perceptions were not nearly as powerful as people had initially thought.
They couldn't do some basic logical operations like Exor and so on.
Interest in these things start to die down.
And in 1973, drought and light hail reports of the British government commissioned a big report on the AI and a very famous scientist mathematician.
Wrote that there's really going to be no progress in this, and his argument was, well, these problems are typically so complex.
It's a commentary explosion of possibilities that they're already going to work on toy problems and never anything else.
And there came the first winter. I mean, the funding completely dropped people's careers ground to a halt, and nobody wanted to fund this anymore.
Then in the 1980s, we had expert systems come back, and a few of you may remember this on excellent lists.
All kinds of companies jump on the bandwagon. There's hundreds of millions of dollars and pounds poured into research,
and great promises were made of what this would all mean, how this would transform our lives.
But then by the end of the 80s, these systems turn out to not work,
as well as people thought they were very expensive to maintain they were very brittle. And so all the funding again stopped.
And as recently as 2007 in The Economist, I found the following quote investors were put off by the term voice-recognition,
which, like artificial intelligence, is associated with systems that have all too often failed to live up to their promises.
And so this is something close to the end of the second age or when people when there was very little funding in this area.
Now, everything changed, obviously the last decade, we wouldn't be here if that were not the case.
And I wanted to locate the big change in 2012. It's somewhat arbitrary, but it's an I'll explain to you what changed this,
this what changed in 2012 and what made CEO Google CEO Sundar Pichai say something like A.I. is one of the most profound things we're working on,
as humanity is more profound than fire and electricity.
Now I can safely say that is probably hype of this, but he's excited, OK, people are excited and there's enormous amounts of money.
Billions and billions are being poured into this area. I have a friend who recently listed a company, the stock exchange,
and he told me by putting artificial intelligence the title, he doubled the valuation of the company.
And so there's a lot of interest in this, and I think for good reason. There's also a lot of hype.
If you go in the newspaper, you find all kinds of wild claims. You know, I will take away your jobs.
It's going to be doctors anymore. Some new, as you would like, and a lot of that is hype cycle and probably not true.
But they're also extremely exciting things that have happened. So one very thing is, don't we maybe familiar with in 2016,
a computer programme from Google DeepMind called AlphaGo beat Lisa Dole, who was the 18 time world champion and in Go.
And that was a big step forward because as opposed to something like chess, which is a relatively constrained game,
goes unconstrained, has an enormous number of possibilities and was thought to be an unsolvable game.
And so this deep learning new age, I had done something really amazing and perhaps even more interesting.
In the summer of 2017, DeepMind released a programme called AlphaGo Zero, which didn't even know anything about Go.
It just learnt the rules OK, except the rules played against itself, so it didn't look at any games that were played by experts.
And after 40 days, it was able to outperform AlphaGo, which had beaten this at all, which had been trained on all the expert games.
And it had taught itself how to play go much better than other computers, uncertain much better than humans.
And the same thing for chess. They did a chess programme that taught itself how to play chess.
Interesting where it started out, it was about chess player. Then it went through all the kind of standard opening moves that you have it learnt them
and discarded them one by one by one until it became better than the world's best.
The best alternative chess programmes, which are built with a lot of expert information.
So this is a programme that teaches itself how to play the game so that without a doubt is an extraordinary achievement and a very exciting thing.
And so the question is. That's why people are excited, but how do these programmes work?
Well, I'm going to give you a very potted introduction or a simple induction to the basic technology behind this,
and I'm going to tell you why I think 2012 is the start of what we call the deep learning era.
So on this very important part of what's changed things is the accessibility of huge amounts of new data.
This is the fifth year computer scientists at Stanford.
I put her in because she was a physicist originally. And actually,
there's a lot of people that were physicists that moved into this field over the last few
decades that have transformed things that she introduced a competition called Image Net,
where there's now 14 million images. The 20000 categories of images are like a cat, a dog, no bicycle.
And there was a competition which was Take your best computer programme, learn a bunch of images,
and then we're going to give you new ones that you haven't seen before and predict where this is or this is a cat or a dog or a bicycle.
And in 2012, a team from the rivers of Toronto, Alex took the main person.
Was Geoffrey Hinton, actually the most famous person in that field of this network, which is called Alex Nets at 60 million parameters.
And it beats all the other comers by of factory 40 percent better accurate 40 percent lower error.
That was a huge step forward. And now we're down to about a two percent error.
And it's all based on these are machine learning networks. Now here I get a little schematic picture of Alex that still looks pretty complicated, OK?
It's a slightly messy system, but. It got people really interested.
Immediately, Google sort of pouring huge amounts of money into its Facebook into Microsoft,
these companies are now rebranding themselves as A.I. companies. And it's all based on these basic technologies called deep learning.
And although this is not complicated, I'll explain to you in a minute more how that more or less works in the academic world.
This grew incredibly so. Geoffrey Hinton here I will mention again a few more times.
If you look on Google Scholar, this is his you see his citations. So this is 2011, 2012, and it's just exploded.
Last year, he had seventy three thousand citations. So these are citations is the number of citations of a world leading.
Scientists are minimal prise winners who have less citations than that in their lifetime.
And last year, this is how many citations you have. This gives you a sense of how many people are working on this field.
Three of the five most cited papers in nature from 2019 were on deep learning,
so there's an enormous excitement in this and the the three great kind of experts in
this field of three great founders of these fields and the current Joshua Bengio.
So I got the order wrong. It's Bengio.
Hinton and LeCun won the Turing prise, which is the equivalent of the Nobel prise in computer science last year.
What's interesting is that these three pioneers worked more or less anonymously, but at the fringe for many years and just lost.
About two months ago, I saw a quote from Hinton where he talked about given try to submit the paper to a conference in AI and he was before, he said.
Hinton's been working on this idea for seven years, and no one's interested. It's time to move on.
So for many years, I mean, Hinton is in his 70s now, so for many years he worked on these ideas.
Everybody told it was crazy.
They worked through the big A.I. winters and in spite of the fact that everyone told them this is wrong and this was they couldn't get funding.
They couldn't get published. They kept at it. And now the revolution has come.
This is a good lesson for us that very often great innovations come from outside.
They may be. The few things is wrong at one time, they actually turn out to be revolutionary at a different time.
And what changed was the availability of large amounts of data like image nets that I was a trained things and of course,
lots of proprietary data and commercial data. And of course, large computers.
And those two things, plus some algorithmic. Innovations allowed an idea which actually comes back from the nineteen fifties, nineteen sixties,
these neural networks to to revolutionise the way that we're doing artificial intelligence today and what's new about them?
I'll explain a few things new about them. What's new about them is they're not simple to use in traditional A.I. systems.
They can be used by students. Secondary school students can use them now and train stuff.
And so it's a huge revolution in physics.
There's a physics world as an article just last year called a machine learning revolution where machine learning revolutionised physics.
Well, we will. See you. Also, be careful to call it revolutions on.
There's definitely a bandwagon, there's hype. I was talking to a colleague this week who said, you know,
one of the tricks in science is to jump on bandwagons and ride them a job, offer them before they crash.
I don't think this was going to crash because we're seeing enormous and exciting applications.
So it's been a long history actually of applications and data analysis.
Particle physics have actually been using neural networks for a very long time to analyse their data.
And it's one of the reasons why I think there's a natural link between physics and this kind of machine learning based on their own networks.
There's a huge amount of stuff in image analysis. We've got people here working on biological physics.
They're using these machines to analyse images or pictures of cells dynamically.
And these these networks are by looking at images, recognising things that are human.
I seemed unable to recognise patterns in the data are being exploited in that way.
It's even used textbooks make quantum many body weight functions and calculates the energy levels of of molecules or because of the body's systems,
the much higher accuracy than was thought before.
There's a beautiful experiments by some colleagues here in Inferiors Davis and others in physics where they looked at
some quantum magnet experiments and they looked at using a machine learning algorithm looking at their data again.
They had discovered a bunch of new patterns in there.
So there's really exciting ways of looking at this and an image analysis of these ways of controlling experiments and the events in the last talk,
we'll talk about that and much, much more. An enormous number of cool examples.
And the. Just to give you my last little kind of.
Big picture argument, so a big picture story, not argue, a big picture explanation,
so what is artificial intelligence we booked this day is artificial intelligence in physics.
What we're really talking about something simple called machine learning, and I'm talking about a subset of that called deep learning.
Artificial intelligence is a catch all phrase for all kinds of computational
methods that you may have a computer intelligence in one way or the other.
There's lots of methods that are based on symbolic manipulation,
or it's a really wide range of techniques inside that is a smaller subset of techniques called machine learning,
which when you saw the quote from Turing,
he was speaking about training a child of computer like a child is a machine learning user use data to train the parameters of a machine.
And there are many different machine learning techniques, a wide variety of different ones.
And the one that really caught everyone's attention is this method called deep learning.
And that big Alex that you saw that solved that basically did the best image that
competition 2012 and is now dominated the images that competitions ever since.
Is a method called deep learning. So what is deep learning? All right.
This is a pretty basic, so this is the quote from touring again with the second part of the quotes.
We've got two parts to the problem a child programme and education process.
So deep learning is based on the following idea I have some input nodes.
I've got some weight, some kind of.
Relationships, which are which I'll describe in two minutes to another set of nodes, those that have nodes and have an output layer,
so the input might be pictures of cats and pictures of dogs, and the output might be this is cat resources a dog?
And very, very loosely. This is inspired by neurones in the brain's neurones in the brain can fire or not.
And so firing would be within the when this notice has a large value and not fire would be as a small value.
And how it connects to the other nodes is the same way a neurone in your brain is connected to many other neurones.
And so these weights are taking that into account. So I want to just draw one or two things here on the board.
I realise you can only see a small amount of the board, but that's OK.
So I'm going to give you a really, really simple example.
So if I have three input nodes like this, OK, those are my input nodes, and I might have a problem that I'm interested in something like.
I ask you three questions. I'm a doctor to figure out if you're ill or see if you need to be quarantined, for example.
So the first question might be, you know, do you have a cough? And so this could be one or zero, right?
The first the first one question one question two would be, do you have a little bit and the degree one or zero?
So yes or no? And the Question three could be, did you just come back from Italy and be one or zero?
OK. And then I might think of some, there's some kind of logic that I would apply to this.
So, for example, if none of these are true. Then the answer would be you don't need to be quarantined, OK?
And maybe if you've just been to Italy, I would point to you no matter what.
And you can kind of even think you can see how this would go on, right? So I could do one zero one, maybe if you sniffle, but you have met Italy.
I will always watch about you. But if you sniffle and you've been to Italy, I will definitely quarantine you, etc.
And I can go one. I'll just I'll just go through it really quickly.
This is not necessarily at some point my logic thing breaks down, but I will not pretend that this is how I do this.
What's what you see? Something interesting, by the way, is I've got one to one two three four one two one one one.
That's zero one. So for each of these inputs?
OK. I have an output, and those are my inputs and outputs.
So what I've actually done in this particular problem is I've got a little function, right?
I function takes this set of inputs and it gives, in this case, a set of outputs.
If I was a well trained doctor to see whether squinting or not,
then this would be my inputs, outputs or perhaps the government would give me this advice.
And that's a function that I then have. And so I met with the train a computer to be able to learn this function.
And this is a relatively simple function. So how would I how would I do this?
Well, I would say I've got three input nodes one to three, which can be zero or one, which.
And then I'm going to have a layer here of further notes.
Maybe lots of notes, and I'm going to call this layer one layer to.
I have an output node, and that I'm going to do is I'm going to make lines here that tell me this node connects to all the nodes in the first layer.
This one does the same. This one does the same.
OK. Simple, incredibly simple. And these nodes do the same to the next layer.
OK. Etc. And I can keep drawing forever, and at the end, the notes all converge on this final decision nodes.
It's all going to draw on, can draw the lines because they keep going on, there's tons and tons of them.
I can have many layers and I want to do this. And then what I do is I'm going to put on each of these layers away, so I'll call this a weights.
This is number one. And this is layer one, right?
This is the first one to three, four or five I call this weight one one.
I call this weights from here. Maybe I'll colour it so you can see it. So these are the weights that are going into number one.
There's three of them weights one one four eight one two and weights one three.
I couldn't quite figure out how to make this go up, so I'm going to jump to the side,
I'm going to hide this little truth table here for a minute or two over here.
So sorry. All right. So what is going to happen to this node?
OK, what's going to devalue this nodes? Well, the value of that nodes? OK.
So the node one in there one.
Is going to have a value, which is the function, some function of the inputs.
What are the inputs? Well, it's x one weights one one plus x two.
Wait. Two to two. Sorry weights two one plus x three weights three one plus maybe an offset.
OK. And this function will typically be something like this would be a a classic one.
And this simple function looks like this. So this is for large negatives ads.
This is zero. At zero, it's a half, and for large positive z, it's equal to one.
All right, so if this inputs is large negative, my nose goes to zero.
Is this if this inputs are large and positive, it goes to +1 otherwise as a value in between?
So it's extremely this is this is one of the many on activation function you could have.
So this is really this thing here. Is this really it's X in W plus b that is the function of an Inter product.
Now I do that for this node, for that node, that note that they're not nodes.
And then I think those nodes and use them for the next layer and I do the same game and I play the same game at the end.
I look at this final node and then I might say if the value is larger one half, then I should quarantine you.
If that's enough. I will quarantine you. OK. So that's basically the the way this works is incredibly simple.
It's almost you wonder, how could something like this if I put a lot, if I put enough of these together, how could that?
Learn how to play go, for example, or how could that learn how to recognise images is quite a striking thing,
but it's incredibly simple, mathematically, unbelievably simple.
You're just adding up numbers, multiplying them and then having a little known linearity at the end.
The threshold? But it is not in the news, it is important because without it, this model just is a linear model.
You can write down as a series, a matrix multiplications and a product of matrices as just a matrix that becomes a linear model of this do very much.
So there's no linearity turns out to be really important. All right.
So that's the first part of the basics. The second part of the basics is that I need to to have some education process.
And there are basically three main classes of education processes.
The first one is called supervised learning and it's supervised learning.
What I do is I have. I have two sets of data, so here's a very famous example called end this.
These are on hand right hand handwritten numbers. And so these were used to train.
In fact, some of the earlier neural networks will be for 2012 that we're using automatic scanning of of of checks, for example.
And so this is zeros. Those are the ones that are too. So I might take this first subset here and then I will take them as inputs to my machine.
And then I will try to play with these waits until for each of these images.
If it's this, I get a zero. If it's this, I get a one, two, three, four or five, so I train it until I get zero error, hopefully.
Or typically I get zero error on my training side. Or I can even make it simple.
If I want to do this example, I would say, Well,
let me pick this as my training set and this so I'm going to give you these four inputs with the correct outputs.
I'm going to ask the automated Optimiser to find weights that will then if handles inputs, give me those outputs.
OK. Then I've trained my system and then I'll test it.
OK, so test it, so I might do it on something more complicated this image.
I would take these ones and see how well I do.
Or I might do it on if I was trying to train extremely simple DR algorithm, I would train it first and then eventually I would test it.
And then once I'm happy that it always gives me the right result,
it gives me the right result with high enough accuracy than I would unleash on the public.
So that's the first of ethical, supervised learning, and most of what I'll be speaking about today later is supervised learning.
Does that make sense? It's a super, super simple thing. One way of thinking about this is that what these things are are a kind of computer programme.
It's fitting some kind of function. And you should have quite written the programme yet because you have to figure out what these weights are.
So this training process is a way of having the machine write its own programme.
OK. So we're thinking about what it's doing. It's writing some programme, seemingly some.
Another kind of learning is called reinforcement learning, which is somewhat similar to this.
You're trying to train these weights, but rather than using label data where you know what the answers are in advance,
you've got a bunch of curated stuff. You you have some kind of process that an agent is going through and some environments.
And every time the agent does well, you say this is a good set of weights of keep something like that.
And if there's value, then you, you penalise it and you keep doing this rewards personalisation technique until eventually the system learns.
Well, that's what AlphaGo zero Alpha Zero does. It trains on games, and every time it wins, it plays against itself.
Without wins, it says that's good times. It loses isn't bad. It's very roughly akin to what we do with our children.
They don't always work quite as well as we fortunate immune system and the lost methods, which which is also super important.
But I won't talk about which today is unsupervised learning. So it's supervised learning.
I might have, you know, two vectors that tell me some property of objects and then all the red ones are here, the blue ones are there.
So I'm trying to learn some decision boundaries. Some line that tells me if my parameters are here, I'm right here as there are blue.
It also provides learning. I have no idea what the the axes are.
I'm looking for patterns of data and I might see that these these are clustered together and those are clustered together.
And so I'll start picking out features of the data, and that's extremely important as well.
If you don't if you've got too much data, you're not quite sure what the right way of thinking about it is.
And unsupervised learning techniques allow you to find patterns there that you wouldn't otherwise see.
Just the basics of how these things work,
and so I think it's absolutely remarkable that a system this simple like what I just showed you maybe souped up with more layers or some tricks,
but effectively this is what you're doing that this can achieve these amazing feats.
And so the big question, I think the very super interesting question for us today is why do deep neural networks work so well?
It's completely remarkable. Why do they give us such incredibly good predictions for all kinds of amazing things
you're going to see on methods of this type being used to find patches in string theory?
They've been used to find, obviously to play games. They've been used in all kinds of image recognition of about six, seven years ago,
Eric Schmidt from Google said We should stop training radiologists because they're all going to become irrelevant
because this image recognition is going to look at your scans and tell you whether you are this cancerous or not.
And already there are on. There are some studies that show that these things work roughly that well,
they can work better than human and best trained radiologist at recognising cancer.
The reason why this hasn't why we have not stopped treating radiologists is because there's a lot more to radiology than just looking at images.
And also, it turns out that these systems are more brittle than you might expect.
So a classic example would be you do this, you train your system on hospital a,
you get a very high accuracy and you break the hospital B and then the accuracy drops by fairly large amounts and nobody quite knows why.
But those are the kinds of problems that we have. And so until we solve these fundamental questions about why they work well in the first place,
we won't be understands why they they have these funny ways of not working well in other ways and other times.
The classic example that's not really well understood is we have a adversarial example, so I can show a computer a picture of a.
Gibbon, sorry about Panda, and by tweaking a few of the pixels cleverly, it will completely be confused and categorised as a gibbon, for example.
There were more frightening examples. There's a recent exploit by a Chinese group called Tencent, where they use.
They did this on Tesla cars. So Tesla cars, like many, many systems in an industry, namely self-driving systems,
use some kind of machine learning to recognise, for example, the lane that you're in.
And they were able to put a few small stickers on the lane and confuse the network so that it thought that it was in the wrong in the wrong lane,
it swerved very quickly. Now before you, before you sell your Tesla very quickly,
the way this works is that you get paid a lot of money by Tesla if you find these kinds of errors.
So you there's there's a whole industry of people that try to find these errors in Tesla.
Tesla will then pay them and then tell them, give them a timeline after which the ramp would be public about it or not.
And so this group did that, but it was quite striking because a human would never do that.
So one of the questions is why do these machines?
Why this is sceptical to things like like adversarial examples, and there's many other interesting questions.
And so although although. We know that these things work very well.
There's really a question of why they work so well, so I'm going to unpack that in a very particular way.
So one really fascinating thing that we've known for quite a while about deep neural networks,
neural networks is that there are, in fact, universal function approximations.
So here's the most recent, probably the most recent theorem by Boris huntin,
but there's a long history of theories before that's really exist for any kind of function from some high dimensional space to the real numbers.
So, for example, this would be from a hydrogen space to a real output, which is either one or zero.
It's there exists a fully connected rather network revenue is just a fancy word for one of these activation functions.
And it's important technically, but not that important for you to understand.
It's basically something like that. And with if you have if you have a width of the of the network,
so if the number of nodes in this layer has to simply be less than and plus four inches, four is the size of dimension of your space.
So you were a three dimensional space.
So if I have a seven dimensional one, I should be able to to produce completely every function in of this of this type.
It's a very powerful kind of theorem. And so that tells us that the whole networks are extremely highly expressive.
That means they can they can fit almost anything you throw at them.
Now why is that kind of interesting? Because it gives us a conundrum, OK, if they're so highly expressive that music can fit any function to the data?
Why do they pick the right function? So let me give you the example here, right?
So I've got. Three bits. All right.
That gives me. That gives me two to the end is a number of different bits I have.
So that gives me eight to eight different possible inputs, how many different functions I have, I can.
The function length is eight. There's two to the eight cities, two to the two to the two to the end functions.
That this can produce, so that's two hundred fifty six.
Have you seen the 56 different functions, so I can the tune of fifty six different ways I can map these bits to these outputs?
Okay, so if I do it and then if I train, so if I train my neural network to give me zero targets, all of these correct.
OK, then there's four left here, right?
So there's 16 possible functions that are all consistent with being correct here,
the 16 different functions here and the system to know what these functions are.
As if I train, it's how does it know? Why is it?
It can do all of them. So why is it pick a good one, which is what typically does?
Does that make sense? It's extremely simple question.
And just to give you a sense of how these numbers grow, the number of functions grows really quickly to the two to the end.
So if I have seven functions, it's going to be two to the hundred and twenty eight, which is ten to the thirty eight functions.
If I have nine, it's due to the thirty eight squared. OK. And if I have this, if I have 10, if it keeps going about eight.
So it's two to thirty eight. Squares have nine two to thirty eight squared squares.
It goes incredibly fast.
So I have roughly if I worked out that if I have nine inputs, OK, and I look at all possible functions, there's tons of that 156 function functions.
That's a lot more than what we currently think. The universe has bits of storage, and if you use every thing you could in the universe to store.
OK, so that's a relatively small system.
Actually, nine questions and I asked you, what are all the possible combinations of ulcers that you could have?
And there are more than that can store in the whole universe. This is what Light Hill was trying to say.
The Coventry explosion, of course, is this even a relatively small problem becomes unbelievably big and.
Intractable, and so he said you're never going to get beyond toy problems, which seems reasonable in this kind of arguments.
Now, fascinatingly and then on that,
we'll be able to reproduce all those very large number of functions because of the universe approximation theorem.
So why does it pick a good function? This is a really interesting question.
And in the field, this was kind of sharpened by a very famous paper by saying it's all being called.
It's sort of been cited, I think, almost 50 times in the last four years, just extraordinary.
And they did define the experiment. They took this image datasets called See Far from Canadian datasets.
So this is aeroplanes and automobiles and birds, cats and deer, et cetera.
And they simply that if you're permitted the labels, so they basically label this rather than label aeroplane, they just picked a random label.
OK. And what they showed was a neural network could learn that new corrupted data relatively quickly.
So not that much more work, and it could reproduce the training that was 100 percent accuracy, zero error.
That's very striking. How could it? How? So it can do that, right, so it can memorise the data.
Obviously, if you then give it new correct images, it gives you zero accuracy, 100 percent error because it hasn't learnt anything.
It's just memorised the links, the labels and images so it can do it, which we we knew theoretically, you could.
But they showed that we could find that solution really quickly, which in itself is exciting and interesting.
And the question is whether why does it generalise? Well, why does it give us good solutions, right?
Given that if I gave it the correct labels, it could just memorise the labels but not have any predictive power.
So why does it not do that when it's given the correct labels, whereas it can do that easily when given incorrect labels?
This is a very big question in the field and a very interesting one.
And as physicists, of course, we're very nervous about these high dimensional systems.
I told you there are 60 million parameters in Hinton's in the Alex Nets from 2012.
There are no models with billions of parameters in them, and they work extremely well.
So why are we worried? Well, as physicists, we've told that you should never have too many parameters, right?
There's a very famous story by Freeman Dyson. He's talked about.
He went to Fermi and he had a more high energy physics model with, I think, five parameters in its.
And he went to Fermi and Fermi. Also, how much data do you have? And he didn't have that much data.
The Fermi said that claim that von Neumann had said to him with four parameters I could fit an elephant with five.
I can make it wiggle its trunk. And so what you're see, we may see the data well, but it is probably not true.
And so Dyson went home not to Cornell with his tail between his legs, but that's an intuition that we teach our students all the time.
Never used too many parameters. If you think if you're these things as a group from 10 years ago that worked out indeed for Norman was right.
You can fit elephant and make a bigger stroke with five parameters with four is not enough.
And so it's just great genius. So I'll give you a very simple toy problem that visualises this in a really simple way.
So here I have a 10 data points. I can fit those 10 data points by, you know, Apollo mule.
Right. And what we tell our students is if you've got 10 data points, don't fit a higher polynomial to it.
So here you see this dash line is a five or polynomial. If I fit a 20 order polynomial to it, I get zero error because I can fit it extremely well.
The data extremely well, but it starts to show very odd behaviour, which intuitively I think is probably not going to generalise well.
Other words, if I give it new data points, it's going to give very bad predictions, right?
And that's what that's what one woman and Fermi were trying to tell Dyson.
Now, here's the fascinating thing we've trained here a bunch of neural networks with one, two,
three, one, two and five hidden layers there, but with them, it doesn't seem to matter very much.
All they feed data like this, that's the green curve. They actually on top of each other in this on this scale.
So the question is why do these networks with thousands of parameters fit the data so smoothly that looks to our eye?
This seems to be much better. So this is a kind of central conundrum in the field.
Why do they work so well?
In fact, there was a famous kind of spat between two younger and another al-Hashimi and two important theorists in this field,
where the first one said, You know, deep learning, machine learning is alchemy, OK?
It works. We have no idea why, and this is the kind of argument they're having.
We have no idea why it's generalised so well. No, I wouldn't be setting them up in this way if I didn't think I had something to say about it.
And so last time when you hear I explains a new idea from our group,
which is actually generalising something that's been around for 50 years in the field of algorithm information theory.
So and very often it is a study of the complexity of single objects.
It's an information theory, but as opposed to sharing information, which is about distributions, it's about single objects.
And so the central thing in this theory is a goal of complexity,
which is formally defined as the length of the shortest programme that will generate a particular string on a universal Turing machine.
So I'll illustrate this to you with a very simple example. Imagine a monkey typing on a typewriter using a typewriter,
or I should say a word processor now for the younger people in the audience who have never seen typewriters.
The point is the result. Let's make it simple as a binary one is only zeros and ones and rookie types on this.
How likely will this monkey type the following one hundreds digit long sequence?
Well, the monkey will type with one half to the power 100 because the monkey what monkeys are not truly random,
but let's assume that we have a random monkey. That's what the monkey is going to do. OK?
There was actually experiment in Portsmouth Zoo where they they gave a typewriter to a monkey.
A bunch of monkeys. And what they found is the monkeys kept tapping the same thing many, many times to preference for K, and then they defaecated on.
And that was the end of the experiment. OK. This is this is a hypothetical monkey.
OK, well, on on a binary keyboard, this is the input output, the probability we will have the power and hundreds.
But what if the monkey was Typekit to some kind of computer programme?
Well, then we might expect that you would get something they might actually type the sequence of prints,
so that's not binaries, assume it's a proper computer.
So the print zero 150 times, let's ignore for a moment the the it's what we want over the size of the keyboard,
but it would be something that scales is to the power. 19.
Are making it doesn't quite work in binary arithmetic, but I'm just doing it because I'll need that binary to stick later.
The point being the if you're typing into some kind of programme that you might accidentally type print zero one many times,
it'll give you a short description and call. The graph shows that mathematically is the right way of describing the complexity of a single object.
There's also the reason why this hasn't been applied is because this common graph complexity links to the halting problem in Turing's machines,
which links to girls on desirability theory. And so there's lots of uniformity can never know whether you have the short programme.
But it's you can approximate, and so what we did is this we said, let's assume that we can approximate anyway.
OK. And then we said to the probability that you get a certain outcome should be something
like one half to the power of the shortest programme that gives you that outputs.
And some countries in there which we could ignore. But the first order, qualitatively, that's what we're saying.
So we're saying this might be a much more general property of maps of Input-Output maps, a map of inputs with an output.
This is inputs outputs that should give me something of that nature.
So the question then is how can we think of the neural network as an input output map?
And we have to be careful here because the input outputs.
But you can also think about it in a different kind of input output map, which is a parameter function that I've got parameters here.
OK. So arbitrarily change. OK. And I've got outputs which are the function that these parameters pick, right?
And so the inputs are the parameters and the outputs are the functions.
Inputs are not the inputs. They're all network. The the parameters that I choose.
And if this coding theorem is correct, then we should see that a poor, randomly picking parameters.
I should get functions that have shortcomings of complexity.
Now, we've one of the great things about being in Oxford is that you have unbelievably talented undergraduates,
so an undergraduate that worked for us over a summer programme and he was thinking about something different.
He's actually working on reinforcement learning. And then one day he came to me said, I've been thinking, I think I've proved something.
And so he's and it turns out that he hadn't quite proved something, but it almost proved something which we have now proven,
which is that for a very simple neural network, it's called perceptron.
We only have inputs then weights and outputs, which it's like a zero layer neural network.
We can prove that for this kind of. Binary problems, OK?
The probability upon random sampling these weights that you get a particular function.
OK. We can't prove that. We can prove that you get a certain entropy.
The functions or a certain number of ones is all equal.
So the probability I get zero ones, which is all zeros, is equal to the probability of all functions that have four ones.
OK. But as you can see, if I have four ones, I can permit them in lots of ways.
There are many functions that give me four ones, but there's only one that gets me all zeros.
So I'm much more likely to get the one with all zeros than I'm going to get this particular one.
OK, because it's one of a big class of functions, so we prove that exactly. And we also do.
We then did some. This is for a very simple system. We then took a neural network with multiple layers.
We randomly picked the functions we took and the seven.
So seven of these seven of these inputs to this one handed to an eight possible inputs is
to turn them into an eight just two to the tenth of the thirty eight possible functions.
OK. And then we just repeat as many times. Also, what one? How often you see the same function again and again?
Well, what we find is the the this plot right here. I just want to point out the scale of this graph.
We know the tens of thirty eight functions. So if the functions were all equally likely to appear, which you might assume is of no model,
then you would get a function with probability of one in tens of thirty eight.
You'd never get the same function twice in the age of our universe, even with the fastest computers by random sampling.
What we find here is that some functions are appearing almost 10 percent and one in one percent of time one in one thousand.
That's minutes. That's thirty five orders of magnitude thirty six automated, more likely than you expect on no model.
And so there are these functions that are appearing with high probability when I randomly pick these these parameters.
And the question is, do I know which ones they are? Well, I just have this theory here.
This theorem that says the probability of getting a function should scale with two to the minus the complexities.
If I put the log with the probability analogue scale here versus the complexity of the function.
In fact, the the function, I basically say. So this would be a slightly more complex function.
A function which is zero one zero one zero one zero one would be simpler function or all zeros would be simpler.
Here I brought the complexity of the function,
and this theory is actually the upper bounds is the upper bounds that we calculate from theory, and it predicts exactly what you expect.
So other words, you're very likely to get these simple functions and very unlikely to get complicated functions.
We've shown this holds much more generally for neural networks.
And so this is exciting because we show that they have this fundamental bias where simplicity.
And what's interesting is you can do other games so I can do a training, OK,
I can train manual network on a complicated function, on a simple function and then ask myself, how well does it do?
So I'll pick half of my inputs to train and the other half the tests.
And what you see is as the target functions as a function of trying to learn, it's more complicated.
Actually, my error goes up. So one thing we see is that these networks do well on simple tasks, but poorly on bad tasks.
And this is one of the top. Here's a random learner where you just pick a random function that fits the test.
And the vast majority of functions that you pick randomly are terrible. Not expectedly.
And so the random learner basically acts like might as well just pick might not learn at all.
And what's interesting is if the network, if you train it on a simple function, it finds functions with this complexity and this error,
so it tends to bunch around the one that you're trying to find and this with the random nerve does.
But if I give it a complicated function, it behaves more or less the same as a random murder.
So that's what you expect. Networks are not magic, right? They just have a bias towards simple functions.
So if I give them simple functions to learn, they should do well. I give you a complicated function.
They're not going to do any better than any other kind of random guessing.
So you might think that this would be very exciting if people believe this is the reason why these things generalise so well,
they generalise well, because just by a simple function, the simple functions give you a good generalisation.
But what we find, what what we find, and I think for good reasons,
people are sceptical and the reason they're sceptical is because you train a neural
network by randomly picking parameters that would be really silly and very slow.
Instead, what you do is you your lost functions,
you calculate how far your function is from the function that you want and then you you use a gradient descent mythical.
So castigating the sense which drives you down down your lost landscape to a minimum.
And that's an optimisation algorithm which is extremely effective and efficient.
And the kind of consensus in the field on the most common argument in the few
would be that there's something magic about study that gives you good solutions.
So people push back on this just because the opportunity probability of a function is high.
Does it mean that you're going to find it that way on on going down gradient descent, which I think is good arguments.
However, we have a kind of qualitative argument says, you know, on this space, you've got lots of different possible functions.
You could find the all fit the data. If that function has a large base in so many different ways, you can find it.
You expect that that it's that high loss, you're still going to be likely to fall into that basement and go down into it.
Right. So this is here are some functions with large based in size.
You're going to probably go down into here where this one has a very small and so it's based on is going to be small.
So, Chris, this undergraduate has done the second piece of work is together with a bunch of other undergraduates, actually.
So I've got three undergraduates working on this project and myself and Guillermo and a few others.
He trained on this image that called mists and that he worked out how likely was do to find a particular function.
So these are the functions with one error out of 100, so a one percent error, there's a function of two percent error.
And this is a function with zero error.
And what we found is that the probability that the function is found a priori is very close to the probability that is found by stochastic.
Great instance. And in fact, so that means that occasionally finds poor functions with a probability that we can predict more or less.
And so this is exciting because we can we think we can find more or less the probability with which these functions are found.
So this is the first time I've presented this data, this data that came from last week.
OK, so it's very hot off the press, but we're excited by we have actually a bunch of other examples of this,
and we think that this may therefore be the explanation.
So this a priori explanation seems to be qualitatively at least what study is doing upon training.
If that's true, then this explains why neural networks work so well.
What it doesn't explain what explains part of expertise is they are biased
towards simple functions in the same way that Occam's Razor is biased towards.
You know, you will learn about very registry that you should you should be biased toward simple solutions.
And that's kind of the deep question that is, why are the things that we look at in nature?
Simple physics we're used to looking at things are simple and so we intuitively think that must be true.
But why would that be true in general?
Well, for this and this dataset, actually, there's it's it's a grid of twenty eight by twenty eight pixels, so 700 eighty four dimensional space.
But you can work out the dimensions of the space by looking at how things scale with size and the dimensions seem to be about 14 or 15 dimensional.
So these images are in a very low dimensional manifold in a very high dimensional space.
They're therefore simple. So a bias for simplicity is good for this kind of image recognition.
But more generally, I think it's a wide open question about the nature of the universe, the nature of of the kind of things we're studying.
Why are they so complicated and why are they simple? And we're claiming that there is a bias for simplicity, which is good.
And if it's good, it should explain exactly why you see what you see.
So conclusions, I think what she's learning is transforming physics, it's not just hype.
I can make that point, but I will be made later today, but I think is very important.
The main reason that hopefully of today and then what I tried to argue to you is a deep learning may work because differing
techniques are work because they have a natural bias towards simple functions and the kind of inbuilt Occam's razor without.
I thank you very much for your attention.