So it's a great honour for me to be here and be for all of you. Thank you very much for coming on.
All of you alumni and friends of the.
Of course, we're very excited about Phoenix. I hope you are one of the great things to be involved in, as Egypt's, you pointed out earlier.
And I'm going to talk about some things that are actually physics, although you may not have considered them to be physics.
So we're taking a number of kind of ways of thinking about the world and applying
them to a very different set of of problems are typically not in physics,
and the question I've given is a very broad, very grand and deliberately provocatively and over the top ground.
Why the world is simple, but I want to start by giving you a some motivation for this question, and that's looking at this very famous story by Borg.
Yes, the famous Argentine writer, a book from 1940 one has the library of Babel, and in this library there are books.
Each book is made of more than 10 pages, with 40 lines of 80 characters,
each using 22 letters plus period space and covers of twenty five characters in total and every book in this library,
every possible book of that length is made. And if you count the number of possible books, it's 10 to the it's basically 10 to the 1.8 million.
So a very, very extraordinarily large number of every possible book is in there.
And this part of the story is that people go through his library trying to find the book that describes
their life because any book for them 10 pages long that describes your life is in the library.
So lots of complicated things are there. The, of course, the point the book is made.
They're also very precious variations of including books. Where are you speaking to an audience of alumni?
And suddenly meteorite comes in and crashes, and that story is also in that library.
Every story is in that library.
And so every book is equally likely many stories in there, including the story of your life, but the chance of you finding it is extremely small.
You could go look. Of course, the question might be, is every story equally likely?
Well, every story is probably not equally likely. There are definitely stories that are relatively simple that can be told in many different ways.
Therefore, you'll find that story repeated many times in the book,
in many books where their stories are very complicated that you need all of them for them 10 pages to really properly tell,
and that's to really only appear probably once. And so I'm going to be interested in this little analogy of all possible sequences of letters.
How many? What fraction of them will carry certain types of information.
And this brings me to the question of why the rules are simple that I've been worrying about for a while,
which is when I take on my computer and I zip a file, I can almost inevitably compress my files enormously.
And I was wondering, is that special or not? Well, that's for the sake of argument.
Take assume that all of my things are encoded in binary strings where there's there are two length strings of length.
One, there are four of length to eight of three. There's two to the end of length each one.
But that's why I have got a string of particular length and there's two to the other possible strings.
How many strings are there that are shorter than that? Because I've got to encode compress something into a string, which is shorter.
Well, for this one particular, there's eight strings of length. Three. But there are only six strings that are shorter than length.
So I can't compress more than six of my strings. And in fact, most of my strings.
I can only compress my one bit. There's only there's there are only two strings that are two bit shorter than three bits.
And so that's the principle that holds up. So the number or the fraction of the strings that I can compress by end bits is
one over two to the power of the number of bits that I want to compress it by.
So if I want to compress a string by 10 bits, they're only one one thousandth of all strings of length of whatever length you have
can be compressed by 10 bits or more and have all compressed my string by 20 bits.
Only one in a million of all strings of that length, whatever length it is, can be compressed by by 20 bits,
which is not very much because when I zipped files pressed by hundreds and thousands of bits.
So the probability the number of strings that can be compressed by an incredibly small fraction of all the strings that are possible.
And the reason is because the number of strings goes as to to the end, it grows exponentially, exponentially.
So it becomes very large, very hyper, astronomically large numbers are very.
Are easy to make, so most strings are extremely rare. So why is it then that most of the things that we compress that we use in our daily life,
why this in principle and why is so much of the things we see in nature?
Why are they compressible this thing collection there to kind of understanding things one way one we can think of or misunderstand something?
You've compressed it into some kind of simpler description, and that's a way of understanding.
So I'm going to give you an intuition. It's going to sound very vague and fluffy, but I'm going to hopefully tighten it a little bit.
So imagine a monkey typing on a computer, OK, I would use a typewriter.
I realise I guess you guys will find typewriters with my students. I don't know where these things are.
So on a word processor and you want to ask this monkey to type in PI? Well, how likely is it if it's a truly random monkey to type in PI?
Well, let's say there and keys on a typewriter. It's one of 10 to the Power X plus one Exige as a party because it got three points.
One force is one extra digit, so the probability is extraordinarily small.
The 50 keys one, if they get three or four songs, 150 square, they get points, etc. So the probability extremely small.
But it's possible in principle you could do it. This trope of monkeys actually goes back to a long history if it goes back to a
very famous mathematician in the world who was 100 years ago pointed this out,
it's about monkeys, but it's interesting. Think about them. In fact, they don't type randomly.
There was an experiment the University of Portsmouth, where they put some monkeys in a in a zoo and gave them a typewriter.
And apparently these kids had a preference for K K, which they kept whacking and defaecated on the typewriter.
And that was the end of the experiment. So these are hypothetical monkeys typing on a typewriter.
Now what if instead of typing though on a word processor, you gave the the monkey a c programmes the monkeys typing on a C programme?
How likely would the monkey then type the first digits of PI?
Well, there's actually a competition for the shortest programme in C that will generate PI.
A bunch of short programmes,
and so this is one of the ones I think it's the it may or may not be the current records under three three characters long.
Correct. If you type this in by accident or by by thinking it will generate the first fifteen thousand pages of pie correctly.
In other words, if a monkey is typing on a computer. He might by accident type, the first time with the religious of this programme,
and suddenly you will get by a very large supply will appear with much higher probability than any other random number.
Another round number, but you can't generate algorithmically. The only way you're going to get it in writing, print that number.
And so basically typing into the computer or typing into a processor, will the first order be the same amount of length?
So that intuition can be sharpens. And the way that that has gone on would be using ideas from from.
Alan Turing, so Alan Turing, very famously in 1936, came up with the universal Turing machine,
which is a computer that can do any possible computation.
Now people don't often is the reason he came up with this was not to make a computer, but to prove that a lot of things were not computer.
In fact, this article, which is on the numbers with the applications to the answer to the decision problem of David Hilbert.
So this is the Dave Hoover's famous decision problem.
The idea that could you come, could you from a set of axioms, have an algorithm that could always decide whether something was true or not?
True? Within that set of axioms and Google,
and then Turing proved that it wasn't by showing that these universal computing machines have one thing that they cannot do.
You cannot prove there is no algorithm that will tell you that a particular input programme will generate an output, which is called holding.
So the computer has to have an input comes in, it runs for a while and then it holds that it gets an output and you can prove that
you can prove that you can't prove that a particular input will generate will,
will make the computer whole. Let's called the whole thing. And ergo, you can reduce that to to any kind of any kind of mathematical logical problem.
Therefore, there are there undesirable? And so based on the idea of true machines, we then see two other great geniuses of growth and cheating.
Who start thinking about the question of what is the complexity of a sequence, so I have a sequence of, say, a binary sequence.
Can I describe this complexity? It turns out that the fundamental,
mathematically profound way of describing them is to say the complexity of a sequence is the
length of the shortest code that will generate that sequence on a universal Turing machine.
In other words, if I have a sequence like this which is zero, many times it's probably quite a short programme.
I can say zero one, 50 times and I'll get that sequence. Where's this sequence beneath?
It may be quite complicated. Maybe it's the only programme that will generate is print that particular number, so it's a complex sequence.
And what's interesting about this definition is that so to define that formula, I need the particular universe to machine.
But because any universal machine can always emulate any other Turing machine by writing a compiler.
In principle, if these things are long enough so I can ignore the compiler terms,
the the complexity of a of a string is independent of the string machine that I use.
So it's a it's actually a property of the string itself up to these compiler terms.
So we say that they are asymptotically these that a particular tree has a very particular code of complexity.
The problem is that because of the whole thing problem, you can never actually calculate correctly and know for sure that it's the of complexity.
Colloquially, what that means if I give you this particular sequence of digits,
you don't know whether that is actually complex or whether there may be something like pie, which generates that as its first number of digits.
And so you never can prove it, but it can define it.
And this gives a whole series of other interesting intuitions.
For example, the definition of a random number is no whose common goal of complexity is its own length or slightly more,
and it was the only way you can generate it by printing it.
If you can generate the number by some other algorithm shorter, then it's not a random number.
That's the definition of a random number morsel of randomness.
Also, the complexity of a set can be much less than the reflexive elements of a set, so the complexity of Baucus's library is extremely short.
Take all old on books and in 410 pages long with 80 characters per page.
Twenty five letters and that's a pretty short.
I can take the entire library of August, whereas your own life, which is described by one of those books,
or maybe several those books, depending on the complexity of your life, that is pretty, pretty complex, right?
So the interesting thing is that you can have the complexity of a set. It can be a much less than the complexity of the individual elements.
This is a lot of kind of things that seem non-intuitive at first, but it become very clear and very precise in this of language.
And the reason I brought this up is because actually, the co-author of Almagro of Complexity was done earlier by another great genius essential genius
called Ray Sullivan of who was trying to effect all of us very heavily influenced by Coin-Up,
a very honest philosopher from the Vienna School who was trying to formalise induction.
And so he was trying to think about how he could do this in a computer, as he said,
Well, if I have a universal truth machine, let's assume that I feed at random inputs.
And for that make life simple. We're going to make a universal truth machine that only takes binary codes as input.
So I give it inputs and I see what it does.
And I asked myself how likely on giving it random inputs, do I get a particular output x that can be some particular string?
How likely will this computer generate that string? Well, it's the sum over all programmes that generate that string on the university machine.
That's what it is. And each programme of length, since it's binary, is the probability I get a programme of linked L as one half to the power L.
So it's the sum of all programmes terms probability that you get the programme, which is a probability of one half to the par.
Now the most likely programme is the shortest programme because the one you're most likely to type by accident.
And so the first thumb in that series is two to the minus K, which is a goal of complexity.
So if I can help with idea before coming off of, it's called Rugova's a name that has stuck by the kind of Matthew principal.
You know, he has more shall be given until he has not.
Even what he has has been shall be taken away. Which is why we know it has caused growth complexity.
And so this is really interesting, this gives you at least a lower bound on the probability that you're going to get
the output by randomly generating and it's given by the chemical complexity.
Now, another great genius there 11, also known as the one of the founders of the idea of p p, it's a great thinker in mathematics.
Computer science proved that not only was there this lower bound by soul,
but also an upper bound up to these kind of pesky order, one terms which are terms that are linked to compilers,
etc. And this is very interesting because this tells us that if I have any kind of system that can be generated by universe through machine,
I randomly give programmes into it.
Then the probability that it produces a particular outputs can be bounded tightly by a half to the power of the of the object.
OK, so that's a very beautiful thing. I think it should be more widely taught because it's a really cool.
It's amazing. Now, the reason why it's not so widely taught is because there are problems in applying this,
and the problems are many systems we care about are not universal true machines.
So in physics, many things are not universally not during universal.
And one of the reasons we know that is because we can take all inputs and outputs.
By definition, we know there if we solve the whole thing problem that can't be a true machine.
And Gulf complexities, by definition, formally on Computable. So that's problematic as I need it in my bones.
And of course, many systems are not in the assets,
and clearly this is only true in the limits of keys that are large enough that I can ignore all of these kinds of things, like my my compiler terms.
So. I've been interested in this for quite a while, and I've got two very brilliant DPhil students come out Dingle and Chico Camargo,
I think Chico sitting in the back is to the fields of work. And so we work very hard on trying to generalise this coding theorem for a known universal
map so we can up the details you can read in this paper that came out just recently on.
But we have a bounce which says that the probability that you get a particular output on random inputs to a computable map,
it's a map of inputs and outputs that are well defined can be still bounded by the same kind of two to the minus ache
with a little squiggle above the K means we approximated K by some good approximation to global growth complexity.
So a compression algorithm is something that gives you good.
And we've got some B, which is a term a constants. That's an offset, basically, which we don't quite know how to fix.
We actually know how to calculate a the first consonant just based on that on the properties of the map, not the properties of the output of the map.
And B we can fix by taking a few measurements. So for this to work, the maps have to be simple.
That is the if my system grows in size, the maps complexity has to grow slowly with system size.
I have to find a good approximation to K. That's a that's a bit of an uncontrolled approximation, but we can try and.
And interestingly, this bound this quite it's an upper bounds, but it's relatively tight on four random inputs via random input into my map.
Then the output should be close on average close to this bounds.
And this only works for non-linear maps because the ones that are interested in. And that doesn't work for particular kinds of maps the or that.
So you see, a map generates mainly, say, pseudo random number generator, which which you use to generate you run the numbers.
They're never really random. They're actually they're conmigo factors relatively short because you run, which generates a short code on,
but they're made so that they will fool a random number generators, which means they're also incompatible.
So you have to have maps that don't have that kind of behaviour. And so we show that a wide range of systems in nature, in fact, behave this way.
So here we have these are oranges, oranges and the tropical speaking about the reiber zone, which is made of RNA.
So those are things so that what you have to then do to make that is you have a strand with
this code of four letters on it and then it falls into a well-defined three dimensional shape.
And then you can study this folding. And we've studied this folding and you can ask yourself if I have a particular strand,
if I just think random strands, how likely out to get a particular shape.
And it turns out that if you randomly pick strands and you look at the shapes that you write out their complexity in terms of complexity,
then the most likely strands you're going to get by randomly picking most likely shapes,
you can get about really picking strands or in fact, simple shapes of compressible shapes.
And interestingly, when you look at nature, the shapes aren't that you find in nature are exactly the shapes that we predict you're going to find,
because those are ones that have short descriptions of complexity,
and they're therefore very easy to make by random mutations because there's many, many sequences mapping to them.
So those are the simple stories for as many different possible ways the same story can be written.
We do the same here, and this is another example we have here. This is a set of couple of differential equations in this particular case, the model.
This is for circadian rhythm, but actually, we just took it as a set of in this case on, I think,
five a couple of differential equations with about 30 parameters, and we just randomly vary the parameters.
So now the input is around, they change the parameters, and I look at the outputs and also self.
Does this set a couple of differential equations that generate a very complicated output or simple outputs?
And you see this line, this black line here is our bond.
The Red Line is a boundary to be to zero, so we ignore B, but so that means there's no free parameters.
One free parameter do a slightly better bond ends up for every particular complex, every particular output.
This is a simple but simple was much more frequently than complicated outputs, and there's bound as tight as boundaries extremely well.
Remember, this is the log of the probability, so there are indeed. Outputs that you get down here that are simple and rare.
OK, but they're very rare, so the vast majority of time with low quality, you're close to the bounds.
And we also can make maps for which it doesn't work at all,
this is a matrix map where the input that's in the matrix that does the computation and then there's outputs.
So a matrix, you think about it, if it's a map, it grows as if it's if you have an inviting matrix is square matrix.
It grows and squared as you grow. So the complexity of the map does not get that grows very quickly with size.
And for that argument, say, Ah, this theory should work, and indeed it doesn't work.
You don't get simplicity.
So for some, this device you need to have at the set of rules that describe how the input process to the output have to be simple,
which is true for, you know, how RNA gets translated, how RNA sequences turn its structure.
There's a set of physical laws that that I wrote that are not really change as you make the artist strands longer.
And perhaps the differential equations not changes because different equations ability to study.
I can do the same with simple financial models, and everything seems to show the simplicity bias.
And just to show you just a few people I know from experience, people will wonder, aren't you looking at some kind of entropy argument?
So this is the this is a compression of the outputs.
And here we have just heard what I mean by the difference, entropy and complexity.
So here's the entropy of strings. This is all strings of length.
Thirty. So the entropy is basically the fraction of 1s and zero.
So it's true the five old ones. OK, that's a very simple string. So strings with low entropy down here also have low complexity of old ones.
Are all zeros, right?
But there are also many strings of high entropy, but low complexity like here that would be a string of five zero one zero one zero one zero one.
It's actually binary. It's entropy, the same fraction of zeros and ones as entropy seems real to be high, even though it's a highly structured string.
And so what we're picking up here is something far beyond engineer picking up these kinds of patterns in nature.
So why are most strings that you see?
Most of these are close to maximum clock, but why do we see so many compressible sequences nature?
One answer may be because the patterns that we're looking at are caused by effectively a process sampling
algorithm and and then they'll be especially biased towards what are effectively simpler outputs.
So going back to my monkey analogy, that was fairly vague and monkeys typing on a typewriter or into a C programme.
Well, what this whole language of information theory tells you is that you can not worry about any particular programme the
monkeys typing in because there's a there is a and you can always write a compiler from one programme to the other.
And I can write and and then I can formalise this in terms of the decoding theorem of it.
What we've done now is gone a bit away from that great generality is something slightly less general, which is maps that are simple,
and we show that we can nevertheless generate a pretty good, a pretty good description of what happens to a wide range of different physical systems.
We lose the lower bounds. OK, but we have a really good upper bound upper bound seems to work.
And so far, we've not really found any system for which it doesn't work in one way or the other.
And there's a lot of other stories behind it, but I won't tell you what I'm going to give you two applications,
so I'll give you two applications one application just evolution.
So evolution happens by randomly changing genotypes that can translated by the laws of physics and biology into a phenotype,
which is the organism of some kind do the other.
So I'm going to put those and then if I have time, I might look at machine learning with deep neural networks, which also effectively does.
It can be thought of as an input output map, but I'm going to hopefully explain why those two things work.
So let's think firstly about some. This is a movie that's the Julia showed of.
Of the self-assembly of the bacteria for motor and the.
What's interesting about that is many silly things, including how on earth is that thing self-assemble?
But Nature designed this particular system by randomly changing the genome types, the genes that made those particular proteins.
And so the question that I got interested in is, how can how does that work?
Because this is a very specific system. It's really hard, actually emulates how on Earth could do by randomly just changing genotypes.
Get something that does this. That forms is very exquisite, three dimensional shape.
And so we came up with a model. Which you call the pull of the Dominos, so these are typical physics model of proteins or proteins,
are the molecular building blocks of the of life, really.
And they form dimer astronomers, customers hexamer all kinds of complicated structures, including that bacteriological motor.
So we're going to replace it with a simple model of little squares on a lattice.
And these scores are interact with each other with particular interactions, and then hopefully they will self-assembly into well-defined shapes.
So here's an example of a polynomial, which will play all roles because no dominoes is to try those three polynomials multiple ones.
And so here I've got a set of particles with letters on the outsides.
I've got a set of rules that tell you who sticks to whom. And it turns out if I have this set of rules, who sticks to who I am,
and I put this number one particle down first and let these other ones just randomly fall in and out of my system,
I'm always going to form this particular structure, and the reason is no. One.
Has eyes on the outside sticks to see the unknown with see in this case is number three.
So these three will stick, nothing else will stick. 3S have on their sides.
They have these. These stick to be the only one with these number two.
So 2S will stick to the outside and etc. And twos have 0s or 0s everywhere.
We don't stick to anything. And so this will deterministic the always self-assembly is of this particular structure.
So that's a very simple model of how self-assembly works in something like that,
flagella mature as well, so these things will always form its particular structure.
Now I can ask the inverse question, which is if I have a shape. Can I find a set of rules that will make that particular shape?
That's a kind of fun little game, and you can use lots of different methods do that include rather theoretical techniques,
but a very clever DPhil student, Ian Johnston, who?
Who decides to try this and he said, well, he said Monday, me, I tried something different.
I just said, make a 16 year, OK. I didn't say we're 60 murders, make any 60.
And so point is a voluntary recreational mathematics.
It turns out there are 30 million seventy nine thousand two hundred fifty five different 16 years.
So I initially thought, Well,
if you just randomly runs an evolutionary algorithm that kind of randomly picks different matrix and matrix interactions,
you should get any particular shape with a probability one in 13 million.
But no, what we found is half the time we got only these structures, these first twenty one,
50 percent of the time by randomly either doing a Darwinian kind of code or just randomly changing things just completely randomly,
just seeing what they make. We got these twenty one out of these 30 million,
and that's actually how I got interested in this coding theorem because I just want to get the problem I gave to to come island to Chico.
Just say, Well, why is this happening? Why are we going to get this very small subsets?
Because what we realise is that subset a special subset, you get these first twenty one, they're highly symmetric.
So there are only in the set of 13 million people. The owner knows five, which have D4 symmetry.
That's the symmetry of the square means you can rotate and flip them and they themselves.
And all five of those are found in this first set of twenty one.
So by randomly looking in the space of algorithms in space, that's fine.
It's making these things. I'm finding all these highly symmetric shapes and thus gave us that gave us the clue to start looking into this eight and
then took to the fields to turn it from a very abstract theory of global growth complexity to something that worked here.
And if you actually look and this whole system, look at probability,
you get a particular output x as opposed to the complexity of the structure complexity being the number of bits.
I need to encode the algorithm that makes that particular structure.
Then the very high probability ones are the simple ones and symmetric structures have are simple.
You don't need very much information to encode a simple structures. They make something that was repeated multiple times.
So symmetry and low complexity are deeply linked to one another.
And then we've worked with some collaborators in Cambridge, and we've discovered that you can look at protein clustering,
the protein database, which is about thirty thousand that are known.
And they've looked at the complexity of those protein clusters, not them onto the same onto the same complexity measure.
And we get almost exactly the same behaviour as we get for a simple domino model that makes is very happy when they're simple.
Models give the same results as nature. And this line here is our bound that we've predicted so are able to predict this upper bound.
And this tells us that not only is this biased towards highly symmetric simple structures there in the polynomials,
but it seems to be there in nature in these protein clusters.
And you can break this down further so you can look at, say, six hours of which there are many in the database.
And this is the complexity down here.
And what you see is a simple six months of very frequent nature and complex six hours are rare and more or less the frequencies that we predict of.
And so we've done a whole series of evolutionary models. I just show you one because I just saw this.
She was in the audience. So Nora Martin was an undergraduate with me.
She worked on Richard Dawkins Bioworks.
Richard Dawkins is a really beautiful little model of shapes that you can make that he uses to show the great power of evolution.
And he or she shows that the probability of complexity of these shapes.
If you if you search this space of algorithms show the same behaviour that we predict for for all the other systems.
And Nora was an undergraduate here. She went to the other place where she's doing this.
So you win some, you lose some. But we have really brilliant students as you as as I think many of you were as well.
It's actually your undergraduate work, and what's interesting thing about this is that actually,
if you look at these rare structures down here that are low complexity and low probability,
it turns out that they're the strings that make those structures.
So this the codes that make the structures of the cells simple so that they have particularly simple codes that make them.
And that's that's interesting. All right.
And then in the last 10 minutes or so, I want to switch gears to something entirely different, which is machine learning.
So you probably see a lot of machinery.
It's been in the news, enormous amounts, and the news has largely been dominated by one particular methods, which are neural networks.
Neural networks are very loosely modelled on the brain the way they work as inputs layer that give you some kind of inputs,
and then you've got a series of little interaction with other with other nodes.
These are weights and say X one is these nodes are either one or zeros.
And this one is one. Then the weights, it multiplies all the ones on this input that are one give a weight to the next layer.
And this was given its next layer, and you can have several of these.
They're called deep neural networks if they have several layers in a row and there's an output that says,
for example, yes or no, this is a cat or is not a cat, for example.
And what's really powerful about this is that they're incredibly good at things like pattern recognition.
So in 2012, very famously, a group from University of Toronto, Jeffrey Hinton's group, completely obliterated the competition in this competition.
Were you given a bunch of images, you're supposed to recognise them.
And then so you train on these images, which means you get a bunch of images that say cats and dogs and horses and cows.
And then you, you're given a whole bunch of them. So you train your computer to do correctly on that set that you've given that you're given.
That means you adjust as little weights so that when you give an inputs of pixels, that is a cat's.
The outputs of this thing is cats. And when this has dog is dog, that's how you train it and then you're given a test.
That's what you haven't seen before of new pictures. And then you run those to your computer and then you see how well you predict.
And these machines are extremely good at this. And so they predict with much higher accuracy knew their methods.
What you what you put in initially that's called generalisation is called
generalisation because you're you're showing that you're able to generalise on new,
unseen data. And this has dramatically changed all kinds of things.
So if you now do you, Google Translate is no longer it's using basically these kind of pattern recognition using large
amounts of data of of text that are translated or put through it and eventually just memorise it.
And you don't even need to know anything about language at all to work on these translation techniques because these machines have become so good.
Now what's very confusing about them is surprising is that they're highly overprivileged, right?
So a typical nerve network has millions and millions, millions of parameters, but you're only feeding it really a few thousand data points, typically.
And we know from experience,
if I think a small number of data points and I have more parameters than I have data points that I start getting nonsense.
I mean, we're taught, we teach this to our undergraduates all the time, you know, don't put in. It's a very famous quote from von Neumann.
Give me four elephants I can four four four three four parameters that can fit elephants.
Give me five and I can make it wiggle its tail.
And so very recently, a paper in science last year where a bunch of A.I. researchers were saying that this is alchemy, right?
There's no reason why we don't understand why this works. There's no theory for it. It works clearly.
But it's, you know, we don't understand and all the kind of classical machine learning theoretical ideas just break down for neural networks.
So I have a very bright DPhil student.
This is a pattern that you hopefully you've noticed lots of bright you feel students who came to me said, Well, let's think about this.
Can we apply these ideas about simplicity, bias to deep neural networks so we can have a little more of a problem, which was I have a system of.
Of seven bits. So there's two to the seven possible different strings and that old Boolean
functions of that subordinate function basically says if this is on and that's off,
then yes, this on this on the left, no, right. So they're actually a very large number.
Then there's two to the 128 different possible Boolean function.
That's what turned to the thirty four. So you might think that if I randomly pick parameters in my Emanual network, in other words,
if I randomly choose these weights and then I ask myself, What function do I produce?
They get each brain function with the probability of one in 10 to a thirty four.
OK, what we see here is here we plots a rank plots of the probability of getting a function and rank.
So here's the first attempt to the functions. And you see these probabilities are much, much, much larger than 10 to the minus thirty four.
So functions certain functions are appearing quite easily by randomly picking them.
And in fact, if you look at the complexity of those functions, these are the low complexity functions.
So we're very excited about this. You know, this is a toy model for neural networks.
Just last night, actually or this morning at 2:00 a.m., your boy sent me a picture of a much bigger network.
This is for a this is actual, proper real network that we used on a four layer deep CNN.
If that interest you, that we use on sea for data and we get exactly the same line.
So we're very pleased that because he's done some very, very clever tricks to be able to work out this probability.
So this is exactly what the neural networks do. These neural networks are highly biased towards simple outputs,
and that may be the reason why when you give them a bunch of data, they don't overpromise.
So even though the number of functions that this thing can generate is extremely large.
So for this Boolean thing, it's tens to throw thirty to the thirty four.
It gets biased by the simplicity bias towards this very small fraction of simple ones.
And we tested this, for example. So here is the generalisation error on the typical supervised learning.
So I've got 64 strings that I train on that then give it six version, which it hasn't even asked.
So I know I have a secret Boolean function I I make I do 64 strings and see what the function would say.
I train my network on it and then I give it the other sixty four strings that I haven't seen yet and ask, Does it find the right Boolean function?
And this error is the fraction of time it gets those things wrong.
And what you see is that, ah, if we randomly sampled inputs, then we get errors that are quite quite close to the to right.
I'll go to small the order a few percent, whereas if we randomly pick functions, for example,
if we just ran the functions and then test and then found functions which exactly
fit the data so we can find lots of functions that will never fit the test.
The training data, because the render is then rated on the test data, they give almost no sense and they give an error of almost 100 percent.
So this Boolean networks are hugely biased towards this very small fraction of simple functions.
And what's also interesting is as you make the target function is more complicated.
So as you generate your data by a more complicated, targeted function, these networks do less and less well.
So this is the error that they make if they're trained as opposed to the complexity of the target they're trying to make.
And what turns out, even though they're extremely good,
if they if you give them complicated functions so we can measure this complexity very accurately,
then they actually they no longer generalise very well.
So the fact that general, as well as because they're looking at simple functions, what we're particularly excited about is that we can apply.
We've now use these ideas in kind of classical learning. Theory is a very big hole,
big field of computer science for lots of textbooks and thousands hundreds of thousands of papers trying to make bounds on how will you generalise?
Because you can make a rigorous bounds, then you know what your maximum error is going to be.
And so we've been able to adapt these these ideas of bias to do a particular technique called PAC base.
Actually, it has not as much to do with base as you might think on. And this is the this is a result that we get for the the error bounds.
And what you see here for these curves on the bottom, these are M. This is a this is a database of of handwritten on handwritten numbers.
So you have to do well. And this is C for which these images and so obviously on C four, you don't do this is your error that you do.
And what we do is we randomise labels. We corrupt the labels on the test sets.
So if I crop the labels, I tell you that a one is actually a five,
then I'm not going to generalise very well because I've got some errors in my tests.
And so as the errors get bigger, my generalisation is less good because I don't give them very good inputs.
And so that's what you see of the error growing as a function of corruption for these three,
these two three, two different two different classes of systems.
And this is our rigorous bounds. So this is the upper bound on the generalisation error.
And what's really I think this and this means this is no longer alchemy.
We actually predicted analytically with no free parameters with the maximum error is that you're going to guess and we're not that we're.
Depends on what we're pretty close to, what these things do, even though we're in this heavily over parameter regime,
this regime, which traditionally machine learning ideas shouldn't work.
And the reason this works is because what we argue or arguing is that the reason why these
neural networks work so well is because they're intrinsically biased towards simple functions.
And so when they give it a whole bunch images, they're actually picking out the simple things that fit the data,
rather than the complicated things that fit the data.
Whereas if you just randomly pick, if you do something kind of standard regression, for example, there's a very high order polynomial,
then you're much more likely to pick a complicated one than a simple one unless you've got a bias in your system.
These things are intrinsically biased,
and because the patterns the the reason why they work well must be because the patterns that they're studying must also be simple,
because if they try to look at complicated patterns, I'll show you they don't work that well.
OK? As we showed you as well. So this tells us that the patterns are studying are in some way the other simple.
So I gave you this very kind of grand claim at the beginning,
why the rules are simple and I don't think I've completely explained it, but I've given you a list.
Some things are simpler because you can think about them searches in the space of algorithms.
So if I think about possibility, space is the space of all possible stories, for example, right?
Then if I randomly look in the books of August, I'm going to find simpler stories much more frequently.
I'm going to find complex stories.
And so even though the probability of a particular very particular sequence of characters in a book is equal, every book is equally likely.
The probability of getting simple stories is much more likely than the probability of getting complex stories.
And thank you very much.