Okay. Good morning, everybody. Assignment three is due now.
I hope that was okay. It probably took a bit more work than the other ones.
The solutions are out and we're happy to talk about them later on.
If you have questions, I thought I'd show you a couple of things on the screen related to that.
So let's see here.
My little code for the ellipse fitting problem there was that we used the backslash at least squares in order to fit in ellipse to some data.
Not a true non-linear ellipse fitting problem, but a linear ized version of it.
And let's see. My code is called a 31 C.
So if I. Run that, then I can click.
I hope you got this to work on your machine.
Just put in a bunch of points and then as soon as I click the right button, I get some kind of an ellipse.
I hope that worked for you. Let me try one more. Let's make it highly elongated.
So you can see it's always an ellipse centred at the origin with this particular formulation.
Obviously one could generalise that. So in this case, it doesn't fit the data too well because the data is mostly on one side of the origin.
Then the other problem was the SVD problem, where you're using the singular value decomposition to find some eigenvalues of a non-trivial problem.
It was the modes of oscillation of a quarter ellipse.
And I hope that went all right for you.
I thought I'd show you online that one of the fun examples is a similar the same mathematics applied to a different problem.
So if you go to the fun examples. Under partial differential equations.
One of them is called Eigenvalues of a trapezoidal drum.
And that's using exactly the same method you used for the ellipse.
So there's a drum. And in order to find, again, modes of it, you can do an expansion in Bessel functions basically,
which satisfy the eigenvalue equation but don't necessarily satisfy the boundary condition.
So you find coefficients to make such an expansion zero on the boundary.
In this case, all the challenge is that that particular corner, because that's the one where the function isn't smooth, it's not analytic.
So you use the right Bessel function for that corner and outcome.
The same mathematics you're used to pictures like what you saw.
And then you fool around with your parameters. And then then you find a value of lambda, for which the smallest singular value is zero.
And that's an eigenvalue of your oscillating drum. Okay.
So that's that. There's one more assignment which will hand out on Friday, and then that's the end of this term.
I guess that will be do in the usual place two weeks later, even though the lectures stop after this week.
Okay. So let me remind you that we've been beginning to talk about optimisation, and this is a vast,
complicated subject, but I like to start with its purest, cleanest version, which is Newton's method.
Last lecture. We talked, first of all, about a single equation, which everybody knows.
Then for a system of equations which is completely analogous, mathematically,
you now have to solve a linear system of equations at each step of the nonlinear Newton iteration.
Then we began to talk about minimising a function which is closely related, defined as finding a zero of the derivative, but not exactly that.
Now you can probably guess that the next step is to minimise the function of several variables.
So that's what I want to talk about now. Newton's method for minimising a function of several variables.
Mathematically, we can think of two or three variables, but of course in applications it might be in the millions depending.
Well, it's. It's like the difference from here to here, only more so.
Mathematically, we have what looks like an equivalent system, but in fact, a lot more complexity comes up.
Let me first of all, remind you that we use lowercase F for minimising.
So we're now going to suppose we have a function lowercase F which maps from our N to R and we're trying to minimise that.
So we seek an x star in our m r n such that it's a local minimum.
So F of x star is a local minimum.
And of course that will be related to having zero gradient, although that's not enough.
But certainly if you're a local minimum of a smooth function, the gradient will have to be zero there.
So let's talk about gradients, which are our first derivatives and then also second derivatives.
So in general, if you have a function like that, the gradient is a vector of partial derivatives, as you all know, I'm sure.
So the gradient of S at a point X in our N is the vector.
Which we would normally write as a column and vector.
So the gradient of F is the column vector consisting of the partial of F with respect to x1, x2, and so on down to X and.
That's the gradient. And then to do Newton's method, we want to find zeros of the gradient,
which means we need to differentiate the gradient in order to extrapolate at each step.
So we now need the second derivative of little less to get the first derivative of the gradient, and that's called the hessian.
So the question of F. Is the Jacoby end of the gradient.
Remember, the Jacobean is a matrix of partial derivatives of a vector function.
Here's our vector function. We now make a matrix of partial derivatives.
So I could call it like this delta F of x, which is the gradient of F differentiated.
And that's an NBN symmetric matrix of partial derivative.
So in the upper left we have partial of F squared with respect to X1 squared.
And down here we would have partial of F squared with respect to x n square.
And of course, down here, partial of F squared with respect to x one and x, n.
And up here partial of f squared with respect to x1xn.
And as you know, it's a fact of mathematics that at least if it's smooth, then it doesn't matter what order you do.
These derivatives so partial with back to end one is the same as partial with respect to one an end.
So it's a symmetric matrix. The question is a symmetric market.
And there already we see something that's a little bit different.
When you solve a system of equations, you have a Jacobean which need not be symmetric.
But when you're minimising, the Jacobean is a special one which is symmetric.
So minimisation is not the same as zero fighting.
So let's write down Newton's method. You can do that without my help, but I'll do it.
So in its pure form. Newton's method for minimising this function of several variables, we would say, given an initial gas.
It's not as always, which is now a vector.
Then we go into our iteration for k equals 1012 and so on.
First of all, we have to evaluate the gradient and the fashion.
So evaluate the gradient and the fashion.
And as you can imagine, as a general matter, those could be very serious challenges,
both in terms of the amount of work involved and in terms of the accuracy.
What method are you going to use for that? Then we have to solve our system of equations which will involve involved the hazard matrix.
So here's the system. Delta of death.
There's our matrix. Times.
The faith step should be equal to minus the gradient of F.
So we solve that symmetric system of equations for s sub k and then we do our update.
We say x, k plus one equals x, k plus s k.
So there's the idea behind a lot of optimisation, but it's only the very,
very beginning of the subject you recognise it's just Newton's method for finding zeros of the gradient.
So I thought, of course we play with that on the computer. We're going to have to bounce back and forth a little bit between machines.
Let's start I think we start on the usual machine.
So if you look at the code called M 21, Pure and Newton men, in that case, I've written down exactly this.
So I've taken a system of two equations written explicitly the function F, the gradient G and the hash and H.
And by the way, those letters are very commonly used.
So we've talked about F people often say G for the gradient and H for the fashion.
So let's see if we go to that code. It's called AM 21.
Let's try it. I hope I'm in the right mood here.
Yes. Okay. So there's a function, the function F and we're trying to find the black dot in two D.
It's pretty easy to see it on the screen. And of course, we'll follow an iteration.
And what we need to do is pick our initial guess.
So for example, suppose I take a to two as my initial guess.
Then you can see the red dot and at every step I'll take a step and you can see that one has not done well.
So we're already it's not converging. We're already seeing the fragility of Newton's method.
You're. I probably figured that would be close enough to get to the black dot.
But Newton's method, it's all about the curve curvature of those contour lines because they're curved sort of in the wrong direction.
Newton's method, which is using that second order information, is going the wrong way.
Let's try again. So let's try one one as our initial guess.
That's got to work, right? Is it working?
The first step is not impressive, is it? But then the second step is looking great.
Now has to look at the numbers here where we're expecting quadratic convergence, of course.
So F is the thing we're trying to minimise and then G is the gradient which we're trying to set to zero.
So let's see if that thing is going to zero. Take another step.
It is smaller. Of course, we're not seeing the red dots anymore.
The gradient is now zero to several digits and now it's ten to the minus sixth, and now it's ten to the -13.
So sure enough, from that very good initial gas, Newton did fine.
And just because I can't resist, let's do a couple more initial guesses. So if I try, for example, two minus two.
Does it work? Looks promising. Yep. Very promising.
But isn't it striking how the magic of Newton's method just doesn't appear until you get very close to the thing you're looking for?
At the moment, it just looks like some crude approximation. Look, it's taken the wrong direction.
Oops. It's gone in totally the wrong direction. So you see how fragile Newton's method can be.
We're going to have to take a better initial guess. Let's take 1.5 and minus point seven.
Will it work? Nope. Looks bad. It's have are.
It's going to. Oh, now we've run. That probably was about to converge.
But I ran into the limit on the number of steps. So you can see that.
When computers came along and people started to optimise.
Newton's method was only sort of an idea, a vision.
Most of the work is that we're going to have to somehow get near that point in order to get our quadratic conversions.
Okay. So I want to say a word about the more practical side of optimisation still without constraints.
We're going to talk about constraints on Friday. We're now talking about the field of unconstrained optimisation.
So let me say a few words about what it takes to get from this kind of Newton's method to practical things.
From Newton's method to practical optimisation.
And I guess I would say that there are two big issues that have to be addressed in this little example.
We saw one of them, which was robustness for bigger examples.
The other one is also important, namely speed. So let me turn to that in a moment.
But first, I want to make a philosophical observation. So the vision.
Is quadratic convergence. Now.
What does that mean? It means that every time you take a step, you double the number of accurate digits.
So you have the error at one step is proportional to the square of the error at the previous steps, which is wonderful, right?
If that's happening, you're in great shape.
If the error at step K plus one is O of the error at step K squared, I want to make a remark about the significance of that condition.
And this is something that certainly puzzled me at one point.
It may puzzled a lot of people. Obviously, it's good if you converge quickly, but what's so special about a square there?
Newton's method, of course, gives you a square when things are functioning well.
But why stop there? Why not have cubic convergence or quartette convergence?
You know, maybe that would have been hard in the 17th century, but now we've got computers, right?
So why not develop something like Newton's method that converges quickly?
Of course, it'll be even less robust. That's a fact. So it will take some finicky care to make it really work.
But it's certainly possible to develop algorithms that converge quickly.
So the question is, why aren't these algorithms famous? And that puzzled me for a while until I realised or somebody told me,
I guess that I'm actually a cubic algorithm is not any faster than a quadratic algorithm.
Let me put it this way. Suppose we had a core tech algorithm.
E k plus one equals o of e k to the fourth.
So the number of correct digits quadruples every step that looks even better.
And mathematically, that too would be possible. You would need fourth derivatives you suddenly be getting.
Tensor is not just matrices. But the point is, this is just like two steps of that, right?
So if you've got quadratic convergence, then e k plus two is o of e k to the fourth.
So the difference between second order and fourth order or any other order bigger than one is only a constant factor.
All you, even you, all you could dream of gaining by going chaotic is sort of having the work.
But of course, you'd never in practice have the work because it would be such a huge effort to write down your core take algorithm.
So there's nothing to be gained in principle by going to different orders.
So all super linear, as we call it.
Convergence rates, which means better than an exponent one.
Are in principle equivalent up to constant factors.
I'm speaking very much in principle. Of course, there are all sorts of practical issues here.
Super linear means e k plus one equals O of e k to some alpha where alpha is bigger than one.
Now, if alpha is equal to one, then it is genuinely slower.
So linear convergence is slower than super linear convergence, but all super linear rates are the same.
I remember when I first became aware of this. There were some methods discovered for computing PI, which were incredibly fast.
They converge quickly, every iteration. You'd get three times as many digits of pi.
And I thought that was an amazing advance over the previous algorithms, which were only quadratic convergence.
But then somebody pointed out to me this observation it's only a constant factor at best.
Okay. Now, in practice, in real optimisation, we exploit this to the hilt and very often we don't achieve quadratic, but we get something.
But with Alpha bigger than one and less than two. So that's realistically what often happens in serious optimisation.
Okay. So now having gone through the philosophy, let me say a word about the practice.
So I mentioned that there are two practical issues we want to talk about.
One is speed and the other is robustness. So to say a word about speed first.
For our two by two problem. It's not an issue.
But if we look at the algorithm that we've written down here, what we see here is the solution of a system of equations.
So that in principle is certainly going to be a cost there.
We have an end by end system, so we expect m cubed work would be the obvious count for how much work that takes.
Then in addition, we have to compute these objects and in particular the hash and matrix has n
squared elements in it and each one is a derivative or a second derivative.
So that may be a very significant cost to them.
So the two issues of speed really are how to compute the derivatives.
And then how to solve the system. And I'll say a word about those.
So in my code, I just wrote down the derivatives, which I computed on paper.
But of course, in general, you don't do that. Now, there are many ideas around for computing derivatives.
The obvious one is some kind of finite difference approximation.
You step forward, you step back, you take a difference that gives you an estimate to the derivative.
So that is certainly a long established idea.
But that's not always the best idea and it's certainly not always the most interesting.
A more interesting one in recent decades is what people call a D, which stands for automatic differentiation.
So the idea here is to use your software to compute the derivatives somehow.
How many of you have done anything with ADD? Okay.
It's very impressive tool that sort of came on board in the 1990s, I guess it was.
It's not a symbolic differentiation. So of course, we all know what it means to write down a formula and differentiate.
That ad is different. It actually takes a computer code and line by line sort of differentiates each line of the code so that
at the end you end up with something which often is sort of the exact numerically computed derivative.
There are various ways to do this. The original idea was to take a code and then transform it to a different code which would produce the derivative.
Then along came object oriented programming,
and a slicker idea was to overload the operations in your code so that you know everything has a new meaning.
So I guess the the o p version of automatic differentiation is one of the standards these days.
Anyway, it started out with relatively small problems and then people found ways to get
bigger and bigger efficiency with bigger and bigger differentiation problems.
Sparse matrices turned out to be a key issue there.
So really the thing that made automatic differentiation take off was a careful treatment of the sparsity in your question.
So this is now one of the major technologies for computing derivatives.
So then how do you solve the systems of equations? Well, of course there's Gaussian elimination, but that's boring.
Now it's a symmetric matrix. So one would hope that you could exploit that.
You would hope it's a symmetric, positive, definite matrix because that means it has a minimum.
If it's not positive definite, you're probably not converging very nicely.
But if it is positive definite, then the symmetric positive, definite version of Gaussian elimination is Solecki factorisation.
But this is not what people normally do for big problems.
So just as finite differences are rather boring. So Gauss elimination is relatively boring.
One hopes to do nicer things like conjugate gradients.
This is such a vast subject. There are corners of it where every possible combination is the optimal thing to do.
A remark about conjugate gradients. When I described it,
I described it to you as a minimisation procedure where you're searching in bigger and bigger spaces for a minimum of a quadratic form.
Well, that was for a linear quadratic form.
But of course, in we're now talking about nonlinear problems.
So conjugate gradient actually has a very natural connection to the larger problem that you're trying to solve.
So this is very important aspect of things.
Now let's take a look. Now let's talk about robustness of it.
Another thing I must mention is that you don't always compute derivatives accurately.
Very often you have some kind of inexact Newton methods where you approximate derivatives.
And the philosophical principle comes into play there.
The exact derivative will give you a quadratic convergence, but often a reasonable approximation will still give you super linear convergence.
So then the other issue is the matter of robustness.
What do we do about the fact that we're probably not close enough to the minimum when we start for Newton's method to converge?
There are lots of ideas here, and one of them is called Modified Newton.
And there are lots of variations on those themes and that theme.
But the simplest one is instead of using the hash an H, add a multiple of the identity to it.
Now, if you think about that, the identity is, of course, a nice, positive, definite matrix.
So one thing to be said is, even though H might not be positive definite, if you add enough of the identity, eventually it will be positive.
Definite. You think of some parabolic surface that might be rising in some directions, shrinking, going down in others.
If you add enough of the identity, you're raising everything up. And now it might be positive.
Definite. Another way to think of it is that when theta is zero, you have a pure Newton's method, and as beta approaches infinity,
you have a steepest descent method, which is a slower algorithm but a more robust one.
Another aspect of robustness is the idea of line searches, which I'm going to illustrate in a moment.
So here the idea is you don't take the full Newton step.
Now, we saw, for example, that. In the example, we show that still up there, sometimes the steps went too far.
Well, maybe not in this one, but sometimes they overshot. Most codes would not do that.
They would have some kind of an exploration where you've picked a direction,
but now you decide how far to go in that direction as you converge to the minimum.
The length of your step converges to the one that Newton's method has recommended.
But when you're far from the minimum, you might take a different length of a step, typically shorter.
And indeed that's related to the other method.
Maybe we should call it B prime of trust regions.
The idea of a trust region is, again, that you don't take a full Newton step, but it's fancier than that.
At every stage in your iteration, you have some sense of what part of the space you feel you have a reliable model of.
And rather than trying to take a step in your whole search space, you can find yourself to a local approximation problem in some local set.
So I'm here at x, k, I'm trying to get here, which is X Star, but of course I don't know where it is.
Maybe I've got this trust region and I only allow myself to go as far as the boundary of the trust region in the next step.
And then of course, I'll update the trust region and hopefully converge towards the thing I'm looking for.
Let's take a look at the table of contents that I copied for you.
So this is one of the handouts. It's a mess, this copy and I apologise about that.
But anyway, it's an attempt to show you the table of contents for the wonderful book by No Saddle and Right called Numerical Optimisation.
So you can sort of make it out, although a lot of the page numbers are obscured.
And I think Chapter six, you can't see the heading of it's called quasi Newton methods in this realm of not using the true lesson.
Just want to give you a sense of some of the ideas on the table.
Now, remember, optimisation has unconstrained and constrain and we haven't talked yet about constrained,
but just looking through the chapters of course is an introduction and then fundamentals, which is more or less what I've been telling you.
Then a big chapter online search methods followed by a big chapter on trust region methods.
So those are the two parallel technologies for robustness, then conjugate gradients, which is a way of speeding things up.
Quasi Newton methods is Chapter six another way of speeding things up, and then large scale problems which have their own considerations.
Chapter eight on Calculating Derivatives. And you can see that it begins by talking about finite differences for the gradient.
Think Jacoby in the lesson How to exploit Sparsity. And then Chapter Section 8.2 is about automatic differentiation.
Then there's this whole business of what to do if your function might not be
very smooth or you don't want to take the trouble to estimate derivatives.
There's a whole business of derivative free optimisation, and I'm sure some of you know the min search code in MATLAB is a derivative free code.
Then there are at least squares problems which are a special case of optimisation of tremendous importance.
Nonlinear equations in Chapter 11 is alluding to finding zeros rather than minima.
And then you can see we're halfway through the book, and then we turn on constraints.
And the back of the sheet of paper is all about how to deal with constraints.
I want to mention a few people. Because Oxford is certainly a place with a lot of expertise in optimisation.
So there are two regular faculty members here, which are Cora Curtis and Rafael Houser in the Numerical Analysis Group.
And these are both optimisers, roughly speaking.
She's non convex and he's convex, which is a big distinction.
And optimisation probably, however, is a bit more theoretical in character as well.
She's also partly theoretical but involved with software.
So he's away this year, but she's a very good person to talk to for advice.
But also there's a visiting professor who visits every week for a day.
Nick Gould So let's see, this is a car, a car, just.
Rafael Houser So this is quite a senior guy and he knows pretty much everything he visits one day a week and is easily found online,
of course, from the Rutherford Appleton Lab.
So if you are engaged in an optimisation problem, it's very easy to arrange to talk to with one of these people.
We also have this term.
Another expert from Belgium, Felipe Twan, is another senior figure who knows everything about optimisation and he's visiting this term.
From the University of Nomura. And I would very much recommend that if your work involves optimisation, it's worth talking to these people.
It's such a highly developed subject. You need to know what people are doing.
Okay. Let's play with some more codes.
So now what I'm going to do is try to show you something about how real codes behave a little differently from the the toy code that I showed you.
And in order to do this, I think because I don't have f men on my laptop, I think we're going to have to switch to the other machine.
Let's see if that works. Okay.
So I'm now going to try the one called f m 22 f min unk.
And I actually haven't tried this here, so it may or may not work.
That's promising. I guess it does have. Oh, yeah.
That's okay. Right. So. We're in the same kind of mode as before.
We're trying to find a zero of this function. And I'm showing you what f men actually does.
So let's start at two. Two. There it is.
Now I'm going to. I've modified things so you can see what's happening.
It's just taken a. I'm going to.
Sorry, I'm going to change something here. Let me try to get it so you can see more.
So there's the picture. And then over here I'll say format compact.
Okay, let's start again. Okay.
So my initial guess was two, two now. So what it does is it.
Begins by trying to estimate some derivatives. So it's just evaluated the function.
But now it perturbs.
Wrong button. Now it perturbs x by such a small amount you can't even see it.
Which means I should say, format long. Shouldn't I try one more time?
Okay. One more time to two. At the next step it perturbs x in order to compute a derivative.
And notice it's perturbed it by about the square root of machine precision.
So the red dot hasn't moved because it's only been perturbed by ten to the minus eight.
Now it perturbs y. So it's estimating the other partial derivative.
The red dot is still the same. And then finally it takes a step.
So you can see it took a pretty good step. Now notice it's just perturbed x again to compute another derivative and then it's perturbed
y y so now it goes much too far and it keeps doing these estimates of derivatives.
So I always have to press the button three times for the red dot to move.
Now notice something that's just happened.
We went from that point to that point and then back to here, which means it hasn't chosen a new direction.
It's doing this kind of line search thing, looking along a line to try to get an improvement.
Now it's probably happy wrong button.
So now it goes off in a new direction. Estimating derivatives, now another new direction.
And now it'll probably be close to quadratic convergence.
Okay. You can see, I don't know if you remember the answers, but it's clearly now converged to many digits.
So that gives a little bit of a hint of how these things work. Let me try another variation.
M22 B now is a code that uses a harder function that we're trying to minimise a famous example called Rosenblatt's function,
and this time I'm giving it a gradient, not making it compute its own gradient.
So let's try that. I'll say I'm 20 to be.
So this is the famous Rosenberg function.
We're trying to find that black dot, but if we get stuck in the valley way over there, it can take many steps for certain algorithms to find that.
So let's give us give ourselves an initial guest that's stuck in the valley, I'll say minus one and one.
So there we are. What will happen now?
We have the gradients. Exactly, I believe. It's fooling around here and then its first attempt goes off in that direction.
So you've got to do a line search so it will realise it's done something stupid, I hope.
Yep. See, this is the line search. It's going back because it doesn't.
It's not going to find a new direction until it at least has got some decrease in this direction.
Keep going. So now it does have a decrease.
That new red dot is slightly lower than the old one, so it now might find a new direction.
Let's see. Yes. See, now it's going off in that direction, trying to find a decrease.
And it's pretty happy now, I guess. Every time I do three, it finds a new red dot.
So it's kind of striking that even a two dimensional function like this and it's not that exotic, a function can be a serious challenge.
Of course, if we didn't have all the statements, it would still find it lickety split,
but there would be a lot of iterations along the way, so optimisation can be amazingly hard.
Let's try M to see. So here we have a more complicated version of the same thing.
Let's give it an initial gas of minus three and one.
Oops. I'm sorry. I'm. You can't see what I'm doing.
I'm typing on the wrong computer. Sorry, I'm 22.
See, there's a more complicated version of the same thing.
Minus three and one. So. What did it do there?
It made a lucky guess. Very, very lucky guess.
Okay, let's try again. Let's try -2.9 and point nine.
Okay. So now you can see. This is what software does.
And even in two dimensions, that must have been about 100 steps that it took.
I found myself intrigued by that. Lucky guess I'm going to try one more.
Does anyone want to propose an initial condition? What's that?
And the gradient was zero. I guess that's right.
Yes. Yeah. Let's go very close.
So let's say -2.99.
I've been tree 10.99. What will that do? Ah.
See, this time the initial guest wasn't so lucky, so it gave up on it.
Interesting. Okay. So that's what optimisation is doing in 2D.
Now, for fun, of course, I can't resist showing you a bit of fun.
And I think for this I can return to the other computer. So.
Right to have fun, of course, likes to deal with functions in one or two or two and three dimensions.
So let me just show you the kind of thing we could do like that.
I could say f, f equals. And then I would make a function of two variables.
And this is the one that we saw a moment ago, one minus X squared plus 100 times R plus cosine of pi x.
If you see a bug, tell me. I hope that's the Roseanne BLOCK function.
Nope, I've made a mistake. Or I do one minus x squared plus 100.
What's that record for? Yes. I don't want them, do I?
Thank you. You're quick. How did you do that? Okay.
And I'll say ethical chip. Fun, too, of this anonymous function on the box.
Minus three, three. Minus three, three.
So Tetfund too now does its usual thing to construct a global representation of that function, and it then uses global methods to do zero fine.
If we plotted, I could say levels equals zero 2500 and I could say contour f levels.
So there's the same function and you can see that.
To have fun has captured it somehow or other. So if I know now say Midvale men pars equals men, two of s.
Tetfund is doing its thing and.
This function has the actual minimum at one one, and you can see that it's got about half the digits.
The minimal value is zero and you can see it's got most of those.
So that's kind of typical that when you're at the bottom of a parabola, the minimal value you can hope to get to machine precision,
the minimum position because of the parabola, you hope to get the square root of machine precision.
So here's an example where the global nature of cheap fun has worked beautifully.
And let's see here. Yeah.
Okay. So we have a couple of more minutes and I want to show you something else on line.
So. So if we go to my Web page or the course Web page, you remember there's this thing called numerical software tools, which I showed you earlier.
And let's now that we're optimisers, go back to the optimisation section of that.
So I'll go down to. Optimisation and there isn't much there.
But these things link to a lot. So niasse.
Is a repository of various things related to optimisation.
It used to stand for network enabled optimisation software.
I think now it doesn't stand for anything. It's just an iOS. But anyway, if I click on that.
I find I met the server. And you can see it's a service for all kinds of things related to optimisation.
It's hosted in Madison, Wisconsin, which is a big centre for these things.
And you can see that it has both software and information.
So there's this optimisation guide. And then it points you in various ways to software.
So let's look at the guide, for example. If I click on NEA's Optimisation Guide.
Then whose phone is ringing?
Is that mine? Drives you crazy, doesn't it? So you can see lots of information here by the best people in the business.
So let's just click at random on optimisation algorithms. Is that somebody's phone?
Whose phone is that? Okay.
So you can see under algorithms, all sorts of things and we can pick one of these at random.
Let's go to Newton methods. Mm hmm.
Yeah. Well, you can do this at home.
Let's just go up a few levels, and I'll show you the other things under optimisation.
I'm not sure this decision tree is still there. Let me see. The decision tree for optimisation software.
I guess it is still there. No, no, wait a minute.
There used to be a tree. How do we get to the tree?
No, I guess it's not in the form it used to be at, so that hasn't been maintained so well.
And then the other one is this MATLAB optimisation toolbox.
Just to remind you that MATLAB does have some optimisation built in the reputation of this toolbox is actually not so great.
So if you're serious about optimisation,
I think prowling around in the office is probably a better strategy than becoming fluent in the MATLAB toolbox.
But of course there is some stuff here for all sorts of things. You can see nonlinear is what we've been talking about.
Linear and quadratic programming is where constraints are very important.
We haven't yet talked about that mixed integer linear programming.
So when you mix together continuous and discrete variables, that's called mixed integer programming.
This wonderful book I keep telling you about, it doesn't have the discrete stuff.
It's all continuous multi objective optimisation when you're trying to minimise several things at once,
least squares, data fitting nonlinear equations and then parallel computing and derivatives.
So those are some of the things that MATLAB offers. Okay.
So I guess we have our last lecture on Friday and then I will be handing out the final assignment then.