It is already recording, and yeah, I think it's time to start with the with our seminar, I'll start into introducing Caroline.
But before, like also like Caroline told me that if you have questions, please do feel free to ask like either like in the chat or Jessica.
Raise your voice. And another question and that is fine and I'm Caroline will be happy to to respond.
I'm now like, let me introduce Caroline.
So this is like the last session of the Oxford Computational Solicits, a machine learning seminar in this season,
and we have Caroline Olear from M.I.T., who is going to speak about causality and policy in the light of drug repurposing for COVID 19.
And Caroline is currently the Henry and Grace Doherty,
associate professor in electrical engineering and computer science and Institute for Data Systems of Society at MIT.
She holds many degrees in maths and biology from the University of Surrey, and that to the studies from UC Berkeley.
Before joining M.I.T., she was a postdoc at the Simons Institute at UC Berkeley, the University of Minnesota, and Footage.
She was also an assistant professor at SD Austria.
Caroline is a member and an elected member of the International Statistical Institute,
and she won an assignment investigator Work as We Fellowship, amongst many others.
And the reason for why we have Caroline here is because she does very exciting research at the intersection of machine learning,
computer science and computational biology, in particular on council in France, generated modelling and applications to.
So, Caroline, please feel free to to Egypt, OK?
Yeah, thank you so much, and thank you very much for inviting me. Very excited to be here.
So we've been working since all of this said a lot and causality in recent years.
And then once COVID 19 started,
we saw many people in the lab were wondering how we could use the kind of research that we have been doing towards this problem.
And so that's when we started working on drug repurposing, which is the kind of problem that says I'm going to be presenting here.
And in particular, I'm going to talk about the mathematics of statistics, the machine learning questions that have arisen just due to these questions,
these biological or medical kinds of questions and really go more into the statistics and machine learning topics
that have been important for for actually coming up with the kind of method that we have been applied here.
So let me start right off with the problem of drug repurposing in particular and in the setting in the context of COVID 19.
But this is, of course, more generally a problem that that could be applied to so many other diseases as well.
So how does drug development work and in particular, how this drug repurposing work?
So drug repurposing is is particularly important when you have diseases where you know you
don't want to have time to go through all of the different trials because these trials,
of course, often fail and they take relatively long. And so drug repurposing means that you're using drugs that have already been FDA
approved and tried to find which ones might also be effective in a new disease context.
This can then speed up kinds of trials because you don't have to go through the safety trials anymore.
You can just directly go to the efficacy trials and try to see whether this new
drug also or this old drug also works in this new and this new disease context.
So how do drugs work generally? Well, drugs usually target a particular protein and then try to make this protein ineffective.
And through that, what you want to do is you want to find drugs that can then push the system back to the normal.
OK, so now if we're thinking about it and in particular when we think about it in these in
these networks and we will see networks also coming up towards the end of the talk.
So, you know, people have used a lot of these fundamentalist grafts in order to come up with
with a drugs that could and to come up with the right targets for these drugs.
So in particular, if you have here, say, in red and in new or these notes that are and maybe these are now genes or proteins, whatever,
whatever you're looking at and these are the notes that in the disease context are differentially expressed as compared to the normal context,
then you know, one of the standard approaches is to define targets that are connected or very
central to many nodes that are differentially expressed in disease control.
Now, of course, you know, if you come already from it, a little bit of a causality background and also otherwise,
you know, it is clear that if your true network in general in biology right there,
that these are all direct relationships where you have like a certain protein or a certain transcription factor turning on another gene, et cetera.
And so if you're a true network, look something like this.
Like this? Direct the graph here. And of course, if you're targeting this node here in the middle.
Well, then although it is connected to all these differentially expressed genes, these genes are not going to change, right?
These red and blue jeans. There is no hope of actually moving the system back to the normal state.
So the directions matter a lot, right? And so causal relationship will actually matter a lot.
You cannot just sit down to write the crap. So in particular, if the graph looks something like this, then maybe this is a good target, right?
Because you at least have a chance of changing all these nodes that are differentially expressed,
and this is in the right way by actually targeting this particular notice.
So directions really matter. And so this is one way where actually causality comes in,
and I'll only talk short about this or we have done a lot of research in terms of actually learning
these diverse graphs or these causal graphs from data on the nodes without knowing the directions.
And so this will be just a little bit of the talk all the way at the end.
I want to talk about something different that there's another problem that also appears in in this drug repurposing studies.
And that's the following.
So, you know, like if we're thinking now of SARS-CoV-2, what in SARS-CoV-2 in particular has an effect on these long epithelial cells?
So now what are the datasets? So often, of course, drug stroke identifying new drugs is still done very much experimentally,
although there are more and more computational approaches and in fact,
also successful drugs that have gone through all of the trials using just that were identified using computational methods.
And so what? What are the datasets that one can use if one is actually trying to use computational methods?
So there are these really large scale drug screens, and I'm going to show some of them or one of them.
In fact, on one of the next slides where you have many, many drugs,
and I'm going to concentrate on these FDA approved ones because we're thinking about drug repurposing.
But in fact, these drug screens, you also have many non FDA approved drugs.
You also have all kinds of milk out, et cetera.
So many, many different kinds of interventions on the system that have been applied to some cell types.
Now, this particular screen that I'm going to look at is a cancer screen.
So the cell types that these drugs were applied to are all different kinds of cancer cell lines.
Now the problem is, of course, that, you know,
now what we want to do is be able to identify drugs that will work in this new context, which is not cancer, right?
And that will. So we want to be able to predict the effects of these drugs on SARS-CoV-2 infects this long epithelial cells.
These are cells that are not in this drug screen. So every time a new disease comes up, you, you would otherwise have to redo the whole drug screen.
And that's of course, infeasible to do, right. So what you really need to be able to do is somehow to transfer these causal effects.
So, you know, the effect of all these drugs or these interventions on some cell lines where they were measured.
Now can you from here predict what the effects of these drugs will be on a different
cell line on a cell line where you know how it looks like in the normal state,
right? But you have not seen any of these drugs applied to it.
And so you would like to be able to transfer it to the. So that's the current and this is again a causal actually question, right?
Because a drug or any knock out, et cetera, is an intervention in the system.
OK, so I have an intervention in the system and now I want to be able to predict what this intervention
in the system will do in a different setting or in a different we call this environment.
OK, so one cell type, one environment and another cell type is another environment.
These same questions appear a lot and policy kinds of questions where you know the effect of a particular policy in a city,
which would there be the environment, then you want to predict what the effect of this policy will be in another city.
So this is known as the causal transport security problem. So it's again a causal problem just because it is an intervention.
And so that's the main question that I want to talk about.
So that's this question here is like, how can you transport the effects of a drug from one environment,
say here a red cell type to a blue cell to predict what this intervention will do and actually do something?
The other question I talked about before is how can you even learn this underlying these regulatory networks?
And this will actually come up. And we have worked a lot on this problem as well,
and this will just come up in the end where we actually want to validate some of the kinds of relationships and cosy relationships that we're putting.
OK, but I'll mainly talk about this causal transport stability problem here.
If you know the effect of different kinds of drugs on some cell types and now you would like
to predict what the effects of these drugs is in the cell type that you actually care about,
which in this case, a source close to infected lung epithelial cells where you have not measures any of the drugs.
And please again, as Gonzalo just said, just just interrupt me whenever you have questions.
OK, so if you come from machine learning numbers, probably one very straightforward approach that you may want to try and in fact,
I know you're in the UK, I have quite a bit to AstraZeneca.
I know that you have tried this. So this is certainly something like very, very standard and it has not worked for them.
And they'll show you why it might not have worked so well for them and how one could maybe get them around this.
So if you come from machine learning, then this is probably kind of an approach that you might say like, well, we should probably at least try.
Right. So we all know this from and often it's used using Gans.
I'm going to use all coders in this whole talking also just because we actually have some theory for it.
So I'm sure you're all familiar with all 10 coders that we have here, this encoder and neural network that maps you from some space.
This can be gene expression. It can be images to this latent space and then the decoder, which makes it back,
and we're going to learn this map just by trying to minimise reconstruction.
Now there are all these like, really nice applications that are usually done with Dan's,
and I'm sure you've seen them where you want to translate an image that you have taken in summer to winter.
Or you have like some people and you want to add smiles to them. Right.
So here you have some people that you've taken, some are smiling and some were also neutral face.
And then maybe here this vector here corresponds to this person here, adding a smile to to her right?
And now here you have a new person and you would like to know how does this person look like with a smile?
You have never taken this picture. Well, maybe you what will work is actually taking this vector here that corresponds to adding a smiling,
different person and just moving it up to this one person here and then a decoding.
This points back into image land and in fact, in this case could come with person with a smile.
Now the question, of course, is can something like this also work when you have drugs?
So here's how you should think of it. It's like we have different kinds of cell types like this blue here.
Just pink, one another blue, another purple one.
For some of them, you get to observe what the effect is of a particular drug, and now you want to predict the effect of this drug on a new cell type.
In this case, this would be, you know, this would be the cancer cell lines where we've seen the drugs on.
This would be the first couple, in fact, that lung epithelial cells, we would like to be able to predict the effects of this drug.
Well, maybe, you know, in the lake and space, I could just take the effects of this drug and then moved over to my new cell
type and then just encode it back out into the vacant space and outsports.com.
The effect this this effect of what this drug will have on the SARS-CoV-2-infected film.
OK, so this has actually worked. But in this particular paper here it works actually quite well just for a well, a few cell types and the few drugs.
And so the question is, of course, is this a general phenomenon?
And as I've already told you, AstraZeneca has tried this and this in general does not work.
In fact, you should also not expect it to work. I mean, there is work by Barenboim and co-authors, they have done and pearl, et cetera.
There's a long route of work, and I've already told you this is known as causal transport stability and Barenboim and Pearl.
They actually have really nice work showing that necessary and sufficient conditions for one causal transport stability works.
Now their work and things are quite stringent conditions.
Now the problem is that their work doesn't really tell us when it will work in this setting here,
because they always assume that, you know, the underlying cause of graft. So if I draw something right, they issue so they assume that.
So you have some causal graph below it and they assume even more that, you know,
the nodes on which the intervention happens and even more that you know, the nodes,
which are the ones that change,
that are that are the ones that are important in order to change environments will say this is the one that differentiates L.A. from,
I don't know, Boston, right? If you want to do a transfer of a policy from one city to another.
So that's that's another knowledge that you have.
So, you know, where do interventions happen and where the nodes are to change the environments or make one cell type into another cell type?
Then based on this graph, we have necessary and sufficient conditions that tells you whether the effect of
this particular intervention can be transferred to this new city or to this.
Until now, you see, we don't have this information, right?
I mean, we don't know happening regulatory networks, we certainly don't know for the drugs in general, like all the nodes that they have an effect on.
And we also don't know the nodes that make a difference between different cell types.
But what is really nice is that they do have a certain sufficient condition, so you should certainly not expect that to work always.
So that's maybe the take away from here. So the question is, when does it work and why does it work sometimes?
And so let's actually look at the large datasets where we'll actually be analysing this.
So this is this large data. As I mentioned that I encourage many of you to look at, I think it has been really, really fun working with this dataset,
if you care about questions like this, about transfer and about interventions and trying to predict the effects of interventions.
So this is a publicly available except the road.
It's a map dataset that has 1.2 million samples where every sample is a one time dimensional vector of gene expression.
So here for can, for just some some genes, well, select the genes and then you have here thousands of different perturbations.
OK, so these are these will only look at these FDA approved drugs, which are many older, many older perturbations that were applied to it.
And as I said, they were applied to different cancer cell lines. So around 70 different cancer cell lines.
And what you see here is every daughter means that you have data, so that's just one thousandth dimensional vector.
But of course, you cannot do all of these experiments, right?
So there is a whole lot of white space out here, meaning that these kinds of combinations were not much.
But, you know, you already have a huge amount of data here on which you can then test whether your method for trying to predict the effect.
You just leave out some validation of whether your method of trying to predict some, some the effect of a perturbation,
actually how well you are able to to predict effects of a corporation that was not being.
OK, so this is the dataset that we're going to use in order to actually check how well that such an Altman colder approach would work.
OK. So look in this data to think a colour the perturbations measure and the effect of the prospective patients measure like a.
Perfect. Yeah, so this is. So it's it's the expression of these one thousand genes that were measured.
So that's what every one of these points here is actually it's really a tensor. So every one of these dots is a 1000 dimensional vector.
So it's quite nice that it's not just a one dimensional thing that it's actually doing.
So there are other drug screens where it's just measured whether to sell dyes or not.
But you're actually you really get to measure this for one thousand financial markets.
And this is just a huge map of the status of how it looks like. So you do see.
So here in colour or the different cell types of seven, we want to hear that you have.
So what you see is that actually and this is known,
so perturbations have a very small effect as compared to the differences between different cell types.
OK, so this in black here, just believe me that these black points here correspond to the orange cell type here,
but the different drugs applied to it. So you see, it's actually quite so.
It's actually a very good baseline is just to predict the effect of a perturbation being the mean of that cell type.
It is quite difficult to have methods that's better than that.
OK, so this is always so that's how you should think of like how the status of Jerusalem.
OK, so let's look a little bit about how these alternate methods work.
And here is something. And this is what we're looking at a little bit into theory, actually.
So it might look a little bit strange now at the beginning, but you know, your standard often code, you're right, you go from.
In this case, we have a 1000 dimensional space where these 1000 dimensional vectors that we're measuring usually
usually are all turned coders in order to get the lower dimensional representation of the data.
And then you train it back to your original space nine hundred one thousand dimensional space now, and we'll see that that doesn't work so well.
And many people have seen for my industry that this doesn't work so well.
Now what what will be proposing to do with something?
Maybe that looks very crazy at the moment because, you know, often cultures are usually used for dimension reduction.
What we're going to do is not use it for dimension reduction,
but we're actually going to go into a latent space that is higher dimensional than the original.
OK, so start off here with your 1000 dimensional vector embedded into a higher dimensional open space and go back to your routine space now.
So, OK, so why is this unintuitive? Well, because we have so many parameters that we could just learn the identity.
So here I could actually just learn the identity map. What I want to show you and I'll give you is a theory.
Why we even tried doing something crazy like this is that we're at the gate.
When you train such an art and culture, you're not going to learn the identity map.
And I will tell you a bit about what kinds of maps we're actually going to be doing.
OK, so so but that's the interesting thing that you actually what you're going to learn is actually something useful.
It seems like. And in terms of intuition, I think maybe intuitively kind of makes sense.
So if you want to.
So in some sense, what you want to do is you want to make effects more linear, right, so that they're more aligned in the making space.
So you want to make effects more linear.
So if you're going into a lower dimensional space where your effects will definitely have to become more non-linear,
more crumbled up than what we were before. So if you're adding up, if you're adding more dimensions,
that's that's hopefully allowing you to actually make things more aligned than what they were before.
At least intuitively, that makes sense. But of course, it depends on what is the maths that is learnt by building.
OK. And so just to show you that something like this is actually happening, we can just check that we can just check it on to status.
After you just leave out someone, you try to predict the effects or you leave out what you're actually going to do is so,
so it will only work if the effects of a drug if you look at different cell types where this particular drug was measured.
It only works if the two drugs are aligned. So what we're just going to measure is the vector between the drug right in the latent
space between a drug in one cell type versus the other cell type in the latent space.
And the correlation between the drug and in one cell type and the other cell type individuals.
And so here this is what you see. So this is the original space.
The correlation and then here is if you go into a lower dimensional latent space in this case,
one dimensional life in space, and you see that basically not much changes.
Right. So this is filled with alcohol. You do have a little bit of an enrichment up, which is good.
This is what you would want to see. Now what happens if you just do a PKA embedding now?
Well, what happens with PKA? In fact, you get quite good enrichment, so you really get very good alignments.
But you're doing you're getting good alignment by throwing away all of your data, right?
So in fact, you can align space by doing aligned data by throwing away data, right?
But then, of course, you don't have any information anymore about the effects of the drug.
And in fact, this is what you see here when you're looking at reconstruction.
So this is one way of getting really good alignment that's just throwing out most of your
data or most of the information in the data and just keeping some of the direction.
So now what you see here is that actually, if you go to this over parameterisation space on, in this case,
higher dimensional latent space than what you started off with, what you, first of all, will see is that it's not learning the identity matrix.
Otherwise everything would have to be nice on the value.
So you do get actually basically the same kind of alignment as what you get with PDA when you're just throwing out all of the information.
But now, of course, with perfect reconstruction. So you're actually not getting rid of any information.
OK. And so the question is, and this is what I want to do now is to tell you a bit of intuition for why did you even try these
crazy Olsen folders where you have like a higher dimensional vacant space than than your Virgenes?
And this is just to show you here. We just did it. I chose this spring for two cell types.
This is just for all different cell types, and you sort of see exactly this kind of stuff.
OK, so let's go a little bit into theory of like, why, why are we even?
Why did we ever try to look at over overdramatise alternate cultures?
And I should say now for all applications we're actually using over something else.
OK, so let's go a little bit into it and give you some intuition for it.
So. So this is just the standard setting. So we'll have our end training examples there, living in R.K. and now in the over parameter settings.
So you can either just have the latent space larger dimensional than the dimension or here.
I will just have like the number of samples, be smaller than than than the inputs I mentioned that I'm taking.
It doesn't matter which direction he's going. And so here we're training our often coder, right, using these end training examples.
So we're just trying to minimise the construction.
OK, so what do we want to do, so let's actually look at this problem, so we're in the overdramatise setting.
Let's actually just get a little bit of intuition for what is happening. So let's just do the linear case first.
So in general, right, this is the map of guilt encoder since this will be highly nonlinear.
But let's just do that. The linear setting first, and let me just have one training example.
That's the great that's the most stringent portrait. So then I would just try to do the following.
So I'm trying to do argaman over a is an hour of patience.
Okay, right? And X is just a one dimensional vector, right?
This is an OK. So I'm definitely over parameterised, and I'm just trying to minimise this so well.
I have so many parameters. First of all, you notice that there are many different solutions to this problem, right?
So one solution is definitely. So definitely, I can get this lost down to zero, right?
And in fact, I can get it down to zero in many different ways. So one solution is the, I guess, the Degan, right?
So a is equal to the identity is definitely a solution. But note that there are many other solutions, right?
In particular, there is a solution of any rank. So, for example, there is a rank one solution which would just project everything onto X.
But you know, you have like any of the ranks of solutions to this problem.
So then the question is what is actually an altered code work? And so now this results actually just work.
The linear setting is very well known. In fact, it is.
If you just do gradient descent started and you know, in your own code, there's usually what you do is you start with very small initialisation.
So in fact, if you start at zero,
it is known that here's what you would learn is in fact the one that the rank one solution or just a projection onto X itself.
OK, and now you can actually generalise this, and this will be to zero in terms of like what is being learnt,
but what will do just to get some intuition as well.
Let's like actually just look at what happens when you apply your unit, art and culture, when you just have when you train it just with one training?
Exactly. So here is one training example these nice Bonny's here you train it on these bunnies and then you put in some random test examples.
And, well, let's see what comes out right when you feed it through your authoring folder and what you see here.
What happens is that always the bunnies come out, OK? So, so actually it.
So I would like you to notice that this is stronger than what we saw before in the linear setting right where rank one solution comes out.
If it's a rank one solution, you wouldn't expect just the bunnies to come out,
but you could also get, for example, the negative of the bunnies come up. So this here is actually a point map, right?
That maps, and it seems like it has learnt a point, Matt,
where it maps anything you put in to exactly the bunnies and not like a linear combination of the buttons.
So. So this is I mean, OK, so many people would say this is not surprising at all.
But I do want to just make sure that you understand that it is really surprising.
So first of all, if you actually use. So you met this really quite deep as well.
So if you use a very shallow network, in fact, you will not learn a point next.
OK, so this is important to notice. So here, in fact, you're again training on one training example.
You put in some random examples and you see that outcomes actually something that looks very similar to what you put in.
Now again, there are papers that say you're learning the identity map.
I'm going to show you this is not the identity not going to will be really important to notice that this is not OK,
but that will come on the next slide. But this is just to show that it is.
It's that you shouldn't be surprised by the results. Also again, as I said before, you know, if you actually this is easy to prove the linear setting.
So if you use a linear encoder and you train, let's say,
on two examples just to show you that this is one example you can actually get the negative of the dog or something if you train it on that.
But if you do hear a linear, then it's very clear that what comes out is always a linear combination of your training.
And so when the linear setting, you should also expect to see the negative example that difference.
OK, so here you get linear combinations. Here you actually get something that looks more like a gigantic map.
So you should be surprised by this kind of results where if you have a quite deep network, in fact,
if you just train it on one example, then it seems like what it is learning is the point.
OK, so then let me give some intuition for actually the map that is being learnt.
And this really comes from this shallow attack.
OK, so the intuition is really, really simple in terms of what we're doing in order to then show what we're actually going to prove.
Well, and also Coder is just so nice, right, that it maps an image to an image.
Well, so if I want to see or understand the bit about what is the map that the alter encoder has learnt?
Well, then let me just iterate the map.
OK, so so if you're in this setting here, right, I put in this this horse here and I get out something that looks very similar to it.
I'm telling you it's not the identity map, because what I'm going to do is put in this horse again.
OK, whatever you got out, you put it in again, you put it in again. And in fact, you'll see that what will come out are actually the rabbit.
That's what we're going to prove. So A.C., it's actually even if you train it on not just one training example, but you train it.
In this case, we trained it on five hundred training examples and these are all training examples.
But then you just add a whole lot of noise on top of it,
and you just put in this example into your alternate colour and this is all ready after a couple of iterations.
So at the beginning, you get something that looks very similar to whatever you put in and you iterated iterate that.
Then you see that in fact, your training example comes out. So what does this suggest?
Well, it suggests that your ultimate coder learns a map where all of the training examples are fixed points, or at least some of them or fixed points.
So you'll see, I mean, I'll show you. So you can prove which ones are fixed points, and so I'll show you on the next slide.
We can actually train these alternate coders to make all of our training examples see fixed points.
And the question is, of course, how large is the region of attraction right around every one of your training examples?
And that's a good question. So here, for example, you have some for some just examples, right?
So here here what we're showing you is that even if you take away like you, just add half of the image,
just be noise in four hundred and twenty one of the examples, you're still inside the region of contraption around your training example.
You're actually outputting the correct training example and not a wrong training. Examples of these regions seem to be quite large.
This is to us an important research question that we don't have an answer to how these regions of attraction actually look like here.
We're just showing you a very simple example in 2D, where we just created the space and looked at where are we converging to?
And this is how these regions look like, and this will depend very much on the nonlinear you're using.
And so this will be one of the open questions that I'll have in the end. I mean, this is super important to understand, right?
In order to use often coders and different kinds of applications, or you may want to have these regions of attraction be slightly different.
And this will really depend on the known use.
OK, so as I said, so what you can actually do with what we cannot prove is that we don't have any other attractors in this whole landscape other than,
you know, just gritting it and seeing that we're always converging to one of the training examples.
What we can prove when you train your network is whether every training example is an attractive because you just look at the
eigenvalues right at at your training examples and then you see that there is more than one so that you are an instructor at this,
at the at the training exam. So here is a network which is actually quite small where we trained it and we can prove that this network
actually has all five hundred training examples in this case from imaging and actually as attractive.
So we don't need very big networks to make all of your training examples be attractive.
OK, so what can we actually prove about all of this? So and this is what we have.
So for now, we only have it for one training example, and this is the following results.
So you have your ultimate coder and with K.
So now here we have something called under suitable conditions.
So actually, all of the nonlinear reviews that are standard use will satisfy these conditions.
initialisation has to be close enough to zero. And then what happens when you give me a nonlinear the initialisation?
Then what's this theorem gives you as a is a is a formula for the maximal eigenvalue of the programme.
So, you know, that's a training example will be a fixed point, an attractive fixed point if this eigenvalue smaller than one.
OK, so from this formula,
what you can then figure out is what kind of depth and what kind of width you need in order to make this training example to.
And so all this says, you know, if you're over parameterised enough in terms of depth and width, then in fact,
this training example will become an attractor for whatever nonlinear event initialisation for.
All right. It's about the phenomenon of the Joneses driving in particular to avert parameterised autumn colours for any kind of and good.
Now this is in fact very particular to overpromise Drysdale. Yep.
And in fact, when you're also plying them, you will see it that in fact, if you have your under parameterised ones in general,
you'll not compete so you'll not converge to training example, you'll just convert.
You can even convert to something like completely that doesn't look like an image at all.
And you can convert it to stuff that looks kind of like. Yeah, yeah.
So it's like you have to factor in the identity map in many states or something like that.
I mean, yes, so let me actually show you how I'm thinking about it in terms of the maps that are burned.
Exactly so. Yes.
So first of all, and these are the kinds of pictures, so in some sense, OK, so we all know right people were Crowdrise neural networks.
They can interpolate training data. But I just want too many papers use interpolation and memorisation as the same as the same thing.
So I want to make a difference between the two because to me, they're not the same.
So interpolation, right, just means that I'm learning some function where as training example will be mapped to itself prior to training,
example will be mapped to itself. So you have zero loss.
OK, so there are many different maps, right, that can interpolate training data so I can have this crazy makeover here.
So this is an ultra encoder that goes from hour one. You should read this to our one.
And say, this is my training example. This is a training example. This is a training.
Examples of this kind of crazy map isn't interpolating map. Every training example is exactly map.
So now this map here in the middle is also an interpolation solution, and this one here is also an interpolating solution.
Now what this map here is is the map that you have basically seen at the beginning where we use the units on one training example,
even if you use it on two or three. What you'll see is that it maps.
This is the point map, right? Where all of these train, all of these test examples in this case,
here would be maps to this example, and all of these fellows here will be mapped to this one.
So this is the point map, right where you just exactly immediately point a map out the training example itself.
And then what we've seen on the previous slide is when you have when you have very deep networks,
this is what is happening when you don't have that deep networks,
but still over parameterised like quite wide networks, then what is happening is this following map here, right?
Where if you iterate the map many times, then this comes up.
Right. But but for now, the the map that is actually learnt is just a map that is contracts at each one of the training.
So it is contracted at each one of the training examples, and once you iterate, well,
if you just iterate this map here and in fact, this map over here, we're sorry.
All right. Ask a clarifying questions. Yes. Why?
Why? Why is it important to have these attractive training points?
Why would I want to have such a map? I think that's a great question.
So here there has actually been and so so you were asking questions about even is this important for generalisation?
And then I'll have to also talk about why, why, what is even generalisation and for optimal coders?
So so what is kind of nice and why we didn't really have to do?
So first of all, it works in our application. Okay.
So that's maybe one way of saying like why you actually care about these things over permettrait guilt and coding.
The other thing is that there has actually been work before where they wanted to make alternate coders be attractive at the training examples,
thinking that this is actually a good property where they added regular risers to
make the map attractive at the at the at the at the at the training examples.
So,
so people have already used these kinds of map with the additional regular or seen that these alternate coders actually work very well in practise.
And so what we're showing here is that you actually don't need any regular writers like just over
parameterisation by itself will give you this property of its being attractive at the trainees.
OK. So this being attractive at the training examples somehow makes the auto encoder work better in some way in many different applications.
Yes. OK, so we've seen it in our drug example.
Others have just added regular razors to actually have this property and have them see that this work actually plays well.
And so we're quite well up, and now we have to talk about generalisation.
I mean, there is no such thing as like, what does that even mean to generalise? Well, for an alternate cruiser, right?
So this is another question that I think is an important one where there should be much more a bit more foundational work, right?
So now people use it in the sense that like oh, and often could generalise as well if I'd learn something close to a density map.
But I mean, I I don't agree with that definition at all, right?
If you want the identity map, you don't have to train. I mean, it's if this is what you want and just don't train.
So that cannot be the right definition of generalisation, but maybe not here.
I'm just proposing something that could be something, but we have nothing to work on it.
But maybe a kind of definition that might make sense is that you should be close to the identity map,
but at the same time, be contract this that you're training examples.
So that might be an alternate and at least you have to train. And then what would happen is that these overdramatise in particular are very wide.
Networks will actually do very well if that's your definition of generalisation.
But I think there should be a lot more, you know, kind of more foundational work on what it actually means to generalise well for all coders.
And certainly not. It shouldn't be that you want to learn gegenpressing. OK, so just one last clarifying question.
Could you please recap the intuition of like why having having training examples as attractive points make it work well in your drug examples?
Oh yeah. So so I haven't even talked about that yet. OK. So for us.
So the intuition and I think by now, we kind of have a proof for what is actually happening.
But the intuition at that point was like, OK, so here what?
We and this war came before our great examples of this work was there, and then we built on it and tried to use it for for drugs.
So the intuition was like, Well, here what you're seeing is that you're going to become attractive at these zero dimensional points.
So you're making points that are similar, more similar to the training examples, but right, because it's going to be attractive.
So we were hoping that maybe this doesn't only hold four points, but it will actually also make lines.
So one dimensional things that are similar in even manifolds like low dimensional manacles that are similar to each other, more similar to each other.
Mm hmm. That was the intuition. OK, so that we make things that are already similar to each other more similar to each other.
And that's what seems to be happening and in at least two drug examples. Right?
That was the intuition. Let's make let's use this over parameter settings so that it makes things that are similar.
And that's kind of what you're seeing, right? When you when you see this right, that's making something like this is your training example.
It was already similar to this training example. So the application of this network will make things more similar to each other.
But I agree that there is quite a bit of a step there,
and this is definitely the kind of work that we're really excited about doing now on the theoretical side.
Why does this actually work? That was the intuition. Yep, and generally, I think the intuition for why you would want something like this is like,
you want it to be contracted out to training examples because you do believe that you're you're training examples are important examples.
Or hopefully, you know, they were not just like, know they are actually hopefully representative of whatever you are expecting to see otherwise.
And hopefully, if you have something close by, well, maybe you may want to make it a little bit more closer to whatever you have already seen before.
So this kind of maps or self regularising. So this was at least at some point the intuition,
and this was work by Joshua Bengio on why they had actually introduced these regular risers to make their all time coders more contract.
Let's hear what this says is you don't need any regular risers. You'll just actually get your contract.
If maps just over pemetrexed, just make your space more jam so that you so that you're actually going to see this.
Cool. Thank you. Great questions. Good.
OK, so this is, as I said, this is the only proof we have. So one training example,
so now everything will be and this kind of goes into like the intuition for what we're doing now
in the lab and all of our applications is always very wide networks and not very deep networks.
And so the intuition somehow already came from here, right?
You have seen it that when you are very deep, what you actually get is these maps that directly you, Matthew, to a training example.
That's probably not what you want, right? You probably don't want that would ever test example you've put in like a training example comes out.
You probably want something that is kind of close to the identity, right? But still attractive that you're training examples.
And it seems like very wide networks and just a little bit of depth is actually doing that.
And so here is the intuition for it, and we don't have a proof, but it would be really nice to actually have a proof for these kinds of things.
So here you actually see these eigenvalues and this is the top eigenvalue and how it changes when you're changing depth and width.
So here what you see is with increasing and this is maybe difficult.
So that's why we have like two different plots, one which you look at the top one,
whatever percent of eigenvalues and here you only look at the top one.
But in all of our simulations and this again, this is just a conjecture.
It seems that what happens with width is that the variance of your eigenvalues shrinks.
OK. What happens with that seems like what it's that the distribution just shifts towards become smaller and smaller.
So what you would want, right? If you want to be close to the identity and still contract,
if what you want is that all of your eigenvalues are just below one but not too small because otherwise you're in this setting, right?
If they're are all like very close to zero, then you just have your points back.
So that's not what you want, right? So that means that what you want this to be very, very wide so that you have a very small variance.
And then just deep enough so that your eigenvalues are just below one so that it is contrasted that the train.
So that's at least the intuition of how we're using these open courses right now in all of our application.
But there is a lot to prove here and now comes a strong example,
and I already gave you that intuition for why we have used it here, just seen that it works well.
And of course, there is a lot to do in terms of proving that this is in fact the right intuition.
Why does it make also similar things more similar to each other,
even if it's not just important, but that was at least intuition for us to try to use this?
Now how do we use it now that we have this, these over parameterised? Often coders actually want to also show you how you use it in this drug example.
So this is a standard approach of like how people do it in a computational drug discovery is often using this kind of signature matching approach.
So here what we have is this is again data that came out after source code started, right?
You have like your gene expression in the normal state.
So these long epithelial cells normal state and then this is what happens when you add the virus.
OK, so this is the disease state. And so this vector here and you see this is double the virus and ACE2 and whatever,
you know, many different kinds of just to see that this is kind of robust.
So this direction always seems to be very similar and this is the disease direction.
So this is the effect of adding the virus does this to your gene expression.
So now this disease, the signature matching what it does is it tries to find a drug that its effect is upwards.
OK, so that hopefully you can just get rid of the effects of the of the disease by adding a drug that moves you back to the normal state.
OK, so now what you want is to find a drug, right?
So so now we'll actually map out all of the effects of the drugs in our latent space, which now is more aligned than what we had before.
And then in there,
now we look at which ones of the drugs is most A. correlated with the effects of the disease and just choose those type drugs that you get out there.
And in fact, if you look at these top drugs, which is really nice, is that in fact, they all in two different classes of drugs.
So, you know, you don't get just like some completely random list of drugs you get like actually two different classes of drugs.
They're all very nicely consistent.
And so whatever sort of drugs, this sort of thing is that many and this is where we also wanted to look at these drugs have.
There are some known targets of these drugs, and these are all actually quite a lot of cancer drugs, which have a lot of targets.
So what we wanted to do is not just have one of these cancer drugs that have a lot of targets,
but also validate or try to predict what is actually the causal target amongst all of these list of targets that these cancer drugs actually have.
So here you have a list of all these genes, right, that are targeted by these cancer drugs.
And we just wanted to know amongst all of these which one would be the putative one that we think is the cause and the one amongst all of these.
Because then maybe you could use a drug that is more specific to that particular
target and doesn't have as many side effects as these drugs that were found here,
which are all these calls, which are all these cancer drugs. And so now this is where we used our previous all of this causal causal structure.
Learning algorithms to figure out which ones have all of these targets would be the one that we hypothesise is that is the causal one.
So what do we do here? So again, what we're trying to do now is to try to actually learn these causal graphs.
And so we just learnt a causal graph.
So when you take out all of the genes that are differentially expressed into disease and amongst all of these targets,
and you also learn the graph with all of these targets here and what you expect,
what would be a good target is in fact a node that is most up that is upstream to most of the disease genes, right?
So you want to target something that is most upstream to all of the genes that are changing into disease?
That's a good target. And so what we're looking for is which one amongst all of these genes that are
targeted by these drugs is upstream of most of the disease associated genes.
OK, so we just learnt these graphs and does have a score for each one of the of the targets from the previous slide.
And what comes out to be the most upstream is the specific one I knew nothing about it came on before starting this,
but it's actually a super, super interesting protein. In fact, it's very stable, so you can do this in different kinds of cancer.
So this is a cell line. Do this also in the CMF datasets.
If you do it in the ones that are actually in our body, these cells, you get out exactly the same risk one protein that is most upstream.
And what is really interesting is actually and this again wasn't put into our model.
One directly binds to SARS-CoV-2 proteins, and in fact, it targets that just targets with one very specifically,
just enter phase two trials, which was quite nice as well. So one of the super interesting protein.
It also has two different. It can have very different effects.
And this is what we have also looked at quite a bit because an individual source who has a very different effects than in young individuals,
it can actually turn on two different pathways. And we think that this might be or hypothesised that this might be going on in.
Individuals versus young individuals and young individuals that might actually be turning on the immune response pathway,
survival pathways, but it can actually also turn on other pathways, which are the fibrosis, the death pathways which actually lead to fibrosis,
blood clotting, etc. exactly the kinds of different outcomes that you'll see in patients.
That is quite an interesting protein that came out.
Of course, a lot of things tend to develop, but at least there is a hypothesised mechanism as well of how this could.
OK, so I am kind of at the end, so five, 20, so this is just an overview that we looked at one causal question,
which is how to predict the effects of interventions, right? In this case, drugs and move over.
So there are all these transport problems in biology that I find really exciting.
So you have like this transport problem of like you have some drugs that you have observed in
some cell types and you want to predict the effects of these drugs and in other cell sites.
I mean, the more extreme case would be right.
I have these drugs that I have observed in mouse, and I would really love to know what these drugs actually do in humans.
Of course, we for now did this just for cell types, and I think that's maybe the more reasonable thing to do.
But then you have also much harder problems, which is, you know, I have some drugs that I applied in one cell type.
Now I would like to be able to transport to a new drug.
OK, so I have these drugs applied in one cell type.
Now I have a new drug that I would like to be able to predict what happens when you apply this new drug that you have never seen before.
We had already seen the drug and we just wanted to apply it for different cell type where we also have some data.
But here you have now a new drug and you would like to transport one or, you know,
you can have all these other transport problems that are really less causal if you want to be able
to transport between very different data modalities in biology or between different time points.
Right. So something that that has always kind of bothered me when we're thinking about,
say, early cancer detection is that if I'm training on data from a pathologist,
I'll never be able to to detect cancer earlier than the pathologist because I'm always using it using the training data from the pathologist.
So I need a way to somehow generate data, right? That corresponds to earlier time points.
So how can I move between different time points, so generate my own training data of how these cells would have looked, I guess, earlier?
So I guess these are other transport problems,
and you see that there is a whole lot of nice machine learning kinds of questions and all these transport companies in biology.
So with that, let me end. So some conclusions. So I think drug discovery requires quite a lot more.
More theoretical kinds of frameworks to look at things in particular.
I think causality is an important framework in drug discovery because drugs are interventions
in the system and this is the only way to be able to predict the effects of an intervention. We actually take a causal approach.
Alton coders are not only extremely useful in biology, which my lab has used them a lot,
but I think also really useful for studying just theoretical properties of neural networks.
Here, you know that they're actually learning these contracted maps.
We're only able to see it because all soldiers are so nice that you can just we applied the maths, right?
It is hard to think of like how this would generalise or to to give you insights about the classification setting.
I think that is a really nice question here.
I sold all of these results and over parameterisation that they showed these remarkable self regularisation properties.
They learnt these contracts with massive training examples.
Now this is super important for you if you're thinking about how just how could memory work,
etc. So this is actually a new mechanism for associative memory.
It's also, of course, it has negative, you know, negative consequences as well.
It probably means that you may not want to share a trained Alton coder between different hospitals, right?
Because you can just put a noise and outcomes or training example.
Certainly means that you have a lot of privacy issues there and a lot of open problems.
I mean, causal transport, the ability is one right and you actually prove that you do get causal.
You get like if you're overcome with tries,
you can actually do this or you can sometimes and hopefully often do this causal transport mobility problem, then I already said classification.
What does this tell us about classification networks would love to have a general definition
of generalisation for all that makes sense and is not just learning identity map.
And I also already mentioned this. We have no idea.
We do know from experiments that these attractor landscape is very dependent on the nonlinearity that you're using.
It will be really important to figure out how this actually looks like and how
could one optimise for different kinds of applications and nonlinear conditions?
So with this, let me end acknowledgements, I wouldn't be able to do this without really an amazing group of people,
students, postdocs and, of course, collaborators, so GV, Shiva, Shankar and all of the biological work we have done.
Mitchell Belkin on the overcommit permafrost also encourages work and then, of course, funding.
And I'll end with just one slide of advertisement. I hope that's fine that we just start at the Eric Schmidt centre at the Brote.
If you like these kinds of problems at the intersection of machine learning and biology, Victor is a really exciting centre going on.
We have a lot of workshops if you're just interested in conferences,
learning more about this intersection and we're always looking for postdocs, et cetera.
So with that, thank you very much. And happy to take any questions.
Thanks a lot. I don't know that question.
Please feel free to ask. Otherwise, I do have a question regarding like you show like these are goading the space imaging,
which like after reading your article, there are many things that are different about inputs slammed into.
I mean, you end up dividing the space into regions like I wonder like what is the relation with with this method with modern life
like way of understanding coding in which you look at these burnard regions and you assign like each point of the space?
I mean, do you partition the space according to how close they are to different centres?
Yeah, exactly. So that's that's what seems to happen when you have sigmoid. Yeah.
So I mean. Well, so that's that would exactly be like a kind of conjecture, right?
So when, when and under what kinds of nonlinearity or other architecture constraints?
I don't I don't know how much that will matter. But like, you know, when will you get something that is just closeness?
Right? So so these pictures might be might be leading us into wrong kinds of directions because these are just two dimensional thoughts, right?
What happens? And just like how you overpayments dry setting, you know, it's not so clear.
So maybe it doesn't depend so much on the nonlinear RV anymore? Who knows?
Right. So so. But yes, but that's exactly the kind of question is like when is there a nonlinear already or is it always the case?
There will just map is the closest one in closeness and what norm?
Mm hmm. Yeah. And do you think like the kind of like this different like behaviour that you may have,
might like, underlie, like different notions of generalisation?
I would think that probably for different kinds of applications, you may want to choose it in different ways.
Also, you may want to have I mean, what I think would be really nice is to have an example like,
can you come up with an example that you will use as a training example that will just have a huge attract,
like huge base enough attraction where somehow all of the nonsense will be matched to?
That would be great, right?
Can you just somehow come up with an example where all the nonsense and for that, you really need to understand it quite well?
Mm hmm. I see. And regarding like the application to.
Gore, that pharmaceutical, do you think what is the current state of technological development there, that La Liga pharma industry,
which has said they are interested in applying this more like a futuristic ways of understanding their role?
What they are super interested. So we'll have joint post-ups. So yes, I think it's actually really exciting time now.
Where is actually really thinking about how to get into machine learning and how to use some of these, you know,
methods where you have some theory, etc. And I think they're getting really excited about these things as well.
So are more and more pharma industries that are investing heavily into ML and actually have really good people.
So I think it is exciting to work with them together if this is an area that you're excited about.
Yeah, it's really nice to see.
I think they realise that something has to change and a lot of money is being put into things that in the end have not been super successful.
So maybe this is a new approach worth investing into as well.
So, yeah, no, I think it's a really exciting time for people in Michigan.
Yeah. So could I ask the question, please?
Hi, my name is Karen Amon, oncologist and on the COVID side with the drugs.
And I just wanted to make a TV. And yeah, so you'll know all of them.
And so in the list of drugs, there is a clear bias towards anti angiogenesis drugs like pazopanib as as they're exit,
and that may warrant further investigation.
And we're looking to just just for my own curiosity into the corporate literature, and I wouldn't be surprised.
And there's also like monoclonal antibodies against TNG Genesis,
which would also have an immunomodulatory role and in the pictographs and oestradiol, although it's sort of take it.
So this one is actually in the original space. So this one was not in the latent space time or so this one was.
That's why I was so surprised. I mean, I actually thought maybe like the people who came up with chloroquine actually just did the simple analysis.
But then people tell me, No, this is all ready to advance in something like this,
because here actually chloroquine amongst all of the drugs amongst all of the list and ZMapp actually
came out to be the most anti correlate that in the original space with with the disease vector,
which I find amazing because in the latent space is not the case.
But they said this is probably not what people have done when they came up with this programme.
Did did anyone look at the Easter dial because COVID supposed to affect men more than women in terms of severe symptoms?
And yeah, so this one, we could check where this one happens to be.
But I mean, it's not one of the top ones yet.
So this would be interesting.
So we haven't I would have to check where this one actually comes out to be in the meeting space and how well it is close.
OK, yeah. Then thanks for the question. Yeah, so since there are no questions.
But thanks a lot, Caroline, this has been very interesting.
Yeah, thank you. This is fun. Hmm. Great.
So to everyone, a great summer and enjoy retirement too.
You too bye. Yeah, thank you. Bye.