[Auto-generated transcript. Edits may have been applied for clarity.] If you separate.
Okay, let us, uh, slowly start. Thank you very much for coming in.
It's such a nice number to this, uh, wonderful place.
It is a great pleasure to introduce Professor Ian Stoker.
Ian is a professor at the UC Berkeley, where he holds Um Su Bo Chancellor chair in computer science.
Well, it is exciting. Um, you, um, has been one of the most influential computer scientist, uh, in the field of distributed systems.
Um, he has been instrumental in developing systems that have shaped how large scale data processing,
data analytics, um, uh, um, AI workloads are built and deployed today.
Some of these contributions include Apache Spark, Apache missiles, um, Sky pilot Ray.
In my time, I was a PhD student to a peer to peer network, sort of popular.
So there he was with the distributed hash tables and the system called code.
And most importantly, also all these amazing projects are open source projects.
Now, Ian has also been a very successful entrepreneur. He has co-founded Databricks and any scale, both companies based on his open source projects.
So, um, we are delighted to welcome Ian here as our Strategy Speaker.
And please join me in welcoming. Yes. Thank you. Thank you, Ivan, for such a kind introduction and very happy to be here.
So today I am going to talk about some research I've done over the past ten years or so.
And I have to start with a disclaimer. I know that you guys here are famous for the theoretical computer science.
I only have one equation at some point in my slides.
So this is very system oriented talk. And without.
Let me start. So over the years I've been involved in quite a few projects, some of them more successful than others.
And in this talk and these are two neurones.
Um, and in this talk I am going to talk about three projects which represent a slice through my work.
Uh, radial alarm and childbirth arena.
And they can be actually, uh, it's it's can be composed into a mini stack while I'm running on top of it and chat about that,
you know, running on top of your alarm. So I'm going to talk about these projects and the template I am going to use for each project.
Uh, is this one I'm going to talk about trends because in many cases the trends are shapes that he said is a work I've done,
challenges the kids what it is, tell you a little bit about the project and about the impact.
So let me start. So they started in 2016 and the trends magazine which are still to today,
is that I demand so are growing faster than a single node capabilities and the AI workloads also becoming more complex.
So this is some more recent data about the growth in computation demands of training.
The top uh, models versus time from 2010 to 2024 is along the Y scale, is a log scale.
And depending how you are on different, uh, different uh, um.
Period. You see that the demands have grown.
Computation demand. Go. Go down between 6.74.2 every year.
So let's say for X every year, uh, at the same time the node capabilities.
And here we are talking about GPUs. Uh, if you look for the same precision it's about 1.35 x every year.
And if you look about in terms of memory capacity, it's about 1.2 x every year.
And the memory, uh, bandwidth between the GPU compute and the memory, GPU memory, it's around the same thing, 1.2 1.22.
So if you put this together, what you see is that compute demands are growing three times every year faster than compute and memory capacities of,
say, GPUs in this case, and of course, much faster than CPUs.
Um, so, uh, and, uh, and a, uh, a corollary here is that even if the demand stops growing today in terms of so,
you are not going to have larger and larger models.
It will still take you maybe decades to still be able to train a model on one chip, which means that distributed computing,
in particular heterogeneous distributed computing, is becoming more and more of a norm.
Right? Is that the exception? The second point I wanted I made is I, I workload are becoming more complex.
And let me start with this kind of all the workload, all the workload.
And it's just a recommendation system. So how do you do your student what you're doing?
I recommend if you use them, you'll get the logs from the user to say interaction with the site.
Uh, you'll do some, uh, you know, Featurization used to do some data pre-processing to extract the main features.
Then you do some training to create, to build the models, to predict, you know, what,
um, what recommendation to give you, like movie recommendation or things like that.
You do fine tuning, so you have a new model. So you are going to as you are going to, um,
play a little bit with different parameters of the model to see which parameters are going to give you the best results.
Um, then you push that in production and then you get logs, new logs, and then you try to improve the model to get better and better results.
Um, so this is what basically happened. And there are many more or gloats in 2015.
And so for this, for the emergence of reinforcement learning, um, if you remember, uh, you know, DeepMind were just here playing Atari games and uh,
then you have AlphaGo, then, you know, being applied to, uh, data centres, reducing,
uh, uh, centres, uh, data centre, uh, energy usage for cooling and many more years.
So these workloads are going again, uh, it becomes more and complex recommendation system reinforcement learning.
Then you have batch inference. And as of one year and a half right now, it was the rise of test time computing.
Um, and post-training. Okay.
So what you so the main challenge here is that when you want to build an end to end application, you applications or ML application in the past,
you have to put together all these components, right, to put them together and you need to scale them that we just discussed.
The demands of the data processing and training these models is growing faster than the capabilities of a single chip or a single node.
Okay. So you want to scale them. And at that time, not even today in big parts, you have distributed systems to scale each of these kind of stages,
like data processing or pre-processing training, uh tuning and uh, batch prediction and so forth.
So batch prediction, by the way, is basically before your use, in many cases,
one of the use cases before you are going to deploy the model in production,
you are going to test on the old logs to see how it performs before pushing it to production.
Okay. Uh, so in order to build your end to end pipeline, your end to end systems, you need to stitch together these components.
Right? And this is challenging for a bunch of, for, for for a bunch of reasons.
Uh, first, it's hard to develop because each of these component becomes with its own API,
it's and it's hard to deploy and manage because you need to manage these different distributed systems.
And this system has its own semantics in terms of failure recovery or failure from failure data consistency semantics and things like that.
Um, it's hard to use resources uh, uh, in a efficiently is because typically you are going to have a cluster for each of these components.
So it's hard to share the resources across, across components.
Right. Uh, so for instance, if you do if you do training,
you have a cluster for training and then you have a cluster for inference when you are done with the training,
that cluster is hard to reuse for other things.
Okay. And finally, it's also slow because you need to move the data between these components and the data.
It's a lot of data. Typically you write on as a kind of a blob store, like S3 or something like that.
Then you have to read it. So he has a lot of overhead reading, writing, serialisation, Deserialization.
Okay, so what Ray provided is one system which is general enough to support all these workloads.
Okay, so this is Ray and has a component that we can see at the bottom is what we call
right core and is a unified computing framework for distributed applications.
And then you have on top of it a bunch of libraries running on core to support these different workloads.
But now you have a system which is going to you can use system for your entire end to end application.
Okay. So that's basically what DBAs. So what is key idea.
So at the behind um right. So what it is is that it takes a procedural language like Python and generalise it to a distributed setting.
That's basically what it is. And it focus on flexibility by exposing the parallel lines to the developers.
Right. So the developers can, um, build the application with various pattern of parallels, like nested parallels and things like that.
And one of the question it's always been asked is that why not the declarative language?
You just said language what to do? And the language A is a.
As a back end, um, the compiler or whatever is going to decide how to do it, right.
Um, I think there are a few challenges with that. First of all, there is no proper, uh, general purpose, uh, general purpose, declarative language.
Uh, and I've been working on Datalog and other languages.
So I've been doing a little bit of work on that.
And, you know, adopting a new language is hard.
Let's face the truth. Um, so and we didn't want to invent one because of these reasons.
And at the same time, when we started, the Andre Python was emerging the lingua franca for AI.
So all I knew, I knew Python and all the libraries like TensorFlow, PyTorch.
There are libraries within Python. Okay.
So what kind of it is there are, you know, in of these kind of procedural languages that out of,
you know, a few concepts I actually that are more I just had as I one which are relevant for me.
Uh, first of all, you have in terms of the compute is like you have function and classes, right?
Classes are stateful operators, right? And function in general.
If you if they say they shouldn't have side effects so that a stateless operators and then you have concurrency obviously.
And this is some a form provided by Python with uh I think io IP which allows to execute
things in parallel uh or overlap for instance computation and communication and other things.
And the other one, which is kind of many more low level.
But it's important is that passing, especially for large amount of data, is the ability to pass a data by reference, right, instead of by value.
Right. Far more efficient is when you have large amounts of data. Okay.
So these are the things we kind of try to capture in Ray.
And we are we we encapsulate this in this fast compute model is what it called.
And so again you are going to have you are going to take classes and and functions and provide um
ability to instantiate transparently these functions and classes remotely as tasks and of course,
right. You know, you've seen a lot of actor languages.
Um, then we have a shared the in-memory distributed objects, which enables passing arguments by reference.
And then we have these futures, in particular distributed futures, which are references to objects arguments as well as the results.
And in some cases these are the results which are created by task or actors even before being scheduling.
So they are existing abstract level. You don't know even where the object is going to be created.
And this obviously it enables concurrency and concurrent execution and parallels.
The record has a very minimal API inside of the all of the functions.
The main functions are core function functions. And I'm not going to read all of them.
I'm going to demonstrate a few of them with a few examples.
So here is a very trivial example. So say you have a function which is computing for one second and return the result.
And if you execute that in traditional Python I'm not talking about multiprocessing.
Python. You you call this function twice.
Is going to take you two seconds to run right. Because you sequential right.
Now what today is doing, you are going to have to have to use this kind of decorator and remote and is doing under the hood.
When you are going to call this, you are going to call f dot remote.
Um, then um is going to take that function.
So here when we execute that and it's going to submit a task for that function
which will be scheduled by somebody in the system can be on a different worker,
one or different note this function is not blocking okay.
So he's going to return immediately. And he's going to return a reference, a future to the result.
Okay, because it's non-blocking. You can execute now the next.
The next call to the function f dot remote. And is going again to get you back, um, a reference to the result.
And finally you get to get the results. Now this is a blocking function.
And under the hood. Do you know, um, that I submitted these tasks to a local scheduler?
Uh, and the local scheduler will figure out where to schedule these, uh, this, uh, tasks for each of the functions.
And then, you know, these are going to be scheduled, executed in this case in parallel because there is no data dependency.
And you are going to get the results. And because these two tasks are executive in parallel, you are going to now the entire, um, program is going to.
Take only one second. Okay. So this is functions.
You also have classes are instantiated as actors.
Again you do that you just need to say array dot remote and then dot remote.
When you instantiate the class and then you call it a method of the class.
By the way this dot remote. Um, it's actually not needed.
The reason we preserve it in the language is to make that developer, the programmer,
aware that this is going to be can be executed remotely on a different machine,
and it can be slower, but also just to, um, give an extra hint to the developers.
It's also has the ability array to which was uh, early on we made this decision, um, to specify the programmer can specify resource demands for, uh,
a particular function or of a particular um, uh, class one is going to be instantiated in terms of the number of CPUs,
GPUs, memory, uh, requirements and things like that.
Okay. This is class. Uh, that's conductors. Then I mentioned that you have a shared in-memory objects store.
Um, so this is, uh, this, uh, uh, shared memory.
Um. It's, um, um, it's, um, um.
It's it's it's it's read is right. Only read and read so you don't update.
It doesn't provide the ability to update, um, a data.
And this makes obviously keeping the consistency much easier.
Uh, of course as a programming level you are going to update it, but under the hood you are just going to create a new version,
uh, next of a new version of the data with, uh, the modified data.
Okay. Um, so and here is another simple example where you have two functions where f and g where uh, where uh and um.
You call F and then. Then you'll pass the result from F to G.
Okay. So again this what happens here is I do not have to call f dot remote.
It's again it's unblocking call. You are going to execute eventually uh Ray is going to instantiate the task associated with F.
And the return is a reference to the result of f this distributed future.
And that is going to be passed to uh, to G.
Right. And the remote is going to uh, G is going to the grey is going to instantiate a task for G,
maybe on another worker on another node and return, uh, the reference distributed future for G.
And then you can you can get the result.
So now and now you have a dependency between f and g is a data dependency that knows that and basically is going in before executing G.
It waits for the reference to be resolved, which means that f should be it should be executed.
It should, uh, finish and produce the result in this case x.
And then now that X is available, rail will uh, transfer x to the node which is G.
And now you are going to finally run G.
The main point here is that in this particular case x it's transferred only once between uh node one and o two which runs fn g respectively.
And the reason I'm giving this example, because you have to to if you want to to contrast with remote procedure call.
Right. The remote procedure call is again a way to run function remotely.
And you have that in many languages and so forth. But in remote in most of the remote procedure calls implementation the data.
Uh, it's the arguments are passed by value, right.
And in this particular case, for instance you have to execute x.
Then you are going to get the value back. You need to execute f which will get the value back x.
Uh, just to pass it to g okay.
And then G will be executed on this.
Now to and in this particular case, as you can see you have two transfers of X in the system instead of one.
But this is again you knew that. I mean that's why the the advantage of using references, passing the argument by reference to avoid extra copies.
And that's true for a single node.
And, uh, it's also true, obviously for a distributed system.
Okay. Um, now I will also, um, try to mention some ongoing work and obviously a lot of in going work.
It's about improving the GPUs. What we are witnessing today, we are moving.
We are have been already moved from a CPU, uh, dominated compute infrastructure to a GPU or accelerator dominated infrastructure.
And we do a lot of work on DirectX to reduce the overhead for that of GPU to GPU communications,
uh, and implementing different collective operations and so forth.
This is one example. Um, uh, basically that I can give you to GPU communication is like in the original Ray, though.
Remember, we have this kind of, um, uh, object storage.
So object storage is implemented in the CPU memory.
So what happens if we want to to transfer some data from one GPU to another like a tensor?
What happens is that this tensor is moved from the GPU memory to the local Ram memory of that.
Now I'm going to go through the object store from to the Ram of the receiver,
and then from the memory CPU memory of the receiver to be transferred in the GPU memory.
So it's kind of quite slow, right. And uh, obviously what we want and there are, uh, a lot of copies, a lot of RPC calls.
So what we want, obviously it's not a GPU to GPU communication, uh, and memory statically allocated and everything, uh, for optimising it.
So one thing I wanted, I give you this figure because initially when I did that and this is still experimental,
we saw that everyone only use direct communication direct transfer between CPU and GPU zero and GPU one.
But there is one instance two out of surprisingly people preferred.
So one is coming on the out of left to go.
These are all traditional way of transferring, uh, the data through the CPU memory.
Do you know I do you know, what is that? What is a case in which you may still prefer?
The left. The slow transfer over the right one. It turns out that when you do Gpio to Gpio transfer, it's a synchronous operation.
Okay. You need also to synchronise the sender and the receivers.
You need to allocate the memory in the Gpio receiver to be ready to receive the data from the sender.
Okay. Now in this case it's more it's more asynchronous.
You need only to be ready to have to allocate the Gpio to be ready to get it to have to allocate the Gpio.
Um, memory. When you have the tensor here on the already it's instantiated the sends out the tensor on the local machine in the CPU or memory.
So the amount of time for which you keep the GPU memory allocated.
To get the tensor is going to be higher when you have direct GPU to GPU communication than when you have slow memory, right.
Again because synchronous. So if your GPU memory it's extremely, extremely a premium and is a bottleneck, you may prefer uh, the left hand side.
Okay, so, uh, there are many, like I mentioned to you, you know, it's Ray is very flexible.
Uh, it's also provide support for a very generous infrastructure.
And here you have, you know, batch inference where you have you read data,
you pre-process the data, or you do something for finance and you write the data.
And, uh, here is the data pre-processing and post-processing can run on CPU as well as the inference can run on GPUs.
And by doing so you can save a lot of cost instead of running everything on GPUs.
And there are many workloads you can support here, or multi-node search and so forth.
Fine tuning, embedding computation. Um, Ray, it's also emerging as a standard, you know.
Well, the not standard but the framework of choice for Post-training.
There are a lot of post-training frameworks being developed over the past year,
many of them using Ray and also using the multi-language AI system or also developed at Berkeley.
Okay. So impact it has been a very popular framework actually.
It was used by OpenAI in their training infrastructures to train ChatGPT for and um, uh, and,
and others and by many other companies to build their AI infrastructure, um, very fast growth, especially in the past year.
This is a number of downloads and this is GitHub status, uh, compared with other distributed frameworks, uh, popular distributed frameworks.
So you can see the growth is accelerated accelerating.
So you saw three I have two other projects I have to finish quickly.
Um, so next is Vlad. Okay. So Vlad, um, it's about inference.
And you know. So what? What kind of change?
So what kind of the trend? So the trend obviously was the large language model.
So what was different about this model from the point of view of training and inference?
It used to be that in the traditional machine learning, for every task, for every use case,
you are going to train a model and then you are going to serve the model.
So training set of large language models are much more general, and they are a kind of provide some do multiple tasks.
So you train a large language model and then you can use it for multiple tasks.
You train wise and you you do the inference for many, many tasks.
Okay. So that means that the inference actually becomes even more important.
And um, you have a lot of this was happening by the in 2023.
This is when I started this research. Um, and um, at that time it was very expensive to serve these, uh, models.
Okay. So you can have, uh, if you remember, each GPU can only serve a handful of requests per second.
That was A100, if you remember, for us, you know, that was a top GPU, uh, in 2023.
Um, okay. And so you have neither, you know, the cost for request was very high.
And why is that? Everyone knows that. So I'm going to go quickly is because, as you know, uh, these models are autoregressive models.
And when they generate a token, they generate a new token based on all the previous tokens.
Okay. So you have this kind of strong dependencies, this attention mechanism.
Uh, okay. And um, so sorry about so generating one token at a time because you have the dependency, it doesn't exploit the GPU parallelism.
Right. The GPUs are fast and have a high, uh, you know, uh, compute capabilities.
But they do that when you can exploit their parallelism. Right.
But now you cannot exploit them because of these dependencies for one request.
So of course, if you cannot exploit for one request, then what is a solution?
Well, I'm going to try to serve multiple requests at the same time.
Right. So you request a batch. You are going to process a batch of requests.
The problem is that now you are going to get out of memory.
So the memory becomes a bottleneck. So why is that.
Well because again of this dependency to generate the next token you need to store it in memory.
And this is how memory is organised. You have the number of zeros as 40GB.
That is at times 3830 billion a lm.
That was a Lama model. So the precision at that time was two bytes.
So you have 26GB with 65% of the entire GPU memory.
And then you have this other activation a few percent and then a KB cache.
So okay we can try to keep the data. Um temporal data to generate the next token.
So in this case we cache you need to keep the embeddings you know for the entire prefix right.
For each request. Right. And each token, each embedding it was like one megabyte.
So one request can have several gigabytes. Okay.
So that's a problem. Okay. So, um.
And when we looked, actually it was this was compounded by the fact that, um, before what happens is a memory was contiguously allocated.
That's a easiest way to allocate memory contiguously allocate memory you are going to need.
Right. And you also need to because it's contiguous.
You need to provide space for the future. Future tokens.
Okay. Of course, you know, if you took, you know, systems 1 to 1, you know,
that list that can lead contiguous memory allocation can lead a lot of can lead to fragmentation.
Um, and you can lead to internal fragmentation because I am going to allocate a space for a request.
But I don't know eventually the length of the output. Right.
And because I don't know, I'm I'm going to have to probably allocate more than I should.
So you have fragmentation there. Um, you also have um, this is a interesting one.
So the interesting one is the following. So let's say that you know, exactly how long is the output okay.
So then I can allocate that all the and then as a block of memory to fit all the output and no more.
Well it turns out that even this is not optimal. So why is not optimal.
Say I have to allocate 1000 tokens for output.
Now these tokens are going to be filled sequentially one by one right.
So initially I have 1000 tokens free I of next token I'm going to generate one token used 999 three and so forth.
So during the process, you know dynamically I am going to waste a lot of tokens which are allocated space which are allocated in the future,
right, which are not yours. Those are going to be used eventually, but for a long time they are not going to be used.
So this is kind of, uh, reserve reserve instance and external fragmentation.
I'm I'm going to not talk much about that.
And at this, this was at that stage, this was how the memory was used.
And the only thing you need to be look at here. It's about the green part.
This out of, you know, the other colour, a different kind of fragmentation.
But this is the effective memory which has been which was used for uh,
by the inference system that at that time, uh, and uh, the most of it, the largest on a 38% by 38%,
you have this kind of oracle is saying here, Oracle is what I told you, assuming that, you know the future, we should acknowledge that in our future.
How long is the request? Still, you are going to effectively use only 38% of the memory if you don't know the future.
The best at that time was 26.8%. The rest was kind of wasted.
So as you know, what is a solution? You know you still have fragmentation and things like that challenges segments.
If you remember, you know, operating system. What do you do that. Beijing.
You have this kind of Beijing which are fixed sizes, you know, a few kilobytes and so forth.
And then the there is no fragmentation, external fragmentation between these pages.
And then, um, you still provide to the application the illusion of, uh, you know, of, uh, nice, uh, address space, um, which is contiguous.
But this address space is going to be mapped to different pages under the hood.
Right? This is exactly what happens here. Right. So this is what memory management you have the process a process.
We they have the mutual memory which is map which is contiguous for each of them.
The virtual memory and the virtual memory under the hood is, is mapped to these pages, which are all over the place in the physical memory.
So the same we are going to do here. And this is a logical KB block.
So it's everything is contiguous each block after another.
And but in the physical uh, memory, uh, you are going to have different blocks are going to be spanned a different location.
A block consists of multiple is a fixed size and consists of a few, uh, of several embedding tokens in this case for right, for tokens.
Okay. So then and then how do you go from the logical to the virtual block space.
Well you have a mapping. You have um, a table, uh, which index?
Uh, each a logical block to a physical block in memory.
And then you have another field which is, uh, field here, uh, which basically says how much of that, uh, block is, has been filled with tokens.
Okay. So in this case, the block one is two.
Um, and now you are going to generate an off next token in this case.
And, and you are going to put in the corresponding block in the physical memory and so forth.
Now and when it is full, uh, the block um, they are allocating another block in the physical memory.
Right. So it's kind of simple. And again what it um, this allows also if you are having multiple, uh, you know,
request and so forth, and ask certainly the same prefix to share as a prefix more efficiently.
Um, so, um, you know, it is you can do parallel sampling multi turn when you have multiple ah, conversation or anything like that.
Okay. Um, now let me tell you because it's interesting, it's, it's very much like paging in kind of,
uh, um, virtual memory and operating system and, but um, there are also a few differences.
So similarities pages are very similar is CPU blocks are very similar with pages.
And you can share pages across processes since like if you have uh, common prefixes for the,
for the request or if you have system prompt, the system prompt, it's used for every prompt, right.
So it can be shared across all prompts from all the, uh, requests.
Uh, but there are some differences. Okay.
So when you when you talk about evictions, you see, and when you evict, when you run out of physical space or memory space, what do you do?
You evict pages, right? To make room to bring a new page unit.
Right. And that's typically a start on uh.
Uh, you know, in, uh, you know, swap space on the disk and, um, then you can bring it back later.
So on, on a slower storage, but bigger storage. Um, so in this case, there are two differences, you see, because the dependency, you know,
when you, you need to generate the next token, you need all the tokens before that.
Okay. So it doesn't make sense to just, uh, page out one block.
Right. You have to position all of them. Right. Okay.
Because you still need to do all of them. And you are going to need to get the next token right.
So you need to do I think you can do it. Request level.
Uh, eviction versus page level eviction, which will free more memory for as a request because the other one also is interesting.
It's like, you know, like I said, in the operating system, you do page in, page out and page in your store is a page on a disk on the disk.
But here because, uh, you know, because the, um, GPU is, um, have so much compute,
sometimes it's more efficient to throw away the, uh, block and to very compute,
to remember, to realise it when you need it,
because the computation is faster than just storing it for in this case can be on the CPU memory and bringing it back.
Okay. This is some of this is fallacies. So what was the result?
Um the result we got um 96% in terms of the memory utilisation at that time.
And this converts directly into support. Right.
So 96% compared to this 26.8%.
It's almost four times higher support because you can fit now in a batch.
And you can process at the same time four times more requests.
Okay. Um, is also is actually this is one of the fastest growing.
Projects I have seen. I was part of it. That's kind of GitHub status of NLM.
Think it's another? Unfortunately, I don't have time to go over it. It's another project we developed, uh, in the lab.
Um, it was a top open source project on GitHub in 2025.
And I also wanted to show these how hard things are in practice.
Um, and this is about, uh, you know,
like what what is saying this is are the number of models which are supported
actually September last year by VLA lab and the number of accelerators it supports.
It's 260 accelerators. And I don't know, it's like 500 model architectures.
It's very, very complicated, very complex. Um, okay.
So yeah, and this is what I want to say here is at this page attention.
It's actually as a techniques via the artefact by page attention.
It's use now by all of these inference uh, uh systems, not only BLM in the industry.
So he's a nice to see that. So the final one and it'll take me ten minutes.
So this as well. Um so it chatbot arena rethinking the VLA as a L alum.
Uh, evaluation. So this is a different one. And this is hopefully, you know, it's like I thought it's a nice study and it's the end of the talk.
So hopefully it's going to be more engaging.
Um, so look, um, it's this is, uh, the Patterson during our webinar and he has this famous quote, for better or worse, benchmark shape of field.
Right. And this is absolutely true about AI. You have all these kind of emission at and CFR and all of these early benchmarks.
And you have more and more now benchmarks right to to uh, to compare and to see which models are better.
So here our study. So, um, if when the time was met, I was opening sort of set of models, the open sort of salama in 2023.
In February, a group of students, uh,
then took that model and took this kind of shared Egypt date that we just shared Egypt as a date as a prompt
sends the results that people are sharing over the web and use that to fine tune right to father train this,
uh, llama, uh, model and call the model vicuna.
Okay. And this was high quality data, shared GPT, because people only share the prompts and the results.
They are, you know, uh, excited.
They are proud about.
Okay, so that was and we affirmed, um, this is the trend here is the rise of chat bots, because you are going to be QnA via a chat bot interface.
That was a chat GPT early days. And it's a question.
It's about how you are going to evaluate it. You have this kind of benchmark with a static benchmark that time.
I mean, you have minutes of thoughts, but they are static.
Static means they are also prone to contamination by contamination.
What I mean is that the model is this large language model, as you know, are are trained on all the data which is available in the internet.
And these benchmarks are available in the internet. They are going to be trained on this benchmark,
and then they are going to be evaluated on the same benchmarks and then do not capture human preferences.
Right. It's not only about correctness, it's about the style and so forth today.
So that's kind of the problem okay. So how do you do it. Now this is a contamination.
It's an example to show you that it's real. This social group is some kind of you know, this is kind of some message I think was sort of on Twitter.
Um, so code force. Right. It turns out that it was ten out of ten solving correctly, um, you know, before 2021 and zero out of ten after that.
And that was, incidentally, the Cut-Off date for data used to train GPT, right?
Of course, you you are trained on, uh, on the solutions, uh, you are going to remember all these solutions.
So that's always the real problem. So you need human evaluation.
So I did it. What do you do as a faculty. Right. Like okay so you'll get some students and I give them some pizza and say okay ask.
And these are the prompts for different models. See what is better okay.
So we realise that this doesn't really scale okay.
And why doesn't scale? Because the assets and the evaluations, uh, take time are not obvious.
It's not like a math problem. This is good and right. And here are two examples.
One one says this question is develop a Python program that reads all the text files
under a directory and returns top five orders with the most number of occurrences.
Okay. You know, so everyone can do that.
And I'm not going to ask you to program is is no, no question.
But I am going to give you two solutions provided by two large language model the two jackpots.
And I'm going to ask you each one is about. How many of your sinks is?
A assistant a. Assistant.
Hey, what about assistant B? Okay, I know that Oxford is very strong on all the other departments.
So here it's like it's a biology. Photosynthesis is a vital.
Process for life on Earth. Could you outline the two main stages of photosynthesis,
including where does it take place within the chloroplast and the primary inputs and output for each stage?
Okay. I'm not going to ask you which out of those, I'm going to give you two answers and I'm going to ask you a Chinese method.
A lot to me. How many? Be.
Okay. But. So now we have a challenge to our, you know, in our hands how we are going to scale this.
Right. It turns out like like what we did is like, well, GPT four was released two weeks before, right?
And they said, why don't you know, students are very inventive, you know, say, why don't you use this?
Uh sympt40mes. Like charge, right.
Say, okay, this is the GPT. This is a question. These are the assets, you know, tell me what is the best on and the.
So it was a question. It was graded from one to, uh, 1 to 10 or something like that.
Okay. And that was a you know, this is a judge, right?
This is um, and you know, the paper about on about that and things like that.
So this is, uh, I think that, you know, this is the first year as a judge, um, in the open,
I think that Microsoft did when they did the internal evaluation of, uh, GPT.
You know, they have a very strong relationship at that time was OpenAI.
But this is the first application in the open. Okay.
And we had to apply it. And, you know, for instance, like for this first question, the Python question, it turns out,
um, that a assistant A didn't handle case sensitivity, punctuation and, and so forth.
So in this case B was better. And in this case, while a site looks much more comprehensive and detailed,
I should again be is better because a confuses which are the inputs and outputs from the two stages.
Okay, so when you look about with this explanation now, it's much easier, right?
It's like to do it okay. But we did that.
But people come back to us and say hey yeah, yeah again.
So it's very early on. It sounds good on this anecdote because anecdote, uh, you know, um, data points.
But really, how does it performance your back in square one to have human evaluation to compare?
Right. And um, again how to scale it.
So human evaluation, you know, like ideally you have for every questions you want to rank the alleles.
I have one question is I have the answer from general rank them right.
And then you do what is a better. Now we know that ranking and choice is hard and easier is to pick some best out of n, right?
Um, but even that is hard. But you know this paradox of choice.
So they use this easier saying you have to pick the best of two.
You cannot do you know less than that and it's the least you can do.
So that's kind of first one angle. Okay. So pick the best answer between two elements.
And now there are many ways you can do it. One you organise a tournament right.
Each for each question and so forth.
The problem is is it doesn't scale and doesn't scale for many, many reasons.
Um, one is because you have to play tennis square, right?
To play, everyone needs everyone but the other one. The bigger problem is that the static.
Right. You cannot enter new steam into competition, right?
Unless it ends. And this leaves sort of starting to, you know, new a new source being released, you know, every month.
Every week then. Okay. But fortunately, there is another way to do it.
And this is our key idea here is that, um, you know, like writing, right?
Like chess. And there are many other similar ratings in other sports.
Almost every sport has one in which you can have a meaningful rating.
With players not playing, everyone is everyone, and it's more related about the strength of the opponent.
If you are going to beat a stronger opponent, you are going to get a bigger bump.
If you are going to be the weaker opponent. You may get a very little bump.
Okay, so that's the basic idea. So this is what they developed. This is kind of chat about arena.
So what we did we provided this kind of interface people.
We provided these models for free. You ask questions you get to access.
So this is you know very early days. So that question is very simplistic.
The answer to this article is so simplistic. And then people can say A is better, B is better in style or both are bad.
And then you compute use. We use a.
Sorry. We use Bradley to remodel.
To compute rating this arena rating. And we rank these models with a proper confidence interval.
Something like that. Okay. So that's what I've done. Okay.
There are many categories here. Uh, for, um, you can rank these models.
We had a lot of, um. Uh, yeah.
And, uh, let me just. Okay. Um, now, now you have user evaluation, and you have them as a judge, right?
So now you can do a proper study. So, you know, this kind of paper is actually this was, uh, in the same paper, like I am of the judge.
And what did we did? Uh, the findings are that the lines are biased when they do make judgement very much like humans.
You have, for instance, positional bias. They prefer the first answer that if you give them question, answer one, answer two.
They prefer the first time. Uh. Verbosity bias. Sometimes they prefer longer answers.
Um, self enhancement bias. They certainly prefer answer from yourself.
Okay. Or the other. Uh, I mean, the same family.
And as in time, they have limited reasoning capabilities.
They are not good at math. Okay. Um, so and then we look about the agreement between two humans, right?
Experts and also between humans and large language models.
And, you know, at the end of the day, it was quite similar.
The human to human preference is 81%. And the human to beat question was, uh, 85%.
Okay. So that's kind of okay. Um, we have many other modalities.
This is one question I asked last night. Please show an image of the Oxford University at 4 p.m. in February.
Okay. It's very, very close to that. Um, I selected a being better.
Um, and. Yeah, this is again on many modalities.
Um, what can you do with the data? I'm going to be very quick here.
Um. Uh, let me just. So here is one idea.
Is that from the leaderboard? So after we do here and we develop a model that's kind of, uh, I think this is,
uh, you can you can look at this equation on the I tell you what is here.
So what you do here, you have a prompt. If I have a prompt, I can generate.
You know, on the fly a rating for different models for that prompt.
So how do you do that? Maybe we never seen that prompt. However, I've seen a lot of prompts similar to that prompt.
So you use, uh, votes to the to the to the to the prompt of the presents.
Other problems which are similar to the new prompt as a proxy to compute the leaderboard for that particular prompt.
And this has many applications.
You can um, you know, given a cost target, you can maximise the arena score or given a arena score you can maximise you can minimise the cost.
Right. And this is different models at that time when I wrote the paper and with this one you know for the same cost you can get 30 more Elo points,
um better models and for the same accuracy I didn't.
I do not score. You can get to X cheaper by using this from the leaderboard and picking the right model.
Okay. You know, it's like it has a you know, it was very rewarding because right now it's like most Foundation labs,
before they release a model, they, they, they, um, uh, evaluate these models also on uh, uh, arena.
Now it's called arena. Okay. And we have a lot of users votes and things like that.
Yeah. Instead of some, uh. People tweeting when the new results come out.
Okay. So I'm going to go very briefly.
So I'm going to have these three, uh slides.
And then I'm going to go to conclusions. Uh so what are the lessons we learned.
I think these are important. I don't want to skip one of these. Lesson number one is that trends do matter, right?
With the emergence of complex AI workloads and heterogeneous distributed systems, what's really important with Vietnam?
The aggressive nature of LMS. So it's a problem changes and problem changes in most of the cases solution is going to change right.
And so chat bots you know looks like out of basically natural labs and complex AI workloads is what actually determined us to work on these systems.
And and actually are the drivers also behind problems that impact the same lesson too?
The simple solution matters, obviously. Um, you know, it's, um, you know, the ray you see is a he has a record, a minimalist API.
Vietnam. The pay attention is quite simple. Uh, as a judge and head to head evaluation using, you know, that, like, writing is very simple ideas.
Maybe obvious, too obvious in retrospect. And the reason this matters is that is easier to understand also by other people, right.
And applies them like I stole your page attention. It's used now by everyone.
And also, you know, like writing, you see a lot of other people doing it or as a judge.
And this is related to the impact, because if you understand something, how is working and how to do it,
you are going is much more likely you are going to do it right versus something you just have to understand.
And uh, the other one, I want to do it. I want to say it is like it's like, um,
you have to plan on for flexibility and rewriting when you develop these kind of systems because you do not have the requirements,
the requirements evolves as you are going to develop the system.
So yesterday we we already wrote the rate for times, uh, which is um, uh,
we lamb who already wrote it once and it's again, it's two and a half year old project and now we are it's another year.
Right. And and and and so forth. Um, so I'm going to skip over the next one and I am going to move to questions.
So, um, so summary is that, um, I do believe in open source everything.
We've done it vertically. It's open source. We actually don't in the lab.
I am we you know yet and in my lab we, we don't file for any patents.
So everything is for public, uh, consumption. And we, you know, I talk today about these three of these, uh, projects, right.
And chat about how, you know, uh, and I do believe that, you know,
we are just scratching the surface and there will be many, many more, uh, challenges we are going to have had to address.
And I hope that most of them will be addressed in, uh, open source using open source technologies.
Thank you.