Stanford CS329A: Einführung in selbstverbessernde KI-Agenten und Scaling-Trends

[Intro Music] (Aakanksha Chowdhary): Welcome, everyone, to Fall Quarter, and welcome to CS329A. I hope you made it to the right class. This class is on "Self-Improving AI Agents." Anyone made it to the right class here? (Students): Yes. (Aakanksha Chowdhary): Yes? Okay. How many of you are here and doing masters at Stanford? Okay. PhD? Okay. Few of you. And then undergrad? Okay. So, masters is the big crowd here. (Azalia Mirhoseini): That's a good balance, yeah. (Aakanksha Chowdhary): It's a really good balance. It's a really good balance. So, we'll introduce ourselves. So, I'm Aakanksha. I have been working in large language models for a while, and I'm a adjunct professor at Stanford, and I also in research at a startup called Reflection AI. (Azalia Mirhoseini): Hi, I'm Azalia Mirhoseini. I'm a- I'm an assistant professor in the CS department, and um- a- Aakanksha and I met each other when we were at Google Google Brain. Google Brain. Yeah. Back in the day, and then Google DeepMind, and I also worked on Claude at Anthropic and Gemini at Google uh DeepMind, and we are so excited to uh teach this course again. This is the second time we're teaching. Of course, we have made some updates, both to the lectures and also the way we are uh kind of conducting everything during the- the- per the course, and we are- we were just so amazed by the amount of interest uh for people to join this course, so we are excited to have you here. (Aakanksha Chowdhary): The course website is at CS329A.stanford.edu. We have updated the website in terms of all the lecture materials and, and the papers that you'll be reading, as well as uh what's the homework schedule, to expect, um the project, proposal, and project structure, as well. So, in today's lecture, we'll start with an overview of large language models. Um, just how the scaling trends have evolved over the last uh 5 years, and then we'll focus more on what we will cover in this class. And then, from there, um, in the second half of the- this lecture, we'll cover the course logistics, and that part you should pay a lot of attention to because that influences your grades. So, let's take a look at an overview of the trends for large language models. So, one of the aspects that has been mind-blowing since uh GPT-3 came out is that- [Long Pause while speaker navigates slides] (Aakanksha Chowdhary): -there is a- that, as you scale up the number of parameters in a model, uh they get better. And what this means is uh that language models like BERT and T5 were good, but as we increase the number of parameters, they got really better. They- they perform much better, and what this led to is something called a "scaling loss" for large language models, uh which are used in building the base model, for the pre-trained base model. And uh as you increase the amount of compute, so there are 3 graphs here. As you increase the amount of compute on the x-axis, the loss function on the y-axis goes down, and- and the more uh compute you have put in, the more uh the test loss goes down, which leads to a better model. So, that was one big aspect that led to GPT-3 and then all the subsequent ChatGPT, PaLM, uh Gemini, and so on. And then, similarly, as you increase the dataset size on x-axis, uh the test loss goes down further on uh y-axis. So, that's another aspect of like- another axis of scaling that leads to better language models, and uh the fourth one is of cour- the third one is of course parameter counts. So, if you increase the number of layers in a transformer or the amount of parameters that are there in the language model, that improves the loss value, and that that leads to a better model. And then, this has been the foundation for a long time, uh until last year where this was starting to hit some kind of a saturation point. [Slide change] (Aakanksha Chowdhary): So, from 2018 to almost 2024, uh this graph is a little bit outdated, but uh from 2018 to 2024, the as the model size has gone up, um almost consistently we have been increasing the- the size of the model. So, you started with something like BERT was 340 million parameters, GPT-2 was 1.5 billion parameters, and then GPT-3 was 175 billion parameters, and then uh say PaLM was 540 billion, and then uh there was a bunch of subsequent models, and GPT-4 is estimated to be there, which is like trillions of parameters. So, you can imagine that there has been an exponential growth in just the model size uh of for large language models. The "large" stands for that they are growing in size, and that has been one of the key areas in which uh the models have been getting better. [Slide change] (Aakanksha Chowdhary): So, why do we care about the models getting bigger? What does this lead to? So, as I was explaining that as the models get bigger, uh they continuously improve in performance on uh national language benchmarks, on various other benchmarks, reasoning benchmarks, and so on. And then, other interesting aspect that I uh will show you in a moment is that they learn how to do few-shot learning. So, earlier, you have to fine-tune the model for specific domain, but just by giving a few examples, the model can follow that template and- and reason in that same vein, which makes it extremely easy to prototype uh things. And then, the third aspect is that, as the models become bigger, they have emergent behavior, where the capabilities like reasoning only emerged in larger models. [Slide change] (Aakanksha Chowdhary): So, just to understand few-shot learning or zero-shot learning here. So, if- if the model is given a question, like- or a task, like "Translate English to French," uh it's able to give uh so this is called a "prompt," like it's given a task "Translate English to French," and then it's asked to translate "cheese" to whatever is the French word. If the model can predict the answer without uh actually being trained on the specific task, then that would be called "zero-shot learning." And then, "few-shot learning," in addition to giving the- instructions of the task, you're also giving a few examples, so in this particular case, it's given "Translate English to French," and it's given a few examples of the translation from English to French, and then it's asked to translate "cheese," and it's able to translate that, and you can see that this capability shows up in the large language models across a large number of tasks, and that has been one of the key areas that GPT-3 and PaLM and uh other models enabled, and then that uh subsequent innovations uh came from reasoning aspects of this. So, on the reasoning side, um as the models have become bigger, they have this emergent behavior of reasoning, which Azalia will tell us a little bit more about. (Azalia Mirhoseini): So, as the models become bigger, not only we have this kind of a predictive scaling laws property that we kind of know how the loss function's gonna go down as we train more compute, data, and parameters in the model, there are new kind of behaviors that appear in the model that they didn't have before, and we never could predict that until we had these bigger models and saw these kind of behavior in them. And one of the most important uh one of those behaviors is the chain of thought behavior. And for those of you who are not familiar with this, here is how it works. So, in a usual prompt prompting of a model, we can ask a model, in this case, a a math problem, um and then we can give it a one-shot example. Here is a math problem, here is the answer. Now, I ask you a similar but not exact, not exactly the same math problem, and the model is expected to answer that. Um and the model might be able to use the- the one-shot example, or it might not be. But we can do better here by not only give an example in, but also provide the reasoning or the chain of thought on how we got to that answer. So, in this case, the question is, a very simple question: "Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many total tennis balls does he have now?" And instead of just saying "11," like you're walking the model through how to get to this. Um, "Roger started with 5 balls, 2 cans of 3 tennis balls is 6, and uh 5 + 6 is 11." Uh and with this example, then the model has become familiar with the process to get to the answer, and it can leverage that to answer new problems. Well, obviously, this is a very simple problem, and any like 1B parameter model nowadays can solve this, doesn't need a- an example or a few-shot or a chain of thought. But the chain of thought property itself is a property that is holding to this day, and it's a very, very important property for the reasoning models and all the thinking models and all the prog- and many- much of the progress that we have seen um in the past year or so. [Slide change] (Azalia Mirhoseini): Now, um this property again appears as we have larger and larger models. Here are 3 models: LaMDA, GPT, and PaLM, and as you can see, the chain of thought, and this is some results on the math, some math dataset. And these models, when they're smaller, like 8 billion parameter for LaMDA or uh G- for GPT around 7 billion, they can't really benefit from chain of thought. Like it doesn't do anything for them. But as they become larger, uh or for a PaLM model again at 8 billion, it can leverage this chain of thought property and learn from the reasoning that is and the- and the process that is provided in the context, in order to solve problems better. Um and not only that, there are other kind of abilities to solve new tasks as we increase the size of the models, like, for examples, models can all of a sudden solve "mod arithmetic" problems better, or solve uh a word unscramble problem, just by having this larger size and this property, you see that all of a sudden, it appears at a certain size. Um and because of all of this, we still, as in frontier companies and labs, are still interested in pushing the- the scaling laws and not only get the natural kind of progression, but also we are be- we will see more and more of these kind of emergent behavior that we are uh that is going to be very important. [Slide change] (Azalia Mirhoseini): Now, um a little bit of history here. So, ChatGPT was launched in uh November 2022. Obviously, it was one of the most successful uh kind of apps or software ever created. The time to reach 1 million users for it was um 5 days, which is significantly uh um faster than any that many of the other famous um software that you can see there or services that you can see there, and there were pieces of of kind of innovation on top of just scaling up the parameter size of the model that were very important to make ChatGPT what it was and to make that kind of leapfrog and over the previous models like GPT-3 and so on. And two- two important piece here were uh the instruction tuning and um the reinforcement learning from human feedback, which which we're going to briefly mention in the next slides, but throughout this course, we're going to learn more and more about them. Um, so let's first, and then uh we're going to learn about all of these in more details, but let's just take a step back and see how the process works. So, the pre-training step is the step with the the beginning of any language model training, uh that's the easiest step and we just train the model to predict the next token out of all sort of text and data and so on. Uh, and then it this is the fine-tuning step that was a key differentiating factor for a model like ChatGPT. Um, and let's start with um uh the first aspect of fine-nu- fine-tuning, which is kind of alignment between human preferences. [Slide change] (Azalia Mirhoseini): So, when a model is pre-trained, it has no sense- it has seen all data on the internet, on the book- in the books, everywhere, but it has no sense of like what is right and wrong and what is- it just have this statistically knows about the state of the world but does it- it doesn't exactly know things or know how to follow instructions. So, the next step in training these models uh is the the the step where we try to steer the AI models to follow the goals and preferences uh and values of humans. Um and this is still a big problem, it's a big issue, we- we haven't like mastered it yet, but as the models become as we progress, models are becoming better and much more uh kind of like uh powerful in the things that they can do. Um so it turns out that we can fine-tune the models to be aligned, for example, uh here are some graphs that show what if we take our base- base model that's just pre-trained, and fine-tune it for things like "sensibleness" or "safety." And these are like it- these are data that we have collected, these are high quality data for kind of showing the model what what we mean by what safe and what's unsafe, or what's sensible and what doesn't make sense, and these are like more curate- like more highly curated data that we pass to the model for fine-tuning and to make the model more aligned with the values or the- or the preferences of humans. [Slide change] (Azalia Mirhoseini): Um [Pause] (Azalia Mirhoseini): So, going back, let's just walk through these process again. So, the first stop when we take a pre-train language model is uh fine-tuning with next-token prediction, which follows the same process as pre-training, uh that's the easiest step and we just train the model to predict the next token out of all sort of text and data and so on. Uh, and then it this is the fine-tuning step that was a key differentiating factor for a model like ChatGPT. Um, and let's start with um uh the first aspect of fine-nu- fine-tuning, which is kind of alignment between human preferences. So, when a model is pre-trained, it has no sense- it has seen all data on the internet, on the book- in the books, everywhere, but it has no sense of like what is right and wrong and what is- it just have this statistically knows about the state of the world but does it- it doesn't exactly know things or know how to follow instructions. So, the next step in training these models uh is the the the step where we try to steer the AI models to follow the goals and preferences uh and values of humans. Um and this is still a big problem, it's a big issue, we- we haven't like mastered it yet, but as the models become as we progress, models are becoming better and much more uh kind of like uh powerful in the things that they can do. Um so it turns out that we can fine-tune the models to be aligned, for example, uh here are some graphs that show what if we take our base- base model that's just pre-trained, and fine-tune it for things like "sensibleness" or "safety." And these are like it- these are data that we have collected, these are high quality data for kind of showing the model what what we mean by what safe and what's unsafe, or what's sensible and what doesn't make sense, and these are like more curate- like more highly curated data that we pass to the model for fine-tuning and to make the model more aligned with the values or the- or the preferences of humans. [Slide change] (Azalia Mirhoseini): Um and because of all of this, we still, as in frontier companies and labs, are still interested in pushing the- the scaling laws and not only get the natural kind of progression, but also we are be- we will see more and more of these kind of emergent behavior that we are uh that is going to be very important. Now, um a little bit of history here. So, ChatGPT was launched in uh November 2022. Obviously, it was one of the most successful uh kind of apps or software ever created. The time to reach 1 million users for it was um 5 days, which is significantly uh um faster than any that many of the other famous um software that you can see there or services that you can see there, and there were pieces of of kind of innovation on top of just scaling up the parameter size of the model that were very important to make ChatGPT what it was and to make that kind of leapfrog and over the previous models like GPT-3 and so on. And two- two important piece here were uh the instruction tuning and um the reinforcement learning from human feedback, which which we're going to briefly mention in the next slides, but throughout this course, we're going to learn more and more about them. Um, so let's first, and then uh we're going to learn about all of these in more details, but let's just take a step back and see how the process works. So, the pre-training step is the step with the the beginning of any language model training, uh that's the easiest step and we just train the model to predict the next token out of all sort of text and data and so on. Uh, and then it this is the fine-tuning step that was a key differentiating factor for a model like ChatGPT. Um, and let's start with um uh the first aspect of fine-nu- fine-tuning, which is kind of alignment between human preferences. So, when a model is pre-trained, it has no sense- it has seen all data on the internet, on the book- in the books, everywhere, but it has no sense of like what is right and wrong and what is- it just have this statistically knows about the state of the world but does it- it doesn't exactly know things or know how to follow instructions. So, the next step in training these models uh is the the the step where we try to steer the AI models to follow the goals and preferences uh and values of humans. Um and this is still a big problem, it's a big issue, we- we haven't like mastered it yet, but as the models become as we progress, models are becoming better and much more uh kind of like uh powerful in the things that they can do. Um so it turns out that we can fine-tune the models to be aligned, for example, uh here are some graphs that show what if we take our base- base model that's just pre-trained, and fine-tune it for things like "sensibleness" or "safety." And these are like it- these are data that we have collected, these are high quality data for kind of showing the model what what we mean by what safe and what's unsafe, or what's sensible and what doesn't make sense, and these are like more curate- like more highly curated data that we pass to the model for fine-tuning and to make the model more aligned with the values or the- or the preferences of humans. [Slide change] (Azalia Mirhoseini): So, going back, let's just walk through this process again. So, the first stop when we take a pre-trained language model is uh fine-tuning with next-token prediction, which follows the same process as pre-training, uh that's the easiest step and we just train the model to predict the next token out of all sort of text and data and so on. Uh, and then it this is the fine-tuning step that was a key differentiating factor for a model like ChatGPT. Um, and let's start with um uh the first aspect of fine-nu- fine-tuning, which is kind of alignment between human preferences. So, when a model is pre-trained, it has no sense- it has seen all data on the internet, on the book- in the books, everywhere, but it has no sense of like what is right and wrong and what is- it just have this statistically knows about the state of the world but does it- it doesn't exactly know things or know how to follow instructions. So, the next step in training these models uh is the the the step where we try to steer the AI models to follow the goals and preferences uh and values of humans. Um and this is still a big problem, it's a big issue, we- we haven't like mastered it yet, but as the models become as we progress, models are becoming better and much more uh kind of like uh powerful in the things that they can do. Um so it turns out that we can fine-tune the models to be aligned, for example, uh here are some graphs that show what if we take our base- base model that's just pre-trained, and fine-tune it for things like "sensibleness" or "safety." And these are like it- these are data that we have collected, these are high quality data for kind of showing the model what what we mean by what safe and what's unsafe, or what's sensible and what doesn't make sense, and these are like more curate- like more highly curated data that we pass to the model for fine-tuning and to make the model more aligned with the values or the- or the preferences of humans. [Slide change] (Azalia Mirhoseini): Um, so the next step is instruction tuning. And by instruction tuning, what we mean is that we show the model again, tradi- traditionally in the past 3 years, this is instruction is instruction-tuning data is like a combination of human-generated data or templates, and then and the and the synthetic data, but the way this instruction tu- fine-tuning works is that we do have instruction and then question-answer pairs uh where the model learns how to follow the questions and answer them. For example, here, an instruction fo- tuning data set could be about the weather, like "Please answer the following question, what is the boiling point of Nitrogen?" And then that this is the answer. Um, or we can have chain of thought uh fine-tuning, where we uh show the model how it can walk through a process to get to the answer, and then uh we have the label, we have the result over the here, and this is like these pairs of like questions and instructions and answers, and again, this this data set is very um it's is there is a lot of effort that goes into this data set, and the quality of the data set and the uh generality of it uh has a lot of impact on the quality of the model that we get at this stage. And after this point, then the model becomes more and more like uh the models we see today, like you can ask them questions, you can go back and forth with them, they they have this sense of like how to perceive a question and how to walk through kind of some sort of process to get to an answer. [Slide change] (Azalia Mirhoseini): And the next step after instruction fine-tuning is this process of um using human preferences and fine-tuning the model based on that, and this process is called RLHF, or reinforce- reinforcement learning from human feedback. Um, the way this is different from the previous step is the way we create the objective function and uh create the data. So, in this case, instead of like just creating some supervised prompt and labels, we are basically creating a reward model out of human preferences. So, companies spend a lot of money uh asking humans, sometimes experts, sometimes just humans, normal humans, uh about questions and the answers generated by the model, and we ask the the humans to rate them, to see which one is correct and which one is not correct, and that's how we can create some sort of a reward model here, um and this reward model is instead, instead of human, then we can use the reward model in order to kind of guide the parameters of our LLM towards uh generating answers that are in line with this reward model. So, basically, we want the the generations by the model uh to be such that this reward model says, "Yes, these are good generations." Um we're going to learn more about this here, but I wanted, we wanted you to have um the high-level idea of what RLHF means, and we can collect, we can have different reward type, different types of reward, for example, we could ask the model to reward a pair of answers based on their correctness, or helpfulness, or specificity, or harmlessness, all sort of reward functions that we can define, and depending on how we want the models to be, and what we care most about, we can weight these different reward models and kind of use that in the RLHF process. So, this step, at a- at a high level, of course there's a lot of effort goes into that, but like pre-training, and then um fine-tuning on higher quality data, instruction tuning and RLHF are the core kind of components that made um a model like ChatGPT, over over something that existed, over all the large models like GPT-3 and prior models, so that made the huge difference in how good and capable the models were. [Slide change] (Azalia Mirhoseini): Um, so that was very cool of until last year or a year- year and a half ago. Pre-training and then fine-tuning were the big pieces. Uh, but since a year and a half ago, it turns out inference is also a frontier for making the models much more capable. Um, so and that opened like a whole set of whole lot of new research and directions and ways that we can make the model better. Um, so let's think, let's talk about it a little bit. Like this is the work we did in my lab last year, last summer, it it was called "Large Language Monkeys." Um, and it's kind of inspired by the infinite monkey theorem and it's related to inference-scaling. Um, how many of you have heard of the infinite monkey theorem? [Audience raises hands] (Azalia Mirhoseini): Okay, that's a good amount. So, the idea of the the infinite monkey theorem, uh what it says is that it's not proven, but what it says is that if you have a monkey and a kind of a typing machine, and then we keep letting the monkey type all day, forever, um after some time, we will have the works of William Shakespeare in in the things that the monkey types. Um, so we were inspired by that to name our project. Um but the idea here was that, let's- let's have the LLM be the monkey, and let's uh ask the model over and over again to solve a given input problem. Uh, you can think of it as parallel sampling or parallel generation. So, instead of asking the model once to solve the input problem, you can ask it many times, and also let's assume you have some sort of a verifier or selection mechanism, so you can pick which one of these generated responses were correct and then output that in your system, that becomes the out- the final output. So, the model instead of generating one answer, generates many, many answers, and then your verifier selects which one of them is correct and then outputs one of the correct ones. Um, you can think of a verifier as say some unit tests, if the problem is a generating code, the verifier can could be the unit test that we run against the code and we see which which one of the code generations passes all the unit tests and so on. Um, the reason this is very- this is a possible possibility here is that models are- do not are not uh there is some variance in the way the models generate responses. It's not deterministic, and you can even control the variation in the model responses with this thing called "temperature." So, you can force the model or encourage the model to generate more different responses as you ask a question. [Slide change] (Azalia Mirhoseini): Um, and here, now let's see some of the results of this repeated sampling. For a bunch of tasks like math and coding benchmarks, we increase the number of samples per problem from one, which is the normal one, uh to 10,000, and here we are showing the coverage or the fraction of problems that are solved by at least one of these samples. And what we are seeing here is that um so the red line, the red dash one, was the GPT-4o model, and while these blue and green lines which are the Llama-8B, Llama-3-8B and 70B were worse than GPT-4o model with one sample, but be- if we increase the number of samples from these models, in all of these cases, they do better than the GPT-4o model. And what this says is that it kind of seems like the model- models already know a whole lot more than what you get out of them when you just ask them once, right? So, this is the kind of the underlying property of inference-scaling because it seems like we can get a whole lot of a whole lot more capability and problem-solving kind of ability from these models, if we just bring in this inference-scaling, here we are showing repeated sampling, but there are many other ways to do that. And we are- So, the reason it's called inference-scaling is that we are not touching any of the parameters of the model, the model is fixed, and we are just like at inference time, we are creating different ways to generate, to kind of produce generations from the model and use them to generate higher quality answers or answers to harder problems for the model. Um, the interesting part here is that for some of these problems, out of these 10,000 solutions, maybe like 3 or 4 of them were correct for a single problem, so it shows that how how kind of um how important it is to to scale the inference to really get to the core capability of the models. (Aakanksha Chowdhary): How does it actually compare- like when you do actually generate all the samples like compared to the GPT-4o, right? Like how does the latency here- is it a trade-off? (Azalia Mirhoseini): Yeah, of course, there is a trade-off. Um, and we- we have we touch on that in the- in the lecture that I go more details towards this. The parallel samples, the good thing about them, that they're- they can be running in parallel. Um, so from a latency perspective, that's less of an issue, but of course there is the frontier of cost trade-offs between how much compute you're spending and- and we will look into that, and it varies, it varies depending on the type of and complexity of the problems. (Aakanksha Chowdhary): Yes, great question again. It's a very hard ta- uh when you have verifiers, it's much easier, when you don't have verifiers, there is this whole set of research on how to train LLM as judge or LLM as reward functions, or LLMs with tools becoming, and we are going to learn about all of these. We have a lecture just dedicated to verifiers. So. (Aakanksha Chowdhary): Okay, so we- you now know about large language models, and then you heard about how they learn how to think, and then they learned how to reason. So, what's next and why does this- why is this course relevant? So, a lot of what this course is about is that large language models as chatbots or as uh reasoning models are basically still single-turn, like or or just like in the chat format, so they're not accomplishing a task for you. They're fine to interact with, but they're not necessarily accomplishing a task for you. What is happened this year, uh almost in the last few months, and it's been surprising is that agents like Cloud Code or Deep Research have really enabled people to do real-world workflows. Uh, so they're agentic workflows which can achieve tasks that you ask them to do end-to-end. So, for example, if you want to go research and figure out, where should I uh rent a home, uh perhaps for the entire uh year, for if I want to take a class at Stanford, the model will actually be able to do a whole bunch of analysis and like go look at a lot of different websites and actually summarize the results and give you pros and cons of different places. This used to not be possible before. So, this uh will cover a little bit of how that that becomes possible and how this course allows you to learn about that. And then, Cloud Code, for example, if any of you are codex, uh which is the coding agent from OpenAI, if you're using that, what you'll see is that just by giving instructions in English, you can modify files or you can like figure out test cases and and what not. So it- it really uh has become a coding productivity tool in in the day-to-day workflows for software engineers at this point in time. So, what- what is the transition from LLM to Agents? So, as I was giving you the example of Deep Research and of uh coding agents, basically the goal- basically now, the model can be given a goal and it will plan out the steps that it will go interact with uh the environment, and based on the feedback, it will correct its steps until it achieves the goals or it will come back and say "I cannot achieve the goal." So, this notion of like having some notion of a goal, uh taking actions towards that goal, getting the feedback, and then deciding when to stop, that's what makes agents uh different from the chatbot times of what we were doing before. And, um this also might require interacting with tools uh which are external to what the model itself is doing, getting some inputs from there, but it still stays on track of whatever task it's uh choosing to accomplish, which might mean that it has to have some form of memory to keep track of the task that it's trying to accomplish. So, what has been uh accomplished? So, like in- in- in several cases, in simpler cases, you can achieve these uh end-to-end goals like Deep Research can like be accomplished end-to-end. But in most scenarios, you're still having very static workflows. So, today, what exists is closer to agentic workflows where you have um some sort of an input where you're giving a model the goal and then uh what this slide is showing you is that one model is perhaps uh giving the output, like a solution, and then another model is judging it and then deciding based on that whether this solution should be accepted. So, that's like one orchestration framework that's possible. Uh, Deep research would be more of the second one where you basically are generating mult- uh you're calling the LLM on multiple possible inputs and then aggregating the uh things to get the output, which will be a summary of the Deep Research results. So, this is a very cartoonish abstraction of what agentic workflows might look like, and you can generalize from there. But, instead of having like very open-ended loop that I was showing you here, where you're supposed to go take an action from the environment and come back and do some feedback, um it's easier for open-ended problems to like construct this graph by hand of like how a human would do it and then like get this feedback, perhaps from an LLM, which is the LLM evaluator here. So, that's one orchestration framework that's possible, and then the second one is where you are calling the LLM on multiple possible inputs and aggregating them, but it's still staying on track of whatever task it's choosing to accomplish. So, that's one orchestration framework that's possible. [Slide change] (Aakanksha Chowdhary): So, typically, the workflows will have things like LLM calls, so you would basically call an LLM with an instruction or some form of like input, and then you're asking the LLM to give you an output. It will have some form of verifiers. We'll cover verifiers in a whole lecture by itself. It will have some form of critics or judges, which is effectively LLM as a judge, uh paradigm. Uh, there might be tool calls. For example, for Deep Research, you have to actually go search the web to figure out what exactly is the content that you should be looking at. So, there might be tool calls of that form or uh getting uh some value around what exactly is the weather, um and similarly, search would be another tool call. And then, you would orchestrate these in in some form of a workflow. So, a simplest workflow here would be prompt chaining. So, similar to reasoning models where the task gets decomposed into subtasks, prompt chaining will simply be like you have chained a bunch of different subtasks that you're supposed to go accomplish to achieve the end-to-end goal. There might be routing. Routing works for complex tasks where you say, "Okay, if this is very complex, then go do this more complicated set of LLM calls and it's less complicated then do this simpler workflow." Um, there might be parallelization where you- like Deep Research is one example where you- you can have multiple LLM calls work simultaneously on researching different keywords that you give as input to the LLM and then uh finally you aggregate the output, or you might break the task into independent subtasks and then combine uh the solution, or you might have this notion of orchestrator where you basically are actually using some sort of a- in simple words, LLM manager, so a central LLM is actually doing the planning. So, in Cloud Code, you actually start to see that that there is some notion of like a plan that it comes up with, and then based on that plan it will like make subsequent LLM calls. You might have an evaluator or a judge. So, instead of like getting feedback from real world like user or from some sort of uh actual running a unit test, it actually might just use LLM as a judge, and we will have some homeworks that will cover this aspect, and then you might have verifiers. Verifiers are things where you can actually verify the outputs. So, in code for example, if you run the code, how do you know that this code is correct? So, typically as software developers we write unit tests, uh so similarly, verifiers might be running some kind of unit test to check whether what LLM generated is correct, and in domains which are verifiable, uh math, code, um and other domains that are more rule-based, uh this verification is a good way to give feedback back to the model so that um it can correct its steps. So, most of these uh workflows mean that the uh the LLM needs to be better at planning, it needs to be better at multi-step reasoning, and it needs to be better at self-improvement. Like, when it makes mistakes, it needs to be able to correct itself. So, effectively, these are sort of new paradigms that the current set of LLMs just with reasoning, were not quite accomplishing, and these are some of the topics we do plan to cover in subsequent lectures. And just to drive the point home around coding agents. So, this is a very simple example. This is actually uh I should have updated this slide for Cloud Code, but basically what this is showing is that you have an LLM agent and uh it's interacting with the computer, these days just with the terminal, you give it an instruction where you might tell it that you want to implement a test, um and then it has some navigation of repository, searching of files, all of these tool calls around like viewing files, editing lines, and then it will go execute a bunch of commands in the terminal, and based on the output, it might be like, "Okay, I need to go edit this other file or I need to go look at this other file." So, if you look at this loop, this very much mirrors um what was- so this was not quite reliable last year and it's just starting to get reliable in what we are seeing in the uh realm of coding agents now. (Azalia Mirhoseini): I think it's- why do you think this is more reliable? Like, is it, is it the- the models or is it the framework? (Aakanksha Chowdhary): I mean, I would say it's both. The models are getting better and better at reasoning, so the models like O1 or Claude 3.5 Sonnet have become much better at reasoning through steps, but the framework is also- there is a lot more structure in the agentic loops. So, in the past, it was just like, "Hey model, go fix this code," and the model was just given a free-form loop. Now, it's more of a structured, uh, like the Agent-Computer interface is much more constrained. So, for example, the model can't just type whatever it wants, it has a strict set of tool calls that it can call, like `view_file`, `edit_file`, `list_files`, and that makes it much more reliable. (Azalia Mirhoseini): Yeah, so, the- the Agent-Computer Interface, you know, the constraints there is what is really improving the reliability, and- and it makes sense, yeah. (Aakanksha Chowdhary): Yeah. [Slide change] (Aakanksha Chowdhary): So, this is basically covering a lot of uh the same set of points, but uh one abstraction that is worth taking away from this particular chart is that even when you give the model a goal, it has to clarify the user intent, so it's like not always obvious to it what the user wants. And then it might go search for relevant files, and then whatever uh set of actions it chooses to take, often times it needs some form of verification. In this particular case, it's based on passing the tests and it might actually generate the tests that it needs to pass which uh is also um which is also mentioned here, but the the key idea is that if you give it a task, how does it come back and how do you know that it's going to complete what you asked it for? Oftentimes the users will not specify the problem well enough, so asking, clarifying the user intent, so that it knows what to go look for and how to verify, starts to become uh important and in models like O3, I think they have seen a lot of these traces end-to-end, so they are able to like do the planning, the reasoning, the multi-step reasoning and like come back with the conclusion, but uh a lot of this um is needed for completing tasks end-to-end or having an end-to-end goal. [Slide change] (Aakanksha Chowdhary): And where this has been super useful is uh if you have repetitive uh repetitive tasks like code migrations or version upgrades, or if you need to restructure the codebase. [Slide change] (Aakanksha Chowdhary): Or if you have tasks that involve data engineering where you have to basically do some sort of extract uh the data and then do some sort of clean- clean-up on the data, or if you have to do some sort of data warehouse migrations. So, a lot of this work tends to be extremely um repetitive and it's much easier to just delegate it to coding agents. Uh, same for unit tests, often times like much easier to like start to give it to uh coding agents. [Slide change] (Aakanksha Chowdhary): Another area where agents have become uh extremely prevalent is in customer support. It's one of the most thankless jobs that uh when- when you are on the customer support side of things you have to do, and using LLMs to do that definitely makes uh streamline the experience in interesting ways. So, one simple set of examples ends up being like you can use them to do live transcription, so that gives you a very nice record, and in- and you're seeing that even for meetings these days. [Slide change] (Aakanksha Chowdhary): Another simple is example is that you can have knowledge assist. So, if you have a database of information, the the customer support agent does not need to know everything. They can consult with the LLM and- and get an answer and surface the relevant article, uh which is better than having uh just search uh like an index and search by itself. [Slide change] (Aakanksha Chowdhary): Smart replies. So, in- in chat, often times this has existed for a while, but you can use an agent to like give uh chat responses, and then having a call summary can help you um you can use the call summary to really uh improve the customer experience by itself as well. So, overall, I think in the customer support area, there are multiple uh companies that are going after this area, but uh there are different segments of the problem that can be addressed using LLMs and they have been super useful and then there's also the end-to-end stuff that's starting to happen here. [Slide change] (Aakanksha Chowdhary): And then the third example that we will use in actually some of the homeworks is that if you have very complex topics and uh you want to provide a comprehensive report, um earlier it used to be that you had to do a lit review and then you had to summarize each paper and then you had to synthesize these things and these days, you can just give it to an LLM and actually do it for you in um study maybe more articles than you would. So, typically, if you give it an example like uh say 2022 Winter Olympics opening ceremony, it will identify what references to go look at, then it will construct an outline of like, "Okay, these re- references are relevant or not," and then summarize each of the references uh as to what is the relevant content and then combine them to create a full-length article, um which is uh very impressive in certain cases and- and you'll get to try it in one of the homeworks. [Slide change] (Aakanksha Chowdhary): And even uh more uh forward-looking, um these research agents are starting to be used for uh as AI scientists. So, basically, they're starting to assist scientists in say uh solving math problems or solving science problems. So, here the LLM is used as a brainstorming thing to come up with ideas, so the idea generation phase, then the experiment iteration phase, it might actually help you iterate on the experiments that you are hoping to work on. This is from the AI scientist paper, and then in the paper write-up phase, it will help you improve uh the- the paper write-up by itself. [Slide change] (Aakanksha Chowdhary): And what's interesting uh is that even though these models hallucinate, just the i- just the notion that they can come up with so many different set of ideas as an AI scientist, it might actually brainstorm or like give you ideas outside what you would have thought if you had just like taken a bunch of courses or or if you're a researcher and you've been in the field for a long time. Sometimes the reading the web uh allows these LLMs to come up with ideas that are way outside the box and can be a very good brainstorming um way. So, this- these have been super useful for in- in the AI scientist style of work. [Slide change] (Aakanksha Chowdhary): Okay, so, let's move to course logistics but before that, do we have any questions? Yes. (Student): This is a question for you that is about verifiers. So, is reasoning and chain of thought baked in training or is it just pump engineering that was accidentally discovering these kind of capabilities in larger models? And if so, are there other emergent behaviors besides uh reasoning in chain of thought that you think might be discovered? (Aakanksha Chowdhary): So, uh I don't think it's baked in by uh design. Uh, it was uh it was basically discovered as in like it was- I mean, we gave it hard problems and then we saw that it was basically- by having reasoning chains it was doing better. So, the GSM8K was the first paper that kind of showed signs of life of this, and then uh with larger model uh like PaLM, we actually saw that this was a very big deal, and one of the examples there was that it could explain jokes, wh- which was very impressive, and then from there, the reasoning models have emerged. So, it was not but it has read all of the web, so it's- it has definitely seen data which is more um methodical and systematic. (Azalia Mirhoseini): But the reasoning models are trained to be reasoning more and more, right? So, is that they didn't, the entire reasoning is not an emergent behavior, the reasoning models are trained to be to be thinking, but like the models are converging, so the models are going to be trained such that they know when to they need a lot of reasoning and when they don't and and to generate answers. But, chain of thought originally was an emergent behavior, they noticed that oh, if we explain things, the model gets better. (Aakanksha Chowdhary): Are there any other emergent behaviors in larger models? (Azalia Mirhoseini): I don't think it's, I would say it as "emergent behaviors" per se, but I think as I was mentioning, we're usually going looking for certain things, right? So, in the agentic workflows what we're looking for is planning. Which is the form of reasoning, we're looking for multi-step reasoning, which we'll cover in class as well, and then we're looking for self-improvement or self-correction. So, all of these capabilities would be nice to have, and like what gets the models there is a set of questions that are worth, like there are papers on this kind of thing, yeah. [Pause] (Azalia Mirhoseini): Yes. (Student): So, to follow up, is there a way to make the number of samples dependent on the problem difficulty? (Azalia Mirhoseini): Is it, is there a way to make the number of samples dependent on the problem difficulty? (Aakanksha Chowdhary): I don't think there's a published piece of work that does that, uh but uh yeah, we just- so we are going to have a follow-up work to that where we you use a reward model and then you can use that. So, assuming the reward model has some notion of complexity, if you haven't solved it, it can guide like more sampling or whether it's repeated or or parallel. Uh, but that's definitely an interesting direction to explore. Yeah. (Student): Is it just parallel sampling or are there any other emergent behaviors in larger models? (Azalia Mirhoseini): I don't think it's, I would say it as "emergent behaviors" per se, but I think as I was mentioning, we're usually going looking for certain things, right? So, in the agentic workflows what we're looking for is planning. (Aakanksha Chowdhary): I think the question is that is it because of the reasoning versus is it actually the act of generating reasoning traces? Or just the asking the model to say it worked in these decomposition at all, like for example if you said "think step-by-step" but don't generate out loud, will they actually generate through the process, which is the big part? Or without saying... (Azalia Mirhoseini): Yeah, um. (Aakanksha Chowdhary): I repeat the question, like do you think the gains due to this asking for a reason versus due to actually the act of generating reasoning traces or just the asking the model to say "it worked in these decomposition at all," like for example if you said "think step-by-step" but don't generate out loud, will they actually generate through the process which is the big part? Or without saying... (Azalia Mirhoseini): Yeah. Um, I repeat the question like do you think the gains due to this asking for a reason versus due to actually the act of generating reasoning traces or just the asking the model to say "it worked in these decomposition at all"? Like for example if you said "think step-by-step" but don't generate out loud, will they actually generate through the process, which is the big part? Or without saying... [Inaudible talk between instructors] (Aakanksha Chowdhary): I think it's- I mean, as long as you're doing actual work and it's not just like reusing the work from the... [Laughter] (Aakanksha Chowdhary): ...the exact copy of... (Azalia Mirhoseini): Yeah, so, the... (Aakanksha Chowdhary): ...the... (Azalia Mirhoseini): ...we had uh, we had students publishing papers out of their projects uh in the last quarter, so in the last, last time we taught the class, so that's something you could, it could be one of you or many of you this time again, uh turning your research project, working more on it and then uh turning it into a publication and at um, uh conferences. (Aakanksha Chowdhary): Um, and so, here is a say the date for for our poster presentation, it's going to be December 12, 4 to 6 PM. So, um we we would want you to be there and present the posters, we will have people from industry joining and you can, that's where you can brag about what you've done and just meet new people. [Slide change] (Aakanksha Chowdhary): Here is the course uh grading rubric, so uh we have 3 homeworks that's 50 percent of your uh of your entire grade, and then another 50 percent is on the project. And the duration of these are kind of like aligned with how much uh how much the grading um grading is allocated to each of these, um and the uh of course expect you to honor the honor code. (Azalia Mirhoseini): Communications with instructors and staff. (Aakanksha Chowdhary): Communications with staff. (Azalia Mirhoseini): Office Hours: will be posted on Canvas. (Aakanksha Chowdhary): Besides office hours, ask your questions only on Edstem. (Azalia Mirhoseini): Edstem: Virtual platform to ask questions asynchronously. (Aakanksha Chowdhary): Gradescope: For submitting Project milestones. [Slide change] (Azalia Mirhoseini): Late Policy. (Aakanksha Chowdhary): All assignments must be submitted by the announced due dates, but you have- (Azalia Mirhoseini): Up to 4 total late days. (Aakanksha Chowdhary): You can use only a maximum of 2 late days per assignment. (Azalia Mirhoseini): There will be a 25% penalty for each additional late day. (Aakanksha Chowdhary): Final project should be turned in on time. (Azalia Mirhoseini): There will be no exception to the policy above. [Slide change] (Aakanksha Chowdhary): Audits. (Azalia Mirhoseini): Audits are NOT allowed. (Aakanksha Chowdhary): But we will have the- the course the videos of on YouTube eventually. (Azalia Mirhoseini): Eventually. (Aakanksha Chowdhary): Yeah. [Slide change] (Azalia Mirhoseini): Q&A. (Aakanksha Chowdhary): And that's it. Any questions? (Audience): No. (Aakanksha Chowdhary): All right. (Azalia Mirhoseini): Okay. Thanks everyone. [Outro Music]