Vollständiges Transkript anzeigen (7.188 Wörter)
[Speaker CJ]
What happens when you type a prompt into an LLM like ChatGPT or Claude or Gemini and press enter? The answer, as it turns out, involves over 80 years of research, billions of dollars, and some of the most elegant math humans have ever invented. Now, to answer this question, I built an LLM from scratch, and I'm going to walk you through that full journey from keystroke to stream response with working code at every step. Now, our story actually starts back in 1950. Claude Shannon, the father of information theory, sat down with his wife, Betty, to play a game. And he showed her a passage of text with the next letter hidden, and she had to guess what came next, letter by letter. And he measured how often she was right. But what he was really measuring is how predictable is written language. And his finding was, given enough context, the next letter is often nearly certain. Now, this means that language has deep statistical structure, and over 70 years later, that's pretty much what an LLM is doing. It's predicting what is coming next. And LLMs are statistical models of language. They extract patterns and relationships from massive amounts of text, and they build a mathematical representation of how words relate to each other.
And people often describe them as next-token predictors. And while that's true, there is a massive amount of machinery involved before you can get to the point of what comes next. And that's really what I wanted to understand. Right? We are increasingly interacting with these AI services on a daily basis, and whether it's at home or at work, and a lot of the details are hidden from us, and most people kind of just hand-wave over, "Oh, it predicts the next token." And I wanted to dive deeper. I wanted to have a better understanding of all this stuff. And so that's exactly what I'm going to break down in this video. You're going to walk away with a deeper understanding of how LLMs work. If that sounds good, let's dive in. My name is CJ. Welcome to Syntax.
Now, before LLMs existed, we had chatbots of many forms over the past 60-plus years, and this really all started back in 1950 when Alan Turing published his paper, Computing Machinery and Intelligence, and this is where he proposed the famous Turing Test, which stated if a machine can converse indistinguishably from a human, can it think? And this really set the target for the next 70 years. Now, in 1966, Joseph Weizenbaum at MIT built a program called Eliza, which was essentially just a pattern-matching program that mimicked a Rogerian therapist. It had no understanding whatsoever. Essentially, whatever the user typed, it would go through a series of if statements to determine how it should respond. And a program like this is known as a rule-based system. Essentially, a programmer has to go in and manually write out all the statements to detect, based on what the user has typed, how it should respond. So, it has no built-in understanding. And famously, Weizenbaum's secretary was using the program, and she actually asked him to leave the room so that she could talk to Eliza privately. And he later wrote, "I had not realized that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people." And we're seeing this today, right? Like, people are making comments about how dependent they are on talking to ChatGPT as if it's their friend or their therapist, even though there's not a real human on the other side.
Now, later in 1972, Kenneth Colby at Stanford built a computer program called Parry, which simulated a paranoid schizophrenia patient. It had internal states for anger, fear, mistrust, and it shifted its responses accordingly. And psychiatrists that sat down to talk to this chatbot couldn't distinguish it from real patients. And later in the year, Eliza and Parry were actually connected together, and you essentially had a therapist chatbot talking to a patient chatbot. After that, there were decades of rule-based systems. Alice was released in '95, and it had 41,000 handwritten patterns. And SmarterChild, some of you might remember, was released in 2001 on AOL Instant Messenger, and this was one of the first chatbots millions of people used daily, and could kind of be seen as a precursor to ChatGPT. It was kind of a "do everything" chatbot. Now, every single one of these chatbots, up until this point, were based on rules, or scripts, or templates, or decision trees. Essentially, a human had to sit down at a computer and program every single response in advance. So, the jump from Eliza to ChatGPT isn't just a better chatbot, it's a completely different mechanism. And before we dive into how LLMs work, let's take a look at how a simple program like Eliza could be written.
All right, so this is the simple chatbot I have set up here. And if I say, "Hello," it will respond just like an LLM. "Hello, how can I help you today?" Or if I say, "Hi," it will respond in the same way. So, all of the code for everything I'm going to show you in this video is linked in the description, and we're going to start here with this simple chat. And you can see I have a list of every possible greeting. And essentially, if the user types any one of these greetings, it will respond with that exact response. "Hello, how can I help you today?" Simple as that. If you were to just come across this chat UI and say, "Hi," you might initially think that there was an LLM on the other side, 'cause it responds in a very similar way to how ChatGPT responds. The next thing is we basically look at the user's messages to determine how we can respond in various other ways. And the first check here is, does the user's message start with, "I feel." So I'll say, "I feel happy today." And then it will respond, "Why do you feel happy today?" So we essentially extract the thing after "I feel." We replace any I's with you, that way if the therapist is responding, they say, "you" instead of "I." And then it simply responds with, "Why do you feel" and the feeling that we extracted. So, it extracted "happy today," and so it said, "Why do you feel happy today?" And I can say something like, "My boss gave me good remarks." And then it says, "Tell me more about your boss." And in the code, we have a matcher for "my." So, if the user's message contains "my," then they're talking about some subject. We extract out that subject, and then respond with, "Tell me more about your" subject. In this case, "my" the subject is "boss." So, as you can see, the code is super straightforward, but if you didn't know what was going on behind the scenes, you might think that there was something smart on the other side. And beyond these matchers, we also have catch-all responses. So, these are just random continuations like, "Please go on," "Tell me more about that," "How does that make you feel?" So if I say something that we didn't match against, like, "He is cool and is a break dancer," it will respond with, "Please go on." So, we didn't have any direct matchers for that particular sentence, it picked a random continuation, and now the user can keep chatting with the bot. So, with just a few if statements, we can make the user feel like there's something smart on the other side. And that's the basics of this chatbot. So, we built a super simple chatbot. It's just a bunch of if statements, but the UI that the user is interacting with is almost exactly the same as ChatGPT or Claude. I mean, I mean, it is, right? It has an input box, the user types, it gets back some response. And I like to kind of abstract this and think about it as as a black box, right? It has some input, perform some process, and then gives some output. In this case, the process is just run the input through a bunch of if statements. But what's interesting about this is we can actually replace that box with an LLM, right? Prompt goes in, LLM does something, answer comes out. And that's essentially what we're going to do throughout this video. We're going to slowly replace the pieces to get smarter and smarter machinery. And I actually like to think about the world of computing and technology in this way. Everything initially is a black box, but we can start to uncover how that thing works. And if we generalize it as inputs and outputs, we can even replace the black box with smarter machinery later on.
Now, before you can prompt an LLM, that LLM needs to be created, right? When you type a question into ChatGPT, that gets sent to a pre-trained model. So that training process has to happen ahead of time, and essentially, that training process produces a model file that then lives on a server somewhere, and that's actually what we're interacting with. And I'm already getting ahead of myself, because first we need to talk about what is a model. Now, LLM stands for Large Language Model. It's actually crazy that I haven't defined that this far into the video, but it has the word "model" in there, and a model is a neural network, which is essentially a system of interconnected nodes called neurons that are organized into layers. And data flows through the input layer, passes through one or more hidden layers, and comes out through the output layer. And each connection has a weight. It's essentially a number that controls how much influence one neuron has on the next. The concept is fairly simple, but the history is wild. And it starts back in 1943. Warren McCulloch was a neurophysiologist, and Walter Pitts, a self-taught logician, published a paper proposing that the brain's neurons could be modeled as simple on-off logic gates. Connect enough of them in the right configuration, and they could, in theory, compute anything. Now, this was just a mathematical thought experiment; nobody could actually build it yet. 15 years later, in 1958, a psychologist named Frank Rosenblatt actually built one, and he called it the perceptron. It was a physical machine, a room-size contraption of wires, and motors, and photo cells, that could be shown images on cards and learn to classify them. The Navy held a press conference. The New York Times reported it as the "embryo of a computer that would be conscious of its existence." Then in 1969, two of the most prominent figures in AI, Marvin Minsky and Seymour Papert, published a book called Perceptrons. They proved that a single-layer perceptron couldn't solve certain basic problems, the most famous, exclusive-or, which is: given two binary inputs, output a 1 if exactly one of those inputs is 1, which is trivially simple for a human, and mathematically impossible for a single-layer perceptron. But multilayer networks could solve exclusive-or. The problem was, nobody knew how to train them effectively. And that breakthrough came in 1986. David Rumelhart, Geoffrey Hinton, and Ronald Williams published a paper in Nature describing backpropagation, which is a method for training multilayer networks. Essentially, you make a prediction, measure how wrong it is, and then propagate the error signal backward through every layer, adjusting each weight to make the prediction slightly less wrong. And then you repeat this billions of times. Now, every neural network you interact with today, whether it's Claude or ChatGPT or Gemini, is trained with some variant of this backpropagation algorithm. And so, let me show you the code, and I'm going to use the exact same problem that killed the perceptron: exclusive-or. Okay, so this example here is called XOR Neural Net, and it will train a neural network in real-time to solve exclusive-or. So if I pass in, "single-layer," this trains a single-layer network, and you can see that it doesn't reach a point, even after 5,000 iterations, where it has the correct output that we're expecting. But if I pass in "multi-layer," this does reach a point where, given these inputs, we're getting our expected output here. And you can see that even around 800 iterations, so after 800 iterations, our loss is extremely low, and then our loss just gets lower and lower from there. And then we really reach a point where the loss isn't much more after 5,000 iterations. So after all of those iterations, we have weight values for our neural network that, given two inputs, either false and false, false true, true false, or true and true, we get the expected outputs. And you can see this isn't perfect, but we do round these values, right? This rounds down to 0, this rounds up to 1, up to 1, down to 0. Now, the main thing that I want you to see in the code is the fact that after we train these neural networks, it actually creates a file in the data folder that contains the weights for that neural network. And so when we talk about a model, it's literally just a file with weight values inside of it. So for the single-layer network, we're connecting those two inputs to one single output. So we need a connection from the first input to the output and the second input to the output. And that connection has a weight, and that's why we see two weight values here. And then the final calculation that happens on the output node also includes a bias value. So the single-layer neural network is literally just three numbers. If we look at the multi-layer weights, we can see it gets a little more complex, but essentially, this neural network has four neurons in that hidden layer, and we need to connect each of the inputs to each of those four neurons. So that's why we see two arrays of length 4. The first array are the weight values for that first input, because that first input needs to be connected to each one of those neurons, so we have 4 connections, and then the second input needs to be connected to those same four neurons, so we have four more weight values. And then of course, four bias values for each of the neurons. Now, those four neurons in the hidden layer need to be connected to one single output, and that's why we see one last set of four weights, because the four neurons will have one connection each, that's 4 total, to the output, plus the bias value. So for this very simple neural network, it's just a bunch of weight values in the model file. And of course, this is very trivial, but every single neural network works in this exact same way. Every neural network you come across is going to have a weight file, a model file, that has all of the weights that were calculated after that backpropagation training. Now, if we take a look at the code, we have our inputs, and then our target outputs. And so every neural network is going to have a certain number of inputs, a certain number of outputs. Our neural network has two inputs, either false false, false true, true false, or true true, and then one single output. And the expected output is false and false is false, false and true is true, true and false is true, and true and true is false. But every neural network, you're always going to need: what are your inputs? And then what are your expected outputs given those inputs? Now, we have our train-single-layer function, and this takes in the number of epochs. You saw in the output we have 5,000 total epochs, and that's basically just what we're defaulting to here, but essentially, that's the number of iterations where we will calculate what are our current outputs, figure out how far we are off from our expected outputs, adjust the weights, and repeat. But that's the total number of iterations. And this example, we're always doing 5,000 iterations. You could also set this up to just keep going until your total loss is within a certain threshold, but this always just does 5,000 iterations. You can see that for those two weight values that we're attempting to calculate, they start off as a completely random value. And basically, for each iteration, we calculate the overall value using those weights and the inputs and the bias, and then we have a given output for that calculation. We then take a look at our target, so what is the value we're looking for, we calculate our error, and then we use calculus, we use the the derivative, to figure out the delta, how far we are off from our expected output, and then we adjust all of our weights and bias accordingly. Finally, after 5,000 iterations, we will have some calculated weight and bias values, and we can determine what our current loss is overall. Now, train-multilayer is very similar, except for the fact that we have to come up with random weight values for every connection. The iteration code is very similar, except we just have to work up through two layers. So first, we do from the input to the hidden layer, and then we do from the hidden layer to the output. So the code is very similar, now we just have two different deltas, right? We have the output delta, that is how far we are off from the output to our expected values, and then the hidden delta, how far we are off from that hidden layer. So from there, we update all of the weights accordingly, and then repeat. So really, the code from single layer to multi-layer isn't that much more complex, it's really just calculating between those layers, and we can add any number of hidden layers we want, the code will be very similar. And so that's the basics of a neural network. You set up your inputs, you set up your outputs, set up your hidden layers, and then just start iterating to adjust those weights, until you reach a point where those weights give you the expected output given these inputs.
Okay, so we've got the basics of a neural network, and as we showed, neural networks have inputs and outputs. And so, essentially, the prompt that you type into your LLM is going to be an input into that neural network. But the neural network doesn't just accept a block of text, or it doesn't just accept your question. That block of text needs to be broken down so that it can basically be turned into inputs for that neural network. And the first step in that process is known as tokenization. Now, a model breaks your prompt into pieces, not words, not characters, it's something in between. A token is the smallest unit of text a language model works with. The word "the" is one token. The word "tokenization" might be split into "token" and "ization," two tokens. A space is often part of a token. A new line is a token. An emoji might be multiple tokens. The model doesn't see words the way you do, it sees tokens. And how does it decide where to split? It uses an algorithm called BPE, or Byte Pair Encoding. And BPE was invented in 1994 by Philip Gage as a data compression technique. It has nothing to do with language models. And the idea was simple: Look at a sequence of bytes, find the pair that appears most frequently, replace it with a new symbol, and repeat. Essentially, it compresses data by finding common patterns. Now, 21 years later, in 2015, researchers Rico Sennrich, Barry Haddow, and Alexandra Birch at the University of Edinburgh realized this same algorithm was perfect for building vocabularies for neural machine translation. Instead of deciding in advance what all of the words are or what your vocabulary is, you let BPE learn a vocabulary by iteratively merging the most common character pairs in your training data. So common words like "the" stay whole, and then rare words get split into subword pieces. And the beauty of this is that one algorithm handles English, Japanese, Python code, TypeScript code, emojis, all without language-specific rules. And tokenization isn't just a preprocessing detail, it has real consequences, right? Token count determines cost. Every API call is priced per token. Token count determines what fits into the context, and every model has a maximum context window, which is measured in tokens. And if you exceed it, something's going to get cut. So, let's take a look at the code for the BPE tokenization. All right, this demo is called the Basic Tokenizer, and essentially you can drop in some text, and then it will show you how it essentially broke that training text down into tokens. So with "The cat sat on the mat," you can see at each merge, the token pairs that it came across. And then finally, we get our overall vocabulary. And so this came across six unique tokens total. And the main idea with BPE is this pair merging, the most frequent pairs are merged. And of course, with a very simple training text, right? It's literally one sentence, it's going to find each word as a unit, as a unique pair. But in the code, we actually can specify how many merges to do maximum. But if we change this to something like 3, and train it on the same small bit of text, you'll see that it actually finds AT as a unique token 'cause "at" appears multiple time in the training data. So "cat," "sat," and "mat." Now, because the word "the" appears twice, it actually got its own token, and then everything else were just extra letters added on. So those appear as individual letter tokens. So, if we dive into the code, one of the first things to look at is this regular expression. And every tokenization algorithm first runs all of the training data through this regular expression to split it up because BPE never merges across word boundaries. So even before we start doing these merges, we need to define ahead of time what are the whole groups we're going to use to actually find the individual merged elements within them. And in this case, our groups are actually words. Um but if you look at the algorithm for GPT-2 or GPT-4, they have a predefined really complex regular expression because they've defined some rules up front of how to split things ahead of time. But our initial step here is to just split on whole words, so the things that we're going to be looking at to actually merge are the individual words by splitting on spaces, essentially. Now, the next step is actually an optimization, and that is we determine the frequency of all of those words. So, our regular expression splits on spaces, and then every single unique piece of text in there we count the number of occurrences. And the number of occurrences is basically going to give us a weight as to how much we actually care about that token in the training data. So, in the simple sentence, "The cat sat on the mat," "the" appears twice, so it's going to have a higher weight than some of the other words that we're looking at. And then we get into our actual training algorithm. Now, from there, we're going to iterate up to our max number of merges. And like I showed earlier, we have this set to 10,000. But you essentially get to decide ahead of time how much iteration you want to do, and that's going to determine how large your vocabulary actually gets. And with a really large max merge size, we're more likely to find all of the unique tokens in in a given training set. And then we have the bulk of the work. So this essentially looks at every character pair to find the most common occurring ones, and also takes into account the weights. So how often that particular word occurs. And so we find the most common occurring pair that has the highest weight, and that becomes a new piece that we're going to then merge on in the next iterations. So we take that best pair that we've found, merge all of the groups, in this case merge all of the words together, and then repeat to find the next most commonly occurring pair. So, to see a more interesting example, I'm going to plug the Bee Movie script into this tokenizer, and we can see it work to actually do all of the merges and find all of the unique tokens in the Bee Movie script. So after the training, this found 2,088 unique tokens, which essentially is all of the unique words in the Bee Movie script. And so you can see it found all the whole words. But if we do reduce our max merges to something like a 1000, and then try this again, we are going to see tokens in our vocabulary that are essentially broken up words. Like you can see the word "according" got broken up into four tokens because we only have a certain token budget size, and the word "according" didn't appear that many times in the overall Bee Movie script. So, that's the basics of tokenizing.
Okay, so we've analyzed a large data set, extracted out all the possible tokens using this BPE algorithm, and that gives us a vocabulary, where each token in the vocabulary has a numeric ID. Essentially, it's the index of that token in the vocabulary. So these IDs are just arbitrary numbers, and they don't tell us anything about the actual meaning of those tokens or how they relate to each other. So we need to turn those tokens into something the model can actually reason about. And the idea behind that goes all the way back to 1884 when logician and philosopher Gottlob Frege coined the context principle, which states, "Never to ask for the meaning of a word in isolation, but only in the context of a proposition." 70 years later, in 1957, a British linguist named J. R. Firth put it more memorably. He wrote, "You shall know a word by the company it keeps." And today, we call this idea distributional semantics, and we've built it into the machines. Every modern language model begins by mapping words into a vast numerical space where neighbors share meaning. And those maps are called embeddings. And that one sentence, "You shall know a word by the company it keeps," is the thesis behind every embedding model ever built. Now, in 2013, Tomas Mikolov and his team at Google built off of these ideas in their paper, Word2Vec. They trained a neural network, an embedding model, to predict words from their neighbors in large amounts of text. It was a simple setup, but when they looked at the vectors the model produced, they discovered structure that no one had taught it. For instance, take the vector for the word "king." Subtract the vector for the word "man," add the vector for the word "woman," and the closest result is "queen." Now, nobody told the model about gender or royalty; these concepts just emerged purely from the statistics of which words appear near other words. And this was essentially Firth's idea, "You shall know a word by the company it keeps," implemented as math. Now, what is a vector? Essentially, it's a list of numbers that identifies a point in high-dimensional space. The simplest version is a vector in 3D space, or three numbers, X, Y, and Z. However, embedding vectors are much larger. The Word2Vec paper used 300 dimensions, and GPT-3's largest model uses 12,288 dimensions. Each number captures some feature the model learned during training, not something a human named. But together, they encode meaning as a position in space. Words with similar meanings end up as nearby points. For instance, "happy" and "joyful" are close together, and "happy" and "refrigerator" are far apart. Now, we can measure exactly how close two vectors are using a formula called cosine similarity, which essentially looks at the angle between two vectors. A score of 1 means they're identical, a score of 0 means they're completely unrelated. Now, modern LLM embeddings are direct descendants of Word2Vec, just scaled up from individual words to entire contexts. So, let's take a look at how to train a simple Word2Vec model. All right, this demo is called Train Embeddings, and you can pass in a couple of words, and it will train an embedding model in real-time, and then show you the comparison of the generated vectors between these words that you pass in. And you can see for each of the words that we passed in, we see the actual generated vector. So these are the embeddings that are generated for each of these tokens. But the cool thing to see with this demo are the analogies that we actually get based on this training set. So we have the the classic word math of "king minus man plus woman is queen," but it also works in reverse. So "queen minus woman plus man gives us king." And then we have a few other interesting examples. So we have the same royalty, where "prince minus boy plus girl is princess." This is a fun one: "kitten minus cat plus dog gives us puppy." And so this is really cool to see that even with our small training set, we're able to get some really interesting word maths that actually make sense to us as humans that understand language. So we have the the classic word math of "king minus man plus woman is queen," but it also works in reverse. So "queen minus woman plus man gives us king." And then we have a few other interesting examples. So we have the same royalty, where "prince minus boy plus girl is princess." This is a fun one: "kitten minus cat plus dog gives us puppy." And so this is really cool to see that even with our small training set, we're able to get some really interesting word maths that actually make sense to us as humans that understand language. So, the main thing you need when you're training a model like this is a corpus, which is all of the text that you're training the model on. So here, we have a list of little over a hundred sentences, and they're all just statements. So, "The cat sat on the mat," "The kitten is a baby cat," she loves her pet cat. And then there's an entire section for royalty. And again, the thing to note with this training data is we're showing it those relationships by putting both "king" and "queen" in similar contexts. So, "The king is a man who rules," "The queen is a woman who rules." A prince is a young man of royal blood. A princess is a young woman of royal blood. So, by having those statements and just swapping out the words, the model will learn that "princess" is associated with "woman" and "young," and "prince" is associated with "young" and "man." So, by having those statements and just swapping out the words, the model will learn that "princess" is associated with "woman" and "young," and "prince" is associated with "young" and "man." So, there's plenty more examples that show us "king," "queen," "princess," "prince" in context, so the model can actually learn those relationships. Now, let's look at the actual training, and in the Word2Vec paper, they actually talk about two different architectures. There's the Skip-gram architecture and then there's also CBOW, which is "Continuous Bag of Words." We implemented the Skip-gram architecture. Now the first step in training Skip-gram is to create pairs of words from the training data. So, you can see here we have this variable called "window size," and you can configure this however you'd like, but we set it to something sensible like 5 or 6. And essentially, every word gets paired with every other word that's five or six, whatever your window size is, away from that given word. So, for example, in the training data, take the word "king," we're going to create pairs of "king is," "king a," "king man," "king who," "king rules." And then we do that with every other word too. So, we have "man a," "man is," "man king," "man who," "man rules." So, we create all of these pairs, and that's then going to allow the model to learn those groupings and see that, "Oh, fairly often in the training data, when I see king, it's very often paired with man." Or when I see queen, it's very often paired with woman. Or when I see king or queen, it's very often paired with kingdom. So, this is the very first step. We build up all of those pairs across all of the training data. So, you can see in the run, we had 107 sentences, and this created 3,970 pairs of words that we'll train on. Now, the other thing to see is the actual weights file. So, just like with the "exclusive-or" neural net, we have a file that contains all of the weights. And the cool thing about Word2Vec is it's one of the simplest neural networks where, essentially, our embeddings is just a list of every word in our vocabulary, and then the vector for that given word. So the first step was to tokenize that input training text. We learned about tokenization in the last section. So, you tokenize it, and then we create a vector, which is just an array of numbers, for every single one of those tokens. So, when you look at this "embedding weights" file, it's literally the entire vocabulary, and for every vocabulary word, we have a vector array. So, just like when we were training "exclusive-or," we need some initial random values for those vectors. So, you'll see we're creating that randomized array for our vocabulary size times the dimension. And in this case, we did a dimension of 32, so every single one of these vectors is of length 32. Now, of course you can make that dimension whatever you want, the more dimensions the more information that it's going to store. But for this, we have 32 dimensions, so that means for every word in our vocabulary we're going to start off with an array of length 32, filled with random values. And then we have the actual training loop. So for each epoch, we're going to take all of our pairs of words, and our first step is to push the target and the context closer together. So, all of the pairs of words that appear next to each other are going to be somehow related. So our back propagation in a sense here, our way of nudging these values, is to say if those two words are close together, we're going to push their vectors closer together. And then we also do the opposite. So, for every vocabulary word that never appears next to that given word in our training data set, we push those vectors further and further apart. So, we kind of have this push and pull on each epoch, where all of the words that are related, their vectors start to get closer and closer together, and all of the words that are unrelated, that is they never appear next to each other, get pushed further and further apart. So, after the model has been trained, we have vectors for every word in our vocabulary, and now we can start to do maths with it. So, we have all of the analogies that you saw in the web UI, we have those set up here, and so, in order to do that math, we need to look up the vector for any one of those words. And so, essentially, once we've trained the model, all of the vectors live here, and then when we're doing inference, when we're just comparing the words or doing math with them, we can literally just look up their vector values. So, this code here says, "for each of those three words we're about to do the math on, pull their vector value out of that weights file, and then just do the math." So take that vector minus that vector, add that vector, and then that gives us our resulting vector. Now, the resulting vector isn't going to be perfect, it's just a vector, but we then compare that vector to every other word in our our vocabulary and we choose the one that's the closest. So that's why you see, like with "queen," that was the closest vector in our vocabulary to the result of performing that math. So that's it for training embeddings. Now, this is a very simple embedding trainer, the data set is really small, and it only creates vectors for individual words. Now, modern embedding models have massive data sets, literally every text written by humans in the entire history of humans, and instead of just embedding the vocabulary, they also embed statements, and essentially context from all of that training data. So you basically take this exact same concept and just scale it up to whole sentences or paragraphs, instead of just individual vocabulary words.
Okay, so we've got tokens, we've got embeddings, but what actually processes them? The answer is a Transformer. And every major LLM that you might use today, whether it's GPT, or Claude, or Gemini, are all a form of a Transformer. And this all started with a problem. By 2016, Google's translation models were built on a type of neural network called an LSTM, short for Long Short Term Memory. LSTMs read text one word at a time, carrying a running memory of what they'd seen so far. They worked, but they were slow. Each word had to wait for the previous one to finish processing. To help with longer sentences, researchers had bolted on an add-on called attention. Now, attention came from a 2014 paper by Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Older translation models compressed a whole sentence into one fixed-size vector before translating. Short sentences were fine, but long ones broke. And Bahdanau's idea was, let the model look back at the input at each step and focus on the words that matter right now. And it worked, but it was still bolted on to the LSTM, so it was still slow. Now, in 2017, Jakob Uszkoreit at Google proposed dropping the sequential part entirely using only attention. And eight researchers built it, all equal contributors, and they titled the paper "Attention Is All You Need." Now, that paper has been cited over 100,000 times, and every one of those eight authors has left Google, and several of them founded billion-dollar companies. Now, the architecture they described is the engine running under everything. And the crazy thing is, like, that paper is only 15 pages long, and it's available for free for anyone to read right now. So, the blueprint for the technology powering a trillion-dollar industry is literally just a PDF that you can sit down and read right now. And they stated, "We propose that a 2-month, 10-man study of artificial intelligence will be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire. The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it." And they thought one summer would be enough. Now, 70 years later, we're still at it, and a single architecture from a single paper powers products used by billions of people. And we might be one paper away from the next shift. Because we don't have to be satisfied with the status quo. The Transformer might not be the final architecture. It's just the architecture of the current moment, that was backed by billions of dollars in research funding. And researchers are actively exploring alternatives. The next breakthrough might look nothing like what we covered today. Now, everything I discussed in this video is based on public research. The papers are public. There are even open-weight models you can download and run yourself. And like I mentioned, all the code I wrote is linked in the description for you to check out. Plus there's plenty of other projects on how to build an LLM from scratch you can check out on GitHub. But, personally, as I've done this research and and kind of dug more into it, the more I feel like LLMs really just are the most sophisticated pattern-matching autocomplete things we've ever built. I really don't think that they have true understanding. Um and I feel that the the less magic there is, the less hand-waving there is, in terms of trying to understand the underlying tech, the closer we'll get to really understanding how we can create better methods of working with LLMs or or better create better understanding of the statistics of language. And I've also thought about how the fact that LLMs are just trained on written text, but human intelligence is more than just text. It's text is just one form of external communication. It's not actually how the brain itself runs. So for me, that actually reinforces the idea that LLMs are predicting the next token based on language statistics, not necessarily like replicating what's happening with human intelligence and what's actually happening in the brain. So, in my opinion, AI is probably better described as something like "alien intelligence" rather than "artificial intelligence," because it's intelligent, but it's not necessarily a fake version of human intelligence, it's something entirely different. So, that's all I've got. Thank you so much for making it to the end of the video. If you have any questions, leave them down below. If I made any mistakes, let me know as well, I'll use the corrections feature of YouTube and I'll add it, so future viewers of the video will see it. And also if there's any topic you want to see me dive deeper into or explore in a future video, let me know as well. All right, see you in the next one.
