Show the full transcript (2,026 words)
You type a question into an AI chatbot, a second later it answers. It feels like magic. But behind that answer is a long journey built by many people over a long time. Today I will walk you through the whole journey from start to finish. I’m Colin. I’ve written software for about 15 years, and I build AI systems for companies. To keep it real, we’ll follow one actual AI model all the way, Kimi K3, from a company called Moonshot AI. They published a long report about how they built it, so we’ll use it like a map. And my friend, Eliduck, is coming along.
Here’s the idea for today: Think of building an AI like raising a student. First, you decide what job the student is for. Then you find books, build a school, design the brain, teach, test, and finally send the student to work. One warning before we start, an AI is not a real student. The picture just helps us see the steps. Let’s walk it step-by-step.
It all starts with a decision. Before anyone writes code, someone has to answer a simple question: What is this AI for and who will use it? Think of it like choosing a school for a student. A future chef and a future pilot need very different lessons. An AI that helps with homework is different from an AI that runs a factory machine. The answer changes everything after it: which books you need, how big the school is, and which exams you give at the end. Kimi K3’s team says they built an agentic model. That’s a fancy word for an AI that doesn’t just talk. It can take steps on its own like searching, writing code, and checking its own work. Here’s something from my own work: most of the time my team doesn’t build a brand new AI at all. We try a ready-made one first. Most of our time goes into the parts around the model: getting the data in, checking it, and making the answers trustworthy.
Once you know the job, the student needs something to learn from. An AI learns from examples, huge piles of text, pictures, and code. Engineers call this data. Think of it as the library our student will read. Two problems show up right away. Problem one: are you allowed to use it? You can’t just take every book you find. Some writing is protected by copyright. Some data has private details about real people. So teams have to check what they may use and keep records. Problem two: real data is messy. Let me tell you about one of my projects: a global parcel company sent my team big files of delivery records. The same address was written in lots of different ways and the fields didn’t match. If you just did the math on it, nothing came out right. Before any AI could help, we had to clean the mess. Honestly, that’s most of the real work. Big AI labs face the same problem at a giant scale. They remove copies, filter junk, and choose good examples. The K3 report describes some of these steps, but it doesn’t share the full recipe or the data itself. Many companies keep their data private. Some research groups do publish theirs, like the OLMo project.
Next, the student needs a place to learn. Our student needs a building and electricity. For an AI, the school is the computers, mostly special chips called GPUs. Imagine 100 chefs, each chopping one carrot at the same time. That’s what a GPU is good at, blocks of small math steps all at once. Training a big model takes an enormous number of those steps, so you need many chips working together for a long time. That’s expensive. Here’s a lesson from planning a training project for a client: the memory on each chip matters a lot. You can split a model across many small chips, but it’s slower and harder than having enough memory in the first place. So teams often rent big machines for training and use smaller ones afterwards. How big was Kimi K3’s school? The report gives some hardware details. For example, it says each of its reinforcement learning experiments was kept within a few hundred GPUs, but it doesn’t give the full size or cost of the main training run. When a number isn’t in the report, I’ll tell you instead of guessing.
Now for the most famous part, the brain itself. Engineers call its design the architecture. It’s the shape of the model before it learns anything. Kimi K3 is huge. It has about 2.8 trillion numbers inside it, called parameters. Think of parameters as tiny knobs. Learning means turning the knobs until the answers get better. Here’s the clever part: for each small piece of text, K3 uses only about 104 billion of those knobs. That’s less than 4%. Imagine a hospital floor with 896 specialist doctors. When a patient arrives, the front desk sends them to just 16 doctors who fit their problem plus two family doctors who see everyone. This design is called a Mixture of Experts (MoE). K3 has a front desk like this on 92 of its 93 layers. So that’s 896 experts on each of those layers, not 896 in total. One limit of the picture, these experts aren’t neat specialists like a math doctor. The model learns its own way to split up the work. K3 also mixes two kinds of memory. Most of its layers keep a small whiteboard that gets updated as it reads. About a quarter of its layers keep a full notebook that they can look back through. The whiteboard isn’t a perfect copy of everything it read, the model learns what to keep. The report says this mix is part of how K3 handles very long inputs efficiently. We’ll open up that design in a later episode.
With the brain designed, the reading begins. This is called pre-training. The model reads a giant amount of text one small piece at a time. These pieces are called tokens. A token is often part of a word, not a whole word. The game is simple, guess the next token. Over and over, a huge number of times. Each wrong guess nudges the knobs a tiny bit. Slowly, the model picks up the patterns of language, facts, and code. K3 also learned to read longer and longer. It started with about 8,000 tokens at a time, then 64,000, then 256,000, and finally about 1 million. It’s like a kid going from picture books to chapter books to whole shelves of books. After this station, we have a base model. It already knows a lot, but it’s like a student who read everything and never had a teacher. Ask it a question and it may just continue your sentence instead of answering. It needs lessons. One honest note, the report doesn’t say exactly how many tokens K3 read in total, so I won’t give you a number. Reading everything isn’t the same as being useful. So next come lessons and practice. This is called post-training. First, come worked examples. People show the model good answers, and it learns to follow that style. That’s called supervised fine-tuning (SFT). It’s like a teacher showing worked examples in class. Then comes practice with rewards. The model tries a task, if it does well, it gets a reward. This is reinforcement learning (RL). Think of training a puppy with treats. And just like a puppy, the model learns whatever gets the treat, even when that isn’t what you wanted. That’s called reward hacking. I’ve seen a cousin of this problem in my own work. We asked a big AI to judge which companies were good customers. It could argue yes for almost anyone and sound convincing. So we stopped asking for its opinion. We made it show receipts, real shipping records. An AI will happily give you what you asked for, even when that’s not what you needed. K3’s team did something interesting here. Instead of one student, they first trained nine expert students, three subjects: general tasks, assistant agents, and coding agents. Each one at three effort levels: low, high, and max. Then they trained one final student to learn from all nine. That’s called distillation. One limit, the final student doesn’t magically get everything the nine experts know. It learns to act like them on the kinds of tasks it practiced.
Then come the exams. Before anyone trusts the model, it takes tests. Engineers call them benchmarks. Hard science questions, coding tasks, long tasks where the AI has to use tools. The K3 report lists lots of scores, and it explains some of how they tested. For example, it gives the settings they used, and it says some scores are averaged over three runs. That matters because an AI’s answers can change a little each time you ask. But remember what a test score is. It’s like a school exam. It tells you how well you did on that exam under those rules. It doesn’t tell you everything you can do. In the real world, two more questions matter: how much each answer costs and how fast it comes back. In my own projects, we test systems like a fair exam. Before we hand a system over, it works out the results without seeing the answer key, then we compare. And if it’s wrong, we find out why.
Before graduation, there’s one more check, safety. Before release, teams test what could go wrong. Could the model be misused? Does it make things up? For example, the K3 report includes a cybersecurity evaluation. It measures how capable the model is at hacking-style tasks because a strong model could be misused. Then comes a big choice, how to release it. Moonshot released K3 as open weights. Think of it like publishing the student’s whole brain, every knob setting, for anyone to download and run following the rules in the K3 license. Here’s the catch with open weights. Once copies are downloaded, the company can’t pull them back. They can release a better, safer version later, but the old copies are already out in the world. That’s why the safety check before release matters so much. And then, graduation day. The first job. Now millions of people ask questions at the same time. Think of a help desk that never closes. This is called serving or inference. Every answer uses computer time and that costs money. So teams work hard to make answers fast and cheap. One trick is like a restaurant that chops the onions once, because every order needs them. If many questions start with the same long text, the computer can save its work on that part and reuse it instead of reading it again every time. That’s called a prefix cache. Speed matters, too. In one of my projects, our AI took 42 seconds to answer. We turned off its think-out-loud mode for that job and changed how we called it. After that, it answered in under one second. Thinking longer can help on hard problems, but it’s slower and costs more. Good teams pick the right amount of thinking for each job. K3 always thinks before it answers, but it lets you choose how hard, low, high, or max.
So that’s the whole journey. Decide the job, collect and clean the data, build the computers, design the brain, read the library, lessons and practice, exams, the safety check, and release, and finally, the first job. Here’s what I want you to remember. An AI model isn’t one magic invention. It’s a long chain of careful steps, and a weak link anywhere, like messy data or a badly chosen reward, shows up at the end. Next time, we’ll zoom in to station four with three simple questions that explain the design of most AI models. If you want to go deeper, the full course rebuilds these stations in code step-by-step. The link is in the description. I’m Colin and this is Eliduck. See you next time.
Video
Video
