Stanford CS336: Language Modeling from Scratch

LinkTatsunori Hashimoto, Percy LiangResource

Stanford University's CS336 course teaches the end-to-end development of language models from the ground up, inspired by traditional operating systems courses. Students build every component: data pipelines, Transformer architectures, Triton GPU kernels, scaling laws, and reasoning-focused alignment.

Key points

  1. Implementation-heavy approach: Minimal scaffolding is provided, requiring students to write significantly more code than in typical AI courses.
  2. Five core assignments: Basics & tokenisation, systems optimisation & Triton FlashAttention2, scaling laws, pretraining data processing from Common Crawl, and alignment/RL for reasoning.
  3. Strict AI tooling policy: General LLM querying for conceptual or low-level questions is permitted, but AI autocompletion (such as Cursor Tab or GitHub Copilot) is strongly discouraged during assignments.
  4. Public resources: Lecture recordings on YouTube, slides, and code files are openly accessible for self-study, accompanied by GPU pricing comparisons for modern accelerators like the B200.

Why it matters

Rather than relying on abstract high-level frameworks, CS336 requires students to engineer the entire stack. This hands-on depth provides an indispensable foundation in performance engineering, distributed training, and dataset curation required for state-of-the-art LLM development.

Analysis

Curated by Percy Liang and Tatsunori Hashimoto, the course syllabus underscores the modern convergence of ML research and systems engineering. Key focus areas like custom Triton kernels, memory management, deduplication of web crawl data, and reinforcement learning for reasoning reflect current production realities in foundation model training.

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Similar in content, found by the AI search.

  • Link:Diyi Yang, Michael Ryan, John Yang

    CS 329Z: Stanford Course on Engineering AI Agents

    Stanford University is running CS 329Z (Engineering AI Agents) for Fall 2026. The course explores the transition from monolithic language models to compound AI systems and autonomous agents. All lecture materials, syllabus details, and reading lists are made publicly available online.

    AI & LLMs· Resource

  • Link:llm-chronicles.com

    LLM Chronicles: Structured Video and Lab Course on Deep Learning and Language Models

    LLM Chronicles is an educational platform designed to teach deep learning and large language models through whiteboard animations and hands-on PyTorch coding labs. The curriculum spans artificial neural network fundamentals, transformers, and recent reasoning-focused architectures.

    AI & LLMs· Resource

  • Link:Percy Liang

    Percy Liang: Research Profile and Initiatives for Open AI Development

    The academic homepage of Stanford computer science professor Percy Liang outlines his core research directions, teaching portfolio, and academic initiatives. Liang focuses on foundational questions in machine learning, radically open model creation through the Marin project, and reproducible research via CodaLab Worksheets.

    AI & LLMs· Resource

  • Video

    Video:Stanford Online

    Stanford CS329A: Introduction to Self-Improving AI Agents and Scaling Trends

    In the inaugural lecture of Stanford's CS329A course, Aakanksha Chowdhery and Azalia Mirhoseini outline the evolution of large language models towards autonomous, self-improving agents. The session explores foundational scaling laws, the emergence of chain-of-thought reasoning, post-training alignment via RLHF, inference-time scaling techniques, and the architectural principles behind agentic workflows in software engineering and research.

    AI & LLMs· Talk

  • Link:Aakanksha Chowdhery, Azalia Mirhoseini

    Stanford CS329A: Self-Improving AI Agents

    Stanford University's CS329A is a graduate course dedicated to self-improving AI agents, taught by Aakanksha Chowdhery and Azalia Mirhoseini. The curriculum covers foundational and modern techniques enabling LLMs to iteratively improve via environment interaction and internal feedback.

    AI & LLMs· Resource

  • X post:CopyRebeldia

    Stanford Lecture on the Mathematical Foundations of LLMs

    A viral post highlights a Stanford lecture on the technical fundamentals behind large language models. The author argues that models like Claude and ChatGPT process numbers rather than text, linking this core competence to lucrative engineering salaries. However, a Community Note corrected the identity of the lecturer and the course shown.

    41.9kbookmarks2.8mviews

    AI & LLMs· Resource

Let's talk about your project

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.