nanoGPT: Minimalist Repository for Training and Fine-Tuning GPTs

Repositorykarpathy/nanoGPTResource

nanoGPT by Andrej Karpathy is a lean and fast codebase designed for training and fine-tuning medium-sized GPT models in PyTorch. Developed as a rewrite of minGPT, it can reproduce GPT-2 (124M). As of November 2025, the repository is officially marked as deprecated by the author, who recommends nanochat instead.

Key points

  1. Concise implementation: Features a ~300-line training loop (train.py) and a ~300-line GPT model definition (model.py).
  2. Model reproduction: Capable of reproducing GPT-2 (124M) on OpenWebText in roughly four days using an 8xA100 40GB node.
  3. Adaptable: Supports both training from scratch and fine-tuning checkpoints such as OpenAI's GPT-2 1.3B.
  4. Underlying stack: Built on Python, PyTorch, NumPy, Hugging Face Transformers and Datasets, tiktoken, wandb, and tqdm.
  5. Project status: Deprecated since November 2025; Karpathy advises using nanochat instead, leaving nanoGPT online for posterity.

Why it matters

nanoGPT stripped away the complexity of large training frameworks, providing a transparent, hackable reference implementation for language models in just a few hundred lines of readable code.

Analysis

nanoGPT was built for practitioners and researchers who need a lightweight, high-performance base to experiment with GPT architectures without unnecessary abstraction. While it reached significant maturity and widespread adoption (over 62,000 stars on GitHub), it has now reached obsolescence. Karpathy notes that the project is preserved only for historical reference, steering current development towards nanochat.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • X post:0xSero

    Weekend Project: Training a GPT-2-Level Model from Scratch

    In a post on X, 0xSero outlines a weekend project aimed at training a GPT-2-level language model using Python. According to the author, the setup requires around 4 to 8 hours and roughly $50 in compute costs, leveraging Andrej Karpathy's educational resources.

    1.2kbookmarks69.1kviews

    AI & LLMs· Resource

  • Repository:josch14/double-descent

    Investigating Double Descent in Fully Connected Neural Networks on MNIST

    A research project from the University of Ulm analyses the double descent phenomenon in two-layer fully connected neural networks (FCNNs) using the MNIST dataset. The work reproduces and expands upon Belkin et al. (2019), evaluating the impact of label noise and regularization techniques.

    15starsPython

    AI & LLMs· Research

  • Repository:deepseek-ai/DeepSpec

    DeepSpec: Full-stack speculative decoding framework by DeepSeek

    DeepSpec is an open-source codebase by DeepSeek for training and evaluating draft models used in speculative decoding. The suite covers the entire pipeline across data preparation, multi-GPU model training, and performance benchmarking.

    7.1kstarsPython

    AI & LLMs· Tool

  • Repository:volotat/mini-AGI

    mini-AGI: Byte-Level Continual Learning on 8 GB VRAM Consumer Hardware

    mini-AGI is an experimental open-source byte-level language model designed to demonstrate continual learning from a single stream of data (batch size 1) on modest 8 GB VRAM hardware. By paging model weights from disk into GPU memory as needed, it decouples parameter capacity from VRAM limits while dynamically scaling its architecture.

    793starsPython

    AI & LLMs· Research

  • Repository:MakazhanAlpamys/Soup

    Soup: Single-YAML LLM Fine-Tuning on Consumer GPUs

    Soup (soup-cli) is an open-source tool designed to fine-tune language models via a single YAML configuration file and a unified CLI command. By leveraging layer streaming, it allows users to train 8B models on consumer hardware with as little as 4 GB of VRAM.

    7.5kstarsPython

    AI & LLMs· Tool

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.