Pretraining LLMs in 4-Bit: NVIDIA Introduces NVFP4 Training Methodology

ArticleNVIDIA et al.Research

A research team from NVIDIA has presented a method for stable and accurate pretraining of Large Language Models using the NVFP4 4-bit floating-point format. To evaluate the approach, they trained a 12-billion-parameter model on 10 trillion tokens—reported as the longest publicly documented 4-bit training run to date. The model matched the training loss and downstream task performance of an FP8 baseline.

Key points

  1. The method applies Random Hadamard transforms (RHT) to bound block-level outliers.
  2. A two-dimensional quantization scheme maintains consistent representations across forward and backward passes.
  3. Stochastic rounding is employed for unbiased gradient estimation.
  4. Selective high-precision layers are incorporated to maintain stability and convergence.
  5. A 12B-parameter model trained on 10 trillion tokens achieved downstream accuracy comparable to an FP8 baseline.

Why it matters

Training frontier AI models demands tens to hundreds of yottaflops, representing massive compute, time, and energy investments. While FP8 is widely adopted, scaling down to 4-bit precision offers further improvements in execution speed and hardware resource efficiency.

Analysis

Quantisation down to 4 bits has largely been restricted to inference and fine-tuning due to instability and convergence failures during long pretraining runs. NVIDIA's architectural choices—specifically combining RHT, 2D quantisation, and stochastic rounding—show that narrow-precision training can scale over massive token horizons without sacrificing accuracy compared to FP8.

Mentioned links

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Article:unsloth.ai

    Unsloth Benchmarks: Faster Speeds and Lower VRAM Usage for QLoRA

    Unsloth's documentation shares benchmark metrics for LLM fine-tuning across NVIDIA GPUs. Compared to Hugging Face paired with FlashAttention-2, Unsloth achieves a 2x speedup, over 70% VRAM reduction, and significantly longer context windows on models such as Llama 3.1 and Llama 3.3.

    AI & LLMs· Resource

  • Repository:NVIDIA/Model-Optimizer

    NVIDIA Model Optimizer: Unified Library for Model Compression and Inference Acceleration

    NVIDIA Model Optimizer (ModelOpt) provides state-of-the-art model optimization techniques such as quantization, pruning, distillation, and speculative decoding. It accepts PyTorch, Hugging Face, and ONNX models and outputs optimized checkpoints ready for downstream engines like TensorRT-LLM, vLLM, and SGLang.

    5kstarsPython

    AI & LLMs· Tool

  • Article:Chris Alexiuk and Chintan Patel

    NVIDIA Nemotron 3.5 Lightning: Compact MoE Model for Agentic Execution

    NVIDIA has announced Nemotron 3.5 Lightning, an open 30B mixture-of-experts (MoE) model featuring 3B active parameters. It is tailored for the high-volume execution layer of long-running AI agents, targeting routine tasks such as tool calling and result validation with low latency.

    AI & LLMs· Announcement

  • X post:Mia

    Running DeepSeek v4 Flash on a Single NVIDIA DGX Spark

    MiaAI-Lab published an open-source launcher allowing DeepSeek V4 Flash 0731 (EXL3) to run on a single NVIDIA DGX Spark node instead of requiring a multi-node setup.

    71bookmarks9.8kviews

    AI & LLMs· Tool

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.