Article:NVIDIA et al.
Pretraining LLMs in 4-Bit: NVIDIA Introduces NVFP4 Training Methodology
A research team from NVIDIA has presented a method for stable and accurate pretraining of Large Language Models using the NVFP4 4-bit floating-point format. To evaluate the approach, they trained a 12-billion-parameter model on 10 trillion tokens—reported as the longest publicly documented 4-bit training run to date. The model matched the training loss and downstream task performance of an FP8 baseline.