Understanding the Difficulty of Training Deep Feedforward Neural Networks

LinkXavier Glorot, Yoshua BengioResearch

This research paper by Xavier Glorot and Yoshua Bengio investigates why standard gradient descent with random initialization performs poorly in deep neural networks. The authors examine the role of activation functions and gradient behaviour across layers, proposing a new initialization scheme that significantly accelerates convergence.

Key points

  1. Standard logistic sigmoid activations are unsuited for deep networks with random initialization because their mean value can saturate upper hidden layers.
  2. Saturated units can escape saturation on their own, but do so slowly, explaining the plateaus observed during training.
  3. Activation functions that saturate less frequently provide clear benefits for deep architectures.
  4. Training becomes more difficult when the singular values of the layer-associated Jacobian matrices diverge significantly from 1.
  5. A newly proposed initialization scheme substantially speeds up training convergence based on these insights.

Why it matters

The paper provides an explanation for why early deep neural network architectures failed to train effectively with basic gradient descent. The resulting technique, widely known as Glorot or Xavier initialization, became a core foundational component in deep learning.

Analysis

By tracking activations and gradient variances across layers, the authors show how saturation and poorly scaled variance halt backpropagation. The work established clear mathematical guidelines for weight initialization, replacing ad-hoc methods with principled layer-variance balancing.

Mentioned links

Found in

Summary generated by AI (Gemini 3.8 Flash, 27 September 2026). It may contain errors – the original source is authoritative.

Related entries

Similar in content, found by the AI search.

  • Article:Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio

    Neural Machine Translation by Jointly Learning to Align and Translate

    In this paper, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio propose an approach that resolves the fixed-length vector bottleneck in encoder-decoder translation models through soft search.

    AI & LLMs· Research

  • Article:Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

    Deep Residual Learning for Image Recognition: The ResNet Framework

    Kaiming He and colleagues introduce a residual learning framework designed to simplify the training of substantially deeper neural networks by reformulating layers to learn residual functions referencing layer inputs.

    AI & LLMs· Research

  • Article:Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

    Identity Mappings in Deep Residual Networks

    The authors analyse the propagation mechanics behind residual networks, demonstrating that forward and backward signals pass directly between blocks when identity mappings serve as skip connections and after-addition activations. This leads to an improved residual unit design that eases training and boosts generalization across deep architectures.

    AI & LLMs· Research

  • Article:Warren Sturgis McCulloch and Walter Pitts

    A Logical Calculus of the Ideas Immanent in Nervous Activity (1943)

    Warren Sturgis McCulloch and Walter Pitts' seminal 1943 paper introduced the mathematical foundations of artificial neural networks. It framed the nervous system as a network of basic logical elements that process weighted inputs against a threshold. The authors proved that these networks could perform all logical functions and equated them with Turing computability.

    AI & LLMs· Research

  • Repository:josch14/double-descent

    Investigating Double Descent in Fully Connected Neural Networks on MNIST

    A research project from the University of Ulm analyses the double descent phenomenon in two-layer fully connected neural networks (FCNNs) using the MNIST dataset. The work reproduces and expands upon Belkin et al. (2019), evaluating the impact of label noise and regularization techniques.

    15starsPython

    AI & LLMs· Research

Back to the knowledge base

Let's talk about your project

Get in touch

Locations

  • Mattersburg
    Johann Nepomuk Bergerstraße 7/2/14
    7210 Mattersburg, Austria
  • Vienna
    Ungargasse 64-66/3/404
    1030 Wien, Austria

Parts of this content were created with the assistance of AI.