梯度消失问题1991

这样做的效果，相当于在n层网络中，将n个这些小数字相乘来计算“前端”层的梯度，这就使梯度（误差信号）随着n呈指数递减，导致前端层的训练非常缓慢。

反向传播使研究人员从头开始训练监督式深度人工神经网络，最初收效甚微。1991年赛普·霍克赖特的毕业论文正式确认了“梯度消失问题”失败的原因。梯度消失问题不仅影响多层前馈网络，还影响循环神经网络。循环网络是通过将前馈网络深度展开来训练，在网络处理的输入序列的每个时间步骤中，都会产生一个新的层。

当所使用的激励函数之导数可以取较大值时，则可能会遇到相关的梯度爆炸问题（Exploding gradient problem）。

In machine learning, the vanishing gradient problem is the problem of greatly diverging gradient magnitudes between earlier and later layers encountered when training neural networks with backpropagation. In such methods, neural network weights are updated proportional to their partial derivative of the loss function. As the number of forward propagation steps in a network increases, for instance due to greater network depth, the gradients of earlier weights are calculated with increasingly many multiplications. These multiplications shrink the gradient magnitude. Consequently, the gradients of earlier weights will be exponentially smaller than the gradients of later weights. This difference in gradient magnitude might introduce instability in the training process, slow it, or halt it entirely. For instance, consider the hyperbolic tangent activation function. The gradients of this function are in range [0,1]. The product of repeated multiplication with such gradients decreases exponentially. The inverse problem, when weight gradients at earlier layers get exponentially larger, is called the exploding gradient problem.

Backpropagation allowed researchers to train supervised deep artificial neural networks from scratch, initially with little success. Hochreiter's diplom thesis of 1991 formally identified the reason for this failure in the "vanishing gradient problem", which not only affects many-layered feedforward networks, but also recurrent networks. The latter are trained by unfolding them into very deep feedforward networks, where a new layer is created for each time-step of an input sequence processed by the network (the combination of unfolding and backpropagation is termed backpropagation through time).

Prototypical models

This section is based on the paper On the difficulty of training Recurrent Neural Networks by Pascanu, Mikolov, and Bengio.

Recurrent network model

A generic recurrent network has hidden states h₁,h₂,…, inputs u₁,u₂,…, and outputs x₁,x₂,…. Let it be parameterized by θ, so that the system evolves as (hₜ,xₜ)=F(hₜ₋₁,uₜ,θ)

Often, the output xₜ is a function of hₜ, as some xₜ=G(hₜ). The vanishing gradient problem already presents itself clearly when xₜ=hₜ, so we simplify our notation to the special case with: xₜ=F(xₜ₋₁,uₜ,θ)

Now, take its differential: （公式略）

Training the network requires us to define a loss function to be minimized. Let it be L(x_(T),u₁,…,u_(T)), then minimizing it by gradient descent gives （公式略） where η is the learning rate.

The vanishing/exploding gradient problem appears because there are repeated multiplications, of the form ∇ ₓF(xₜ₋₁,uₜ,θ)∇ ₓF(xₜ₋₂,uₜ₋₁,θ)∇ ₓF(xₜ₋₃,uₜ₋₂,θ)⋯

Example: recurrent network with sigmoid activation

For a concrete example, consider a typical recurrent network defined by xₜ=F(xₜ₋₁,uₜ,θ)=W_(rec)σ (xₜ₋₁)+Wᵢₙuₜ+b where θ =(W_(rec),Wᵢₙ) is the network parameter, σ is the sigmoid activation function, applied to each vector coordinate separately, and b is the bias vector.

Then, ∇ ₓF(xₜ₋₁,uₜ,θ)=W_(rec)diag (σ '(xₜ₋₁)), and so （公式略）

Since |σ '|≤ 1, the operator norm of the above multiplication is bounded above by ‖W_(rec)‖^(k).

Solutions

To overcome this problem, several methods were proposed.

RNN

For recurrent neural networks, the long short-term memory (LSTM) network was designed to solve the problem (Hochreiter & Schmidhuber, 1997).

For the exploding gradient problem, (Pascanu et al, 2012) recommended gradient clipping, meaning dividing the gradient vector g ‖g‖/gₘₐₓ ‖g‖>gₘₐₓ. This restricts the gradient vectors within a ball of radius gₘₐₓ.

Batch normalization

Batch normalization is a standard method for solving both the exploding and the vanishing gradient problems.

Multi-level hierarchy

In multi-level hierarchy of networks (Schmidhuber, 1992), pre-trained one level at a time through unsupervised learning, fine-tuned through backpropagation. Here each level learns a compressed representation of the observations that is fed to the next level.

Deep belief network

Similar ideas have been used in feed-forward neural networks for unsupervised pre-training to structure a neural network, making it first learn generally useful feature detectors. Then the network is trained further by supervised backpropagation to classify labeled data. The deep belief network model by Hinton et al. (2006) involves learning the distribution of a high-level representation using successive layers of binary or real-valued latent variables. It uses a restricted Boltzmann machine to model each new layer of higher level features.