Oriented stochastic loss descent algorithm to train very deep multi-layer neural networks without vanishing gradients
作者：
Highlights：
• A new back-propagated gradient descent optimization algorithm without a vanishing problem is proposed.
• The proposed algorithm is called oriented stochastic loss descent (OSLD).
• OSLD updates in the opposite side of its gradient sign by a tuned ratio of random loss.
• OSLD is competitive to Adam and is compatible with most backpropagation architectures.
• OSLD is stable and opens more choices in front of very deep multi-layer neural networks.
摘要
•A new back-propagated gradient descent optimization algorithm without a vanishing problem is proposed.•The proposed algorithm is called oriented stochastic loss descent (OSLD).•OSLD updates in the opposite side of its gradient sign by a tuned ratio of random loss.•OSLD is competitive to Adam and is compatible with most backpropagation architectures.•OSLD is stable and opens more choices in front of very deep multi-layer neural networks.
论文关键词：Vanishing gradient problem,Deep learning,Backpropagation,Gradient descent algorithm,Optimization algorithm,Deep multi-layer neural network
论文评审过程：Received 27 November 2020, Revised 3 June 2021, Accepted 9 August 2021, Available online 11 August 2021, Version of Record 23 August 2021.
论文官网地址：https://doi.org/10.1016/j.knosys.2021.107391
