A weight initialization based on the linear product structure for neural networks

作者：

Highlights：

• We consider the neural network training from a nonlinear computation point of view.

• A new linear product structure initialization strategy has been developed for training neural networks.

• Theoretical analysis shows that the LPS initialization yields a low probability of dying ReLU.

摘要

•We consider the neural network training from a nonlinear computation point of view.•A new linear product structure initialization strategy has been developed for training neural networks.•Theoretical analysis shows that the LPS initialization yields a low probability of dying ReLU.

论文关键词：Weight initialization,Linear product structure,Neural networks,Nonlinear computation

论文评审过程：Received 16 March 2021, Revised 25 September 2021, Accepted 29 September 2021, Available online 19 October 2021, Version of Record 19 October 2021.

论文官网地址：https://doi.org/10.1016/j.amc.2021.126722