Projection based weight normalization: Efficient method for optimization on oblique manifold in DNNs

作者：

Highlights：

• We propose to constrain the incoming weights of each neuron to be unit norm to address ill conditioned problem in DNNs.

• Constrain ing weight s can be formulated as an optimization problem over the Oblique manifold.

• We propose a simple yet efficient method referred to as projection based weight normalization (PBWN) to solve the optimization problem.

• PBWN has the property of regularization and collaborates well with the commonly used batch normalization technique.

• Extensive experiments on several widely used image datasets show the consistent performance improvement over the baseline DNNs.

摘要

•We propose to constrain the incoming weights of each neuron to be unit norm to address ill conditioned problem in DNNs.•Constrain ing weight s can be formulated as an optimization problem over the Oblique manifold.•We propose a simple yet efficient method referred to as projection based weight normalization (PBWN) to solve the optimization problem.•PBWN has the property of regularization and collaborates well with the commonly used batch normalization technique.•Extensive experiments on several widely used image datasets show the consistent performance improvement over the baseline DNNs.

论文关键词：Deep learning,Weight normalization,Oblique manifold,Image classification

论文评审过程：Received 31 July 2019, Revised 23 February 2020, Accepted 24 February 2020, Available online 27 February 2020, Version of Record 5 June 2020.

论文官网地址：https://doi.org/10.1016/j.patcog.2020.107317