Differentiable neural architecture learning for efficient neural networks

作者：

Highlights：

• We build a new standalone control module based on the scaled sigmoid function to enrich the neural network module family to enable the neural architecture optimization.

• Our DNAL method produces no candidate neural architectures but one, thus drastically improving the learning efficiency, i.e., costing 20 epochs for CIFAR-10 and 10 epochs for ImageNet.

• It is applicable to conventional CNNs, lightweight CNNs, and stochastic supernets.

• Extensive experiments confirm that our DNAL method achieves excellent performance on various CNN architectures, including VGG16, ResNet50, MobileNetV2, and ProxylessNAS, over the task of CIFAR-10 and ImageNet-1K classification.

摘要

•We build a new standalone control module based on the scaled sigmoid function to enrich the neural network module family to enable the neural architecture optimization.•Our DNAL method produces no candidate neural architectures but one, thus drastically improving the learning efficiency, i.e., costing 20 epochs for CIFAR-10 and 10 epochs for ImageNet.•It is applicable to conventional CNNs, lightweight CNNs, and stochastic supernets.•Extensive experiments confirm that our DNAL method achieves excellent performance on various CNN architectures, including VGG16, ResNet50, MobileNetV2, and ProxylessNAS, over the task of CIFAR-10 and ImageNet-1K classification.

论文关键词：Deep neural network,Convolutional neural network,Neural architecture search,Automated machine learning

论文评审过程：Received 22 July 2021, Revised 26 September 2021, Accepted 22 November 2021, Available online 22 January 2022, Version of Record 8 February 2022.

论文官网地址：https://doi.org/10.1016/j.patcog.2021.108448