Multi-level knowledge distillation for low-resolution object detection and facial expression recognition

摘要

Recently, remarkable object detection and facial expression recognition (FER) approaches have been made by researchers. However, all of these models are trained and tested on high-resolution images without considering that low-resolution images are more common in practical application. Therefore, to relieve this issue, in this paper we aim to propose a knowledge distillation approach to transfer the learned high-resolution features from a teacher network to a simpler structured student network trained on low-resolution inputs. In our approach, instead of transferring knowledge from the same layers of the teacher and student network, we chose multi-level knowledge of the teacher network to supervise single-level output of the student network. Furthermore, we do not directly use the knowledge from the teacher network, instead, before knowledge transfer we concatenate different level features of the teacher network and figure our what kind of information is important and what is redundant and set different weight value to them. Then we use these knowledge with different weights to guide the output of single layer student network to extract abundant features from the low-resolution images. To evaluate the effectiveness of our proposed approach, we apply this approach to two models for object detection and facial expression recognition tasks. Through our experiments, we find that, in object detection task, the CornerNet achieves an accuracy of 40.6% on the original MC COCO dataset, while this index drops dramatically to only 34.2% on the resolution degraded images. By comparison, our proposed model trained by our knowledge distillation approach achieves 35.4% and 33.4% on the original and resolution degraded datasets, respectively. At the same time, compared to CornerNet the number of layers of the proposed network has been reduced by about 60%. Furthermore, in the task of facial expression recognition and image classification, the similar experimental results can also be observed.