Place perception from the fusion of different image representation

Highlights：

• We propose a multi-task deep neural network to realize the indoor place understanding and recognition together, which imitates and learns the process of place perception in a human-style.

• From the perspective of multi-modal information transformation and complementation, we propose an image captioning model to automatically generate natural language descriptions from place images, which is an additional information source to assist the decision-making in place recognition.

• We propose a multi-modal feature extraction and fusion architecture based on a mixed-CNN-LSTM network that gathers both visual and linguistic features corresponding to instance-level and concept-level information, respectively.

• We validate the effectiveness of the proposed strategy of using natural language descriptions to place perception through experiments on four public image datasets.