Multi-task framework based on feature separation and reconstruction for cross-modal retrieval

Highlights：

• We introduce feature separation into traditional cross-modal retrieval task to deal with information asymmetry between different modalities, and use different loss functions to supervise different parts of the feature vectors.

• We introduce image and text reconstruction tasks for specific information of images and texts, forcing the accuracy of feature separation operation and improving the quality of specific information.

• We use the multi-task learning framework, integrate cross-modal retrieval tasks, image and text reconstruction tasks, and further improve the performance of cross-modal retrieval tasks through joint training.

• We conduct extensive experimentation on MS-COCO and Flickr30K datasets. Our empirical results demonstrate that feature separation and specific information reconstruction can significantly improve the baseline performance of cross-modal image-text retrieval.