CMAFGAN: A Cross-Modal Attention Fusion based Generative Adversarial Network for attribute word-to-face synthesis

摘要

Face synthesis based on attribute words is a novel and challenging topic in computer vision, which has various application potentials in public security and multimedia. Existing attribute vector-to-face (V2F) synthesis methods mainly generate faces based on attribute label vectors that lack rich semantic feature information, which leads to low-quality generated face images. To address this challenge, we advocate attribute word-to-face (W2F) synthesis, using attribute-word sequences that contain rich semantic information as input. A novel Cross-Modal Attention Fusion based Generative Adversarial Network (CMAFGAN) is proposed to generate faces from facial attribute words. CMAFGAN is highlighted by two blocks, cross-modal attention fusion (CMAF) and word feature transformation (WFT), which are proposed to explore the correlation between image features and the corresponding attribute word features. Experimental results on the CelebA and LFW datasets demonstrate that our CMAFGAN achieves state-of-the-art performance, effectively improving the quality of the synthesised faces. In particular, the consistency between the predicted images and input attribute words (R-precision) on the CelebA and LFW datasets achieved 61.24% and 64.46% respectively, which is significantly better than previous methods. In addition, CMAFGAN achieves comparable or better performance than the current best methods of text-to-image synthesis (R-precision 83.41% on caltech-ucsd birds-200-2011, CUB).