Co-Attention Fusion Network for Multimodal Skin Cancer Diagnosis

作者：

Highlights：

• We propose a novel co-attention fusion network for precise multimodal skin cancer diagnosis by designing two new blocks: the co-attention (CA) block and the attention fusion (AF) block.

• The CA block leverages a cross-modal attention mechanism to achieve the cooperation of dermoscopy and clinical images, which can enhance the representation ability of the extracted features through the mutual guidance between modalities.

• The AF block employs an attentional feature fusion mechanism to fuse multimodal features, which can generate more fine-grained fusion features.

摘要

•We propose a novel co-attention fusion network for precise multimodal skin cancer diagnosis by designing two new blocks: the co-attention (CA) block and the attention fusion (AF) block.•The CA block leverages a cross-modal attention mechanism to achieve the cooperation of dermoscopy and clinical images, which can enhance the representation ability of the extracted features through the mutual guidance between modalities.•The AF block employs an attentional feature fusion mechanism to fuse multimodal features, which can generate more fine-grained fusion features.

论文关键词：Skin cancer diagnosis,Convolutional neural networks,Multimodal fusion,Attention mechanism

论文评审过程：Received 7 November 2021, Revised 22 July 2022, Accepted 20 August 2022, Available online 24 August 2022, Version of Record 16 September 2022.

论文官网地址：https://doi.org/10.1016/j.patcog.2022.108990