Context-Enhanced Representation Learning for Single Image Deraining

摘要

Perception of content and structure in images with rainstreaks or raindrops is challenging, and it often calls for robust deraining algorithms to remove the diversified rainy effects. Much progress has been made on the design of advanced encoder–decoder single image deraining networks. However, most of the existing networks are built in a blind manner and often produce over/under-deraining artefacts. In this paper, we point out, for the first time, that the unsatisfactory results are caused by the highly imbalanced distribution between rainy effects and varied background scenes. Ignoring this phenomenon results in the representation learned by the encoder being biased towards rainy regions, while paying less attention to the valuable contextual regions. To resolve this, a context-enhanced representation learning and deraining network is proposed with a novel two-branch encoder design. Specifically, one branch takes the rainy image directly as input for learning a mixed representation depicting the variation of both rainy regions and contextual regions, and another branch is guided by a carefully learned soft attention mask to learn an embedding only depicting the contextual regions. By combining the embeddings from these two branches with a carefully designed co-occurrence modelling module, and then improving the semantic property of the co-occurrence features via a bi-directional attention layer, the underlying imbalanced learning problem is resolved. Extensive experiments are carried out for removing rainstreaks and raindrops from both synthetic and real rainy images, and the proposed model is demonstrated to produce significantly better results than state-of-the-art models. In addition, comprehensive ablation studies are also performed to analyze the contributions of different designs. Code and pre-trained models will be publicly available at https://github.com/RobinCSIRO/CERLD-Net.git.