Efficient photorealistic style transfer with multi-order image statistics

摘要

Photorealistic style transfer concerns rendering the style of a reference image to a content image with the restraint that the stylized image should be realistic. While the existing methods have achieved promising results, they are prone to generating either structural distortions or inconsistent style due to the lack of effective style representation. In this work, to represent the inherent style information effectively, we propose a two-branch learnable transfer mechanism by considering the complementary advantages of the first-order and second-order image statistics simultaneously. Instead of directly using these two image statistics, we design a learnable transfer branch to implement the second-order image statistics learning to capture the consistent style and improve the efficiency. We further use a multi-scale representation branch to retain more structural details of the content image. In addition, a lightweight but effective adaptive-aggregation mechanism is proposed to fuse the features across different branches dynamically to balance between the consistent style and photorealism. Qualitative and quantitative experiments demonstrate that the proposed method renders the image faithfully with photorealistic results and high efficiency.