Dual-frame spatio-temporal feature modulation for video enhancement

Highlights：

• The proposed architecture combines the concept of spatio-temporal multi-resolution feature modulation with a multi-receptive encoder and domain-based feature filtering modules to learn domain-specific features. The architecture provides temporal consistency with recurrent feature merging, achieved by providing feedback of the previous frame output.

• The extensive experiments for de-hazing and de-raining with veiling effect databases and its comparisons with state-of-the-art methods are provided. We also provide results for nighttime weather degraded video enhancement to highlight the benefits of our method compared to SOTA methods that are based on depth estimation, which does not work very reliably under such conditions.

摘要

•As video enhancement is a pre-processing step for applications like video surveillance, traffic monitoring, autonomous driving, etc., it is necessary to have a lightweight enhancement module. Therefore, we propose a dual-frame spatio-temporal feature modulation architecture to handle the degradation caused by diverse weather conditions.•The proposed architecture combines the concept of spatio-temporal multi-resolution feature modulation with a multi-receptive encoder and domain-based feature filtering modules to learn domain-specific features. The architecture provides temporal consistency with recurrent feature merging, achieved by providing feedback of the previous frame output.•The extensive experiments for de-hazing and de-raining with veiling effect databases and its comparisons with state-of-the-art methods are provided. We also provide results for nighttime weather degraded video enhancement to highlight the benefits of our method compared to SOTA methods that are based on depth estimation, which does not work very reliably under such conditions.