Dual-frame spatio-temporal feature modulation for video enhancement

Current video enhancement approaches have achieved good performance in specific rainy, hazy, foggy, and snowy weather conditions. However, they currently suffer from two important limitations. First, they can only handle degradation caused by single weather. Second, they use large, complex models with 10–50 millions of parameters needing high computing resources. As video enhancement is a pre-processing step for applications like video surveillance, traffic monitoring, autonomous driving, etc., it is necessary to have a lightweight enhancement module. Therefore, we propose a dual-frame spatio-temporal feature modulation architecture to handle the degradation caused by diverse weather conditions. The proposed architecture combines the concept of spatio-temporal multi-resolution feature modulation with a multi-receptive parallel encoders and domain-based feature filtering modules to learn domain-specific features. Further, the architecture provides temporal consistency with recurrent feature merging, achieved by providing feedback of the previous frame output. The indoor (REVIDE, NYUDepth), synthetically generated outdoor weather degraded video de-hazing, and de-raining with veiling effect databases are used for experimentation. Also, the performance of the proposed method is analyzed for night-time de-hazing and de-raining with veiling effect weather conditions. Experimental results show the superior performance of our framework compared to existing state-of-the-art methods used for video de-hazing (indoor/outdoor) and de-raining with veiling effect weather conditions. The code is available at https://github.com/pwp1208/PR2022