Improving Super-Resolution Methods for Real-World Requirements
COM2 Level 2
MR3, COM2-02-26
Abstract:
Deep Super-resolution (SR) restores high-resolution (HR) images or videos from low-resolution (LR) ones and predicts lost details. While existing SR methods have attained impressive quantitative scores on benchmark datasets, these high benchmark performances may not accurately reflect the true quality of super-resolved images, especially in real-world scenarios. This gap between benchmark and real-world performances relates to a lack of consideration for real-world requirements. To address this issue, our research focuses on two aspects: optimizing aims for true image quality improvement and learning techniques for better processing real-world degradations.
First, conventional optimization aims are not comprehensive enough to improve the true performance of super-resolution. Existing methods are either objective-focused, producing high-fidelity but blurry outputs, or perceptual-focused, producing sharp patterns but with hallucination. Although both fidelity and clearness are important for the true image quality, considering them simultaneously is hindered by the trade-off between corresponding optimization aims. To address this problem, we correlate the objective and perceptual qualities to low- and high-frequency bands accordingly to mitigate the conflicts between optimization aims. The proposed method is the first end-to-end pipeline that achieves high objective and perceptual qualities at the same time.
Second, the standard SR setting assumes LRs are simply downsampled versions of HR images. However, real-world scenarios have many other degradations, e.g. noise, blur, and JPEG compression, without known formulations. As paired real-world data is typically unavailable, many existing methods synthesize LR images with extensive synthetic degradations, e.g. Gaussian blur, Gaussian noise, etc, to generalize the SR model to diverse input conditions. However, learning from diverse input conditions challenges the effectiveness of architecture designs tailored for standard settings. We investigate the prevalent Transformer architectures for real-world video super-resolution (VSR) by comparing spatial and channel attention w.r.t their sensitivity to input degradations. A state-of-the-art VSR model is built based on our findings.
Learning SR models through complex synthetic augmentations on training inputs achieves blind generalization. Nevertheless, it sacrifices the performance on the specific degradation domain. Therefore, we study how to train models specialized for a real-world domain without direct supervision. To tackle this problem, we propose two works following unsupervised and self-supervised learning respectively. Unsupervised super-resolution uses real-world LR images and external unpaired HR images. We study from a distillation perspective and introduce a novel pairwise distance distillation framework. Through our framework, an SR model specialized in synthetic degradation adapts to target real-world degradations by distilling intra- and inter-model distances across the specialized model and an auxiliary generalized model.
Self-supervised super-resolution only requires the real-world LR data. Existing methods derive self-supervision from low-resolution images by creating pseudo-pairs or enforcing a low-resolution reconstruction objective. These methods struggle with insufficient modeling of real-world degradations and the lack of knowledge about high-resolution imagery, resulting in unnatural super-resolved results. Our fourth work strengthens awareness of the high-resolution image to improve the self-supervised real-world super-resolution. We propose a controller to adjust the degradation modeling based on the quality of super-resolution results. We also introduce a novel feature-alignment regularizer that directly constrains the distribution of super-resolved images. Both the unsupervised and supervised methods can be applied to off-the-shelf SR models and improve their performances toward a target real-world domain.