: Learning from Imperfect Visual Data

Abstract:

Modern deep learning models require a large amount of data. However, learning from real-world data is usually imperfect, suffering from label noise and imbalance. Besides, collected data are too limited to capturing the real world, which means deep models can just learn from closed-set data but may encounter novel samples during inference. Imperfect learning hurts the generalization ability and reliability of deep learning models and thus limits their application to real-world scenes. In this paper, we focus on three imperfect learning scenarios, i.e. learning from noisy labels, learning from imbalanced data, and learning to detect out-of-distribution data. We aim to enhance imperfect data learning by improving learning targets. First, we focus on learning from noisy learning targets, where the targers are usually perturbed by noise introduced during annotation or learning target generation. We study a typical regression task, crowd counting, which contains both kinds of noise. Specifically, factors such as biased dot annotations and mismatched Gaussian kernels used to generate learning targets introduce deviations from the true learning targets. We start by analyzing regression-based and classification-based counting approaches and find that standard continuous regression is highly sensitive to these errors, while classification can tolerate the label noise by discretizing the learning targets. However, discretizing learning targets inevitably introduce discretization errors. To eliminate the discretization error, we design a global count loss. Combining discretization and global count loss, a novel discrete-constrained regression is proposed to overcome the label noise in crowd counting. DC-regression shows state-of-the-art results in three challenging crowd counting datasets. Besides counting, DC-regression is also suitable for age estimation tasks.

Then, we delve into imbalanced learning, where the label distribution is imbalanced. Specifically, the majority of training samples lie in a head range of target
values, while a minority of samples span a usually larger tail range. Besides imbalance, tail classes may also suffer from data insufficiency, leading to over-fitting on
the limited training samples. We focus on imbalanced learning in regression. By selecting the class quantization, one can adjust imbalanced regression targets into
balanced classification outputs, though there are trade-offs in balancing classification accuracy and quantization error. To improve regression performance over the entire range of data, we propose to construct hierarchical classifiers for solving imbalanced regression tasks. The fine-grained classifiers limit the quantization error while being modulated by the coarse predictions to ensure high accuracy. Standard hierarchical classification approaches, when applied to the regression problem, fail to ensure that predicted ranges remain consistent across the hierarchy. As such, we propose a range-preserving distillation process that effectively learns a single classifier from the set of hierarchical classifiers. Extensive experiments and theoretical analyses have verified the effectiveness of HCA.

Finally, we extend our research on data imperfection from in-distribution data to out-of-distribution data that appears during model inference. Specifically, we focus on learning to detect out-of-distribution samples that do not semantically belong to any training class. Current out-of-distribution (OOD) detection methods, especially post-hoc methods, rely on in-distribution (ID) pre-trained models. Existing literature shows how OOD and ID performance are correlated, i.e. stronger models with better ID performance tend to perform better in OOD detection. However, significant performance discrepancies exist between model versions, and sometimes models with better ID accuracy deliver worse OOD performance. In this study, we systematically investigated this issue and identified two main factors—label smoothing and mixup that, while improving in-distribution accuracy, lead to a decline in OOD detection performance. We provide empirical and theoretical explanations for this phenomenon and propose a solution that enhances OOD Detection while maintaining strong ID performance.