Representation Learning for Deep Regression

Abstract:

Regression and classification are two fundamental tasks in machine learning, with the choice between the two depending on the categorical or continuous nature of the target output. Yet, most works focus only on classification and, curiously, solving regression-type problems as classification tasks often lead to better performance. This raises the need for a deeper understanding of regression, including its differences from classification and why converting it to classification often improves performance. Our research tackles this from a representation learning perspective. We find that, compared to the classification with cross-entropy, typical regression losses 1) lag in its ability to learn high-entropy representations, leading to lower mutual information between the learned representation and the target output. 2) do limit to reduce the conditional entropy H(Z|Y) of representation Z conditional on the target Y, even though it is vital for generalization performance. 3) require ordinality with respect to the target, leading to continuous representations, and thus leading to fundamentally different representation topologies. The first two findings shed light on why classification losses benefit representation learning for regression, while the third reveals the effective representation topologies for regression. Furthermore, we introduce a fully controlled synthetic dataset that enables systematic analysis of deep regression models by independently manipulating factors such as input complexity and label imbalance. This dataset provides a rigorous and interpretable benchmark for advancing our understanding of regression behavior. In summary, this thesis explores the differences between classification and regression representations, providing a deeper understanding of both. The insights gained benefit a wide range of regression tasks.