Deep Neural Network Calibration
Dr Stefan Winkler, Adjunct Associate Professor, School of Computing
COM1 Level 3
MR1, COM1-03-19
Abstract:
Recent advances in machine learning have led to the development of large neural networks with exceptional recognition performance in fields like computer vision and natural language processing. These networks are increasingly employed in safety-critical applications such as autonomous vehicles, medical prognosis, facial expression recognition, and weather forecasting. Despite their popularity, deep neural networks have a tendency to produce overconfident and uncalibrated predictions, particularly under out-of-distribution (OOD) shifts. This thesis tackles the critical challenges of deep neural network calibration and proposes strategies to enhance model calibration, robustness, and overall performance.
For the first work, we propose MaxEnt loss for calibrating deep neural networks. Using statistical constraints derived from the training set, we show that MaxEnt loss delivers better model calibration without sacrificing training accuracy. We introduce three forms of statistical constraints that serve as an additional signal during supervised learning. We also include a numerical framework that automatically tunes the Lagrange multipliers controlling the weight of these constraints. Our experiments demonstrate that MaxEnt loss remains robust in terms of accuracy and calibration across both synthetic and in-the-wild distribution shift benchmarks. Furthermore, we analyze the ordering of the model’s feature norms under increasingly shifted inputs. MaxEnt loss is non-restrictive and seamlessly complements many ad-hoc calibration techniques, such as temperature scaling and label smoothing.
In the second work, we propose a soft OOD calibration benchmark for the field of facial expression recognition (FER). We argue that to effectively evaluate model calibration against OOD shifts, test labels should not be restricted to traditional one-hot encodings. Specifically, our benchmark aims to replicate commonly encountered OOD shifts through carefully selected image corruptions, providing corresponding soft labels that accurately reflect the underlying uncertainty and information loss caused by each corruption type. Our experiments discuss how common image corruptions exacerbate the task of FER, and how calibration can be used to improve the overall performance of five state-of-the-art FER algorithms.
In the third work, we investigate the optimal integration of recent neural network calibration methods. By framing the task as a multi-objective optimization problem, we explore how different state-of-the-art calibration techniques can be effectively combined into a single framework. We incorporate seven unique calibration algorithms, providing both theoretical and empirical evidence for their equal and weighted combinations. Finally, we address critical issues such as computational efficiency and component synergy through extensive experiments on both in-distribution and out-of-distribution computer vision tasks.