Trustworthy Machine Learning: Abstention, Detection and Safe Generation
COM1 Level 2
Video Conference Room, COM1-02-13
Abstract
The past decade has seen remarkable progress in machine learning, with deep neural networks trained on large-scale datasets achieving or surpassing human-level performance across a wide array of tasks, from object recognition to game playing and medical diagnosis. The emergence of ``foundation'' models has further expanded the scope of deep learning, enabling generative capabilities in domains such as image, video, and language synthesis. Despite these advances, critical challenges remain that hinder the safe and trustworthy deployment of these models in high-stakes applications. This thesis aims to address some of these challenges by developing post-hoc methods that improve the safety and reliability of deep generative and discriminative models. We focus on post-hoc methods because foundation models are increasingly used across diverse downstream tasks without modification to their original training, making post-training interventions a practical and scalable approach to enhancing model trustworthiness.
The first part of the thesis addresses safety in image generation. The recent proliferation of large-scale text-to-image models has led to growing concerns that such models may be misused to generate harmful, misleading, and inappropriate content. To address this issue, we propose a technique inspired by continual learning to selectively forget concepts in pretrained deep generative models. This approach, termed Selective Amnesia, allows users to control how a concept is forgotten. Selective Amnesia can be applied to conditional variational likelihood models, which encompass a variety of popular deep generative frameworks, including variational autoencoders and large-scale text-to-image diffusion models. Experiments with various models demonstrate that our approach effectively induces forgetting on a variety of concepts, from entire classes in standard datasets to celebrity and nudity prompts in text-to-image models.
We then turn our attention to another important aspect of trustworthy AI systems: robustness to out-of-distribution (OOD) samples. Neural networks are known to be overconfident when making predictions on inputs that lie outside their training distribution, making OOD detection an essential step to mitigate mispredictions. Traditionally, unsupervised methods typically utilize deep generative models for OOD detection. However, such approaches require a new model to be trained for each inlier dataset. In our next work, we investigate whether a single model can enable OOD detection across a variety of tasks. To address this, we propose Diffusion Paths (DiffPath), a method leveraging a single diffusion model originally trained for unconditional generation to detect OOD samples. We introduce a novel technique of measuring the rate-of-change and curvature of the diffusion paths connecting samples to the standard normal. Extensive experiments show that with a single model, DiffPath is competitive with prior work using individual models on a variety of OOD tasks involving different distributions.
While generative OOD methods such as DiffPath provide a principled way to detect outlier samples, they are most effective on far-OOD settings where inputs fall well outside the training distribution and reside in low-likelihood regions. However, these methods are less suited for near-OOD scenarios, such as covariate shift, where the input distribution shifts subtly while the label space remains unchanged. In such cases, classifiers may still produce seemingly confident predictions, even when they are incorrect, making detection alone an insufficient safeguard.
To address this, we shift our focus from generative models to discriminative ones and adopt a post-hoc selective classification strategy. Instead of attempting to detect covariate shifts directly, our approach allows the model to abstain from making predictions when it is uncertain, thereby reducing the risk of critical errors. We formulate this strategy within the Neyman–Pearson framework and demonstrate that the optimal rejection rule corresponds to a likelihood ratio test. This perspective not only unifies a wide range of post-hoc selection methods but also leads to new techniques that outperform existing baselines across vision and language tasks involving covariate shift.
Collectively, these contributions support the development of trustworthy AI by introducing practical, post-hoc methods that improve safety in content generation and reliability under distribution shifts of machine learning systems.