Trustworthy Machine Learning: Mitigating Overconfidence and Hallucinations
COM1 Level 3
MR1, COM1-03-19
Abstract:
Building trustworthy machine learning systems is essential for deploying deep models in real-world applications. Despite strong performance, current models often suffer from overconfidence and hallucinations, raising concerns about their reliability.
Our first study explores how public foundation models can be used to improve confidence estimation without fine-tuning. We introduce a neighborhood agreement metric that compares internal representations across models, leading to more accurate confidence assessments.
The second study proposes a self-supervised probing method to examine how a model’s predictions relate to semantic properties. This approach helps detect when a model is overly confident and enables adjustments that better align confidence with actual accuracy, improving decision-making reliability.
With the rise of large language and vision-language models, hallucination has become a major obstacle, particularly in safety-critical domains. Our third study addresses object hallucinations in vision-language models, showing that CLIP provides a more reliable signal of correctness than likelihood-based measures. We develop a CLIP-guided decoding strategy that reduces hallucinated outputs and improves generation safety.
Finally, we investigate how vision-language models handle conflicts between visual and textual inputs. We identify a “blind faith in text” tendency, where models over-rely on text even when it contradicts visual evidence, resulting in severe performance drops under corrupted text. We analyze contributing factors such as prompts, model size, and token order, and show that fine-tuning with text augmentation alleviates the issue. Our analysis suggests this bias arises from the imbalance between text-only and multimodal data in training, pointing to the need for more balanced data and careful treatment of modality interactions.
Together, these studies advance the goal of trustworthy machine learning by improving confidence calibration and reducing both hallucinations and text bias in multimodal models.