On the Detection and Mitigation of Prediction Bias And Other Failures in Language Models
COM2 Level 4
Executive Classroom, COM2-04-02
Abstract:
In this thesis, we tackle the task of detecting and mitigating prediction biases and other failures in language models. Prior work shows that Transformer language models have some understanding of the building blocks of language, but the self-attention mechanism and prompt designs can also elicit biases in the pre-training texts. Throughout the proposed works, we investigate logical failures of IS-A transitivity, biases that resemble human cognitive biases, and an output accuracy bias of language models. With a series of detection and mitigation methods, we reveal the challenges in identifying different types of language model biases and failures, and highlight the pros and cons of the detection/mitigation methods.
The practical implication is to mitigate inherent language model biases in a post-hoc manner, and push forward more accurate and fairer classification results of language model inference, without being strictly constrained by the original sources of bias, and without updating the model’s underlying parameters.
First, to see if language models respect important logical relations such as transitivity, we present a novel nearest neighbor probing classifier to examine the IS-A relation and its transitivity. Due to a lack of word sense-level IS-A probing dataset, we build a probing dataset from WordNet. Results on BERT’s contextualized embeddings show that BERT’s predictions do not fully obey the transitivity property of the IS-A relation. In other words, having known A is-a B and B is-a C, BERT does not always know that A is-a C, and it only knows 82.4% of the time. (Note that the statistic is for 3-tuples investigated in this thesis.) Our minimalist probing setting reveals an important logical inconsistency failure in language models.
Secondly, the observed imbalanced class accuracy in the binary IS-A classification task can be viewed as a prediction bias. This leads to the following question: does the accuracy imbalance consistently exist in multi-class classification tasks? If it does, for those multi-class prediction biases that are hardly easy to trace back to hidden states, we need more dedicated quantitative metrics to describe them (and ultimately, to debias them). As for bias detection methods, although probing turns to be effective for detecting logical failures of transitivity, probes may fail—prediction biases occurred during inference may not always trace back to explicit internal representations, and probes are not able to directly measure output-level prediction biases as bias scores, thus may not directly engage with debiasing. To overcome this issue, we propose targeted bias metrics to quantify biases at language model outputs. We propose the availability bias score and the Framing Effect Paraphrase (FEP) score. On a drug–-drug interaction extraction task, our bias measurements reveal an error pattern similar to the availability bias when the labels for training prompts are imbalanced, and show that a toning-down transformation of the drug–-drug description in a prompt can elicit a bias similar to the framing effect, warning users to distrust when prompting language models for answers. Further to cognitive biases, we quantify a prevalent output-level performance bias—the accuracy bias among classes—when using LLMs for text classification. We reconceptualize the accuracy bias as the Contextual Oddity Bias (COBias), measuring the averaged pairwise class accuracy difference. Our measurements reveal COBias’ wide existence in GPT, Llama, and Gemma models, across general-domain and biomedical-domain text classification tasks. For example, GPT-2-XL’s COBias across seven text classification benchmarks reaches 50.3%.
Thirdly, based on the quantifiable COBias metric, we present a series of three post-hoc debiasing methods to directly rectify LLM outputs, without updating LLMs’ underlying parameters. This is achieved by viewing the debiasing problem as a combinatorial optimization problem, particularly nonlinear integer programs, for: (i) selecting class-specific weight coefficients, (ii) choosing fine-grained sample-level correction functions, or (iii) combining corrections across both class and sample levels. The weights or functions are selected using an optimization dataset (analogous to a training dataset in gradient-based learning) and are directly applied to correct biased LLM class probabilities at test time. These efficient optimization procedures achieve significant COBias reduction and overall accuracy improvement, with optimization completing in seconds to minutes. These post-hoc debiasing methods are model-agnostic and demonstrate robust within-task generalization across diverse prompt configurations. The methods are summarized as below.
Class-level debiasing: This debiasing method adjusts the predicted probability of each class by multiplying it with a class-specific weight coefficient determined through optimization based on nonlinear integer programming. All samples belonging to the same class are corrected uniformly using a single coefficient. This class-level debiasing method enables the direct optimization of non-differentiable and highly non-linear performance metrics—such as accuracy and the proposed COBias metric—without requiring surrogate losses. In addition, it allows the probability reweighting coefficients to be selected from a discrete set, eliminating the need for continuous parameter learning. Under varied LLM prompting strategies, empirical evaluations show that the method achieves a 61% relative reduction in COBias and an 18% relative improvement in accuracy, averaged across five LLMs from GPT, Llama, and Gemma families and across seven text classification benchmarks.
Sample-level debiasing: This method provides fine-grained control over probability correction by applying a distinct membership function to each class, rather than using a single global reweighting coefficient. Each membership function defines how the original predicted probability should be adjusted based on its value, allowing some samples to be strongly corrected, others slightly adjusted, and some left unchanged. This method also enables probability-level interpretability: during optimization, a membership function is selected for each class based on the distribution of the original probabilities. Although not semantically tied to class labels, these functions reveal how under- or over-predicted probabilities are systematically corrected, making the process transparent and visualizable. That is, one can inspect the chosen membership function to understand exactly how much a probability value is being adjusted and why. In experiments on the text classification benchmarks, this sample-level correction further improves accuracy while reducing COBias over class-level debiasing.
Ensemble debiasing: Instead of committing to a single debiasing strategy, ensemble debiasing dynamically chooses between weight-based correction (broad, class-level) and membership-based correction (fine-grained, sample-level). This integration leverages the complementary strengths of class-level and sample-level debiasing to further elevate low-accuracy (weak) classes while avoiding over-correction of already strong classes. The optimization process automatically determines, for each class, whether sample-level corrections are necessary or whether class-level adjustments are sufficient. Quantitative analyses show that the average proportion of classes using membership correction relative to weight correction falls between 0 and 1. Specifically, while most classes apply weight corrections, the lowest-accuracy classes adopt membership correction, indicating that membership based adjustment is essential for lifting weak classes and should remain available as an option during optimization. By flexibly applying the appropriate correction granularity to each class, ensemble debiasing achieves state-of-the-art accuracy on benchmark text classification tasks while significantly mitigating class performance disparities.