Towards Trustworthy Language Models

Abstract:

Language Models (LMs) are essential in Artificial Intelligence (AI), unifying natural language tasks like text classification, summarization, and dialogue generation. Large Language Models (LLMs) further enhance these capabilities, improving contextual understanding, accuracy, and responsiveness, advancing AI toward artificial general intelligence. The adoption of LMs and LLMs has transformed downstream applications in many fields like education and finance. Technically, LMs predict the next token based on the learned probability distribution, typically using a multi-layer Transformer architecture. Trained on large datasets, they capture linguistic patterns and knowledge to generate fluent language to solve various tasks. However, this training paradigm does not guarantee the trustworthiness of their outputs, limiting their real-world application.

Trustworthiness in LMs involves three levels of concerns. During the training stage, LMs are unaware of the distribution shifts between training and testing data, exhibiting poor out-of-distribution (OOD) generalization. This lack of robustness manifests as spurious correlations in LMs and non-robust prompt optimization in LLMs. During the inference stage, LLMs lack self-awareness and show significant overconfidence in incorrect responses. This poor calibration of confidence makes it hard to detect untrustworthy outputs. Finally, in the alignment stage, LLM responses should align with human preferences to be both harmless and helpful. However, jointly optimizing these two objectives remains challenging due to their conflicting nature, making it hard to achieve multi-aspect trustworthiness.

To tackle these issues, we focus on improving the current LM training paradigms, specifically addressing the information gap (data containing critical trustworthiness information) and the method gap (approaches to incorporate the trustworthiness information). Based on this, we detail the proposed solutions.

Firstly, fine-tuning LMs for machine reading comprehension often leads to spurious correlations, especially in complex contexts like tabular data, causing accuracy decrease on test data without such correlations. While counterfactual training is a promising solution, constructing faithful counterfactual tables is costly. To reduce spurious correlations in a more economical fashion, we propose using hypothetical questions to highlight genuine context understanding. We introduce a hypothetical training framework that uses hypothetical questions to guide the model’s gradient toward counterfactual answer changes, promoting causal relationships instead of spurious correlations. This approach achieves superior generalization results on OOD datasets.

Secondly, optimizing prompts for LLMs using labeled data from one distribution and applying them to a different distribution may lead to performance drops. Thus, we aim for robust prompt optimization against distribution shifts. We incorporate unlabeled target group inputs to introduce target group information with lower cost and real-world applicability. We propose to iteratively adapt knowledge from the source group to generate precise target group labels as feedback to perform joint prompt optimization on the two groups. This approach improves performance on the target group while maintaining source group performance.

Thirdly, LLM confidence estimation is a promising self-evaluation approach for identifying hallucinated answers. Existing confidence estimation approaches only concentrate on LLM-generated answers, leading to poor calibration with high confidence in incorrect answers. To address this, we expand the answer space by including comprehensive candidate answers beyond LLM generations, where answers with stronger validity help counterbalance overconfidence in incorrect answers. We instruct the LLM to reflect on and justify each candidate answer, aggregating these justifications to provide a more calibrated confidence score for the evaluated answer. This approach significantly enhances calibration and can be integrated with existing methods for further improvement.

Lastly, achieving trustworthy LLM responses requires alignment with multiple human preference aspects, such as harmlessness and helpfulness. However, the inherent preference conflict between these objectives makes joint optimization challenging. To address this, we propose constructing Pareto-optimal responses, that is, responses that outperform given response pairs across all objectives, enabling conflict-free fine-tuning and guiding the model toward better performance. To obtain and leverage these responses effectively and efficiently, we introduce a self-improving framework that allows LLMs to self-generate and select Pareto-optimal responses for self-supervised preference alignment, achieving Pareto improvement over various baseline approaches.

In summary, this thesis focuses on enhancing the trustworthiness of LMs from three aspects: (1) robustness during the training stage, including the spurious correlation issue for LMs and non-robust prompt optimization for LLMs; (2) calibration during the inference stage, specifically addressing overconfidence in incorrect responses; and (3) jointly achieving harmlessness and helpfulness during the alignment stage by handling preference conflicts.

Our solutions center around the information and method gaps to incorporate critical information and features for trustworthiness. Extensive experiments demonstrate the effectiveness of our solutions. We attribute this effectiveness to the insight that the core challenge in LLM trustworthiness arises from subtle differences between LLM generation and human reasoning. The identified information and method gaps specifically target these subtle differences to mitigate their impact. We anticipate that trustworthiness can emerge in a manner similar to other emergent abilities, guided by next-step fundamental principles that align LLMs toward more nuanced, human-like thinking and behavior.

In the future, we plan to explore the following directions.

First, we aim to investigate spurious correlations in the era of generalizable LLMs, focusing on hallucination and causal relationship understanding. Second, we plan to explore confidence estimation for open-ended generation where ground-truth answers are unavailable. Third, we will examine addressing multiple trust issues simultaneously. Finally, we aim to extend our study to additional trust issues, such as privacy, adversarial robustness, and fairness.