Enhancing Sequence Model Reliability: Studies on Noise Robustness, Demographic Fairness, and Attention Sink
COM1 Level 3
MR1, COM1-03-19
Abstract:
Modern Transformer-based sequence models achieve remarkable performance under controlled conditions, yet their deployment in real-world scenarios frequently exposes critical vulnerabilities. Developing robust and reliable systems requires a dual-pronged approach: addressing vulnerabilities in how models process external data and interpreting the internal architectural artifacts that govern their stability. This thesis investigates methods to enhance model reliability across two sequence modeling domains—audio and language—through the lenses of data, learning method, and model architecture.
The first thrust of this research focuses on external reliability by examining singing voice transcription (SVT) models. We identify and address two major performance shortcomings that emerge in unconstrained environments. First, to mitigate the lack of noise robustness in audio-only systems, we introduce a multimodal approach leveraging both audio and video data. We demonstrate that visual information significantly reduces performance degradation, enabling the model to remain reliable despite background music or environmental noise. Second, we uncover a clear group bias, where SVT models consistently perform better on female singers than on male singers. To address this demographic unfairness, we propose a novel note-conditioned adversarial learning method that encourages gender-invariant representations, yielding an improved fairness-utility trade-off.
The second thrust investigates internal reliability within auto-regressive language models (LLMs). We focus on critical failures where models like Gemma exhibit catastrophic performance degradation without a token, and LLMs experience severe perplexity surges when the initial token is evicted during long-context generation. These vulnerabilities are intrinsically linked to the attention sink phenomenon, where significant attention is consistently assigned to the first token regardless of its semantic meaning. However, the underlying reason for when it emerges and why LLMs need such phenomenon remain poorly understood. Through extensive analysis from both mechanistic interpretability and training recipe ablation perspectives, we reveal that the attention sink functions as a key structural bias. Stemming from the inner dependence on attention scores caused by softmax normalization, it acts as a "no-op" that stores non-informative attention scores while remaining functionally essential for maintaining performance reliability. Ultimately, this phenomenon reveals that Transformers naturally develop internal mechanisms to prevent representation collapse, providing critical design principles for new attention operations. This work directly leads the design of attention biases and inspires the gated attention mechanisms currently adopted in frontier LLM architecture.