Advancing Adaptation and Alignment in Large Language Models

Abstract:

Large Language Models (LLMs) have led major advances in natural language processing and artificial intelligence, achieving state-of-the-art performance in language understanding and generation. However, challenges remain in ensuring their robustness and alignment with human intent, particularly under distribution shifts. To address these challenges, we first explore test-time adaptation via multi-source models that evolve through user feedback. We formulate this as a stochastic decision-making process and introduce Co-UCB, a dueling bandit-based approach for efficient model selection during adaptation. This enables models to learn and improve continuously during deployment. We then study post-training alignment via offline reinforcement learning, introducing Preference-Guided Reflective Sampling (PRS). Unlike traditional random sampling, PRS iteratively refines outputs through a tree-like structure guided by user preferences, improving sampling efficiency and personalization. In multi-agent settings, we propose Tree Search-based Orchestrated Agents (TOA) to enhance synthetic data generation. TOA dynamically coordinates multiple LLMs using Monte Carlo Tree Search, optimizing response quality as inference compute scales. TOA consistently outperforms static workflows and single-agent methods in alignment performance. In addition, to improve instruction-following reliability, we propose Selective Instruction Following, allowing the model to refuse to answer when response quality is predicted to be low. This is powered by Self-J, a self-training framework that uses unlabeled data to train judge models aligned with GPT-4 evaluations. Finally, we unify PRS and Self-J to enable self-improvement, where response quality scores guide more effective data sampling and model learning.