Reinforcement Learning for Language Models that Chat,Reason and Act

Abstract:

This thesis advances the theory and application of reinforcement learning (RL) for large language models (LLMs), addressing fundamental challenges in sample efficiency, reasoning, and agentic capabilities. We present a cohesive body of work that spans from theoretical foundations to practical algorithms and scalable infrastructure.

First, we tackle the problem of sample-efficient alignment of LLMs with human preferences. By formulating online alignment as a contextual dueling bandit problem, we develop SEA (Sample-Efficient Alignment), a principled algorithm based on Thompson sampling. SEA integrates an epistemic reward model, policy-guided search, and mixed preference learning to actively explore the most informative feedback. Extensive experiments demonstrate that SEA achieves superior sample efficiency and final performance compared to existing methods, often by a factor of 2-5 times.

Second, we provide a critical analysis of the "R1-Zero-like" training paradigm for enhancing LLM reasoning. We demystify emergent behaviors by showing that key attributes like self-reflection often pre-exist in base models. Furthermore, we identify and rectify a critical optimization bias in the popular GRPO algorithm, introducing an unbiased variant, Dr. GRPO, which improves token efficiency by preventing the generation of progressively longer incorrect responses without sacrificing performance. Our minimalist recipe produces state-of-the-art reasoning models with significantly reduced compute.

Finally, to bridge the gap from single-turn reasoning to multi-turn agency, we introduce GEM (General Experience Maker), a unified and scalable environment framework for RL with agentic LLMs. GEM provides a diverse suite of multi-turn tasks, tool integration, and a standardized interface compatible with major training frameworks. Using GEM, we benchmark RL algorithms and show that a simple REINFORCE variant with Return Batch Normalization (ReBN) is a strong, flexible baseline for complex, long-horizon tasks. We further demonstrate that the discount factor (γ < 1) is crucial for learning optimally efficient behaviors, a capability lost in prior simplified formulations. Beyond single-task training, GEM enables systematic studies of continual reinforcement learning, revealing that online RL induces sparse, task-specialized parameter updates that support forward transfer and mitigate catastrophic forgetting across heterogeneous tasks.

Collectively, this work provides novel algorithms, critical insights, and foundational tools that advance the sample-efficient alignment, reasoning, and agentic capabilities of large language models.