Reward Shaping for Reinforcement Learning

Abstract:

Reinforcement Learning (RL) seeks to optimize agents' policies to maximize environmental rewards, yet most native reward models in real-world environments are inherently sparse, delayed, and coarse, presenting a fundamental obstacle to efficient learning. Reward shaping (RS) has emerged as a promising solution by enriching the native rewards with auxiliary signals that provide fine-grained guidance and informative feedback. This dissertation systematically investigates several RS frameworks, aiming to improve learning efficiency and stability in environments with natively limited reward structures.

We begin from the perspective of exploration by introducing Random Reward Perturbation (RRP), a novel strategy that bridges noise-driven exploration and reward shaping, laying the foundation for our subsequent work. RRP injects zero-mean Gaussian noise into native rewards, with the noise gradually annealed throughout training to ensure a smooth transition from exploration to exploitation. Our theoretical analysis shows that RRP effectively increases policy diversity and expands the agent's exploration range.

Building on this concept, we further consider the states' latent value and propose RL with an Assistant Reward Agent (ReLara), a dual-agent RS framework. It comprises a policy agent for behavior optimization and a self-learning reward agent that generates auxiliary signals. The signals from the reward agent evolve from initially randomized outputs, which act as noise to promote early-stage exploration, to stable task-aligned values guiding late-stage exploitation. Moreover, leveraging the generalization capability of ReLara, we extend it to multi-task scenarios by designing a Centralized Reward Agent (CenRA) that distills and shares knowledge across tasks for effective transfer and adaptation.

Despite ReLara's advantages, its dual-agent architecture incurs high computational overhead, and the reward agent's black-box nature reduces interpretability. To overcome this, we propose Self-Adaptive Success Rate based RS (SASR), a lightweight and interpretable mechanism. SASR shapes rewards using success rates, which are modeled by dynamically evolving Beta distributions. These distributions initially exhibit high uncertainty to encourage exploration and gradually shift to more confident estimates to support exploitation. Kernel density estimation with random Fourier features are applied to derive the Beta distributions, ensuring scalability in high-dimensional and continuous state spaces.

While exploration by RRP is effective in many cases, it lacks directional guidance in extremely long-horizon tasks, where structured exploration proves essential. Lastly, we introduce Dual Random Network Distillation (DuRND), a unified framework that combines the strengths of novelty-based and latent value-based RS branches. DuRND employs two random network modules to simultaneously compute a novelty reward for directed exploration and a contribution reward to evaluate task progress toward completion, dynamically guiding agent behavior across different learning phases with low computational overhead.

We validate all proposed methods on challenging sparse-reward benchmarks across both high-dimensional and continuous control domains. Experiments consistently demonstrate superior performance over several state-of-the-art baselines, highlighting the effectiveness and versatility of our approaches.