首页 > AI前沿 > Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?

Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?

arXiv自然语言 2026-11-11 20:31 19 阅读 查看原文

Reinforcement learning (RL) has become central to post-training large language models (LLMs). However, popular RL methods like GRPO incur non-negligible overhead by computing both old-policy and current-policy likelihoods to form importance sampling ratios.

In this work, we propose Likelihood-Gated Policy Optimization (LGPO), which enforces a soft trust region constraint via likelihood-based gating, eliminating the need to compute old-policy likelihoods. Empirically, we show that removing the importance sampling correction term does not harm training stability, whereas removing the trust region mechanism can lead to collapse.

Moreover, ratio-