Crafting effective reward signals remains a central challenge in reinforcement learning (RL) for complex reasoning.
Existing methods for automated reward optimization search with a fixed proposer which never learns from its costly trials.
We present Dual-Evolving Reinforcement Learning (DERL), a dual-training framework for autonomous discovery of reward structures.
An outer-loop proposer composes a reward from task primitives, and an inner policy trains with this reward.
The proposer itself learns by policy gradient from the validation performance of the inner policy.
Thus, DERL utilizes past-trial feedback to shape its reward-proposal distribution, an empirical meta-gradient of task success.
Validation
We validate DERL on embodied (ALFWorld), scientific (ScienceWorld), and mathematical (GSM8K, MATH) reasoning.
On multi-turn agent benchmarks, DERL outperforms strong baselines, significantly on out-of-distribution tasks.
In mathematics, DERL improves on the outcome reward where directly added primitives hurt.
Further Analyses
Further analyses under matched compute show that our proposer effectively captures the meta-gradient to discover useful reward signals.