首页 > AI前沿 > How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis Strategies

How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis Strategies

arXiv机器学习 2026-10-08 23:55 3 阅读 查看原文

Pretraining scaling laws treat training data as a one-dimensional quantity: a token count.

Agentic reinforcement learning (RL) inherits this scalar abstraction, but modern synthesis pipelines now generate tasks, environments, and trajectories independently and at very different costs, turning data into a structured design space.

This raises a question pretraining never had to answer: given a synthesis budget, which axis should be scaled?

We address this question end-to-end with three tightly coupled contributions.

First, we decompose agentic data into three cardinality axes (task, environment, trajectory), plus a reward-density control variable, and fit per-axis scaling relations on different large language models (LLMs) across math+code, SWE, web, and UI/tool application domains.

Across all model sizes, model families, domains, and RL algorithms (PPO and GRPO) we measured, the scaling exponents order as environment $>$ task $>$ trajectory.

When environment diversity is decomposed into world and environment-structure diversity, its advantage is mainly driven by the former, while the latter is just comparable to the task diversity.

Second, we use the fitted laws as a common ruler to price synthesis methods, capturing each method's efficiency as a per-axis discount factor and identifying critical synthetic-to-real ratios at which collapse begins; these ratios differ by an order of magnitude across axes.

Third, we cast budget allocation as a constrained optimization under our laws and discounts, deriving a Chinchilla-style compute-optimal recipe and validating it at held-out scale.

The empirical results demonstrate that following the recipe yields up to $1.7\times$ compute efficiency over the dominant ``scale-the-trajectories'' practice.