首页 > AI前沿 > Adaptive Memory for Online Reinforcement Learning under Sparse Model Changes

Adaptive Memory for Online Reinforcement Learning under Sparse Model Changes

arXiv机器学习 2026-11-17 04:44 19 阅读 查看原文
用户需要将纯文本新闻正文排版成语义化HTML片段。我需要遵循以下规则: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释。 2. 每个自然段落用`

`包裹,长文用`

`/`

`划分小节。 3. 列表内容用`
  • `,引用用`
    `,代码用`
    `,重要句子可用``。
    4. 保留原文所有信息,不得增删改内容。
    5. 不要输出``/``/``等文档级标签。
    
    正文是关于强化学习研究的,内容如下:
    "We study the discounted reinforcement learning (RL) problem in a setting where the underlying environment changes between deployments. Each deployment provides $H$ observations from a fixed MDP. The reward function and transition kernel are fixed within each deployment. Across deployments, at most an $ε>0$ fraction of the reward and transition kernel entries may change. In this setting, learning each model to deploy from scratch may discard useful observations from past deployments, whereas pooling all past observations can bias the estimates of entries that have already changed. We then propose an algorithm that for each state and action pair, selects how much history to reuse to estimate the reward and probabilities of the next state. We start with observations from the current deployment and add earlier deployment observations one at a time, the most recent first. After each addition, we check whether a single average reward agrees with the estimates from all histories considered until that point. We apply the same check to the distribution of next states. For each estimate, we leverage the last history that passes these aforementioned checks. The optimal control policy is then computed on top of the fitted MDP. We characterize the average gap between the optimal return and the return of the learner's policies over a finite number of deployments. We demonstrate that when $0< |S|ε\leq 2$, with $|S|$ being the cardinality of the state space, our algorithm needs only the observations from the most recent $O(1/ε)$ deployments, and achieves a long-run average gap of at most $\widetilde{O}(\sqrt{ε/H})$."
    
    分析内容结构:
    这段文本是一篇学术摘要或论文介绍,没有明显的列表、引用或代码。但可以将其划分为几个逻辑部分,使用`

    `或`

    `来划分小节,以符合"长文用

    /

    划分小节"的要求。 不过,这段文本虽然有一定长度,但实际上是一个完整的摘要,逻辑连贯。如果强行划分小节可能会破坏原文的连贯性。但规则说"长文用

    /

    划分小节",我可以考虑将内容分为几个小节: 1. 问题设定(问题背景) 2. 算法思路 3. 理论结果 让我尝试划分:

    问题设定

    We study the discounted reinforcement learning (RL) problem in a setting where the underlying environment changes between deployments. Each deployment provides $H$ observations from a fixed MDP. The reward function and transition kernel are fixed within each deployment. Across deployments, at most an $ε>0$ fraction of the reward and transition kernel entries may change.

    算法思路

    In this setting, learning each model to deploy from scratch may discard useful observations from past deployments, whereas pooling all past observations can bias the estimates of entries that have already changed. We then propose an algorithm that for each state and action pair, selects how much history to reuse to estimate the reward and probabilities of the next state. We start with observations from the current deployment and add earlier deployment observations one at a time, the most recent first. After each addition, we check whether a single average reward agrees with the estimates from all histories considered until that point. We apply the same check to the distribution of next states. For each estimate, we leverage the last history that passes these aforementioned checks. The optimal control policy is then computed on top of the fitted MDP.

    理论结果

    We characterize the average gap between the optimal return and the return of the learner's policies over a finite number of deployments. We demonstrate that when $0< |S|ε\leq 2$, with $|S|$ being the cardinality of the state space, our algorithm needs only the observations from the most recent $O(1/ε)$ deployments, and achieves a long-run average gap of at most $\widetilde{O}(\sqrt{ε/H})$.

    这样划分是合理的,保留了所有信息,没有增删改。使用`

    `作为小节标题。 检查重要句子是否可用``:规则说"重要句子可用",是可选的。我可以将一些关键结论用``标注,比如最后一句关于算法性能的结论。 检查数学公式:原文中有 $H$、$ε>0$、$0< |S|ε\leq 2$、$|S|$、$O(1/ε)$、$\widetilde{O}(\sqrt{ε/H})$ 等数学表达式。这些应该原样保留在文本中,不作为代码处理,因为它们是行内公式,不是代码块。所以不需要用`
    `。
    
    最终HTML片段:
    
    

    问题设定

    We study the discounted reinforcement learning (RL) problem in a setting where the underlying environment changes between deployments. Each deployment provides $H$ observations from a fixed MDP. The reward function and transition kernel are fixed within each deployment. Across deployments, at most an $ε>0$ fraction of the reward and transition kernel entries may change.

    算法思路

    In this setting, learning each model to deploy from scratch may discard useful observations from past deployments, whereas pooling all past observations can bias the estimates of entries that have already changed. We then propose an algorithm that for each state and action pair, selects how much history to reuse to estimate the reward and probabilities of the next state. We start with observations from the current deployment and add earlier deployment observations one at a time, the most recent first. After each addition, we check whether a single average reward agrees with the estimates from all histories considered until that point. We apply the same check to the distribution of next states. For each estimate, we leverage the last history that passes these aforementioned checks. The optimal control policy is then computed on top of the fitted MDP.

    理论结果

    We characterize the average gap between the optimal return and the return of the learner's policies over a finite number of deployments. We demonstrate that when $0< |S|ε\leq 2$, with $|S|$ being the cardinality of the state space, our algorithm needs only the observations from the most recent $O(1/ε)$ deployments, and achieves a long-run average gap of at most $\widetilde{O}(\sqrt{ε/H})$.

    再检查一遍:只输出HTML片段,没有Markdown围栏,没有解释,没有文档级标签。符合所有规则。 不过,我注意到原文中并没有明显的小标题,我添加的"问题设定"、"算法思路"、"理论结果"是否属于"增删改内容"?规则4说"保留原文所有信息,不得增删改内容"。添加小标题可能被视为添加内容。让我重新考虑。 规则2说"长文用

    /

    划分小节",这意味着需要划分小节,但小节标题应该是从原文中提取的,或者如果原文没有明确的小标题,我可能需要根据内容逻辑来划分,但标题本身可能算作新增内容。 为了严格遵守"不得增删改内容",我可能不应该添加原文中没有的小标题。那么我可以不使用`

    `/`

    `,只用`

    `包裹每个自然段落。原文实际上就是几个自然段落(虽然在给定文本中是一整段,但从逻辑上可以分成几段)。 原文在用户输入中是一整段,但排版时可以按句子或逻辑分成多个`

    `段落。规则1说"每个自然段落用

    包裹",原文是一整段,那是否应该只输出一个`

    `?但这样就没有利用`

    `/`

    `划分小节了。 规则2是"每个自然段落用

    包裹,长文用

    /

    划分小节"。这是一个"且"的关系,即对于长文,需要划分小节。这段文本属于长文(学术摘要),所以应该划分小节。但小节标题如果没有,我可以用