首页 > AI前沿 > Eyes in the Update? Learning Where Rewards Land in Multimodal Reasoning

Eyes in the Update? Learning Where Rewards Land in Multimodal Reasoning

arXiv机器学习 2026-11-17 17:11 9 阅读 查看原文
用户需要我作为专业的新闻排版工程师,将一段纯文本新闻正文排版成语义化的HTML片段。我需要仔细分析要求: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释 2. 每个自然段落用

包裹,长文用

/

划分小节 3. 列表内容用
  • ,引用用
    ,代码用
    ,重要句子可用
    4. 保留原文所有信息,不得增删改内容
    5. 不要输出//等文档级标签
    
    现在看一下正文内容:
    
    这是一篇关于Gaze-RL方法的学术新闻/技术报道,包含多个段落:
    
    1. 介绍多模态训练从指令微调转向强化学习的背景
    2. 指出当前方法的局限性(奖励分配问题)
    3. 提到近期研究的进展
    4. 提出具体问题
    5. 介绍Gaze-RL方法
    6. 说明该方法的特点和优势
    
    这段文字可以分为几个小节来组织:
    
    - 可以设置一个

    标题介绍背景 - 然后是段落内容 - 再设置

    或

    介绍Gaze-RL方法 - 最后是结果 但原文没有明显的标题,我需要基于内容合理划分。用户说"长文用

    /

    划分小节",这段文字不算特别长,但可以合理划分。 让我考虑如何排版:

    多模态强化学习的发展与挑战

    Multimodal post-training has moved from broad instruction tuning toward reinforcement learning for image and video reasoning, where outcome rewards can improve final-answer accuracy. Yet these rewards are assigned to complete generated trajectories, while visual evidence is used only at sparse and uneven positions inside the response. Recent work has begun to make multimodal RL more token-sensitive through grounding signals or external token-level weighting.

    Gaze-RL:基于表征的奖励重分配方法

    We ask a more specific question: can the allocation of the same trajectory-level reward be inferred from the model's own multimodal computation, rather than from an auxiliary annotator, perception module, or process label?

    We propose Gaze-RL, a representation-driven policy optimization method that redistributes completion-level advantage through a latent reward landing field. The field is computed from internal cross-modal geometry and local predictive uncertainty, assigning larger update weight to positions that are both visually anchored and decision-bearing while remaining mean-one, so the total credit of each rollout is preserved and only its positionwise allocation changes.

    方法优势与实验结果

    Gaze-RL requires no process supervision, auxiliary perception module, or architectural modification, and improves both Qwen2.5-VL and InternVL3.5 across a broad range of image and video benchmarks.

    等等,用户要求"不得增删改内容"。如果我添加了

    标题,这些标题内容是原文中没有的,这算不算"增删改内容"? 让我重新理解规则:用户说"长文用

    /

    划分小节",这意味着需要添加标题。但"保留原文所有信息,不得增删改内容"指的是正文内容。标题作为结构划分是排版需要,不算修改正文内容。 但标题的文字需要合理,不能编造原文没有的信息。或者,我可以把关键句作为标题,或者用概括性的标题。 实际上,更保险的做法是:如果原文没有明确的小节标题,可以不强行添加

    ,或者只对明显有层次的内容添加。但用户明确说"长文用

    /

    划分小节",说明期望有标题。 让我再想想——标题是结构性的,不是正文内容。我可以使用概括性的标题,这些标题是对我的段落内容的总结,不算"增删改原文内容"。 但为了最安全,我可以: - 使用原文中的关键短语作为标题 - 或者使用非常概括的标题 让我尝试用原文中的术语作为标题: 方案1:

    背景与挑战

    ...

    问题提出

    ...

    方法:Gaze-RL

    ...

    结果

    ...

    这些标题是概括性的,属于排版结构,应该可以接受。 或者,考虑到用户说"重要句子可用",我可以在正文中对关键概念加粗。 让我重新组织一下。原文是一个连贯的文本,可以分为3-4个语义块: 1. 背景和现有问题 2. 提出的问题 3. Gaze-RL方法描述 4. 优势和结果 我觉得可以这样:

    多模态强化学习的发展与问题

    Multimodal post-training has moved from broad instruction tuning toward reinforcement learning for image and video reasoning, where outcome rewards can improve final-answer accuracy. Yet these rewards are assigned to complete generated trajectories, while visual evidence is used only at sparse and uneven positions inside the response. Recent work has begun to make multimodal RL more token-sensitive through grounding signals or external token-level weighting.

    研究问题

    We ask a more specific question: can the allocation of the same trajectory-level reward be inferred from the model's own multimodal computation, rather than from an auxiliary annotator, perception module, or process label?

    Gaze-RL 方法

    We propose Gaze-RL, a representation-driven policy optimization method that redistributes completion-level advantage through a latent reward landing field. The field is computed from internal cross-modal geometry and local predictive uncertainty, assigning larger update weight to positions that are both visually anchored and decision-bearing while remaining mean-one, so the total credit of each rollout is preserved and only its positionwise allocation changes.

    实验结果

    Gaze-RL requires no process supervision, auxiliary perception module, or architectural modification, and improves both Qwen2.5-VL and InternVL3.5 across a broad range of image and video benchmarks.

    这样看起来结构清晰。但我不确定标题文字是否算"增删改内容"。为了更安全,我考虑另一种方式:使用原文中的句子片段或不加标题。 但用户明确要求"长文用

    /

    划分小节",所以应该添加。标题作为排版结构元素,不改变正文内容,应该是允许的。 让我再考虑:用户说"保留原文所有信息,不得增删改内容"。这里的"内容"应该指正文信息,标题是结构标签,是排版的一部分。我添加的标题文字是对段落的概括,不改变原文句子。 我觉得这个方案是合理的。 另外,是否需要加粗更多内容?用户说"重要句子可用"。原文中的"Gaze-RL"是方法名称,可以加粗。也许"latent reward landing field"也是关键术语,但不要过度使用。 让我最终确定HTML:

    多模态强化学习的发展与挑战

    Multimodal post-training has moved from broad instruction tuning toward reinforcement learning for image and video reasoning, where outcome rewards can improve final-answer accuracy. Yet these rewards are assigned to complete generated trajectories, while visual evidence is used only at sparse and uneven positions inside the response. Recent work has begun to make multimodal RL more token-sensitive through grounding signals or external token-level weighting.

    研究问题

    We ask a more specific question: can the allocation of the same trajectory-level reward be inferred from the model's own multimodal computation, rather than from an auxiliary annotator, perception module, or process label?

    Gaze-RL:基于表征的奖励重分配

    We propose Gaze-RL, a representation-driven policy optimization method that redistributes completion-level advantage through a latent reward landing field. The field is computed from internal cross-modal geometry and local predictive uncertainty, assigning larger update weight to positions that are both visually anchored and decision-bearing while remaining mean-one, so the total credit of each rollout is preserved and only its positionwise allocation changes.