首页 > AI前沿 > Learning to Think with Images Faithfully via Adaptive Data Selection

Learning to Think with Images Faithfully via Adaptive Data Selection

arXiv自然语言 2026-11-19 22:43 13 阅读 查看原文
用户现在需要把给定的纯文本新闻正文排版成语义化的HTML片段,首先得先理清楚结构。首先原文是一个整体的新闻,首先看有没有可以分的小节?不对,原文是一段?不对,等下看原文内容,哦,原文是一段?不对,再仔细看,原文的内容是一段,但首先,有没有需要用h2或者h3的?不对,先看规则,每个自然段落用p包裹,长文用h2h3划分。首先,原文的内容是关于TwI的,首先开头可以先?不对,等下原文的内容是:首先第一句讲TwI的作用,然后转折说存在的问题,然后分析原因,然后提出方法,然后讲实验结果,最后结论。不对,有没有自然的段落?哦,原文的用户给的正文是一段?不对,看用户给的正文:是一整段?不对,等下看:“Thinking with images (TwI) empowers... without directly supervising the reasoning process.” 哦,是一整段?不对,不对,再仔细看,有没有可以拆分的?不对,规则说每个自然段落用p包裹,那如果原文的自然段落是一整段的话?不对,等下,不对,再看,哦,不对,是不是可以把内容合理划分?不对,规则说保留原文所有信息,不得增删改。哦,等下,原文的自然段落就是一段?不对,不对,再读:首先,第一部分是讲TwI的优势和存在的问题:“Thinking with images (TwI) empowers vision-language models (VLMs) to overcome perceptual bottlenecks by actively cropping and zooming into fine-grained visual details, promising both higher accuracy and interpretable, region-grounded reasoning. However, our analysis shows that the improved accuracy of TwI models does not guarantee faithful tool use; they frequently invoke tools without actually relying on the zoomed region, undermining the reliability of tool-assisted reasoning.” 这是第一部分?然后接下来是分析原因和解决方法:“We trace this unfaithful behaviour to the training data: when questions are answerable without zooming, the model can answer correctly while ignoring the returned evidence, and this problem is exacerbated during training as the model becomes better at answering without tools. To keep zooming useful throughout training, we adapt the training data to the model's evolving capability and train on questions where correct zoomed evidence can still help the model answer correctly. This selection is supported by a data synthesis pipeline that localises small objects and constructs verifiable questions about their fine details, and complemented by an adaptiveness reward that favours tool-free answers when zooming is unnecessary.” 然后是实验结果和结论:“The experimental results show that our method substantially improves reasoning faithfulness and achieves state-of-the-art performance on visual search benchmarks (+11.6% on V*, +7.9% on HR-Bench 4K, and +9.4% on HR-Bench 8K). On multimodal hallucination benchmarks, our model also outperforms the compared TwI baselines. These results suggest that improving TwI faithfulness does not sacrifice accuracy and that both can be learnt from the right training data without directly supervising the reasoning process.” 哦,不对,原文是不是自然的三个部分?不对,用户给的正文是一整段,但是作为新闻,可能需要用h2划分小节?比如h2可以是问题与发现?不对,不对,规则说不要增删改内容,所以h2的内容应该是?哦,不对,等下,h2是划分小节,但是不能自己加内容啊?哦,不对,哦,等下,是不是原文的内容可以直接用p,然后如果有需要的话,h2用原文里的?不对,哦,等下,我错了,规则是保留原文所有信息,不得增删改内容,所以不能自己加小标题的内容?哦,那如果原文没有小标题的话,是不是就不需要用h2h3?不对,再看规则:“每个自然段落用

包裹,长文用

/

划分小节”,哦,长文的话划分,但如果没有合适的原文里的小节标题,是不是就不用?不对,等下,再仔细看原文,哦,原文是一段,那是不是就一个p?不对,不对,再看,哦,不对,原文的内容是不是可以分成几个自然段?不对,用户给的正文是连续的,但是作为排版,是不是可以按照语义拆分?不对,规则说保留原文所有信息,不得增删改,所以拆分段落的话,不能改变内容,只是把原来的一段拆成几个自然段?哦,不对,自然段落是原文里的,用户给的正文是一整段,那是不是就一个p?不对,等下,再看用户给的正文,哦,等下,我再读一遍: 原文正文: Thinking with images (TwI) empowers vision-language models (VLMs) to overcome perceptual bottlenecks by actively cropping and zooming into fine-grained visual details, promising both higher accuracy and interpretable, region-grounded reasoning. However, our analysis shows that the improved accuracy of TwI models does not guarantee faithful tool use; they frequently invoke tools without actually relying on the zoomed region, undermining the reliability of tool-assisted reasoning. We trace this unfaithful behaviour to the training data: when questions are answerable without zooming, the model can answer correctly while ignoring the returned evidence, and this problem is exacerbated during training as the model becomes better at answering without tools. To keep zooming useful throughout training, we adapt the training data to the model's evolving capability and train on questions where correct zoomed evidence can still help the model answer correctly. This selection is supported by a data synthesis pipeline that localises small objects and constructs verifiable questions about their fine details, and complemented by an adaptiveness reward that favours tool-free answers when zooming is unnecessary. The experimental results show that our method substantially improves reasoning faithfulness and achieves state-of-the-art performance on visual search benchmarks (+11.6% on V*, +7.9% on HR-Bench 4K, and +9.4% on HR-Bench 8K). On multimodal hallucination benchmarks, our model also outperforms the compared TwI baselines. These results suggest that improving TwI faithfulness does not sacrifice accuracy and that both can be learnt from the right training data without directly supervising the reasoning process. 哦,确实是一整段。那现在要排版成语义化HTML,规则是每个自然段落用p包裹,那这里如果原文是一个自然段落,那就是一个p?但是有没有需要用strong的地方?比如重要的句子,比如开头的TwI的作用?还有问题的部分,还有结果的部分?比如: 首先,第一句里的“Thinking with images (TwI) empowers vision-language models (VLMs) to overcome perceptual bottlenecks by actively cropping and zooming into fine-grained visual details, promising both higher accuracy and interpretable, region-grounded reasoning.” 这里可以把核心的作用部分加strong?不对,重要句子可以用strong,比如问题的部分:“our analysis shows that the improved accuracy of TwI models does not guarantee faithful tool use; they frequently invoke tools without actually relying on the zoomed region, undermining the reliability of tool-assisted reasoning.” 这个是重要的发现,可以加strong?还有最后结论的部分:“These results suggest that improving TwI faithfulness does not sacrifice accuracy and that both can be learnt from the right training data without directly supervising the reasoning process.” 也可以加strong? 等下,但是要注意不能增删改内容,加strong只是格式,不改变内容,对吧。 然后,有没有需要用h2或者h3的?比如如果划分小节的话,但是原文没有小标题,那是不是不需要?因为不能自己加小标题的内容,对吧?因为规则说保留原文所有信息,不得增删改,所以不能自己加h2的文字,对吧?哦,对,所以如果没有原文里的小标题,就不需要用h2h3,对吧? 然后,有没有列表?没有。有没有引用?没有。有没有代码?没有。所以就是用p包裹,重要的句子加strong。 等下,但是再看规则,长文用h2h3划分小节,这篇算长文吗?但是如果没有合适的小节标题,总不能自己加吧?比如不能自己加“研究背景”“问题分析”这种,因为属于增改内容了,对吧?对,因为规则说不得增删改内容,所以不能加原文没有的文字,所以h2h3如果没有对应的原文内容,就不需要用。 那现在,把原文的一整段放在p里,然后重要的部分加strong。比如: 首先,开头的TwI的作用,是不是重要的?然后问题部分,然后方法部分,然后结果部分,然后结论部分。 等下,再仔细看,比如:

Thinking with images (TwI) empowers vision-language models (VLMs) to overcome perceptual bottlenecks by actively cropping and zooming into fine-grained visual