首页 > AI前沿 > Whispers from the Neighbors: Rethinking Caption Consensus as Inference-Time Evidence for Vision-Language Retrieval

Whispers from the Neighbors: Rethinking Caption Consensus as Inference-Time Evidence for Vision-Language Retrieval

arXiv自然语言 2026-10-25 00:59 13 阅读 查看原文

An image is rarely described only once.

Retrieval benchmarks and many collections attach several captions to each picture, yet dual encoders still judge every image-caption pair in isolation. We ask what these neighboring captions contribute when a frozen encoder consults them at inference time. Fusing image similarity with similarity to an image's own captions improves text-to-image R@1 by 3.4 to 16.8 points across nine frozen encoders when the captions come from the same annotators as the queries. The gain falls to 1.9 to 7.2 points when an off-the-shelf captioner writes them, and to at most one point when they come from a different annotation source; a paraphrase-trained text encoder strengthens the text side further.

  • 3.4 to 16.8 points improvement when captions originate from the same annotators as the queries across nine frozen encoders.
  • 1.9 to 7.2 points improvement when captions are produced by an off-the-shelf captioner.
  • At most 1 point improvement when captions derive from a different annotation source.

Refinement and Evaluation

A natural refinement, scoring the part of the image that the captions leave unexplained, appears to add much more under a leave-one-out evaluation. We show that this advantage arises because the correct candidate's supports are built differently from those of the negative candidates, with set size and query removal each contributing about half, and that under a fixed support index the residual reduces to a reweighting of what the captions already provide.

Why the Advantage Occurs

Support Construction

We show that this advantage arises because the correct candidate's supports are built differently from those of the negative candidates, with set size and query removal each contributing about half.

Under a fixed support index the residual reduces to a reweighting of what the captions already provide.

Benchmark Observations

On compositional benchmarks, the captions help by matching candidates to human descriptions of the image, without improving image-text grounding.

The code is available at: https://github.com/nhatkhangcs/clip_consensus