Retrieving information from visually rich documents requires matching queries against pages that contain complex evidence in tables, figures, and layout-dependent structures.
Current vision-language retrievers address this but incur query-time costs that grow with corpus size and are impractical to scale.
We propose SCOUT, a cascaded two-track retrieval framework that separates retrieval performance from exhaustive search.
The visual track first uses a lightweight parent retriever to filter the corpus to a compact candidate pool, then applies a stronger multi-vector reranker only to those candidates.
In parallel, a semantic track retrieves over hierarchical page and document summaries generated offline to provide a signal complementary to visual matching.
The outputs are fused through cross-modal query refinement, which uses semantic scores to adjust the visual query representation before final ranking.
We evaluate SCOUT on 4 major benchmarks against 6 baselines and observe a significant query-time latency improvement up to 80.7x (28.8x on average) relative to the best baseline, while exceeding state-of-the-art retrieval performance by up to 7.8% NDCG across all benchmarks.