Visual Words Meet BM25 Sparse Auto-encoder Visual Word Scoring for Image Retrieval

Abstract

Dense image retrieval is accurate but opaque and compute-intensive at scale. We present BM25-V, which applies Okapi BM25 scoring to sparse visual word activations from a Sparse Autoencoder (SAE) on Vision Transformer patch tokens. Visual word frequencies follow a Zipfian distribution across the gallery, making BM25's IDF weighting the principled scoring choice for suppressing pervasive, uninformative visual words and amplifying rare, discriminative ones. BM25-V retrieves high-recall candidates via sparse inverted-index operations at a fraction of the cost of dense search: Recall@200 ≥ 0.993 across all benchmarks, enabling a two-stage pipeline that reranks only K=200 candidates instead of all N, recovering near-exact accuracy (−0.2 pp average across seven benchmarks). An SAE trained once on ImageNet-1K transfers zero-shot to seven fine-grained benchmarks without fine-tuning, and every retrieval decision is attributable to specific visual words with quantified IDF contributions.

Key results

Method overview

BM25-V bridges text retrieval and visual search through three ideas:

1. Sparse visual words. A frozen SigLIP2 ViT extracts 729 patch-level features (1152-dim each). A Sparse Autoencoder projects each patch into 18,432 dims and applies ReLU + top-k sparsification, keeping only k=16 active visual words per patch.
2. Term frequency via sum-pooling. Sparse patch vectors are sum-pooled across all patches into one image-level vector; activation magnitude serves as TF — visual words that fire across many patches get higher TF.
3. BM25 scoring with IDF. IDF is computed from the gallery, down-weighting pervasive background features and amplifying rare discriminative ones. Scoring follows the standard Okapi BM25 formula via sparse matrix multiplication on inverted posting lists.

The two-stage pipeline combines BM25-V (fast sparse first stage, top-200 candidates) with dense cosine reranking, recovering near-exact dense accuracy while providing interpretable attribution.

Main results

Cross-domain retrieval: SAE trained on ImageNet-1K, applied zero-shot to all target datasets. The two-stage system matches full dense retrieval within rounding (−0.2 pp average R@1).

† Dense-only methods (no interpretability). R@1 reported. Two-stage exceeds full dense on Flowers and DTD.

Zipfian distribution of visual words

Visual word frequencies follow a power-law (Zipfian) distribution with exponents α ∈ [1.20, 2.32] — steeper than natural language (α ≈ 1). Most visual words are rare and discriminative, while a small head of common words acts as visual stop words. BM25's IDF weighting is the principled response: it suppresses the common head and amplifies the discriminative tail.

Interpretable retrieval attribution

BM25-V is interpretable by construction: the score for any (query, retrieved) pair decomposes as a sum of IDF-weighted terms, one per shared visual word. Unlike dense retrieval where the match is an opaque inner product, BM25-V attribution is exact and requires no approximation.

Cars-196

CUB-200-2011

Describable Textures (DTD)

Visual word semantics

Each SAE dimension encodes a specific visual concept. For high-IDF dimensions, collecting the strongest-activating crops from the reference set reveals visually coherent patterns — confirming monosemantic encoding.

Citation

@article{han2026bm25v,
  title   = {Visual Words Meet BM25: Sparse Auto-encoder Visual Word Scoring for Image Retrieval},
  author  = {Han, Donghoon and Park, Eunhwan and Seo, Seunghyeon},
  year    = {2026},
  note    = {Preprint}
}