Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?

Abstract

Recent multilingual vision–language encoders cover hundreds of languages in a single model, yet on two state-of-the-art instances retrieval on low-resource languages (LRL; e.g. Swahili) trails high-resource ones (HRL; e.g. English) by 30⁺ pp. We ask where in the trained encoder this gap is located. Prior modality-gap and cross-lingual subspace work suggests a linear language direction at the output crowds out alignment-relevant geometry. We falsify this: LEACE drives the linear language classifier from >99% to near chance and iterated INLP to 37–50% while LRL retrieval moves within ±1.5 pp and all tier means within 2.2 pp, tracking random controls. The linear bias is a symptom, not the cause. Instead, the alignment-causal factor lies along the encoder's forward path: the EOS (end-of-sequence) hidden state's per-language trajectory diverges with depth. Substituting the EOS with its parallel English value three blocks before the projector lifts Swahili from 22.1% to 69.1% on one encoder (and reproduces on the other); three controls rule out pooled-position tautology and English specificity. A front-layer trunk that pulls each language's projection toward the parallel-content centroid corroborates the diagnosis at training time, recovering +9.6 / +17.1 pp on LRL XM3600 retrieval (1,000-image subset), with consistent gains across three further benchmarks while preserving HRL performance.

Key numbers

The question

MetaCLIP-2 and SigLIP-2 cover hundreds of languages in one dual encoder, but image–text retrieval still shows a 30⁺ pp Recall@1 gap between high-resource languages (HRL-6: en, fr, de, es, zh, ko) and low-resource ones (LRL-5: bn, fil, hi, sw, te), reproduced on XM3600, Flickr30k-200, XTD-200 and Babel-ImageNet. Benchmark-level analyses report that the gap exists; they do not say where along the encoder's forward path the representations diverge. We localise it by intervention.

1. The linear language direction is a symptom, not the cause

Language identity is trivially linearly extractable from the output text embedding: a 13-way logistic-regression probe reaches 99.8% on MetaCLIP-2 and 99.7% on SigLIP-2 (chance 7.7%). Averaging the six HRL embeddings of a parallel caption also lifts retrieval by +17.4 pp. The natural hypothesis is that this direction crowds out alignment-relevant geometry.

It does not. LEACE erases the direction to chance and retrieval barely moves — and moves exactly as much as a same-rank random orthogonal ablation. Iterated INLP up to rank 128 attenuates the classifier to 37–50% with the same null result.

LEACE drops the language classifier from ~99.8% to 3–5%, yet no cell moves more than 1.5 pp and LEACE is indistinguishable from a random projection of the same rank. A linearly visible feature is not, by itself, evidence of a causal one.

2. The causal factor is the EOS forward-path trajectory

We probe the forward path with single-row activation patching. At layer ℓ, the target caption's EOS hidden state is overwritten with its parallel English value, every other token row is kept, and the frozen back-half runs unchanged. Retrieval is then scored against the fixed image gallery.

We headline ℓ = N−4, which still passes through three unmodified back-half blocks. LRL retrieval rises from 44.1% to 73.4% (+29.3 pp) on MetaCLIP-2 and from 21.7% to 66.6% on SigLIP-2 — on a causal-attention and a bidirectional-attention encoder alike. Three controls separate this from a pooled-position tautology:

• Mid-token patch. Patching a non-EOS row at ℓ = N−1 leaves LRL retrieval essentially unchanged — the effect is specific to the pooled row.
• Random-English EOS. An English EOS from an unrelated caption collapses retrieval below baseline (44.1 → 7.7 on MetaCLIP-2). The rescue needs parallel content, not an English-distributed vector.
• Norm-matched Gaussian. Destroys retrieval entirely (HRL ~80 → 8.5%, LRL → 1.6%), so the effect is signal-specific rather than norm-driven.

The rescue is also not English-specific: a parallel French EOS lifts Swahili to 78.7%, and the mean of the six HRL EOS states lifts it to 88.0% — above the English single-source reference. That is what a trajectory account predicts: a multi-HRL centroid sits closer to the interior of the parallel-content cone than any single EOS.

3. Trunk calibration: the same diagnosis, at training time

Patching needs an oracle parallel caption at inference. As a general-input test of the same diagnosis, we soft fine-tune only the first M text-encoder blocks (M=4 for MetaCLIP-2, M=3 for SigLIP-2) so each language's projected output moves toward the parallel-content anchor, while the back-half, projection head and vision tower stay frozen. Training uses text-only parallel-translation groups over a 1M-caption CC12M subset; ~50M / ~40M parameters are trainable.

Per-language XM3600 retrieval

Image-to-text R@1 (%) on the 1,000-image XM3600 subset, frozen vs. trunk-calibrated. Bold marks a lift ≥ +5 pp. HRL performance is preserved or slightly improved throughout; SigLIP-2's weak German baseline also rises +10.1 pp.

Cross-benchmark replication

Trunk-calibration Δ in tier-mean performance (pp). Metric is i2t R@1 for the retrieval benchmarks and top-1 accuracy for Babel-ImageNet. Every cell is positive, so the gain is not XM3600 overfitting.

4. What calibration does to the geometry

Per-layer cross-lingual CKA on 1,000 image-aligned XM3600 captions shows the predicted reshaping. In the frozen encoder, the HRL–LRL curve sits about 0.10–0.12 below the within-HRL curve near the projector input — the geometric counterpart of the retrieval tier gap. After calibration, HRL–LRL CKA rises from 0.458 to 0.556 at ℓ=23 on MetaCLIP-2 and from 0.302 to 0.451 at ℓ=26 on SigLIP-2.

The trajectory divergence itself falls at every depth, with the tier gap shrinking from 0.119 to 0.022 on MetaCLIP-2 and from 0.143 to 0.045 on SigLIP-2. Across the ten non-English trained languages, that same frozen divergence correlates negatively with a WebLI-100B pretraining prevalence proxy (Spearman −0.88 / −0.64) — corroborating context, not a second claim, at n=10.

5. It is not specific to the MetaCLIP-2 / SigLIP-2 pair

We replicate the frozen gap, pooler-row patching, and the same trunk recipe on three architecturally distinct encoders that differ in pooling regime, text-tower scale, and pretraining objective: AltCLIP (XLM-R-Large, CLS-pooled, 24L), NLLB-CLIP-L (NLLB-200 encoder, language-code-pooled, 24L), and mSigLIP (SigLIP-base, sticky-EOS, 12L).

The HRL > LRL gap holds across every pooling regime and text-tower family tested. AltCLIP's large lifts follow from its very weak frozen LRL baseline; NLLB-CLIP-L's smaller lifts follow from its MT-pretrained high baseline — yet a 16.8 pp gap persists even there.

Takeaway

Linearly extractable language identity at the output and the alignment-causal mechanism are different phenomena. Concept erasure removes the identity direction with little change to retrieval; the causal factor is the per-language forward-path trajectory of the pooled hidden state. For multilingual dual-encoder design, this argues for intervening on the trajectory rather than the output embedding — per-language drift is already expressed at the depth-M EOS hidden state, which makes the front blocks a natural correction site while the back-half stays frozen.

These measurements also form a retraining-free diagnostic checklist for new multilingual vision-language encoder releases: per-layer cross-lingual CKA, LEACE on the output embedding, EOS-patch sweeps at ℓ ∈ {N−4, N−1} with parallel-EN / random-EN / mid-token controls, and per-language back-half sensitivity.

Scope. What we establish is an interventional localisation within trained encoders, not a complete causal account of why the divergence arises during pretraining. Tokenisation, attention regime, optimisation dynamics and data prevalence all remain live upstream candidates. Trunk calibration also narrows but does not close the tier gap, and it requires parallel-text supervision that passes an MT-quality screen.

Citation

@inproceedings{han2026where,
  title     = {Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?},
  author    = {Han, Donghoon and Moon, SungHyun and Zhakatayev, Aidyn and Cha, Junghun and Lee, SeungJae},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026},
  note      = {To appear. Preprint: arXiv:2608.30725}
}