You CAN Teach an Old Model New Tricks Domain Adaptation via Complementary Subspace Expansion

Abstract

Production retrieval requires three properties jointly: domain adaptation, general-retrieval preservation, and backward compatibility with deployed indices. Existing methods achieve at most two: PEFT breaks the index and degrades general retrieval; BCT preserves the index only approximately; L2C preserves general retrieval but still modifies the deployed embedding. We introduce CoSE (Complementary Subspace Expansion), a side-branch LoRA whose frozen pathway is structurally untouched: legacy z_old entries are never rewritten, and a jointly-trained z_new channel is appended alongside them. At α = 1 the deployed score recovers the frozen ranking exactly; for α < 1, domain gains require computing z_new per document but the legacy index stays valid throughout migration. A three-term joint loss converts the frozen+LoRA concatenation into Pareto-efficient retrieval. Across four domains (Korean broadcast, medical radiology, art paintings, remote sensing) and 3 seeds, CoSE at 6.4M trainable parameters stays within ±0.048 of frozen on Flickr/COCO/ImageNet at α = 0.5 and is domain-competitive at val-tuned α* — the only configuration we evaluate that simultaneously delivers structural BC, general retention, and competitive domain quality from a single checkpoint.

Key numbers

The problem: the index is the expensive part

In multimodal retrieval, the index built from a deployed embedding model is often the largest asset in the serving stack — re-embedding a billion-document corpus can take weeks of GPU time. Adapting that model to a new domain therefore carries a second cost beyond fine-tuning: index migration. Current methods force a trade-off.

Qualitative positioning at 6.4M trainable parameters (✓✓ strong, ✓ acceptable, ≈ partial, ✗ fails). CoSE is the only configuration whose score family contains an exact frozen-ranking endpoint while still moving toward domain quality.

1. Why expansion is necessary

Two short results bound the design space. First, exact compatibility within the deployed d-dimensional channel is trivial: if an adapted query encoder reproduces every score against a full-rank legacy index, it must equal the frozen encoder. Any non-trivial adaptation that keeps legacy scores exact has to expand the score channel.

Second, a direct-sum normal form: any expanded representation from which the frozen embedding can be read out by a fixed linear projection is, up to a change of coordinates, [ zold ; h(x) ]. CoSE's concatenation is therefore not an ad hoc trick — it is the canonical shape of auditable backward compatibility.

2. Side-branch LoRA architecture

Because the first d coordinates are produced by a forward pass identical to the base encoder, every score against a legacy document is exactly the frozen-model score (index preservation by construction). On Qwen3-VL-Embedding-2B (L = 28, d = 2048, rank 16) the side branch has 6.4M trainable parameters, 0.30% of the base.

At inference, a single scalar blends the two channels:

s_α(q, d) = α · ⟨z_old(q), z_old(d)⟩ + (1 − α) · ⟨z_new(q), z_new(d)⟩

For every α ∈ [0, 1] this is a positive-semidefinite kernel, so α indexes a family of valid retrieval kernels rather than a heuristic interpolation. α = 1 is the frozen model exactly; α = 0.5 is the unweighted concatenation and a zero-tune default; an operator can pick α* with a 1-D sweep on a small validation set — no retraining.

Training uses three terms, each anchoring one operating point: InfoNCE on z_final (the α = 0.5 blend), InfoNCE on z_new alone (the α = 0 endpoint), and a Matryoshka loss so z_new can be truncated for cheaper storage. The α = 1 endpoint is anchored for free by the frozen pretraining.

3. Results across four domains

We adapt Qwen3-VL-Embedding-2B to Korean broadcast/cultural imagery (HAN extended), medical radiology (ROCO), art paintings (SemArt) and remote sensing (RSICD), and track general retrieval on Flickr30K, COCO and ImageNet zero-shot. All learned methods use 6.4M trainable parameters; numbers are 3-seed means on the test split.

Domain R@1 is avg(i2t, t2i). Migration R@1 queries with the adapted encoder against a legacy frozen index (for CoSE, the expanded index, with α = 1 guaranteeing exact frozen recovery). General columns are ImageNet zero-shot top-1 and COCO/Flickr avg R@1. Selected baselines shown; the paper also reports BCT-cosine and a parameter-matched L2C on every domain.

CoSE is domain-competitive, not domain-SOTA: at α* it trails the per-domain leader by 0.012–0.027 on three domains (within noise) and by 0.072 on ROCO. What it buys is a distinct frontier point. LoRA's domain peaks cost up to 0.363 ImageNet top-1; CoSE at α = 0.5 stays within ±0.048 of frozen on every general cell. And for shifted single-encoder baselines, querying a legacy index captures only 27–61% of their self-gain, because query and corpus encoders drift apart — CoSE's migration column equals its domain column by construction.

4. One checkpoint, a whole operating curve

LoRA, BCT and L2C each produce one point; CoSE exposes the full curve at deployment time. The same checkpoint also supports partial branching (sharing the first K layers cuts forward overhead from 2× to about 1.5× at K = L/2 for ≤ 0.04 R@1 on three of four domains) and Matryoshka truncation of z_new for cheaper storage. The legacy z_old slice is preserved in every variant.

5. Why train jointly?

A natural baseline is to ensemble the frozen model's score with a separately trained LoRA. It does not work as a substitute: the val-tuned ensemble weight collapses to α ≤ 0.05 on three of four domains, essentially recovering LoRA along with its general drift. A standard LoRA is a low-rank perturbation of the base (cos ≈ 0.85), so the two branches make the same mistakes and averaging is redundant.

The standalone InfoNCE on z_new is what makes CoSE different. It forces the side channel to be retrieval-valid on its own, which emergently places z_new at cos ≈ 0.5 to z_old — partially decorrelated, so blending behaves like an ensemble. Removing that term (HAN extended, seed 0) collapses α = 0.1 retrieval from 0.489 to 0.185 and z_old/z_new overlap@1 from 26.0% to 2.4%.

Takeaway

When existing indices are large and re-indexing is costly, CoSE offers a backward-compatible route to domain adaptation that keeps general retrieval intact. The base encoder is never modified, α = 1 recovers the legacy ranking exactly, and α < 1 adds a complementary domain score without rewriting old coordinates — so migration becomes a gradual α-sweep instead of a hard cutover.

Scope. CoSE does not eliminate re-embedding: for α < 1, z_new must still be computed per document. Full branching costs 2× forward compute and 2× embedding width, reducible but not to zero. Each run specializes z_new to one domain, and all experiments use Qwen3-VL-Embedding as the base encoder. Where latency, storage, or pure peak-domain accuracy dominate, single-embedding methods are cheaper.

Citation

@inproceedings{han2026cose,
  title     = {You CAN Teach an Old Model New Tricks: Domain Adaptation via Complementary Subspace Expansion},
  author    = {Han, Donghoon and Moon, SungHyun and Lee, SeungJae},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026},
  note      = {To appear}
}