DH Donghoon Han

EMNLP 2024 Β· Industry Track

MERLIN Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank

* equal contribution

arXiv 2407.12508 Code

Abstract

Users typically issue short, under-specified queries; videos contain rich multimodal context. This mismatch between query and content is a primary driver of retrieval errors, and one that prior work β€” focused on cross-modal interaction, foundation-model scaling, or probabilistic modeling β€” largely ignores. We introduce MERLIN (Multimodal Embedding Refinement via LLM-based Iterative Navigation): a training-free pipeline that uses an LLM and a multimodal LLM to simulate clarifying interactions, gradually refining the query embedding toward the user's intent. On MSR-VTT, MSVD, and ActivityNet, MERLIN delivers substantial gains in Recall@1 over strong retrieval baselines.

The query–video gap

Illustration of the discrepancy between short user queries and rich video content
A short user query rarely captures the full context of the target video. MERLIN closes this gap iteratively, without retraining the underlying retrieval model.

Pipeline

MERLIN architecture: question generation, video QA, aggregation, embedding refinement
MERLIN runs over an off-the-shelf retriever. At each round, an LLM asks a clarifying question grounded in the top-1 video; a multimodal LLM answers from sampled frames; the answer embedding is fused into the query via SLERP.
  1. Question generation. An LLM uses the top-ranked video's metadata to generate a clarifying question β€” eliciting context the user didn't initially provide.
  2. Human-simulating agent. A multimodal LLM answers the question by analyzing frames sampled at 1-second intervals; an aggregation step fuses frame-level answers into a video-level response.
  3. Iterative embedding refinement. The current answer embedding is combined with the running query embedding via SLERP (spherical linear interpolation) with Ξ±=0.8 β€” biasing toward earlier embeddings to prevent query drift.

Main results

Recall@1 / Recall@5 / Recall@10 from round 0 (baseline retriever) to round 5 (after five MERLIN iterations).

Dataset R@1 (R0 β†’ R5) R@5 (R0 β†’ R5) R@10 (R0 β†’ R5)
MSR-VTT 44.40 β†’ 78.00 67.60 β†’ 94.20 76.20 β†’ 96.80
MSVD 52.39 β†’ 77.61 77.16 β†’ 94.48 84.78 β†’ 97.31
ActivityNet 56.58 β†’ 68.44 84.77 β†’ 91.95 91.73 β†’ 96.63

+76% relative R@1 on MSR-VTT, +48% on MSVD, +21% on ActivityNet β€” without training a new model.

Why iteration helps β€” and how it can hurt

Average rank improvement across QA rounds
Average rank of the target video improves monotonically across rounds for Ξ± = 0.8.

The fusion coefficient Ξ± controls how much weight the running embedding keeps versus the new round's answer. At Ξ± = 0.8, gains accumulate stably across rounds. At Ξ± = 0.2, the query is dragged by each new answer β€” initial rounds help, then performance declines as the embedding drifts away from the user's original intent.

Case studies

ActivityNet

ActivityNet retrieval case study

MSVD

MSVD retrieval case study

MSR-VTT

MSR-VTT retrieval case study

Citation

@inproceedings{han2024merlin,
  title     = {MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline},
  author    = {Han, Donghoon and Park, Eunhwan and Lee, Gisang and Lee, Adam and Kwak, Nojun},
  booktitle = {EMNLP 2024 (Industry Track)},
  year      = {2024}
}