Many dense-retrieval papers I read during my Master's used millions of English query-document pairs, mature tooling, and benchmarks refined over years. My thesis setting had none of those advantages.
I worked on dense retrieval for languages with few training pairs, a thin web corpus, and tooling built around multilingual models. A model accepting the language as input did not guarantee good ranking quality for its queries.
Where the evidence thins out
A multilingual encoder may have seen text in a language without learning a strong representation of relevance in that language. Tokenization can fragment rare words, and limited training data leaves fewer examples that pull related queries and documents together. In retrieval, a document landing near the correct cluster still fails if it appears at rank 40 instead of rank 1.
I spent much of the project building training and evaluation sets because no benchmark provided them. We had to define relevance under a small annotation budget and inspect labels closely; with a small evaluation set, a few mislabeled pairs could move the reported metrics substantially.
I had underestimated the evaluation work. Improving a low-resource retriever required a credible way to judge relevance in the target language, not only a model that could encode its text.
Newer multilingual benchmarks make that work visible. MIRACL covers 18 languages with relevance judgments produced by native speakers. Mr. TyDi showed that multilingual dense retrieval did not beat BM25 uniformly, while hybrid signals helped in some settings. Benchmark construction determines which improvements can be measured.
Relevance study
The same documents, a different first answer
Follow a query into two illustrative rankings, then inspect relevance independently of position.
Scenario
Illustrative rankings of invented English glosses. These are not thesis queries, documents, or model results.
Sequence
- Keep the intentCurrent
- Read the ranked listUpcoming
- Inspect relevance separatelyUpcoming
- Keep metrics distinctUpcoming
Keep the intent
The query asks when a bridge is closed. A shared topic is insufficient: the answer must describe closure.
Read the full explanation
Near misses first
Illustrative rankings of invented English glosses. These are not thesis queries, documents, or model results.
- Keep the intent. The query asks when a bridge is closed. A shared topic is insufficient: the answer must describe closure.
- Read the ranked list. The first two documents share the setting but do not answer the query. The relevant document is third.
- Inspect relevance separately. Read each candidate span against the query. The closure notice answers it; the opening report and weather report do not. In a real language-specific evaluation, keep the original text and record the reviewer’s reason.
- Keep metrics distinct. This one illustrative query only shows the rank of the first relevant document. The article’s 81% retrieval accuracy and MRR of 57.83% are distinct historical evaluation results, not outputs of this example.
Intent match first
Illustrative rankings of invented English glosses. These are not thesis queries, documents, or model results.
- Keep the intent. The query asks when a bridge is closed. A shared topic is insufficient: the answer must describe closure.
- Read the ranked list. The same relevant document now ranks first. Changing the order does not change any relevance judgment.
- Inspect relevance separately. Read each candidate span against the query. The closure notice answers it; the opening report and weather report do not. In a real language-specific evaluation, keep the original text and record the reviewer’s reason.
- Keep metrics distinct. This one illustrative query only shows the rank of the first relevant document. The article’s 81% retrieval accuracy and MRR of 57.83% are distinct historical evaluation results, not outputs of this example.
Near misses first
Illustrative rankings of invented English glosses. These are not thesis queries, documents, or model results.
- Keep the intent. The query asks when a bridge is closed. A shared topic is insufficient: the answer must describe closure.
- Read the ranked list. The first two documents share the setting but do not answer the query. The relevant document is third.
- Inspect relevance separately. Read each candidate span against the query. The closure notice answers it; the opening report and weather report do not. In a real language-specific evaluation, keep the original text and record the reviewer’s reason.
- Keep metrics distinct. This one illustrative query only shows the rank of the first relevant document. The article’s 81% retrieval accuracy and MRR of 57.83% are distinct historical evaluation results, not outputs of this example.
Intent match first
Illustrative rankings of invented English glosses. These are not thesis queries, documents, or model results.
- Keep the intent. The query asks when a bridge is closed. A shared topic is insufficient: the answer must describe closure.
- Read the ranked list. The same relevant document now ranks first. Changing the order does not change any relevance judgment.
- Inspect relevance separately. Read each candidate span against the query. The closure notice answers it; the opening report and weather report do not. In a real language-specific evaluation, keep the original text and record the reviewer’s reason.
- Keep metrics distinct. This one illustrative query only shows the rank of the first relevant document. The article’s 81% retrieval accuracy and MRR of 57.83% are distinct historical evaluation results, not outputs of this example.
Change the ranking. Audit the labels.
Blend two illustrative retrieval signals and inspect how one relevance correction changes this query’s metrics.
Query: “When is the bridge closed?”
Score = 0.75 × lexical + 0.25 × semantic
Bridge opening ceremony
The bridge opened last spring. A public ceremony marked the first crossing.
Original annotation: Opening history does not give a closure time.
The English gloss is invented. A language-specific evaluation must keep the original text and a qualified reviewer’s reason.
Bus service timetable
The last bus crosses the bridge at 21:00 each evening.
Original annotation: A bus schedule is not the bridge closure schedule.
The English gloss is invented. A language-specific evaluation must keep the original text and a qualified reviewer’s reason.
District weather bulletin
Heavy rain is expected near the river bridge tonight.
Original annotation: Nearby weather alone does not establish bridge access.
The English gloss is invented. A language-specific evaluation must keep the original text and a qualified reviewer’s reason.
Road authority closure notice
The bridge will be closed from 18:00 Friday until 06:00 Monday for repairs.
Original annotation: An explicit closure interval answers the query.
The English gloss is invented. A language-specific evaluation must keep the original text and a qualified reviewer’s reason.
Flood access order
The bridge is closed whenever the river gauge exceeds four metres.
Original annotation: A conditional closure rule also answers when access stops.
The English gloss is invented. A language-specific evaluation must keep the original text and a qualified reviewer’s reason.
Five invented English glosses with fixed, illustrative lexical and semantic scores on the same 0–1 scale. The blend is a teaching model, not BM25 or a trained retriever. Real hybrid systems must reconcile score scales. Labels change metric judgments without changing ranking scores. This is one query’s reciprocal rank, not mean reciprocal rank; the article’s 81% accuracy and 57.83% MRR are separate historical measurements.
- MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse LanguagesMIRACL uses native-speaker relevance judgments across 18 languages, making the annotation work part of the benchmark itself.
- Mr. TyDi: A Multi-lingual Benchmark for Dense RetrievalMr. TyDi shows that multilingual dense retrieval did not uniformly beat BM25 across its non-English tasks.
- SWIM-IRRecent direction for low-resource retrieval: synthetic training data across high- to very-low-resource languages when human pairs are scarce.
Where it landed
81% retrieval accuracy and an MRR of 57.83% on our evaluation. I'll resist the urge to dress those up. Against what English-language systems post on mature benchmarks, they're modest. Against the baselines available for the setting, they were real progress. And more importantly, the failure analysis told us where the remaining errors lived, which is worth more than the score itself.
The habit I kept from the project is to inspect the training pairs and relevance judgments before changing the retriever. Those inputs determine what the metric can reveal about a new architecture.
A model can accept text in a language while returning poor search results for the people who use it. Product claims about language coverage should therefore include retrieval quality measured in that language.
I chose the problem because language technology learns from the text available online, and that supply is distributed unevenly across languages. Without deliberate datasets and native-speaker evaluation, poorer retrieval quality becomes the default for speakers whose languages have less text on the web.