Why Vector Search Fails on Fine-Grained Entities

About

Who likes Birch Trees? In a pool of 50,000 short profiles, two people do. Most embedding models can’t reliably find them, and a plain keyword index usually can.

That is the weak spot in the standard RAG recipe: chunk documents, embed each chunk, store the vectors, search with approximate nearest neighbors. It works well for broad questions and stumbles when the answer hinges on one small detail.

This is the first of four posts on an evaluation we ran for enterprise search. The question for this one: why does a single vector lose the detail a query turns on, and how much of the problem is the model versus how we search it? What to do about it comes in the next posts.

A benchmark built to break it

LIMIT (Weller, Boratko, Naim and Lee, of Google DeepMind and Johns Hopkins; ICLR 2026) is deliberately boring. Each document is a person and the things they like. Each query asks who likes one thing, like the Birch Trees question above. One of its two right answers:

“Olinda Posso likes Bagels, Hot Chocolate, Pumpkin Seeds, The Industrial Revolution, …”

The full set has 50,000 documents and 1,000 queries, with two relevant documents per query. Only 46 documents are ever relevant, but they can form 1,035 unique pairs, and the 1,000 queries cover almost all of them. The retriever has to tell nearly every pairing of those same 46 documents apart, not just find the 46.

The paper argues that a fixed-size vector can only produce a limited set of top-k results, even when the vectors are optimized directly against the answers. Across the single-vector models they tested, the best Recall@100 on the full set was 18.9% (Promptriever, 4,096 dimensions), and Recall@2 never passed 3.2%. Their BM25 scored 85.7% Recall@2 (Table 5).

The paper tests models in isolation. We wanted to see how this plays out in a real search engine, so we ran it ourselves.

Our setup

One machine (Intel Arc iGPU, 16-thread CPU, 62 GB RAM), OpenSearch 2.19 with Lucene HNSW, the full 50,000 documents and 1,000 queries. We report Recall@2: with two relevant documents per query, 1.0 means both come back at ranks one and two.

Costs in this post are wall time multiplied by the on-demand price of an AWS c7i.4xlarge, $0.714 an hour. They are estimates from our timings, not a bill.

BM25 first

Before any neural model, plain BM25 scored 92.95% Recall@2. It took 21 seconds for all 1,000 queries, about 0.02 seconds each, and cost about $0.004. The query’s rare word appears in exactly the right documents, so a keyword index does well here.

The paper reports 85.7% for BM25 and we got 92.95%. The analyzer and setup differ, and we didn’t isolate which difference accounts for the gap.

Same index, two searches, very different answers

Late-interaction (ColBERT-style) models keep one vector per token instead of one per document. On this benchmark they do far better than single-vector models, but only if you search them the right way.

We ran GTE-ModernColBERT against the same index in two ways. Radial search keeps every token vector above min_score: 0.8. Bounded search keeps the top 25 per query token (k: 25, ef_search: 100).

Bounded search is roughly 90 times faster. The smaller mxbai model went from 15 hours 17 minutes to 223 seconds, roughly 250 times faster. Radial search takes about a minute per query, which rules it out for a live product, and by our estimate cost about $11.94 for the 1,000 queries against $0.004 for BM25.

Where does the minute go? We timed each stage:

OpenSearch accounts for 60.15 of the 60.21 seconds. The model is not the slow part. Bounded search spent 0.62 seconds in the same stage.

Why bounded search collapsed

We took the first 200 queries and checked, for each query token, whether its top 25 neighbors contained a relevant document.

Take “Who likes Alligators?”. In the smaller mxbai model, the two relevant people rank first and second for the word pieces “ig” and “ators”, so they collect two token scores and finish first and second overall. In GTE, neither person appears in any of the nine token lists, so both score zero. The common tokens (“who”, “likes”, the punctuation) fill their top 25 with unrelated people at nearly identical scores.

Our reading is that GTE’s higher-dimensional token vectors spread out more and push relevant matches past the cutoff. That is a hypothesis from two models, not a general rule.

A threshold can empty the results

One more setting caught us out. We ran Qwen3-Embedding-0.6B, a single-vector model with 1,024 dimensions, in the same OpenSearch cluster with a similarity cutoff of 0.8, using the plain question as the query. For 997 of 1,000 queries, nothing came back. MAP was 0.0000.

The cutoff is part of that result, and we did not run an unthresholded top-k baseline for this model, so it is not a score for Qwen3. It is a warning about cutoffs. One possible cause is that similarity scores for short, similar-looking queries cluster below 0.8, but we haven’t measured the score distribution.

What this tells us

1. A single vector can’t keep every pairing apart. LIMIT shows the cap is structural, not a training gap.
2. The same model can look broken or excellent depending on its search mode. A model’s score without its search mode is incomplete.
3. A keyword index is a hard baseline. 92.95% Recall@2 in 21 seconds for under a cent sets the bar for anything slower.
4. The slow late-interaction run was slow in OpenSearch, not in the model. Time each stage before blaming the model.

Limitations

This is one dataset with one query shape (“who likes X”), built to stress embeddings. Real enterprise queries are messier. We have no clean unthresholded top-k run for the single-vector model, so our own evidence for it is the 0.8-cutoff result above plus the paper’s numbers. The hardware is a single small machine, so absolute latencies will differ elsewhere. We tested a handful of models, not the field.

What is next

The next post asks what happens when a language model doesn’t retrieve at all and instead writes the query for the search engine to run.

Why only 46 documents?

LIMIT needs about 1,000 queries that each point to a different pair of documents. The number of unique pairs you can form from n documents is n × (n − 1) / 2:

• 45 documents give 45 × 44 / 2 = 990 pairs, just short of 1,000 queries.
• 46 documents give 46 × 45 / 2 = 1,035 pairs, the smallest set with enough.

So the authors use 46 relevant documents and vary which two each query should return. Instead of needing thousands of relevant documents, they test whether a retriever can separate almost every pairing of a small, fixed set. The other 49,954 documents are distractors that mention the same kinds of attributes (paper, dataset construction section).