About
Who likes Patience? In LIMIT, two people do. Our system also returned Patience Schmiel and Patience Moloney, whose names contain the word and who like something else entirely.
The last post showed IRL reaching 98.35% Recall@2 on LIMIT, almost all of it from the lex clause. The question here: where does a rewritten query with phrase matching go wrong, and where does it beat plain keyword search?
We looked at the ranked results for all 1,000 queries. The numbers below come from our reading of the result files, so they may differ from the metric scripts by a query or two.
What happens before the query runs
The model’s only job is to return a keyword for the question. Ordinary code builds the query around it. For “Who likes Birch Trees?” the finished query, in the notation we have used so far, is:
(or (knn "Who likes Birch Trees?") (lex "Birch Trees"))
The knn clause keeps the question exactly as the user wrote it, because that text goes to the embedding model. The lex clause holds only the keyword, and only the keyword is lowercased. Between the model’s output and OpenSearch, five things happen to the keyword:
- Extraction. The prompt asks the model for the keyword lowercased and hyphenated, in a fixed form, but the code does not rely on that: step 2 enforces it. Code pulls the keyword out of the output, and if the output doesn’t match, it tries any parenthesized group, then the first line, then a stopword filter on the original question.
- Sanitizing. The keyword is lowercased, quotes and punctuation are stripped, and words are joined with hyphens. Apostrophes inside a word are kept.
- Repair against the original. The keyword is realigned onto the question’s own tokens, which restores apostrophes and word breaks the model dropped. Tokens follow the same rules as OpenSearch’s standard analyzer.
- Parsing. A small grammar reads the expression. It rejects anything with other than exactly one
knnclause, and the error quotes the offending rewrite. - Phrase building. In phrase mode, a multi-word keyword such as
apple-juicebecomes a singlematch_phraseon “apple juice”, which requires the words to be adjacent and in order. The clause runs against the document’s text field, lowercased.
Two details matter in what follows. Hot Chocolate works because the joined keyword becomes one adjacent phrase. And the clause searches one field that holds the whole profile, name included, which is where the failures start.
Where phrase matching wins: Hot Chocolate
For “Who likes Hot Chocolate?”, BM25 searches the whole question. Its top result, Evelina Licitra, scored 5.51. Her profile contains “hot” twice and “chocolate” twice, in items like Hot Springs and Dark Chocolate, and never the phrase “Hot Chocolate”. The next five results also contain both words, and none of the six contains the phrase. Neither relevant document appears in BM25’s top 100.
IRL’s lex clause searches for the phrase “Hot Chocolate”. It returned two documents, and both are relevant. BM25 scores the two words separately, so profiles that hold both words in different items outrank the ones with the phrase. Only the adjacency picks out the right two.
What the misses look like
In 24 of the 1,000 queries, at least one of IRL’s top two results was wrong. They fall into three groups.
| Cause | Queries | Example |
|---|---|---|
| The word is in a person’s name | 15 | “Who likes Patience?” returns Patience Schmiel |
| A longer item contains the phrase, and the results stop at 100 | 6 | “Who likes Forests?” returns profiles with “Mangrove Forests” |
| A longer item contains the phrase and outranks the right one | 3 | “Who likes Delight?” returns a profile with “Turkish Delight” first |
Names sit in the same field
LIMIT’s documents begin “Name likes …”, and the name sits in the same text field as the items. A phrase match on “Patience” cannot tell a person from a preference. The other 14 are Risk, Sage, Love, Ivy, Muenster, Windows, Socks, Sanders, Sailing, Jade, Calamari, Barley, Rosemary and Gouda. Each returned a profile whose owner is called that, such as Mathilda Risk or Burl Gouda. In 12 of the 15 queries one of the top two was still a correct document. In the other three, such as Patience, both top results were people with that name.
Longer items hide the short one
In six queries neither relevant document made the top two. The phrase appears inside a longer item that many profiles share:
| Query | What filled the 100 results |
|---|---|
| Who likes Forests? | Mangrove Forests (100 of 100) |
| Who likes Venus? | Venus Flytraps (100 of 100) |
| Who likes Clouds? | Fluffy Clouds (100 of 100) |
| Who likes Honey? | Honey Bees (100 of 100) |
| Who likes Snow? | Snow Leopards and Making Snow Angels (100 of 100) |
| Who likes Apple Cider? | Apple Cider Vinegar (99 of 100) |
For Forests, the scores run from 3.181 at rank 1 to 3.108 at rank 100, so the documents are close to tied. The system returns 100 results, and the two relevant profiles, which hold only “Forests”, did not make the cut. For three other queries, a longer item ranked above the right one, as with “Zip Ties” and “Bow Ties” for “Ties”.
Ideas we haven’t tried
- Separate the name from the items. Indexing the name as its own field would remove the 15 name collisions by construction. It is a property of LIMIT’s layout, and many real corpora have the same shape, such as author names alongside article text.
- Prefer an exact item to a longer one. An exact-item match would need to outrank a phrase buried in a longer item, which plain phrase scoring does not do.
- Return more than 100 results. For the six worst queries this could recover the documents if the list were long enough, at the cost of a longer list.
Limitations
This is one dataset. The failure causes come from LIMIT’s synthetic names and item lists, and a real corpus will have different ones. The knn clause did not help on any of these queries, and we haven’t tested it on a query where the lexical clause fails.
The authors of the paper behind our SEAL and MINDER runs, Bracher and Vakulenko, built a harder variant called LIMIT-H. It adds a near-duplicate negative for each relevant document, with overlapping words and a different meaning, and BM25 and generative retrieval both drop sharply on it. The misses above look like the same kind of problem on a smaller scale. We haven’t run IRL on LIMIT-H, so we can’t say how it does there.
What is next
The rewrite step costs 2.6 seconds per query and does one job here: pull the entity out of the question. The next post asks whether a language model is needed for that.