Inside IRL: Turning Questions into Search Queries

About

Given “Who likes Birch Trees?”, a small language model we run writes (or (knn "Who likes Birch Trees?") (lex "Birch Trees")). A parser turns that into an OpenSearch request, and the engine does the retrieving.

The previous post showed that single vectors cannot separate fine pairings and that plain BM25 is a hard baseline. The question here: should a model return the documents itself, or turn the question into a query that the search engine runs? We compare both on the same benchmark, with the same hardware and cost method as before.

The short answer on LIMIT: rewriting each question into a structured query for OpenSearch scored 98.35% Recall@2, and almost all of that comes from one of the query’s two clauses.

Two ways to use a model in search

Generative retrieval, such as SEAL and MINDER, replaces the index lookup with a model. It takes the query and generates words or phrases that exist in the corpus, and an FM-index scores the documents that contain them. An FM-index is a compressed index over the whole corpus text that can find every occurrence of a phrase without scanning the documents; section 3.1.1 of Bracher and Vakulenko explains how SEAL uses it, and the Wikipedia article covers the data structure itself.

Query transpilation keeps the search engine in charge. The model reads the user’s question and writes a short query in an intermediate language. A small parser turns that into an OpenSearch request. The model never sees the documents.

The LIMIT paper does not include either method. Bracher and Vakulenko later evaluated SEAL and MINDER on LIMIT and found that they beat BM25, which is what prompted us to test them. We ran both ourselves on the same machine as everything else in this series, so the numbers below are our runs. They land on the same Recall@2 values that paper reports for the default and beam-search settings (SEAL 0.791 and 0.917, MINDER 0.858 and 0.958).

The language

We call it IRL. It is written as an S-expression, the parenthesized prefix notation used by Lisp: an operator followed by arguments, nested to any depth. The version we tested has four operators:

For “Who likes Birch Trees?” the model writes:

(or (knn "Who likes Birch Trees?") (lex "Birch Trees"))

The knn clause keeps the full question for the embedding model. The lex clause keeps only the entity, so the keyword side ignores “who” and “likes”.

We chose S-expressions because the grammar is tiny and easy to parse, and an unbalanced parenthesis is easy to detect. We did not compare against JSON or YAML output from the same model, so we make no claim that this syntax beats those.

The pipeline

rewrite -> parse -> embed -> codegen -> opensearch -> serialize

1. Rewrite. Qwen2.5-1.5B-Instruct turns the question into an IRL string.
2. Parse. The string becomes a tree.
3. Embed. Qwen3-Embedding-0.6B embeds each knn text. The model never writes vectors.
4. Codegen. The tree becomes OpenSearch query JSON.
5. OpenSearch. The engine runs it.
6. Serialize. Results are written out.

We timed each stage over 1,000 queries:

The model call is 98% of the time. OpenSearch took 3 milliseconds. Compare that with the 60 seconds per query that OpenSearch spent on radial late-interaction search in the last post: with a short IRL query, the engine is the cheap part.

The rewrite step is the only one worth optimizing.

How it compared

All rows use the same 1,000 LIMIT queries, the same machine (Intel Arc iGPU, 16-thread CPU, 62 GB RAM) and the cost method from the last post. Recall@2 is the share of the two relevant documents found in the top two.

IRL scored the highest Recall@2, 98.35%, with a MAP of 0.9869. It is not the fastest: BM25 is more than a hundred times quicker, and both beam-search variants of SEAL and MINDER answered faster than our rewrite step. At Recall@10, SEAL (0.9985) is slightly ahead of IRL (0.994), so if you only need the right documents somewhere in ten results, the gap closes.

The closest rival is MINDER with beam search at 0.958 Recall@2, about 2.5 points lower. The same paper reports 0.988 for an oracle MINDER setting, where each relevant document is indexed with the exact test queries it answers. That needs the answer key in advance, so no live system can do it, and we did not run it. It does show how close generative retrieval can get on LIMIT.

Which clause earns the score

The query has two clauses, so we ran each alone on the same 1,000 queries, with the rewrites from the same Qwen2.5 model:

The lexical clause produced the whole result. The knn clause, with its 0.8 cutoff, returned nothing for 997 of 1,000 queries, the same cutoff effect as the Qwen3 run in the last post. Adding it to lex changed no score.

This fits what LIMIT is built to do. Each query turns on one rare phrase, so once the model has pulled out “Birch Trees”, a phrase match finds the right two documents. The gain over BM25 comes from the rewrite: BM25 searches the whole question, including “who” and “likes”, while lex searches only the entity. The knn clause is in the grammar for queries that LIMIT doesn’t contain, and we haven’t tested those.

Building the index

Indexing costs differ as well:

The generative methods built their FM-index in seconds, so on this corpus the cost of keeping them fresh is not the obstacle. Their cost shows up at query time. The knn clause is the expensive part of IRL’s index: without it, IRL needs only a text index and builds in 3.7 seconds.

What this tells us

1. A model that writes a query can match or beat one that retrieves. On LIMIT, 98.35% Recall@2 against 95.8% for the best generative run, using a 1.5B model that never touches the corpus.
2. On LIMIT, the rewrite is what earns the score. Pulling out the entity and phrase matching it beat the whole-question BM25 run by about five points. The dense clause added nothing here.
3. The search engine stays the system of record. Documents are added, deleted and filtered in OpenSearch as usual, and nothing about the corpus lives in model weights.
4. Latency is the rewrite. Everything after the model call took about 62 milliseconds.
5. BM25 remains the cheap option. It gets 92.95% in 21 seconds for under a cent.

Limitations

This is one dataset with one query shape, “Who likes X?”, where the right rewrite is almost always the entity in the question. Real enterprise queries need filters, dates and access control. IRL’s grammar can express those as more operators, and because the transpiler writes the final request, security filters can be added by code instead of by the prompt. We have not built or tested either, so treat them as a direction. The rewrite is also easy on LIMIT, because every query has the same template. Messier questions would make rewrite errors matter more, and we haven’t measured that. For SEAL and MINDER we used the default and beam-search settings from that paper and did not tune them further. The hardware is one small machine.

What is next

The next post looks at the six queries IRL still misses and one case where phrase matching beats BM25.