Author here. I'm a Tech Lead at Georgian (a growth-stage VC).
Short version: I used a few retrieval benchmark datasets, and experimented with a 21 embedding model leaderboard. I held the benchmark corpus and relevance labels fixed, and transformed the queries: paraphrase, terse, verbose. I kept the “information need” the same under each query transformation, but the leaderboard order changed more than I expected.
I reproduced published benchmark scores for benchmark dataset + embedding model pairs, and then used that same setup with transformed queries as the input. I observed the reordering under each rewrite style to be consistent across 4 independently-generated rewrites (gemini-2.5-pro, claude-opus-4.8, gpt-5.1, llama-3.3-70b) rather than showing up in just one.
Let me know your thoughts on this … or in general let me know about any experience with putting an embedding model into a production retrieval system and being surprised by poor performance.
I hadn’t thought about this before, but when should you decide to optimize your search for agents vs keep it optimized for human queries? Should you provide each a different endpoint altogether?
Probably yes - and I think my results suggest that optimizing for each would not only mean different endpoints but also different embedding models behind each. One model may measure best for short human-authored queries, while a different model may provide better results for longer LLM-authored query forms.
Author here. I'm a Tech Lead at Georgian (a growth-stage VC).
Short version: I used a few retrieval benchmark datasets, and experimented with a 21 embedding model leaderboard. I held the benchmark corpus and relevance labels fixed, and transformed the queries: paraphrase, terse, verbose. I kept the “information need” the same under each query transformation, but the leaderboard order changed more than I expected.
I reproduced published benchmark scores for benchmark dataset + embedding model pairs, and then used that same setup with transformed queries as the input. I observed the reordering under each rewrite style to be consistent across 4 independently-generated rewrites (gemini-2.5-pro, claude-opus-4.8, gpt-5.1, llama-3.3-70b) rather than showing up in just one.
Code and query score files are here so you can check the tables without a full re-run: https://github.com/georgian-io/query_form_experiment
Let me know your thoughts on this … or in general let me know about any experience with putting an embedding model into a production retrieval system and being surprised by poor performance.
I hadn’t thought about this before, but when should you decide to optimize your search for agents vs keep it optimized for human queries? Should you provide each a different endpoint altogether?
Probably yes - and I think my results suggest that optimizing for each would not only mean different endpoints but also different embedding models behind each. One model may measure best for short human-authored queries, while a different model may provide better results for longer LLM-authored query forms.