Hi All, I'm cofounder and CEO of Quissly, author of the research piece published here.
We tested 5 different search engines for ecommerce, and given we win across almost all benchmarks presented, you should read it with a grain of salt. We ran 1259 queries and 99,168 LLM judgements to get to the results presented in the report. Everything is reproducible from scratch, but we didn't use T=0 so you might not get identical results for the parts where LLMs where used, and you will need your own API keys for specific Engines.
While we used LLM as a judge with the hope of staying objective, we rejudged everything with Gemini 3.5 (it only became available when we were at the finish line of there search) and the overlap of scoring was 86.1%, so we ended up using Gemini 3.5 as the main judge in the report as it was the latest model. To further eliminate any doubt, we also used Claude Sonnet as an alternative judge, and got overlap of 82.2% and kappa of .745. The winner and ordering basically held up consistently across all judges, and for anyone, who wants to redo it, note that it cost us $353 to do rejudging, although it finished under 2hrs - all the code needed for this is in the repo.
There are a few metrics, where we lost against our competitors and all of that is also reflected in our blog post, research report, and the interactive dashboard available on our website - quissly.com/benchmarks/search . We lose once to Luigi's Box in nDCG, we lose once to Algolia in generating junk search results, and we are within statistical noise at 7 cells against Doofinder in the zero results comparison.
I don't see Google Cloud's AI Commerce Search in the benchmarks?
It used to be called Vertex AI Search for Commerce (leave it to Google to rename a product every year...) It's clunky, but it's usually the winner in every AB test I've seen when measuring for revenue per visitor on large catalog ecommerce sites.
Hi All, I'm cofounder and CEO of Quissly, author of the research piece published here.
We tested 5 different search engines for ecommerce, and given we win across almost all benchmarks presented, you should read it with a grain of salt. We ran 1259 queries and 99,168 LLM judgements to get to the results presented in the report. Everything is reproducible from scratch, but we didn't use T=0 so you might not get identical results for the parts where LLMs where used, and you will need your own API keys for specific Engines.
While we used LLM as a judge with the hope of staying objective, we rejudged everything with Gemini 3.5 (it only became available when we were at the finish line of there search) and the overlap of scoring was 86.1%, so we ended up using Gemini 3.5 as the main judge in the report as it was the latest model. To further eliminate any doubt, we also used Claude Sonnet as an alternative judge, and got overlap of 82.2% and kappa of .745. The winner and ordering basically held up consistently across all judges, and for anyone, who wants to redo it, note that it cost us $353 to do rejudging, although it finished under 2hrs - all the code needed for this is in the repo.
There are a few metrics, where we lost against our competitors and all of that is also reflected in our blog post, research report, and the interactive dashboard available on our website - quissly.com/benchmarks/search . We lose once to Luigi's Box in nDCG, we lose once to Algolia in generating junk search results, and we are within statistical noise at 7 cells against Doofinder in the zero results comparison.
Happy to answer any questions.
I don't see Google Cloud's AI Commerce Search in the benchmarks?
It used to be called Vertex AI Search for Commerce (leave it to Google to rename a product every year...) It's clunky, but it's usually the winner in every AB test I've seen when measuring for revenue per visitor on large catalog ecommerce sites.
what the judge disagreement rate looked like between runs?