I've been building embench.ai for a while. It started as an internal tool to compare embedding models on our own taxonomy/data for clustering and matching problems. I enhanced it into a playground where I can try different search and retrieval methods on the same corpus. I was more curious to see why something fails than getting a final score.
Currently in the playground you can:
- Compare embedding models / similarity metrics on your texts
- Compare rerankers
- Embed & Search: semantic vs keyword/BM25/grep vs hybrid (and related modes) on a pasted corpus, with optional labeled relevance so you can inspect precision/recall/MRR
It's still deliberately pretty manual for now.
What I'm trying to figure out next: how is agentic search going to evolve and evaluate before putting something in production and measure when it is. Detect drift, and iterate — eventually beyond text (images/video), though I haven't gone multimodal yet.
Would love to get feedback on this, if this is useful especially for the ones building search agents or "deterministic" retrieval stacks on proprietary data/taxonomy?
Or anyone building agents seeing this as a tool you can outsource the measurements to so just have your coding agent cook the recipe.
Thanks!
I've been building embench.ai for a while. It started as an internal tool to compare embedding models on our own taxonomy/data for clustering and matching problems. I enhanced it into a playground where I can try different search and retrieval methods on the same corpus. I was more curious to see why something fails than getting a final score.
Currently in the playground you can: - Compare embedding models / similarity metrics on your texts - Compare rerankers - Embed & Search: semantic vs keyword/BM25/grep vs hybrid (and related modes) on a pasted corpus, with optional labeled relevance so you can inspect precision/recall/MRR
It's still deliberately pretty manual for now.
What I'm trying to figure out next: how is agentic search going to evolve and evaluate before putting something in production and measure when it is. Detect drift, and iterate — eventually beyond text (images/video), though I haven't gone multimodal yet.
Would love to get feedback on this, if this is useful especially for the ones building search agents or "deterministic" retrieval stacks on proprietary data/taxonomy? Or anyone building agents seeing this as a tool you can outsource the measurements to so just have your coding agent cook the recipe. Thanks!