The benchmarks page seems interesting and something I can use to help make an informed decision. Can you talk about how you're measuring some of these? I imagine it needs to involve some human input.
Turn-taking specifically does not need a listening panel, but it is measured mechanically. 200+ real human clips, and we score end-vs-wait decisions: did the model decide the caller finished speaking, or just paused mid-thought. The best detector gets 94.0% of those right; a plain VAD silence timer gets 46.9%. Results are published here: https://benchmarks.speko.ai/turntaking
You are right about human input for naturalness, that one we did not automate away with yet. We run blind A/B listening rounds with native speakers.
Does this include a turn taking API? It'd be great to have one API that could do "Conversation in a box". One of the biggest annoyances is daisy chaining many models together for turn taking, dumb models for immediate responses, with smarter models returning and taking over after.
Yes, on the hosted side (agents platform): full sessions come with VAD and turn-taking handled - we set them up and tune them for your use case, so that is the closest thing to conversation in a box. If you run your own orchestration, the gateway is just the routing layer and turn-taking stays in your framework - in our own stack we run Pipecat's Smart Turn in-process and tune the commit threshold on real calls. We also share our benchmarks here: https://benchmarks.speko.ai/turntaking
On fast dumb models answering while a smarter one takes over: we are experimenting with exactly that split - a small fast model holds the conversation while a larger one works behind it. Today it runs as two pinned routes, not one packaged API. Most turns in a phone call do not need a frontier model, and the fastest models on our LLM board are all small, so this is where routing earns its keep. We publish benchmarks on LLMs here: https://benchmarks.speko.ai/llm
The main difference from gateway is we help with picking the right voice stack, which seems to be a big problem for users: we benchmark the models continuously and route based on those measurements for your language and constraints, and the boards are public at https://benchmarks.speko.ai/
Second difference is where it runs. Our gateway is open source and runs in your own container, including with self-hosted livekit/pipecat. You get a temporary token before the session starts, and then your orchestration connects directly to the provider.
Vapi is a managed platform: you use their infra to use the voice AI stack. In our case you can have your own infra and switch between models, so you are not locked into a vendor. A lot of teams we talk to build their own infra as they mature, and that is where the router comes handy.
Which model best allows me to transcribe speech that uses a lot of domain-specific terms? For example, when I say "Claude Code", it often gets transcribed as "Cloud Code", and I have to go back and edit or do a second pass with a traditional LLM (which can introduce additional errors).
I’ve had a lot of success in the past with fine tuning STT using synthetic data.
I was doing it for Veterinary (ambient recording -> SOAP notes) which has tons of complex domain-specific language AND it is critically important to get right.
“CPR” transcribing as “see pee are” just doesn’t cut it in that industry.
Synthetic-data fine-tuning is the other credible answer to domain vocabulary. Curious whether you re-benchmark the fine-tune when new base models ship?
agree with omneity here. Whisper's initial-prompt trick is exactly that, and several hosted vendors have equivalents (custom vocabulary / keyword prompting). Domain vocabulary is where STT models separate the most in our runs. for example, on medical terms the field spreads from about 8% to 19% WER across models: https://benchmarks.speko.ai/blog/what-a-voice-agent-hears.
We often find that models that wins on clean speech are often not the one that wins on your terms, so test with your own vocabulary rather than a headline number.
Open weight models are getting really good. Specko appears to be focussing on paid services which has its role to play but I have a benchmark in development for Open Weight models here if that is helpful: https://orcabot.com/benchmarks/open-weight-tts.
Still shoring up the details but you can try a sample of all the options and see how they compare in terms of model size, peak RSS, real time factor etc.
Just canceled my WisprFlow subscription a few days ago to switch to a open source, free, local alternative. In my case handy.computer with the Cohere model.
cool app, and agreed that on-device keeps eating the single-user cases, we are seeing dictation and translation are exactly where local models shine. We benchmark the open models on the same boards as the hosted ones, but there are still a few: https://benchmarks.speko.ai/open.
production phone agents are a different shape today: the call terminates server-side, three models plus turn-taking under one latency budget, and per-language quality still swings a lot from our tests
Ever since OpenAI launched their improved voice mode, I've been looking for a capable LLM with builtin voice-in and -out. Llama 4 was supposed to be it but turned out to be a dud. I haven't followed the topic closely lately, did I miss anything? Are there capable (!) open weights omni models that allow low latency voice chat? If so, what software do you use them with? Can you use a PWA on your phone? WebRTC? WebTransport?
Actually, we measured exactly this recently. The strongest open-weights speech-to-speech model we have run is NVIDIA's NemotronLabs VoiceChat 11B - no provider serves it, so we hosted it ourselves and ran the same scripted call every model on our board gets. Remarkably stable, 40+ sessions with zero errors - but by turn thirteen it was answering nine tries in ten without saying a word. Stable engine, but degrades on long calls.
Wider picture: we rebuilt the speech-to-speech test around a hard 20-turn concierge call, and today only the closed models finish it cleanly. Board: https://benchmarks.speko.ai/s2s
And jakswa's Gemma datapoint matches what we measure, the fastest rows on our LLM board are the small models like Gemma 4 is performing incredibly well for voice agents
FWIW, I have seen a couple phone apps using react native and sending directly to backends using a socket. Seems to avoid some overhead but I don’t have first hand experience with the client side.
What purrcat259 said, and I think that the page should explain it, we will add a tooltip.
for some languages CER is more relevant than WER. Thai and Mandarin have no word boundaries, so we score them by character, and Japanese gets a reading-based CER.
there is a good progress on on-devise models, but not ready for production yet to fit in devices. But as soon as there is are some good results, we are going to benchmark them and put in https://benchmarks.speko.ai/
> Typical production voice agent is an ensemble of three models: STT, an LLM, and TTS.
To use a claudism, I would like to push back on this. The industry is very much moving towards one-model-does-all end to end trained similar to LLMs and VLMs. Mostly for latency reasons and partially because the results for the end to end trained models are just so much better than those using three pieces architectures.
I think most of the value prop is in automatic evals, not routing specifically. A better pitch for you would be "the LM Arena of voice models" rather than comparing yourself to openrouter because the value add is rather questionable. For TTS specifically, the current SOTA for production systems are all using prompt based voice gen i.e. instead of having 10 different Tacotron models trained on 10 different models, these days it's all a single large model and the "style" is a prompt in the system prompt. The input is usually something like
<System prompt>
Speak in a deep smooth voice similar to a documentary narrator
</System prompt>
<Text to Narrate>
Speko is the ultimate evaluation platform for voice agents. We do automatic evals.
</Text to Narrate>
It's the same for voice cloning too, you just pass the reference speech as an input file for all generations. A lot of systems don't have any separate style vector extraction step or model-specific fine-tuning anymore.
So something like OpenRouter for voices offer questionable value given that stakeholders usually make this sort of decisions once at the start of the project. On the other hand if you can offer automatic evals and figure out which prompts give the most similar results across different voice providers, that would offer a lot more value. It would be nice to be able to switch from e.g. Grok voice agents to ChatGPT voice agents knowing that the output style won't change too much. There are many companies now with evals as a core business model: LM Arena, Artificial Analysis, Prompt foo (before they got acquired and pivoted to security only) so many take a look at them.
Fair pushback. On end to end: we measure those too, same methodology: https://benchmarks.speko.ai/s2s. If the single models win, we route to them the same way, so we do not care which architecture (s2s or cascaded) wins. For now, what we see in production so far is that most teams still want to control each piece: swap the STT for medical vocabulary, keep the LLM, keep the voice.
On "promptfoo of voice models": that is closer to how it started. At my last company we ran these evals manually, we would even hire native-speaking raters, benchmark, switch if it wins. The evals are the value, agreed. The routing is what makes them actionable: teams told us swapping always looked like an R&D project, so scores alone did not change what ran in production.
On prompt-based voice gen and reference-audio cloning: agreed, that is what we see too. It makes continuous measurement more important: the same style prompt behaves differently per language and per content type, so we rank the voices themselves, tagged by use case: https://benchmarks.speko.ai/tts-voices
> The industry is very much moving towards one-model-does-all end to end trained
I've worked with hundreds of enterprises on voice AI and voice agent solutions. In my experience, this isn't true. Or rather I should say, the people actually paying for voice agents (i.e enterprises) are not moving towards STS solutions in a meaningful way. The composability, observability, and reliability profile of STS systems is not amenable to enterprise criteria. Not to mention costs.
The benchmarks page seems interesting and something I can use to help make an informed decision. Can you talk about how you're measuring some of these? I imagine it needs to involve some human input.
https://benchmarks.speko.ai/turntaking
Turn-taking specifically does not need a listening panel, but it is measured mechanically. 200+ real human clips, and we score end-vs-wait decisions: did the model decide the caller finished speaking, or just paused mid-thought. The best detector gets 94.0% of those right; a plain VAD silence timer gets 46.9%. Results are published here: https://benchmarks.speko.ai/turntaking
You are right about human input for naturalness, that one we did not automate away with yet. We run blind A/B listening rounds with native speakers.
Does this include a turn taking API? It'd be great to have one API that could do "Conversation in a box". One of the biggest annoyances is daisy chaining many models together for turn taking, dumb models for immediate responses, with smarter models returning and taking over after.
Yes, on the hosted side (agents platform): full sessions come with VAD and turn-taking handled - we set them up and tune them for your use case, so that is the closest thing to conversation in a box. If you run your own orchestration, the gateway is just the routing layer and turn-taking stays in your framework - in our own stack we run Pipecat's Smart Turn in-process and tune the commit threshold on real calls. We also share our benchmarks here: https://benchmarks.speko.ai/turntaking
On fast dumb models answering while a smarter one takes over: we are experimenting with exactly that split - a small fast model holds the conversation while a larger one works behind it. Today it runs as two pinned routes, not one packaged API. Most turns in a phone call do not need a frontier model, and the fastest models on our LLM board are all small, so this is where routing earns its keep. We publish benchmarks on LLMs here: https://benchmarks.speko.ai/llm
What is the difference with Livekit Gateway? https://livekit.com/blog/introducing-livekit-inference
Or even something more managed like Vapi?
The main difference from gateway is we help with picking the right voice stack, which seems to be a big problem for users: we benchmark the models continuously and route based on those measurements for your language and constraints, and the boards are public at https://benchmarks.speko.ai/
Second difference is where it runs. Our gateway is open source and runs in your own container, including with self-hosted livekit/pipecat. You get a temporary token before the session starts, and then your orchestration connects directly to the provider.
Vapi is a managed platform: you use their infra to use the voice AI stack. In our case you can have your own infra and switch between models, so you are not locked into a vendor. A lot of teams we talk to build their own infra as they mature, and that is where the router comes handy.
Which model best allows me to transcribe speech that uses a lot of domain-specific terms? For example, when I say "Claude Code", it often gets transcribed as "Cloud Code", and I have to go back and edit or do a second pass with a traditional LLM (which can introduce additional errors).
I’ve had a lot of success in the past with fine tuning STT using synthetic data.
I was doing it for Veterinary (ambient recording -> SOAP notes) which has tons of complex domain-specific language AND it is critically important to get right.
“CPR” transcribing as “see pee are” just doesn’t cut it in that industry.
Synthetic-data fine-tuning is the other credible answer to domain vocabulary. Curious whether you re-benchmark the fine-tune when new base models ship?
Which open source STT models have you had success with for fine tuning?
agree with omneity here. Whisper's initial-prompt trick is exactly that, and several hosted vendors have equivalents (custom vocabulary / keyword prompting). Domain vocabulary is where STT models separate the most in our runs. for example, on medical terms the field spreads from about 8% to 19% WER across models: https://benchmarks.speko.ai/blog/what-a-voice-agent-hears.
We often find that models that wins on clean speech are often not the one that wins on your terms, so test with your own vocabulary rather than a headline number.
Good old Whisper allows you to enter a prompt with domain specific terms and it will use them for transcription.
Gemini 3 flash series is quite good, also elevenlabs scribe v2
Seems to be useless, the state of art for all categories is local on-device, voice model vendors are just rent seekers for those who know no better.
Open weight models are getting really good. Specko appears to be focussing on paid services which has its role to play but I have a benchmark in development for Open Weight models here if that is helpful: https://orcabot.com/benchmarks/open-weight-tts.
Still shoring up the details but you can try a sample of all the options and see how they compare in terms of model size, peak RSS, real time factor etc.
You're not the customer. This is for people building products that support thousands of users.
Linux on desktop is great for you, but this is a tool for people delivering solutions.
> You're not the customer. This is for people building products that support thousands of users.
That's their dream. Your dream. The AI dream. Many would say it's AI psychosis.
1) what if I told you I can leverage local models and serve thousands of users?
2) you know nothing about me
3) of course I am not! I do know better
Just canceled my WisprFlow subscription a few days ago to switch to a open source, free, local alternative. In my case handy.computer with the Cohere model.
You mean handy.computer?
Yup, my bad, fixed.
I made a completely free 100% on device translation app https://apps.apple.com/us/app/arda-translate/id6778970560 and hard to image a world where TTS and STT will not be done locally in the future
cool app, and agreed that on-device keeps eating the single-user cases, we are seeing dictation and translation are exactly where local models shine. We benchmark the open models on the same boards as the hosted ones, but there are still a few: https://benchmarks.speko.ai/open.
production phone agents are a different shape today: the call terminates server-side, three models plus turn-taking under one latency budget, and per-language quality still swings a lot from our tests
Ever since OpenAI launched their improved voice mode, I've been looking for a capable LLM with builtin voice-in and -out. Llama 4 was supposed to be it but turned out to be a dud. I haven't followed the topic closely lately, did I miss anything? Are there capable (!) open weights omni models that allow low latency voice chat? If so, what software do you use them with? Can you use a PWA on your phone? WebRTC? WebTransport?
Actually, we measured exactly this recently. The strongest open-weights speech-to-speech model we have run is NVIDIA's NemotronLabs VoiceChat 11B - no provider serves it, so we hosted it ourselves and ran the same scripted call every model on our board gets. Remarkably stable, 40+ sessions with zero errors - but by turn thirteen it was answering nine tries in ten without saying a word. Stable engine, but degrades on long calls.
Wider picture: we rebuilt the speech-to-speech test around a hard 20-turn concierge call, and today only the closed models finish it cleanly. Board: https://benchmarks.speko.ai/s2s
shared some write-up: https://benchmarks.speko.ai/blog/can-s2s-replace-the-cascade
And jakswa's Gemma datapoint matches what we measure, the fastest rows on our LLM board are the small models like Gemma 4 is performing incredibly well for voice agents
FWIW, I have seen a couple phone apps using react native and sending directly to backends using a socket. Seems to avoid some overhead but I don’t have first hand experience with the client side.
Gemma 4 (both E4B + 12B) performed really well as ears+brains. I mostly comment because I too am always scouting for a nice local all-in-one model.
What does WER/CER stand for? I see it as some kind of key metric under each model but not clear what it's measuring
Word Error Rate and Character Error Rate.
Common metrics to track how coarsely or finely accurate voice AI is
What purrcat259 said, and I think that the page should explain it, we will add a tooltip.
for some languages CER is more relevant than WER. Thai and Mandarin have no word boundaries, so we score them by character, and Japanese gets a reading-based CER.
Looks awesome, can't wait to try it for some Filipino workflows when it's available!
thanks! actually, we have the filipino already, can you check out and share your feedback?
the concept is interesting I must say - good luck !
Thank you!
any progress with on-device models??
Most of the open weight models are can be used on device: https://orcabot.com/benchmarks/open-weight-tts
there is a good progress on on-devise models, but not ready for production yet to fit in devices. But as soon as there is are some good results, we are going to benchmark them and put in https://benchmarks.speko.ai/
cool!
> Typical production voice agent is an ensemble of three models: STT, an LLM, and TTS.
To use a claudism, I would like to push back on this. The industry is very much moving towards one-model-does-all end to end trained similar to LLMs and VLMs. Mostly for latency reasons and partially because the results for the end to end trained models are just so much better than those using three pieces architectures.
I think most of the value prop is in automatic evals, not routing specifically. A better pitch for you would be "the LM Arena of voice models" rather than comparing yourself to openrouter because the value add is rather questionable. For TTS specifically, the current SOTA for production systems are all using prompt based voice gen i.e. instead of having 10 different Tacotron models trained on 10 different models, these days it's all a single large model and the "style" is a prompt in the system prompt. The input is usually something like
It's the same for voice cloning too, you just pass the reference speech as an input file for all generations. A lot of systems don't have any separate style vector extraction step or model-specific fine-tuning anymore.So something like OpenRouter for voices offer questionable value given that stakeholders usually make this sort of decisions once at the start of the project. On the other hand if you can offer automatic evals and figure out which prompts give the most similar results across different voice providers, that would offer a lot more value. It would be nice to be able to switch from e.g. Grok voice agents to ChatGPT voice agents knowing that the output style won't change too much. There are many companies now with evals as a core business model: LM Arena, Artificial Analysis, Prompt foo (before they got acquired and pivoted to security only) so many take a look at them.
Source: we have been building TTS systems for over a decade too https://narrationbox.com
Fair pushback. On end to end: we measure those too, same methodology: https://benchmarks.speko.ai/s2s. If the single models win, we route to them the same way, so we do not care which architecture (s2s or cascaded) wins. For now, what we see in production so far is that most teams still want to control each piece: swap the STT for medical vocabulary, keep the LLM, keep the voice.
On "promptfoo of voice models": that is closer to how it started. At my last company we ran these evals manually, we would even hire native-speaking raters, benchmark, switch if it wins. The evals are the value, agreed. The routing is what makes them actionable: teams told us swapping always looked like an R&D project, so scores alone did not change what ran in production.
On prompt-based voice gen and reference-audio cloning: agreed, that is what we see too. It makes continuous measurement more important: the same style prompt behaves differently per language and per content type, so we rank the voices themselves, tagged by use case: https://benchmarks.speko.ai/tts-voices
> The industry is very much moving towards one-model-does-all end to end trained
I've worked with hundreds of enterprises on voice AI and voice agent solutions. In my experience, this isn't true. Or rather I should say, the people actually paying for voice agents (i.e enterprises) are not moving towards STS solutions in a meaningful way. The composability, observability, and reliability profile of STS systems is not amenable to enterprise criteria. Not to mention costs.
exactly, we see the same thing, around 95% cases are still cascaded, even tho STS has been improving a lot
totally agree with this
cool! we have something similar implemented: https://github.com/fastrepl/anarlog/tree/main/crates/owhispe...
looks cool, checking it out!
The link, since it seems to be missing?
https://speko.ai/
Yes, i added it. that's the right link.
Awesome. Thanks for sharing!