WebRTC doesn't give up on late audio. This isn't factually correct. WebRTC has NACKs[0]. The receiver (AI Inference server) decides when to either accept the gap in RTP packets or if it sends the NACK and waits.
This article doesn't describe how Voice AI systems I work on operate. For speech-to-speech I am working at 20ms/200ms chunks and doing inference. So I don't want buffers
What I find frustrating about these sales pitches that conflate the different parts of WebRTC. WebRTC is protocol, sdk/software and a API in the browser. So if one part has an issue it is important to explain it with detail.
It's like attacking HTTP by saying 'HTTP is bad because this server doesn't have Y setting'.
But the default WebRTC client is the one that browsers use. If Y setting isn't in that client or even if it is in the client and not enabled in the browser, it's basically as if WebRTC didn't have that at all. It would be like saying 'SSH is bad b'c OpenSSH doesn't have Y setting' b/c the vast majority of clients/servers are OpenSSH.
I've had some experience with audio on the web, back in the day I built a weird collab music streaming system in the browser and presented at NIME. Back then AudioWorklets were new and I didn't understand WebRTC. The initial design used Raspberry Pis and IceCast. However, the setup was not only difficult to me but beyond the abilities of my target audience. I was more familiar with web dev and figured why can't something similar work for the browser, truly naive. After lots of learning through trial and error and understanding how these types of streaming worked, I ended up with something in hindsight worked pretty simply and was beautifully naive.
It took audio input, fed it into a wasm opus encoder, output audio in "segments" of audio, and uploaded via typical HTTP. No WebSockets or WebRTC or QUIC (I don't even think I used HTTP2). The only reason it wasn't just a full encoded recording was because I wanted to make it continuous and streamed. I basically learned this via HLS, but also didn't understand HLS nor was it easy to encode and stream data (in my mental model) from the browser to the server.
On the server and streaming to the client, it just collected the segments and told clients when they were ready, via polling, similar to HLS. The client would download, wasm decode, and add the audio to a buffer. The thing is that because of the nature of the project it didn't really have much to do with one continuous stream, but it did take sequences of the segments from multiple user's uploads and combine them for the different audio channels.
Most of the argument I think for what I'm seeing for voice AI is the code complexity, server complexity, interruptions, latency, drop outs, that kind of stuff, and I guess that's all good for companies with huge numbers of users and wanting to follow the work done at frontier labs. Also everyone wants the technology they sell to be the ones companies use.
Just weighing in that there could be cases where you record some audio, encode it, upload it, transcribe it, feed to an LLM, TTS the response and download it. You can stream it in segments or not to reduce latency. You can interrupt it with UI/DSP/NN and detect the end of a message similarly. Maybe for most smaller companies that would be enough? It feels like reading these different takes is that they create problems while solving problems, and maybe make the adoption of voice AI harder (and therefore slower) than it could be. I'm not any less naive than I was before though, but often its people like me tasked to implement this stuff.
Summary: Long-winded advertisement for MoQ. Author mentions that WebRTC has some limitations and Google needs to address them. Until then, anyone (including OpenAI) trying to use WebRTC for human<->bot communication may run into issues.
So their argument is that you can send voice generated by the AI faster than realtime and buffer it. This is not entirely true since you can interrupt the AI agent. So you would need some additional client side logic to handle that. Meh not sure if this tradeoff of fixing potential network issues, for more client side logic is worth it.
Their other argument is that you can retry the sending of your voice to the AI. Which WebRTC already does. (NACK, FEC, Opus RED)
The moq people, and they are loud and obnoxious, have this incredible ability to ignore the practical experience that led to the defaults of various systems being what they are while believing the solution is to expose every possible to dial to them so they can eventually rediscover the same results, which presumably they will smugly and obnoxiously broadcast as new discoveries to celebrate their brilliance.
It's like that famous saying about a certain group of people: you can always rely on them to do the right thing, after they've tried every other possibility. They could at least exhibit some level of self awareness.
If I were "Sean-Der," I would not reply. The "being wrong on the Internet causes smart people to write smarter stuff that's interesting to read" discourse is over. You have to own how much of a waste of time it is.
Sean is not the only developer who has used WebRTC, and conferencing is not the only thing MoQ can be used for. I'd really like to see more devs weighing in, especially those with WebRTC experience
WebRTC doesn't give up on late audio. This isn't factually correct. WebRTC has NACKs[0]. The receiver (AI Inference server) decides when to either accept the gap in RTP packets or if it sends the NACK and waits.
This article doesn't describe how Voice AI systems I work on operate. For speech-to-speech I am working at 20ms/200ms chunks and doing inference. So I don't want buffers
[0] https://webrtcforthecurious.com/docs/06-media-communication/...
What I find frustrating about these sales pitches that conflate the different parts of WebRTC. WebRTC is protocol, sdk/software and a API in the browser. So if one part has an issue it is important to explain it with detail.
It's like attacking HTTP by saying 'HTTP is bad because this server doesn't have Y setting'.
But the default WebRTC client is the one that browsers use. If Y setting isn't in that client or even if it is in the client and not enabled in the browser, it's basically as if WebRTC didn't have that at all. It would be like saying 'SSH is bad b'c OpenSSH doesn't have Y setting' b/c the vast majority of clients/servers are OpenSSH.
You can have opus in-band FEC as well, though he mentions it in the article too.
I've had some experience with audio on the web, back in the day I built a weird collab music streaming system in the browser and presented at NIME. Back then AudioWorklets were new and I didn't understand WebRTC. The initial design used Raspberry Pis and IceCast. However, the setup was not only difficult to me but beyond the abilities of my target audience. I was more familiar with web dev and figured why can't something similar work for the browser, truly naive. After lots of learning through trial and error and understanding how these types of streaming worked, I ended up with something in hindsight worked pretty simply and was beautifully naive.
It took audio input, fed it into a wasm opus encoder, output audio in "segments" of audio, and uploaded via typical HTTP. No WebSockets or WebRTC or QUIC (I don't even think I used HTTP2). The only reason it wasn't just a full encoded recording was because I wanted to make it continuous and streamed. I basically learned this via HLS, but also didn't understand HLS nor was it easy to encode and stream data (in my mental model) from the browser to the server.
On the server and streaming to the client, it just collected the segments and told clients when they were ready, via polling, similar to HLS. The client would download, wasm decode, and add the audio to a buffer. The thing is that because of the nature of the project it didn't really have much to do with one continuous stream, but it did take sequences of the segments from multiple user's uploads and combine them for the different audio channels.
Most of the argument I think for what I'm seeing for voice AI is the code complexity, server complexity, interruptions, latency, drop outs, that kind of stuff, and I guess that's all good for companies with huge numbers of users and wanting to follow the work done at frontier labs. Also everyone wants the technology they sell to be the ones companies use.
Just weighing in that there could be cases where you record some audio, encode it, upload it, transcribe it, feed to an LLM, TTS the response and download it. You can stream it in segments or not to reduce latency. You can interrupt it with UI/DSP/NN and detect the end of a message similarly. Maybe for most smaller companies that would be enough? It feels like reading these different takes is that they create problems while solving problems, and maybe make the adoption of voice AI harder (and therefore slower) than it could be. I'm not any less naive than I was before though, but often its people like me tasked to implement this stuff.
Summary: Long-winded advertisement for MoQ. Author mentions that WebRTC has some limitations and Google needs to address them. Until then, anyone (including OpenAI) trying to use WebRTC for human<->bot communication may run into issues.
Quite interesting. I found the part about tokens with different priorities super cool.
Native apps should probably just run local voice recognition AI
Size is the tough part. These models require 256GB+ of vram, but other then that I think it's possible!
Transcription models certainly don't require that ram. Parakeet even runs the CPU while generally outperforming Whisper.
This is for communicating with agents and inference, not just dictation.
So their argument is that you can send voice generated by the AI faster than realtime and buffer it. This is not entirely true since you can interrupt the AI agent. So you would need some additional client side logic to handle that. Meh not sure if this tradeoff of fixing potential network issues, for more client side logic is worth it.
Their other argument is that you can retry the sending of your voice to the AI. Which WebRTC already does. (NACK, FEC, Opus RED)
The moq people, and they are loud and obnoxious, have this incredible ability to ignore the practical experience that led to the defaults of various systems being what they are while believing the solution is to expose every possible to dial to them so they can eventually rediscover the same results, which presumably they will smugly and obnoxiously broadcast as new discoveries to celebrate their brilliance.
It's like that famous saying about a certain group of people: you can always rely on them to do the right thing, after they've tried every other possibility. They could at least exhibit some level of self awareness.
This seems like an ad for their voice SaaS?
Just ask "what is le capital of la France" no more confusion then.
Unhinged indeed.
If I were "Sean-Der," I would not reply. The "being wrong on the Internet causes smart people to write smarter stuff that's interesting to read" discourse is over. You have to own how much of a waste of time it is.
Sean is not the only developer who has used WebRTC, and conferencing is not the only thing MoQ can be used for. I'd really like to see more devs weighing in, especially those with WebRTC experience
I don't understand, mind rephrasing?
I respond because incorrect info will cause developers to make bad decisions/build the wrong thing.
as someone who works at openai you already know that developers are not making any decisions anymore haha