What does that mean "running in real time" here? This seems like a meaningless phrase in this context. What would this look like if it wasn't running in real time?
For example, a video decoder running in real time means it can output x seconds of output data within x seconds of wall time. But there's no time constraint on an LLM.
What does that mean "running in real time" here? This seems like a meaningless phrase in this context. What would this look like if it wasn't running in real time?
For example, a video decoder running in real time means it can output x seconds of output data within x seconds of wall time. But there's no time constraint on an LLM.
From the thread on X it’s only 4.7 tokens/sec which personally I would find unusable for “real time” work