> Looking ahead
> Although Hy4 is one of the strongest open-weight models available today, a notable gap remains between it and top proprietary models, particularly on challenging internal benchmarks for research-level mathematics, long-horizon coding, model self-improvement, and advanced visual question answering. We will continue to steadily scale pre-training and post-training to narrow this gap.
I like that they acknowledge that. For working with ai as a tool open models are already great. For millennium problems and doomermaxxing they are not there yet.
Verrrry thorough in its work and therefore verrrrry slow. It spent a long time with each question, running about 26 queries for each question, and in some questions it ran 64 queries. One of the queries that it wrote also took about an hour to run, which affected its overall exam time.
Last year the open weight models were the cheap & good alternatives. Then in September/October that cost story got rewritten by the GPT-6 series and Claude's 5 series.
Hy4 brings open models back on the cost/accuracy frontier, but its behavior is too thorough + slow.
Another thing we're seeing is that post training open models is starting to see results. Particle-1 from last week is also on our frontier, which is post-trained on DeepSeek V4.1 and Fireworks Ember-1 from a few weeks ago was post-trained on Kimi K3.
Take these numbers with a grain of salt: this exam is just for short horizon, low-context data analysis tasks, not full-fledged software projects.
Slightly worse than Kimi 3.1M Pro (which just released open weights today) but much cheaper even excluding the current 50% off promotion, and Tencent promises to make it even cheaper once it goes open-weight. Interesting price economics.
Kimi 3.1 Pro is 2.8T vs this at 770B. I find that the jump in parameter count gives less tangible benefits like better "taste" and long context coherency, but it depends on the problem if that matters obviously.
increasingly feeling like the entire US economy hangs on whether or not Ant/OAI can stay ahead of the Chinese models.
If the gap closes, we do zero austerity, and interest rates are high, I cannot see a way out of a major economic crisis, even if we do start to see dramatic productivity wins.
More on-topic (and maybe more positive?), but this model feels a bit more expensive than I would have expected!
It's arguable that there is a threshold of quality that matter and US and China are both there already. I guess we'll see. Many of us on here are software engineers and I think we've seen that while Opus 4.7 (I think) was some sort of outlier/miracle, today there are several options to choose from. Depending on the task, the harness, etc Astra/Fable are only better when the work demands their level of capability - which much of software engineering does not.
Videos 1 and 4 on vision and computer use are wild. They depict the model finding a 2m feature from a large satellite image, then determining where a photo was taken. Is it just me or do the Chinese model releases have less concern about looking like a military weapon? I don't think it's a bad thing necessarily, it seems honest. Also the music they choose for it is hilarious.
China hit maximum tech dystopia a long time ago. Now they can enjoy tech because they know its already maximally invasive and there is nothing they can do.
Last time I went to China, a friend left her phone in a taxi (I guess a "didi") near a train station. She ran to the train station's police station, pestered them into looking at the recordings, got them to tell her the license plate of the car, and figure out how to call the driver. Her phone was back 20 minutes later.
If you're going to be surveilled 24/7, at least you gotta make it work for you.
They can enjoy tech because people see tech improving every aspect of their lives. Nowhere in the the world have people ever seen the rate of technological progress that people in China have. Imagine going from having practically no electronics, to everyone having a cellphone, self driving cars, high speed rail across the whole country, drone delivery, and so on. Naturally, people who've seen technology dramatically improve every aspect of their lives would be optimistic that further technological progress will bring even better life.
Meanwhile, in the west, the standard of living has been eroding for the majority of the population. Technological progress is associated with layoffs, loss of agency at the workplace, and diminishing life prospects. So, it's no surprise that there is growing opposition to AI.
The price numbers look weirdly specific, I wonder where they came from.
Personally, the thing I'm most excited about is the FrontierCode 2.0 score, which is a benchmark I trust a bit more - 31 seems to be roughly on par with 6.1 Sol and Opus 5.5 medium
We've moved past per token cost a long time ago. Smaller models use more tokens. What's interesting is the cost per task, on that front deepseek isn't particularly good.
It highly depends on the task. So far, on the tasks I personally benchmark, DS v4.1 Flash @ deepinfra.com is always cheaper because their cached input price for it is so very low and their cache hit rate is so very high.
> Looking ahead > Although Hy4 is one of the strongest open-weight models available today, a notable gap remains between it and top proprietary models, particularly on challenging internal benchmarks for research-level mathematics, long-horizon coding, model self-improvement, and advanced visual question answering. We will continue to steadily scale pre-training and post-training to narrow this gap.
I like that they acknowledge that. For working with ai as a tool open models are already great. For millennium problems and doomermaxxing they are not there yet.
Landed on the cost/accuracy frontier in our data analytics benchmark: https://plotly.com/blog/tencent-hy4-plotly-data-analytics-be..., right in between GPT-6.1 Sol and GPT-6.1 Terra.
Verrrry thorough in its work and therefore verrrrry slow. It spent a long time with each question, running about 26 queries for each question, and in some questions it ran 64 queries. One of the queries that it wrote also took about an hour to run, which affected its overall exam time.
Last year the open weight models were the cheap & good alternatives. Then in September/October that cost story got rewritten by the GPT-6 series and Claude's 5 series.
Hy4 brings open models back on the cost/accuracy frontier, but its behavior is too thorough + slow.
Another thing we're seeing is that post training open models is starting to see results. Particle-1 from last week is also on our frontier, which is post-trained on DeepSeek V4.1 and Fireworks Ember-1 from a few weeks ago was post-trained on Kimi K3.
Take these numbers with a grain of salt: this exam is just for short horizon, low-context data analysis tasks, not full-fledged software projects.
Slightly worse than Kimi 3.1M Pro (which just released open weights today) but much cheaper even excluding the current 50% off promotion, and Tencent promises to make it even cheaper once it goes open-weight. Interesting price economics.
Kimi 3.1 Pro is 2.8T vs this at 770B. I find that the jump in parameter count gives less tangible benefits like better "taste" and long context coherency, but it depends on the problem if that matters obviously.
increasingly feeling like the entire US economy hangs on whether or not Ant/OAI can stay ahead of the Chinese models.
If the gap closes, we do zero austerity, and interest rates are high, I cannot see a way out of a major economic crisis, even if we do start to see dramatic productivity wins.
More on-topic (and maybe more positive?), but this model feels a bit more expensive than I would have expected!
It's arguable that there is a threshold of quality that matter and US and China are both there already. I guess we'll see. Many of us on here are software engineers and I think we've seen that while Opus 4.7 (I think) was some sort of outlier/miracle, today there are several options to choose from. Depending on the task, the harness, etc Astra/Fable are only better when the work demands their level of capability - which much of software engineering does not.
Videos 1 and 4 on vision and computer use are wild. They depict the model finding a 2m feature from a large satellite image, then determining where a photo was taken. Is it just me or do the Chinese model releases have less concern about looking like a military weapon? I don't think it's a bad thing necessarily, it seems honest. Also the music they choose for it is hilarious.
China hit maximum tech dystopia a long time ago. Now they can enjoy tech because they know its already maximally invasive and there is nothing they can do.
Last time I went to China, a friend left her phone in a taxi (I guess a "didi") near a train station. She ran to the train station's police station, pestered them into looking at the recordings, got them to tell her the license plate of the car, and figure out how to call the driver. Her phone was back 20 minutes later.
If you're going to be surveilled 24/7, at least you gotta make it work for you.
They can enjoy tech because people see tech improving every aspect of their lives. Nowhere in the the world have people ever seen the rate of technological progress that people in China have. Imagine going from having practically no electronics, to everyone having a cellphone, self driving cars, high speed rail across the whole country, drone delivery, and so on. Naturally, people who've seen technology dramatically improve every aspect of their lives would be optimistic that further technological progress will bring even better life.
Meanwhile, in the west, the standard of living has been eroding for the majority of the population. Technological progress is associated with layoffs, loss of agency at the workplace, and diminishing life prospects. So, it's no surprise that there is growing opposition to AI.
That makes a lot of sense.
The price numbers look weirdly specific, I wonder where they came from.
Personally, the thing I'm most excited about is the FrontierCode 2.0 score, which is a benchmark I trust a bit more - 31 seems to be roughly on par with 6.1 Sol and Opus 5.5 medium
How does it do with tasks anthropic doesn’t like? This is my main use case for open models.
Not cost competitive. The performance difference is not commensurate with the price.
We've moved past per token cost a long time ago. Smaller models use more tokens. What's interesting is the cost per task, on that front deepseek isn't particularly good.
https://artificialanalysis.ai/#intelligence-efficiency-tabs
It highly depends on the task. So far, on the tasks I personally benchmark, DS v4.1 Flash @ deepinfra.com is always cheaper because their cached input price for it is so very low and their cache hit rate is so very high.
Wild that this is the first question I ask at this point. After that it’s a question of how it stacks intelligence wise vs glm or ds.
OAI / Ant are in for a reckoning.