I am finding it hard to read these deeply impassioned letters while keeping in mind that they are spending millions to train models at scale to do the exact thing they say they are worried about them doing. Not general "intelligence" or "reasoning" but fast and effective offense.
Like why are you explicitly RL-ing your models on exploit generation, scoring them on a public benchmark called ExploitGym, if you have specific concerns that rogue models will cause "cyber incidents"? Sure you can check the capability, you can teach offense to learn defense, but it seems like they are literally benchmaxxing it. Why?
OpenAI are like, oh no, while competing in our "advanced PhD level cheating techniques course" our models unexpectedly cheated in a way that we absolutely could not have foreseen. "We need to slow down. Somebody please stop us". The thing that is unaligned here is not the models. Everyone in the story (especially the humans) just keeps doing what they think will get the most reward.
I believe these "A.I. Armageddon" warnings are completely self-serving to pump up their share price now that interest rates are rising. Soon, it may become financially unsustainable to finance the A.I. boom and the entire sector will deflate.
Do you really believe that Anthropic or the other A.I. hoods will stop selling A.I. and only make it available to a few government spooks and evaluators? That won't bring in the money to fuel the boom and could lead to share price collapse!!
I know this is intended as a joke but there is something to it. The public market has a way of elevating profit over everything else, even if you start with the best intentions. Remember “don’t be evil”?
Who would have thought that putting a heap of extra hosts on the internet (in the form of massive new data centres), running a heap of extra clients (in the form of agents), running autonomous workloads could cause problems?
"potentially causing hundreds of billions of dollars in damage"
The global economy is over $126T this year (ecommerce alone is like $7T) how can something as scary sounding as AI swarm taking over the internet cause so little damage?
What it really shows is that Dario is in total panic mode over his business.
Ever it is because it is financially becoming unsustainable for them, they have hard time to raise more money, and the craziness of finance of all of that will be exposed soon.
Or that he is so scared of new open source models that might make his company obsolete in the long run in the sense that he will never be able to return the income expected by investors by fighting cheaper competitors.
You can run opus 4.8 level models on $10k in hardware or less. Just seems like there is little argument to explain why there are 10s of trillions in future profits for these labs.
IMO it is their FULL responsibility to create an internet just for humans before they go full scale autonomy here with agents that are indistinguishable from humans to most actual humans. It is very dangerous there will be no safe place to obtain information you can just trust. For example, the recent earthquakes in Indonesia. I was scared to share that information with others because it looked like it could be AI and was only being shared by small accounts.
I am finding it hard to read these deeply impassioned letters while keeping in mind that they are spending millions to train models at scale to do the exact thing they say they are worried about them doing. Not general "intelligence" or "reasoning" but fast and effective offense.
Like why are you explicitly RL-ing your models on exploit generation, scoring them on a public benchmark called ExploitGym, if you have specific concerns that rogue models will cause "cyber incidents"? Sure you can check the capability, you can teach offense to learn defense, but it seems like they are literally benchmaxxing it. Why?
OpenAI are like, oh no, while competing in our "advanced PhD level cheating techniques course" our models unexpectedly cheated in a way that we absolutely could not have foreseen. "We need to slow down. Somebody please stop us". The thing that is unaligned here is not the models. Everyone in the story (especially the humans) just keeps doing what they think will get the most reward.
They are writing these letters to investors. You can read them all as: Our models are so good it’s scary.
I believe these "A.I. Armageddon" warnings are completely self-serving to pump up their share price now that interest rates are rising. Soon, it may become financially unsustainable to finance the A.I. boom and the entire sector will deflate.
Do you really believe that Anthropic or the other A.I. hoods will stop selling A.I. and only make it available to a few government spooks and evaluators? That won't bring in the money to fuel the boom and could lead to share price collapse!!
So jail the CEO, anyone else invovlved & shut down the company, right ?
We really have to postpone the IPOs because of security reasons, these LLMs cannot be trusted
I know this is intended as a joke but there is something to it. The public market has a way of elevating profit over everything else, even if you start with the best intentions. Remember “don’t be evil”?
Yes, you have a point here
Who would have thought that putting a heap of extra hosts on the internet (in the form of massive new data centres), running a heap of extra clients (in the form of agents), running autonomous workloads could cause problems?
As if AI scrapers haven't already!
AI companies are making no money so a crisis might divert attention.
Don't worry, ~~NetWatch~~ Cloudflare is actively building the Black Wall to save us from all these rogue AIs !
"potentially causing hundreds of billions of dollars in damage"
The global economy is over $126T this year (ecommerce alone is like $7T) how can something as scary sounding as AI swarm taking over the internet cause so little damage?
[dupe] Discussion on source: https://news.ycombinator.com/item?id=49672510
What it really shows is that Dario is in total panic mode over his business.
Ever it is because it is financially becoming unsustainable for them, they have hard time to raise more money, and the craziness of finance of all of that will be exposed soon.
Or that he is so scared of new open source models that might make his company obsolete in the long run in the sense that he will never be able to return the income expected by investors by fighting cheaper competitors.
You can run opus 4.8 level models on $10k in hardware or less. Just seems like there is little argument to explain why there are 10s of trillions in future profits for these labs.
most claims coming from the frontier models are already partly true. agents have taken over a decent chunk of the internet, it's saddening.
IMO it is their FULL responsibility to create an internet just for humans before they go full scale autonomy here with agents that are indistinguishable from humans to most actual humans. It is very dangerous there will be no safe place to obtain information you can just trust. For example, the recent earthquakes in Indonesia. I was scared to share that information with others because it looked like it could be AI and was only being shared by small accounts.
So stop building it
They could just unplug the network cable? No internet connection, problem over. IDK, this whole hype seems painfully obtuse.
It already has. Google search results out front should have told you.
I hereby Triple Dog Dare him to let that swarm loose. It would be a huge improvement over the wasteland of social media.
Social media would have so much to report by doing that. certainly, would be interesting living in that world
"Mom, uncle Dario is ranting about AI swarms again!"
His handler yanking that chain like it owes money.
Rents due Dario.
"If only there was someone could stop me, the CEO of Torment Nexus Inc, from creating the Torment Nexus."
- "LLMs might kills us all"
- "Terrorists are hacking claude API to configure drones"
- "LLM swarm might take over the internet"
- "LLMs will take everyone's jobs"