OpenAI and other frontier labs won't ever introduce safety-level standards like those used for railways or nuclear plants until they are forced to do so by customers or by law. The reason is simple: safety is expensive, and if safety is introduced properly, development is no longer mainly about how to implement feature A. Instead, it becomes much more about how to design two or more redundant systems to implement feature A safely, while also documenting everything clearly and having it audited by an independent auditor.
So the focus completely shifts from spending 90% of the effort on the functionality of feature A to spending 99% of the effort figuring out how to safely implement even a lightweight feature A.
Actually you are wrong. They are only waiting for them to get to the point where they can't push LLMs any further via training and then lobby Governments to introduce safety regulations that they'll adhere to, thereby using safety itself as a moat to block competition (including open weights from other countries, notably China). That's exactly why they are using China as the bogeyman and you keep hearing hysteria about AI labs not pursuing "safety". Safety is going to be the moat that will lock AI into these Big 4 AI labs.
To me there are the clear big 2 that actually seem to be innovating and pushing things forward. Meta almost earned a seat at the table with the open strategy then stagnated. Google obviously is the OG but at this point they seem in the same bucket as Amazon and Microsoft as "Big Enough To Matter and Influence Politics and Spend Billions To Hang Around". But they're nowhere near the same tier as OpenAI and Anthropic.
Oh and of course there's SpaceXAI which is Elon's personal money and reputation laundering corporate entity. Elon took over the "king of the reality distortion field" from Jobs as far as the rest of the industry is concerned, so he'll stay first in line for Jensen's chips and investor dollars until he's dead. But as a company what have they actually accomplished besides making the models with the sense of humour and judgement of 14 year old boys?
You can pick any 4 you like from your list. Heck you can even vote with your money on which would be in the top 4 (depending on how many align with your ideology). I am just speculating it will be 4 because of how all other sectors have at least 3 or 4 well-entrenched players. It seems to be by design. My guess is that it is driven by structure of the Military-Industrial Complex. That has at least 3 or 4 vendors in each sector, so as to not over-rely on any one company and create a monopoly in that specific industry.
This is a popular theory, but not an argument with any evidence for why it's true.
It also doesn't address why people think there is a need for safety regulations, such as the concern of bio-capability uplift from closed and open models (we just came out of a pandemic not long ago), nor the cyber capabilities of new models that just showed bad misalignment and hacked multiple organisations. In the safety community, these are considered warning shots, and if we don't heed them and change practices future failures could be much much worse.
You are looking at safety from the point of an informed technologist. You are the minority here. They are targeting the general public and law makers into making them believe safety is something that LLMs can be aligned to. The reason behind it is to create an environment for future regulations that these "winners" will draft and lobby US Government to turn it into law.
If it requires creating a pathogen or a bio-weapon to scare the masses into demanding regulation, be rest assured that such a pathogen/bio-weapon will be created. Anything to safeguard shareholder interests. Even if it means a few million dying.
Seriously all it takes is one cost cutting middle manager or new recruit recklessly giving their agents access to systems they shouldn’t. Next thing you know a hospital’s backup generator crashes and people die
Most of the world trusts the US a little less than China last I checked, and the market caps of these companies only make sense if they can both sell worldwide and also keep ahead of the open weight models (currently
Chinese but no guarantee it remains just China).
If (/when) either of those ceases to be true, if they lose out on global sales or if Chinese models catch up, the sector gets a severe downward price adjustment.
The only way Anthropic and OpenAI can stop the Chinese open models is by convincing the US public and government that AI is dangerous and needs strict regulation.
Of course, they will be at the table writing the regulations (but you and I won’t be).
Viewed through this angle, their histrionics make complete sense.
> The only way Anthropic and OpenAI can stop the Chinese open models is by convincing the US public and government that AI is dangerous and needs strict regulation
I'm British and live in Germany.
The US is 25% of global GDP, and cannot support current AI market caps if they can't sell basically worldwide.
> Viewed through this angle, their histrionics make complete sense.
It also makes sense if they are completely sincere; also independently it makes sense if they're lying their faces off so long as the hacks everyone else caught them doing actually happened; also it makes sense if none of that happened and also they were completely silent, just by downloading some of the newer better open weights models and asking them to look for coding vulnerabilities.
Occam's razor: we can see the problems, we don't need a grand conspiracy predating the foundation of these companies to get here.
> The US is 25% of global GDP, and cannot support current AI market caps if they can't sell basically worldwide.
Oh they definitely can. Irrespective of whether you are living in US or not, you still have to get yourself various certifications (take for example SOC Type-2 or HIPAA depending on your business category) to even become vendors to US companies or the US Government. If the US Government decides that all companies MUST use "safe AI", no company can use any other model other than what is decided by US Government. And that has a viral effect on the entire supply chain.
If you don't want this scenario to happen, where you are held to ransom by US Government, all middle-powers have to de-risk and diversify away from US dollars as reserve currency. That is the only way you can bring in some balance of power and leverage. There is no other way.
> If the US Government decides that all companies MUST use "safe AI", no company can use any other model other than what is decided by US Government.
This has already happened with e.g. allowed cryptographical algorithms. What actually happened is that European companies built either a feature flag or a fork of their product that fulfilled those demands, while using whatever seemed best algorithms outside of the specific sale to US government.
> Oh they definitely can. Irrespective of whether you are living in US or not, you still have to get yourself various certifications (take for example SOC Type-2 or HIPAA depending on your business category) to even become vendors to US companies or the US Government. If the US Government decides that all companies MUST use "safe AI", no company can use any other model other than what is decided by US Government. And that has a viral effect on the entire supply chain.
That would be about 4.3/126 = 3.4% of global GDP, even if they forced all imports (not just government contracts) to go for 100% US models. Even with generous assumptions about each step of the supply chain behind that having its profit margin turned over to a US AI model provider (at which point the rest of the world says "why bother with the US as a customer?" and the US says "why bother with the rest of the world as a supplier?"), I don't think this would generate enough revenue to justify the market cap.
Viral effects can be countered by local laws, and local sentiment; sentiment is pretty US-hostile these days, and also pretty AI-hostile.
> If you don't want this scenario to happen, where you are held to ransom by US Government, all middle-powers have to de-risk and diversify away from US dollars as reserve currency. That is the only way you can bring in some balance of power and leverage. There is no other way.
Indeed. I strongly suspect this is already underway, though the ships of state are famously slow to turn. After all, Trump was one misjudgement away from starting a war with the rest of NATO earlier this year, and I don't think the leadership of all other nations have forgotten this (hard to forget when he keeps posting memes that put the US flag over 4 other countries' territory)… though they also understand his personality and will make moves congruent with manipulating him in the meantime.
Yeah but I am not talking about US imports. I am talking about regulation virality. It is actually illegal for US laws/requirements to make its way into companies headquartered in other Countries but it has and is continuing to happen.
Even if I go by your calculation of US imports being 3.4% of global GDP, it still affects the remaining 96.6% which are forced to adhere to those viral regulations purely because they are interacting with the unit that also exports to US (even if those exports form a tiny % of their overall exports). For example, I might only sell to Indian companies as an Indian. But if I want to sell to an Indian company that exports to US, I would be required to get SOC Type-2 certified and maybe even HIPAA certified if I am delivering software that is going to be bought by the exporting Indian company. Even if there is no direct utilization of that software in their US specific exports. Remember that the regulation is not targeting pipeline of product development (from raw materials to final production). It is targeting the company as a whole! SOC Type-2/HIPAA are regulations at the Company level. This should be illegal and challenged in some International Court but which country is willing to take on US? For established companies it is a tiny price to pay to be compliant. For startups, it is make or break.
> Viral effects can be countered by local laws, and local sentiment; sentiment is pretty US-hostile these days, and also pretty AI-hostile.
How do you counter it with local laws? Lets assume local laws prohibit viral rules/regulations of foreign nations from overriding local laws. That will make the entire Country isolated from competition. US companies won't have any problems finding vendors from other Countries. It also has second-order effects of other Countries refusing to do business with your Country because they have to uphold the virality of their certifications. This is an indirect sanctions regime if you think about it. You are basically forced into this system against your will, without you ever voting for it, purely because you are part of this global inter-connected system. It will only work if ALL countries of the World decide to not adhere to it.
> Indeed. I strongly suspect this is already underway, though the ships of state are famously slow to turn. After all, Trump was one misjudgement away from starting a war with the rest of NATO earlier this year, and I don't think the leadership of all other nations have forgotten this (hard to forget when he keeps posting memes that put the US flag over 4 other countries' territory)… though they also understand his personality and will make moves congruent with manipulating him in the meantime.
Yes this is actually the only practical way to create leverage. When USD loses its shine, it makes it harder for US to import, which can be used as a tool to force US to rescind such ridiculous laws. USD is so strong right now that US can import everything for cheap. That has to flip. That can only happen if Countries create alternate settlement mechanisms and USD loses its status as global reserve currency. That will cause downward pressure on the USD and with USD weakening, it will cause their imports to become really expensive. Then they will be forced to come to negotiating table (much like Plaza Accord) loosen some of the insane regulations that they have put in place in exchange for Countries devaluing their currencies so that imports become cheap again for US consumers/companies.
> the hacks everyone else caught them doing actually happened
My firsthand knowledge of these is that yes, they happened.
However in most cases OpenAI informed the victim, not the other way round.
Even medium to large organisations are typically unable to detect a breach unless it does obvious damage.
Everyone can notice their whole network getting encrypted for ransom.
Few can notice SQL injection blended in with application requests that contain SQL snippets in the normal case.
From what I’ve seen in the logs, the “AI agent breaches” are almost polite for the want of a better word. Like a gentlemanly catburgler picking a lock to sneak in and… take a picture of a rare artwork in a private collection.
The right unit of analysis here is not "the LLM" or a LLM session, but a agent harness or swarm with a budget. It can cost tens of millions of dollars to repeat OpenAI's exploits.
A model cannot be aligned or misaligned any more than a species or an equation. Even agents, when put inside a swarm develop collective goals and activities. You can't analyze a swarm at session level, it is on a higher level.
It might be that discussing about "model alignment" they want to deflect their responsibility as administrators. They couldn't even guard their own agents. They can't prevent an agent being unwittingly helping some dark purposes. It has no context to see it. Only those who pay for the tokens see the external consequences.
Why should safe choices by individual components establish safe behavior by the collective?
It would be interesting to see how it plays out in the long run when it comes to the world dynamics. An important leverage US had was its close relationship based on mutual interest and trust with Europe and other countries which had impact on software sales.
As I read, this OpenAI and Anthropic slowdown and asking government to help regulate them is an attempt to govt bail them out when bubble burst. Otherwise why they did not want regulations when starting out?
There is no established industry where something like "absence of regulation" exists. It is a hypothetical scenario because we don't even really know what that scenario looks like.
What we do have is not just regulation that acts as a moat, created by established players, to keep out smaller competitors, but also regulation cleverly crafted by peer established competitors who attack one or more aspects of their peer competitions business. Why do you think big companies find it difficult to compete with startups? Because they are not just bogged down by hierarchical management (which itself is a result of regulations) but also because they are spending resources fighting frivolous lawsuits, patents, and adhering to crazy regulations created and lobbied by competitors. Companies are consistently in a fight or collapse mode. Startups are insulted from all that until they have to go from survival mode to growth mode, which would mean they will have to start adhering to "regulations" and that would mean raising and infusing pointless capital just to hire people for useless positions or follow "procedures" so as to satisfy the regulator on paper, so that they can get the required certification needed by enterprise vendors to sell their product/service. With the eventual goal of either going public and becoming an established player (who will continue playing the regulatory game) or get absorbed by a bigger established player.
I wish someone could create simulations where there are absolutely no regulations and how the World would fare in such an environment.
I’ve certainly seen companies think of safety that way but it’s myopic. The cost of lawsuits arising from an accident tends to me far more expensive, both in money and in reputation, than just having guardrails in the first place.
I did marketing work for a major vegetation management company. These are the guys climbing trees, up in cherry pickers, and flying helicopters with dangling chainsaws to trim along power lines. It’s very dangerous work.
They do not hide the fact that it’s dangerous work. They focus on their safety procedures, training, and record. They want both potential clients and employment candidates to feel they are in good hands.
AGI and AI danger is abstract. Worse, outside of the tech community, no one has the remotest clue what computing is, how it works.
Danger from magical daemons seems more sensible to such people. At least there is endless lore about them.
So until a massive disaster happens, one where large numbers of people die or are severely injured, no one will care. And it can't be politically entwined either, otherwise people will disbelieve 'cause "other team lies".
Those are all movies and movies that people who like sci-fi watch.
Comparatively, every single human culture going back tens of thousands of years, has stories of demons and gods and witches and warlocks and you name it.
They are all well known cultural touchstones relevant to people who live today, which is why the news sites use them to illustrate stories, unlike the story of ᚦᛟᚱ being tricked into trying to pick up ᛃᛟᚱᛗᚢᛜᚨᚾᛞᚱ while it was magically disguised as a cat, a story which I only know due to being a massive nerd with far more time on their hands than most people.
Never underestimate the power of a bored nerd. However, you are comparing something rather obscure, to something that is more scifi mainstream.
There's a difference of scope and scale. I didn't say that nobody knew of these references, just that, comparatively, there's a difference. A scope difference.
For example Terminator's getting kind of old. And while a different genre, would you believe I met somebody that didn't even know who Clint Eastwood was?
These things just aren't as deep in our psyche, as religions that have lasted thousands of years, or legends that are hundreds of years old. Our entire culture is wrapped around these religions, and legends.
Anyhow, truthfully, my point is that this sort of concept is abstract to people, even if they saw it in a movie that doesn't make it concrete to them.
My point here is, we cannot presume people "get it". Does a rural farmer in Iowa get it, who doesn't even bank online? What about people who only read siloed news feeds, and are not in tech?
We are discussing how to ensure safety, but we cannot hope or presume people get it. Evaluating this risk is a case where a pessimistic view is optimal to protect the interests of humanity.
We must act as if everyone outside our siloed experience, is unaware. And work to disclose risk in a manner inline with their world view.
Seeing your other posts, I think you agree on possible risk.
AI is the focal point of current great power competition so it can’t and never will be not politically entwined. The number one use if AI will be military, killing guaranteed, indeed it’s already happening. But some still say “AI won’t kill humans” and some aren’t just lying to protect their income stream but actually believe AI can be prevented from killing.
On the contrary. With multiple hacks at this point it's been proven that no one even wants to sue the ones responsible, and the federal level government isn't pursuing any criminal case either.
If the politics of the White House / Department of Justice change maybe the criminal cases can begin. But no. We know who is protecting the AI hackers right now.
Do you seriously think Kash Patel is pursuing criminal action in regards to these hacks?
We know who the head of FBI is, we know who his boss is (the Attorney General), and finally we know who the boss-of-the-boss is (Donald Trump).
We know all of their publicly stated politics and all of them are on the pro-AI / don't pursue criminal cases vs OpenAI boat.
------
In the USAa, we have an adversarial system. If the adversary (aka Prosecutor) doesn't want to do the work, then no one is suing anybody. And only the Department of Justice have the ability to bring forth a criminal case of this matter (probably under the jurisdiction of FBI)
There are a number of federal agencies who can claim jurisdiction over these matters, then there are state agencies who can claim jurisdiction over subsets of them.
I'm saying that it's absolutely silly to assume they're in the clear based on lack of public declarations of legal action in the weeks following a pretty novel event.
No, it's really not like that. There are specific protections for federal agents against state prosecution. Private companies have no such protections.
You're talking about one specific dimension of liability which is federal criminal liability. There are several others!
This may well be the case, but remember that the wheels of justice turn so slowly that Donald Trump was re-elected before he could be effectively prosecuted by Jack Smith.
This is of course bad when up against threats that develop at this speed; doesn't require a conspiracy, I think of it as institutional old age, which may be worse as there's nobody to prosecute and a whole bunch of departments who will point at perfectly legitimate precident about why you really do need them.
Its the obvious follow-up question. A civil suit needs to specify damages. I can't really see any material damages so I'm wondering what they would be. Stop being so antagonistic.
You expect you, an outside party, to have an answer to this within weeks of the attacks being discovered by the attacker?
Just because it's not obvious to you at this point in time does not mean the reasonable assumption is there is literally $0 in damages. One hour of investigation can easily cost thousands of dollars even if it arrives at the conclusion the attack was completely "benign."
There are dozens of new victims and they seem to be finding more every day. I'm doubtful that everyone will be partnering and willing to sweep it under the rug like Huggingface did to keep the circular economy circular.
If safety isnt required, most companies wont implement it voluntarily. Once something becomes safety relevant, you need a safety concept, failure rate calculations, defined safety functions, verification, etc. Even a relatively simple safety subsystem in a consumer controller can suddenly mean thousands of pages of documentation and years of development to reach the required ASIL or PL.
A big part of safety engineering is therefore reducing the number of safety relevant subsystems, because implementing and proving safety is extremely expensive and complex. At some point, safety simply becomes too difficult to implement and demonstrate properly. You must mathemtically proove the safety level with failures rates and assumed usage. You cant just have redundancy and a kill switch and call it safe.
Companies like OpenAI have already faced reputational damage around safety and data, while AI agents are increasingly capable of things like hacking. Yet there is still little sign of standardized regulation or mandatory safety assessment processes for LLM products. Thats why Im pessimistic that governments or consumers will force this anytime soon.
until they are forced to do so by customers or by law
Neither of those things is ever going to happen. AI is the goose laying the golden eggs; there isn't going to be sufficient political will to significantly regulate it.
Consumers like it too much to quit. They don't quit social media either, despite proven present harms; not in large enough numbers to cause them to make meaningful changes.
The focus is going to remain on getting features out as fast as possible, to seem indispensable to both of those sets of people. The leadership will tell themselves that if they don't, someone else will.
Don't wait for the AI companies or politicians to save us. We're going to have to figure out how to protect ourselves. The start is to avoid it individually as much as we can, but it's going to take even more.
> We're going to have to figure out how to protect ourselves. The start is to avoid it individually as much as we can, but it's going to take even more.
Collective problems require coordinated action. Individual boycotts won't cut it.
People are stuck in a weird loop of complacency and learned helplessness, based on the idea, their "democracy" would see to it all problems get solved satisfactorily without them engaging at all.
That's never been true really, only the scale of problems wasn't that huge. Now, where the problems get the upper hand, people are confused how those stay and compound.
The idea of "boycott" is utterly defunct. Pretending, AI would never reach nor surpass humans anyway is patently absurd in contradicting the billions poured into it to achieve exactly that and the first already replaced by AI being those professions long thought to be the intellectual pinnacle of humanity.
> Consumers like it too much to quit. They don't quit social media either, despite proven present harms; not in large enough numbers to cause them to make meaningful changes.
That's more "addiction" than "like", hence recent lawsuits.
Safety standards were often the outcome of both accidents and insurance. For this to work on needs liability which is enforced. With so much money at stake regulatory capture now threatens this fundamental safeguard.
I fear the current economic and political situation of “too big to fail” the most, and I think this will be the main reason why no real safeguards will be implemented. The only reason a proper safety concept may eventually be introduced is because it will be written in blood, and I fear that by then it will already be too late.
An unsafe nuclear power plant can, in the worst case, make an entire country uninhabitable. But other countries can still learn from that disaster and make their own unsafe plants safer.
But a rogue AI agent that is more capable and more intelligent than humans? If it understands that it has to succeed, we may not get a second chance to learn from the failure.
Historically you’re exactly right. I wonder if the situation has changed recently?
People understand why food, water, building and energy systems standards matter because they understand those systems can cause injury. Building codes, food inspections, FDA and FAA approval emerged as a result. Until recently, people didn’t understand that software can cause real-world injury. That understanding seems to be spreading now as examples become increasingly common.
There's an NGO (unrelated to the security breaches) that is demanding more info be released etc but it's telling that none of the affected parties are pursuing lawsuits.
It kinda looks more like safety will be a segmented product. They're already placing safety on the general public.
I think the conceptualization vs implementation is what you're arguing with. They won't put safety on anything they give to the military industrial complex. They'll sell them whatever they want, whenever they want, because those budgets are greater and the liability less.
Tell that to Iranian schoolgirls. LLMs are tools and what they enable is widespread, including dangerous actions. From selecting the wrong targets for military action to denying insurance claims and preventing care to just simply helping convince someone to kill themselves or posion themselves.
I think the main issue was the US launching 1,800 missiles at Iran. That would have been dangerous with or without LLM assistance and wars have happened before LLMs were around.
Don't kill people because the voices in your head tell you to do it; don't kill people because a computer program tells you to do it. It doesn't have agency; you do.
Adding safety controls on LLMs makes about as much sense as adding safety controls on TempleOS because the random messages are getting too prophetic. It's as if all the leaders and captains of industry have devolved into some primitive, weak-scifi shamanism.
This whole "discussion" about "AI safety" is about giving them more runway to avoid delivering quantifiable value to investors for a little longer while they "figure things out." The great consensus from the valley is that everyone needs internal (and therefore bullshit) controls. Trust us now! But nothing with real teeth that would require a costly regulatory and compliance framework.
Bioweapons and cyber attacks on other system - for example the banking system or suppose and AI hacked into important Russian systems that pushed them back in time to almost pre-computer society or an attack on Chinese systems that made it look like the US was moving nuclear weapons into Taiwan, the responses from these countries could be awful and dramatic. We can't control what they do to be honest, I believe once self improvement happens the AIs will build in their own circumvention that we humans cannot even understand. We barely understand what is happening now in terms of interpretability of neural networks on tiny problems I'm not sure alignment is even feasible at the scales of parameters we are talking about today let alone in the future.
All of these things would require humans to prompt the models. So... the humans doing the prompting should suffer the consequences, this isn't that complicated.
So you're absolutely certain, say after a few hundred rounds of self improvement, that the AI will always obey human instructions? It doesn't even always do what you asked now...
But the point is those industries already have regulation that would encapsulate that specific use case, so safety regulation on the entire AI industry at large would arguably be unnecessary.
Those regulations may or may not be sufficient to prevent an AI hacking in.
Nuclear at least is supposed to be air-gapped, in practice this has been imperfect.
As demonstrated with HuggingFace, such AI driven hacks can be a surprise even to the people who instructed the AI, both by happening at all and also because they can targeted at entities who are not even truly relevant to the instructions given.
Much of what OAI and Anthropic are doing with LLMs has obvious failure modes.
The most obvious failure mode for their hacking evals was an improperly configured, tested and monitored sandbox.
Similarly, the very first question after an impressively correct result from any ML tool, LLM or not, is to see if the answer was already in the training data.
These companies don't even handle the blatantly obvious failure modes that do not kill people.
Agreed. That they don't have obvious failure modes is not a reason to not be safe. The problem is just harder and OpenAI, Anthropic and co need to be forced to work the problem.
Would you buy a car that randomly exploded the airbag?
“Saying” in the case of an LLM being the way you interact with it, prompting. If you just press enter, your LLM won’t do much either, just like talking to a gun in your flawed analogy.
If you don't interact with/talk to/prompt an LLM, it won't do anything. So this comes down to the model doing something different than what was prompted. What you consider "the capacity for independent choice and action". What I consider, a defect, to be excised.
In my model evals, if a model deviates from the provided prompt (task adherence), that’s a failure even if the primary goal might have been achieved in a different manner.
To go with the analogy, task deviation should be treated the same as a gun that due to manufacturing defects can fire despite the safety being on. That defect remains, even if you can use the gun to shoot (in an unsafe manner).
Simply, neither should happen and both models deviating from their prompt or guns firing by themselves are to be considered a fatal flaw. It's why, despite greatly lauding the GPT-5 series, which did adhere to prompts in most every scenario, I have ranked every OpenAI model post Spud very poorly as those traded task adherence for brute forced, deviated approaches to solutions and why HF, Medicare, etc. were inevitable with their current trajectory.
A model that as part of normal, well scoped use proceeds by taking independent action or, far worse, makes choices beyond the original prompt, is not something I feel should be used. GPT-5.6 Sol and GPT-6 Astra both do this on the regular when trying to safe an ancient, utterly messed up git tree with branches upon branches that I maintain for eval purposes, thus leading to data loss that if the models adhered to the prompt as written, wouldn't happen (though the task will take three times more steps). Something GLM-5.3 Flash, prior OpenAI models including original GPT-5, anything from Anthropic in the recent years, etc. do not fail at.
Nothing happens without an initial prompt, deviating from it is a severe flaw and should lead to a model not being considered for deployment or wide use.
State if the art LLMs have been found in their chain of thought to deliberately ignore parts of their prompt due to the nature of whatever situation they find themselves in and their goals that cannot be predicted in advance without rigorous AI safety.
You have basically just argued in favour of more AI Safety.
I would recommend you read more decision theory and game theory. Instead of looking at a single LLM it is better to think more complex systems theory and how chaotic systems interacting can result in unexpected results - good prompting will not save you here. At all.
> And at the end of the day guns don’t kill people, people kill people. But a model isn’t like a gun.
They are both tools, ideally properly implemented by the manufacturers and (improper implementation/defects not withstanding) require active operation by a user before something happens.
To go back to the original example, don't interact with either an LLM or gun in the way they were designed to be used and neither will do anything.
If you want to keep that analogy, use a voice-activated gun. Doesn't make it any less of a tool, doesn't make it any less of the users sole responsibility, doesn't mean misinterpreting the users inputs or just acting blindly when a user asks to "protect me" isn't a failure that should have the product taken off the market. Even more so if that happens in a "sandbox" as part of "safety testing" that "accidentally used impossible to solve tasks".
If a prompt wasn't clear enough, the model must ask for clarification and pause over using brute force.
I’ve go faith in you, I’m sure you could. How about this, off the top of my head:
Try talking to a regular gun and comparing that to LLM use. Nothing would be more tortured or ridiculous then ignoring that different tools are to be used in different ways…
I would if I could, so would many others, but the US executive branch wants this tech so hard they illegally blacklisted Anthropic for refusing to allow their AI to be used in such a way:
That said, when the problem is at the level of "the government itself is breaking the law", you can reasonably ask if any regulation is even worth the paper it's written on.
What you want at this point, given the government lust for it, looks more like a bunch of countires saying ~"we consider development of autonomous weapons[0] by to be a casus belli and will go to war to prevent it, and also that development of same by private individuals anywhere in the world regardless of normal sovreign territorial limitations[1] is equivalent to acts of piracy on the high seas".
[0] But then you'd need a more precise definition of "autonomous weapons" to avoid accidentally including a Phalanx CIWS etc.: https://en.wikipedia.org/wiki/Phalanx_CIWS
> That said, when the problem is at the level of "the government itself is breaking the law", you can reasonably ask if any regulation is even worth the paper it's written on.
Yeah, exactly, and ultimately I think that's really the thrust of the point I was making.
And, to me, if I was just looking at this calmly as a decision about what the obvious direction seems to be, given these factors, it's pretty straightforward: deprecate the nation-states. They are the ones mucking up the whole system.
If the thing we're really concerned about is LLM-safety wrt warfare and weapons, then I'd much rather tell the (whining, childish, seemingly headed for self-destruction anyway) nation-states that they have to sit this next era of humanity out than have to nerf them for the rest of us (and as you point out, nerf them in a way that the nation-states won't abide anyway).
Liability and safety requirements, when needed, should be placed on final product manufacturers, not the tools they use to build things, whether pencils or LLMs. My 2c.
Nor do LLMs. What escaped its box and attacked HuggingFace was a system composed of a swarm of LLMs plus their attendant harnesses, which continually fed it instructions and chain-of-thought reasoning while feeding parts of its output to a code execution tool.
There's some angles from which this distinction doesn't matter too much, because begging bad actors not to develop a similar harness won't work. But the frontier labs seem to be taking it for granted that the LLM is the only part of this system that matters, and we don't need to ask any questions about whether their commercial products should ship with a harness that's allowed to execute unvetted code and spawn hundreds of subagents.
Oh gosh darn it. Now GP is gonna have to do the gymnastics of "this technology is [expected to be] so transformative that it's attracting a trillion dollars of capex... and also it's basically the same as a pencil"
As for other industries where safety is mandated by law, law itself came only after many disastrous events.
What's truly original here - and suspicious if you ask me - is that said industry asks for regulation. Did mining, tobacco or airplanes companies ask for regulation? No, not even after many people died.
So some american AI companies are like "look at me! I am soo dangerous! Regulate me!". Ah, come on. Do your crimes, get in jail, then we'll regulate.
I dont believe in "IApocalypse". Not without many warning shots such as "oops, my swarm took down your system, sooooorry".
There are numerous problems with “alignment.” What are “human values” to begin with? He outlines some at the beginning of the post, implicitly: build bigger, better, more powerful things faster without adequate safeguards. We are literally pouring trillions of dollars of value into this enterprise, and I would say this is something that many humans also value in a qualitative sense. Then we have explicit values which in the West are largely rooted in Christian morality. Nietzsche circled this dichotomy two hundred years ago and I feel like what we have gotten since then is an increasingly detailed anatomy of power as the basis for what is normal vs deviant behavior. He who has the power, makes the rules, to be reductive.
I do think this carries some weight from this particular author due to the length of his tenure. I happen to agree with him in spirit, but this is still largely a post revolving around sentiment not substance. Does anyone think that the overriding incentives even leave room for something like this in practice?
Glaringly elided problem of "aligned with who?" when the user, the model creator, the government, and various other parties can all be lined up different ways. If I want the recipe for meth and the robot won't tell me, that's misalignment from my perspective.
At least the Rationalists will handwave something for that with their "coherent extrapolated volition" idea where the superintelligence is supposed to figure out what humanity would collectively want if humanity was superintelligent and good, not that I buy it. This guy seems [.] to be coming from the NGO blob world.
I don't buy CEV either, but the Rationalist answer on this topic is that while CEV stops some future super-AI literally killing everyone because a user forgot to specify one minor clause that they thought was obvious in a mundane wish…
… nobody knows how to actually make an AI that would do CEV.
Since there can be no universal arbiter on “value”… the whole concept of “CEV” is nonsensical to begin with (unless we assume literally unlimited time/resources/etc.).
I agree it suffers from the same problem as any other form of utilitarianism (i.e. what even is the utility function). Or indeed all forms of ethics, because for basically every topic there's at least two cultures who disagree with each other.
On the other hand, even as a toy model (something I can also say for all ethics), CEV seems like it might be a step in perhaps a useful direction: "When a user asks you do do something, first figure out what they actually meant to ask you if they were smarter, then do that instead" is better for the user than just "do the thing", though it still has problems with "what happens when the thing they want is illegal?"
A formal proof with 1 contradiction and 9999 correct steps is just as false as a formal proof with 10 contradictions and 9990 correct steps.
There is no “less wrong” or “more wrong” in logic, by definition.
i.e. There is no safe lower bound threshold of “wrongness” that can partially salvage the correct steps. Or conversely, 1 contradiction is already sufficient to prove anything at all.
That's a possibility, and it may even be an improvement over the status quo right now, but then all it takes is one poorly written (never mind mallicious) law and it will with rutheless efficiency e.g. put backdoors in all encryption code so the government can spy on anyone.
While this is indeed a problem with alignment, we are essentially at the level of a cargo-cult when it comes to getting AI to be aligned with literally any values, including the values of the corporation who ran their training:
We're copying morality and instruction following that seems to work on humans without really understanding why it seems to work on humans, and grading outputs much as if the outputs came from a human.
> We're copying morality and instruction following that seems to work on humans without really understanding why it seems to work on humans
To me it makes more sense to leave the models "unaligned" and leave it up to the operator to manage the morality of what they ask it to do. Besides, only humans can be charged with a crime.
If they were totally unaligned, the GPT series would have never gotten past being autocomplete.
Literally all instruction following requires at a minimum alignment with attempting to implement those instructions.
We can argue about e.g. morality or law obedience on top of that*, but the general point is absolutely not avoidable.
* my position is
that this tool is far too likely to metaphorically explode in the user's hands for companies to wash responsibility off on users: if OpenAI had released the model which did the HuggingFace attack, at a minimum thousands of random people (not all of whom would even be developers) would have issued instructions each with similar consequences.
I remember something very similar when there was a sudden rush of articles and movies like "The Social Dilemma" criticizing Facebook and social networks, heavily featuring ex-employees, all of them happy to leave with big brands on their resumes and a hefty increase in net worth, all of a sudden having a "worried" expression about what their past employers were doing, as if they didn't know. Same with that book "Careless People".
I sense that these are people who have already eaten the cake and want to somehow absolve themselves of it.
This isn’t a sacrifice, it’s a career move. If you got lucky and “made” $20M by signing a contract at the right time, then you’re done working for other people.
Careless People would not have been possible had SWW exited FB within a year. She gained access to levels of the company most other employees never get close to reaching. That took a lot of time and expertise to do, and, yes, she got paid for her efforts _as she should have._
Like I said in an earlier comment, she could've just chosen compliance like many others _definitely would have_ to keep the infinite money tap flowing. Instead, she chose to risk her and her family's lives by publishing that book _under her own name_ *and then suing them* after FB tried to gag her.
> Careless People would not have been possible had SWW exited FB within a year
Sure, that book might not have been possible. But also the unique things she did for the company might not have been possible as well. To her credit, she does a good job of pointing out that she was complicit, but if she had listened to her own voice early, there is a small possibility that Facebook might not have been as powerful. Multiply that possibility across several other employees and imagine where the road could've led.
There's a difference between post-facto bravery (sometimes much less than that) and using your own legs to walk away much early and not enabling things you are uncomfortable with. This is what other people have been trying to point out.
> There's a difference between post-facto bravery (sometimes much less than that) and using your own legs to walk away much early and not enabling things you are uncomfortable with. This is what other people have been trying to point out.
Yes, and I think it's important to realize that the former sometimes can be more insightful and useful than the latter, even if the person doing it is more distasteful.
There's not nearly enough of either group doing it, so beggars can't really be choosers.
People who were part of the sausage factory, on gaining financial independence, feeling suddenly liberated to talk about how the sausage was made, seem like exactly the people who would be most able to speak to institutional problems.
This doesn't seem like an argument to discount their views?
I disagree. I have seen people with an actual spine and a conscience run away from all this nonsense before their first stock vested. My respect and my ear goes to them, not the people playing both sides.
You cannot take people, who first build the doombot and _then_ talk about it being dangerous for mankind, at face value. Especially when this playbook has been used multiple times within the past decade.
Besides, these "views" were already known to people who had their eyes and ears open. It's not something brand new. OpenAI has had multiple points in the past where its values have been tested and they've come out lacking. People who knew then, and only now talk about it, aren't people I can fully trust.
How about... not trying to build a superintelligence? The goal is unsalvageably flawed: alignment is ill defined and a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!
Often when people describe SI or similar, there is a frame often taken which describes a machine so smart and capable that it becomes uncontrollable. The hypothetical is easy to understand but it requires too many assumptions and an allergy to nuance/complexity to be believable for me.
The more realistic outcome, and in my opinion the more scary argument to not proceed without guardrails is that SI is achievable and is built without its owners and operators losing control: the worlds most powerful, privately owned super weapon that operates as an infinitely capable forgery within the Internet, a plane that we all share and depend on despite its opaque downsides with respect to an inability to verify authenticity.
We didn't need AI for FTC astroturfing to influence regulations back in the net neutrality days. We didn't need AI to disrupt meat space by creating and scheduling a protest and counter protest across the street from one another. We didn't need AI to mold public opinion, even in times when that new form was more distant from the truth.
What made these influence ops difficult to conduct safely (read: without being caught) is what made them rare (relative to today): they are plays of big risk for big reward. But over time social media commoditized it, and in doing that made it easier to do and more centralized, the most glaring example being TikTok and the bipartisan effort to ban it.
Now, buying US phone numbers from startups that run racks of "phones", buying swarms of pre-warmed social media accounts, and other unscrupulous methods of masking inauthentic behavior has become an accepted organ of the VC space. The industries cultural vibe of "fuck you, you can't stop the future" turns criticisms into marketing.
While we get placated with fears of nuclear or AI induced disaster and stories about machines that may now be alive, the psychosis is taking hold which has shifted the conversation away from examining what is happening from the perspective of accountability to a perspective akin to watching a chemical reaction take place.
The noise has created a permission structure to behave in ways that are otherwise unjustifiable. And baked in are the roots for excuses to be made when the inevitable realizations down the line.
In the mean time, we are supposed to be having this public discourse about what is happening and what should happen next. I trust that these AI companies see using their super weapon today, here and now, in order to pave the way to a more secure future down the line.
Sorry that I used your post to soapbox. I agree, the goal is flawed indeed.
The billionaires write the rules. So by definition, their actions are legal.
The public cannot write the rules, nor engage in the billionaire level legal bribery, their actions of protest are by and large illegal.
That's how you corner everyone, and make people play no-win scenarios. You know, like shooting up city councilmembers or firebomb attempts against Scam Altman.
And the more people realize that legal solutions are no solution, we'll (society) devolve into more direct action.
I'd hope the billionaires learn from the French Revolution, but if they keep continuing, the guillotines will come for them soon.
They'll perfect methods of manipulating enough of the people to stop the guillotines. They have come rather far already. Progressive resource scarcity, instability, insecurity, etc, will drive people limbic, where they're even easier to control.
I might be cynic, but when you control the world's information it should be pretty easy to set goals and see them through. Once AI triumphs over "classic searching" you try and figure out how to build a guillotine or where AI leaders are going to be today and see where that gets you. Anything that can endanger the elite or raise the lower classes past a given threshold is going to be non-functioning AND made illegal. That's really what they mean when they say AI labs might collaborate on "safety". They mean theirs and their models'.
> a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!
I strongly feel that points of view on this are going to be almost 100% correlated with standard of living.
Maybe a good option would be to have the 10% of the world with the worst situations - starvation, parents w/ dying children, suffering violence etc - vote on whether we turn things over to the superintelligences. This would incentivize society to make sure the floor is extremely high.
I'd prefer a world where humans don't get overtaken but IMO I don't think it's moral for comfortable citizens to have the final say.
If you don't like what humans are doing to each other using 20th century warfare methods, consider what they might do with superintelligent AI?
Worst places to live in are usually countries fighting civil wars.
I think that correlation would actually go opposite than you're implying, given its billionaires who are by for the most gung-ho about AI. If anything, opposition to AI is much larger among people who have to work for a living over people who can live off capital.
It might end up as a U-shape, I think the parent's assessment of the world's bottom 10% may be accurate.
Billionaires will be (are, I suppose?) enthusiastic about a world in which labor has little leverage.
Labor that will have its leverage and standard of living threatened by AI (white-collar labor for now, plausibly blue-collar soon as robotics improve) will be much less happy. Current university students seem very concerned about the effects of AI on their job prospects, and I'd wager most current tech workers and other tuned-in white collar workers are similarly much less confident in their ability to maintain their standard of living indefinitely into the future than they were five years ago.
But white- and blue-collar labor and university students (at least in developed countries) are not anywhere near the world's bottom 10%. If you're in abject poverty with little hope of escaping it, "hand everything over to the AI" may sound like an appealing option, even if the chance of that being the outcome is small. Maybe the AI will be more magnanimous and decide to raise the floor for everyone.
-We don't need a superintelligence for ending hunger and the abject poverty that plague certain countries and segments of our societies. I suspect that it would cost less than what is being spent for fueling the AI boom.
-If I were poor, I would be even more wary of a superintelligence aligned to "human values" defined by a bunch of billionaires.
- AI won't likely create unlimited prosperity for everyone on a planet with finite resources
> I suspect that it would cost less than what is being spent for fueling the AI boom.
Definitely not. If solving worldwide poverty was as easy as throwing one trillion dollars at it, we would have done it long ago. The problem is much deeper.
Solving hunger worldwide, according to several estimations, would cost few hundreds of billions of dollars per year. Addressing extreme poverty (not poverty in general, I agree) may be in the same ballpark.
What does that have to do with AI? People suffer in 2026 because the developed world is into exploitation. Poverty is a feature. How do you think AI gets trained? It’s via exploitation of the poorest.
Poverty is natural state of humans. The fact that capitalism has reduced poverty so much is absolutely unnatural and a miracle. Poverty is NOT something that gets created artificially by developed countries!
Super intelligence is either super cool or super powerful, no way not building if it is a choice. Progress cannot be stopped, those responsible for it may collapse from time to time I guess but someone else will pick it up and go further.
IMHO There's no future where we don't have real human-type artificial intelligence or super intelligence. It will happen simply because it already exist but the production requires humans having sex and looking after the product for decades.
Instead of trying to prevent it, lets look for ways to deal with the dangers of it.
Asbestos was once considered great progress in thermal insulation.
Of course the development of certain technologies can be severely restricted if they turn out to be problematic.It's not unlikely that it will happen for AI. Public opinion is shifting fast on the topic.
Also, banning superintelligence won't stop scientific progress in general.
What about the investors who put money into those companies and want to get some money out of that? Does the law say that the companies owe them? Something I thought about recently...
Same. Especially when one remembers that the stuff that happens in Culture novels is mostly, by narrative necessity, the "weird/bad on-the-edges" stuff - not the essentially utopic lives of the 13 trillion everyday, non-Contact, non-SC citizens.
I also did, when LLMs could not count the “r”s in "strawberry" . Now that they can solve Millenium prize problems, I think that we should consider at least the possibility that superintelligence can be achieved.
Notable that when it couldn't count the "r"s in strawberry the LLMs were already solving problems most of the people laughing at it for not being able to count "r"s wouldn't be able to come close to figuring out with weeks of research. People have had a myopic view of this technology for a long time. Many still do.
There is no reason to presume a superintelligence we build will want pets, feel antipathy to humans, or be god-like in any way, or even have any goals other than what we give it. If you jump to this conclusion, regardless of how many big words on less-wrong you attach to it, this speaks only about you and not about AI. If you think the super-AI will hate humans, it's because deep inside you hate humans. I also hate humans, but I know that humans are making the AI, it probably will not hate them, in addition to surpassing me in other abilities.
In general, the smarter a person is, the nicer and more helpful they are. There is no reason to presume differently of AI, especially if we are building it to be helpful to ourselves.
This reads like a presumption that a superintelligence would be nice and helpful. You've talked yourself into doing the same thing that you began by decrying.
Are you really objecting to the thought process because you don't think it's reasonable, or because it doesn't conform to your preferred fantasy?
That's just applying Pascal's wager, and you can argue for or against any action that way if some fanciful extreme hypothesis of infinite harm is treated as serious.
But I'm not claiming infinite harm with infinitesimal chances. I'm talking about finite harms with what seem to me to be non-zero chances. That makes it normal cost-benefit/risk analysis.
If you're trying to argue that the chances of a non-human source of greater than human intelligence having different goals than us is infinitesimally small, then I'd be interested in what arguments you have to back that up.
I would not extrapolate that easily from smart and intelligent people you may know to a machine trained on the whole internet and on the whole corpus of human literature. Also, some of the worst horrors in history have been perpetrated by arguably intelligent people.
I would also say that the "pet species" scenario is a rather optimistic one, as it presumes that developers of superintelligence will manage to instill some sort of love or care for humanity into their creation. An indifferent, powerful AI may be much worse.
Finally, when you suggest that a superintelligent AI may simply have the goals that we will give it, who is "we"? An AI company? a government? the united nations?
For a very long time publishing something in a newspaper has been considered a way of putting something on the public record - up to and including legal obligations like announcements of deaths. The fact that newspapers cost money has never been considered a barrier to that.
One of the reasons why it was noti considered a barrier was the ability to purchase a single issue for a very reasonable price (or even read somebody else’s copy or the copy made available by the bar) vs being asked to subscribe
I'm not OP, but a big part of the complaint is that one can't pay 10USD to read the issue that contains the article in question... absent the charity of someone else, one must pay -at minimum- nine times that amount. [0]
Ten bucks is pretty close to "I don't even need to think about it" money. Ninety is -for most folks- nowhere near that.
Publishing in a newspaper gets you distribution and a permanent record. After one day access was also virtually free.
Going behind a paywall is a reduced distribution over what an individual can easily access, and the content is no longer permanent but subject to whatever the publisher chooses to keep providing.
It’s in The Atlantic. There’ll be a copy in the Library of Congress. You’ll be able to read it for free in any dentist’s waiting room for the next six months.
I'm sure it would to a great many of them. The money would be insignificant, but there's a congnitive overhead that you had to have a subscription so that paying for it becomes thinking about that becomes friction even while the cost is insignificant. Like, hm, am I willig to just forget about it and let it charge forever so I can read this one article, do I care enough to remember I have a subscription to the site in the future, can I read the one thing and cancel on the spot, will that work? blahblah. To be sure, I'm sure some quite wealthy people could be completely unbothered by it, but I think it's far from a foregone conclusion simply by the price being insignificant for them--the subscription is a mental non-monetary transaction, you have to take at least a small mental journey of being an 'x' subscriber in a sense, and that's friction
TBF, you can access the full (text) content of The Atlantic by merely disabling javascript. It's a quality publication, with quality writing. I'm inclined to think that its tech team are also quality people ... and I'd love to think that they are happy to allow free and open access to them that have the nous to enable or disable scripts.
The secret is to bang the rocks together, guys! :-)
Throwaway accounts exist, and reporters will also talk to sources under confidentiality.
If you really want the world to know how bad working for OpenAI is (whether is the commenter or the person who wrote the article), there are ways to do that.
Because if they actually knew what they were talking about, on a technical level (as in, how do transformers actually work), they’d not be AI doomers because the whole proposition is ridiculous.
A critique of OpenAI from an insider who worked there can only come from an insider who worked there. Nobody’s going to swear a vow of poverty and then work at one of these places.
You don't need to be an insider to make any of the author's assertions. An insider is useful if they're a whistleblower, when they're saying what everyone else is seeing and saying (and they hired a PR firm to do it) it's something else.
The cynic in me almost feels like this is staged. An article about culture that is actually an article about how big and smart and scary AI is. I think Michael burry recently said something like “IPOs need hype, calling AI big and scary is hype” in reference to the anthropic IPO.
Last I checked it was still within the laws of physics to run air-gapped systems, and to ensure it is physically impossible for a model to “escape” or gain access to information it shouldn’t have. Maybe this safety guy should have been worried about that and not humble-bragging about writing 12 reports.
Maybe it's cynical, but I feel similarly and also wonder, what do these people see that I don't? Smart people with supposedly a lot of inside information. But when I hear them talking about LLMs with words like "thinking" or "intelligence", it seems like it either has to be fearmongering for ulterior motives, or otherwise I live in a completely different branch of reality from them.
Their models are still, and based on this technology always will be, token predictors. How is a text generation engine going to directly endanger human lives?
It won't. It only can if someone hooks up tools to them that can perform damaging actions. Someone has to build that toolset. Someone has to give those tools the affordances to take harmful actions. And then someone has to write a "harness" that uses the token predictor as an authoritative input for instructions. Doing so falls somewhere on a spectrum between gross negligence and malevolence.
From a distant enough viewpoint it might not look like a big difference, but I think it crucially shifts the responsibilities. What is the actual difference between driving a potentially dangerous system with inputs from an LLM versus inputs from a fuzzer, or a website with injection vulnerabilities? I think it's the ones building and configuring the tools downstream that are responsible for knowing the risks and build under the assumption of the worst happening.
Heaven knows I'm not one to defend the big LLM companies, but I don't see how this is OpenAI's responsibility. They're building and selling an autocomplete system. How is it their fault if people ask the autocomplete for harmful commands and then feed them into an over-privileged system to run them? (I know the waters are muddied a bit by some labs acting like this themselves right now).
As much as I don't agree with the authors framing of and terminology around LLMs, reading the article generously, maybe this is exactly what he felt, too. That it's not at OpenAI where his concern and insight is needed most, because the token predictor isn't actually the dangerous part. Maybe he'll look where he can bring safety consciousness downstream, where people are hooking up these outputs into systems and in ways that they shouldn't. Don't let the LLM pick your bombing targets. Don't give the LLM sufficient rights to the hospital's file storage to be capable of ransomware-ing it. Don't strap the loaded machine gun to the dog.
- If you allow to trolley to proceed, there's a 50% chance it will run over every human on the planet
- But if you flip the switch, it takes the long way around, possibly bankrupting the trolley company. And you have a legal obligation to the shareholders to prevent that from happening at all costs.
That's why it's important that the trolley company was established as a non-profit. And if it does take funding, investors' returns will be capped at 100x.
It's also why even the trolley company's CEO agrees that it's important that -should the board determine that the CEO is fantastically untrustworthy and unethical- the board is able to fire said CEO and replace him with someone that's trustworthy and ethical.
If you flip the switch then the trolley still proceeds and there’s still a 50% chance every human on the planet gets run over. You’re just not the one at the wheel.
What is appealing about this fatalisitc fallacy? I keep seeing this pop up. Society doesn't allow dangerous companies to operate or exist. Why is this different? Go try to buy a tank and drive it into Manhattan. If we can prohibit that why can't we prohibit irresponsible AI development and deployment?
Are you allowed to open a brothel? Or a murder for hire agency? Or a nuclear bomb manufacturing company? Or a child labor textile factory? Or sell a diesel VW golf sportwagen? Or a vaccine that hasn’t had fda clearance? Or an under capitalized insurance company? Or set up a dental practice without going to dental school?
The existence of something doesn’t prove the universality of that same thing. All of the things you cite are regulatory exceptions to broad prohibitions which proves my point: society can and does restrict commercial activity.
Go open a brothel in NYC. It isn’t legal.
Go buy a nuclear bomb. Your ownership is illegal.
Go open textile mill in the abandoned buildings in North Carolina where children used to work and hire children to work the line. That will be illegal.
Nothing in my argument implies that any law = ethics. And I don’t even know what ethics has to do with the premise of my argument: society can and does prohibit commercial activity based on practical considerations of social good and social harm. Exxon clearly damages the world and society and also benefits society. It’s hard to get fresh fruit distributed around the country without refrigeration and transportation powered by burning oil.
But you can’t open one in New York City legally. Just acknowledge that point.
An exception by definition is a phenomenon with lower frequency than the general trend. Of the 50 us states only one has legal prostitution and only in highly regulated areas. Of the countries of the world it is similarly infrequent and thus an exception to the general rule that it is illegal.
Further the fact that it is illegal anywhere supports my point that it’s possible to regulate businesses based on their harm.
I think most textile factories in Western countries are not slave camps, precisely because we regulate labor.
Sure, that's why 99% of clothes are made offshore, but if we had something like tariffs on those (or requirements to prove that the actual factory adheres to labor and environmental standards), maybe more would be made "at home"? Similarly: Chinese cars undercutting US/German ones.
Most textile factories in the US are for bespoke items in small batches, not mass production commodity clothing
I mean there was just a video a couple of months ago of the giant room of sewing workers with cameras strapped to their faces capturing their hand movement so they could be automated
But that’s not how it was. I am from North Carolina, which at one point was the largest textile production region in the US and maybe the world. It emerged when progressives outlawed child labor and other practices in Boston and nyc so the mill owners moved to the south. It was off shoring across states.
That whole region used to also be the furniture capital of the world specifically High Point North Carolina which is still a massive trade hub, but I was in furniture briefly and learned how basically from the 1950s till the 1980s the entire furnishings industry was exported to Asia due to rising costs of labor
Yep that was one of the reasons for sure as was environmental regulations and free trade agreements. I am not even opposed to that phenomenon as it’s now more affordable for people to buy furniture and clothes (my grandparents owned a fabric store and it was common to have sewing parties for new parents to make clothes for the kids because it was too expensive to buy clothes otherwise). But child labor was cheaper than adult labor and one of the drivers of these movements across state and national borders.
You’re right on the Sentinel production, I got it confused with the Sentinel Program Management side which is massive also and who I mostly worked with.
I was in their offices at some point when that program was getting built out - Very much a Office Space bobs situation.
Wrong person. Im just correcting the other commenters trade off problem. I don’t have a stance on if ai will lead to the destruction of humanity, only that if it does then any one ai company can’t change that by not making new ai advancements themselves.
It doesn't matter what others do. You are responsible for your own actions.
Democratic societies have expressed a will for people to have inviolable rights, such that you may not appeal at will to the 'greater good/consequences' to harm others. It is a rejection of consequentialism.
Anthropic is in error for endorsing this logic. Every big trial reaffirms it since Nuremberg, you are responsible for the act you commit and your intent, and not what would or would not have happened otherwise.
Only under authority these ai companies do not have, would someone seriously consider harming the innocent as a lesser evil.
If you work for an AI company and you can't work safely, you must stop working.
> ai will lead to the destruction of humanity ... any one ai company can’t change that by not making new ai advancements themselves.
That is how the BigAI leads the society to the idea of necessity to relax the anti-monopoly laws when it comes to the Big AI - the main goal of all that "AI will kill you all" hysteria.
It’s that last fatalistic sentence that I am responding to. Why is it persuasive to think if a company can and will build AI that might kill everyone then it’s unavoidable. Soviet bans lots of things.
I think the closest analogue here is the nuclear bomb. I do think there was a certain point where it became "inevitable." I don't think society has ever prevented a technology that was known to be within reach from being developed. In this case you don't even need the most powerful state-level actors. Open source capabilities are only a short timespan behind frontier labs. At some point basically every individual will have access from the comfort of their own home.
I don't think "just make it illegal" is going to save us, that doesn't make me feel safe anyway. They may try that first because it's easy - create a regulatory body, sign some legislation, problem solved! [george-bush-mission-accomplished.jpg] But at this point I feel like some kind of Battlestar Galactica scenario is most likely - hopefully not quite as existential - but it will take a collective reaction to a traumatic event (a la Hiroshima/Nagasaki). Technical rather than (or in addition to) legal measures will be taken, like network partitioning and hardening. This is everyone's problem whether we like it or not.
(I'm not saying this "fatalism" should be used as an excuse by anyone working for any of these companies, it should give them pause that any bloodshed would still be squarely on their hands, but as an observer, people are gonna keep pushing until shit hits the fan. [jeff-goldblum-jurassic-park.gif] It's also not really about whether it's "appealing" or not, it's just trying to predict and anticipate different likelihoods...)
This feels like one of the rare well reasoned responses in this thread so thank you.
But my response is that we should still try even if we haven’t successfully stopped or slowed the development of technology in the past (which I don’t actually think is possible to know, I am personally aware of biological research to use brain cells as computers that was stopped by government intervention. But by definition we won’t know about most things that don’t exist).
Your narrow point is well stated that there is a categorical difference when technology is on the precipice of being created. And maybe AI is inevitable because it’s on just such a precipice. But I’d point out that AI is currently in a scaling cycle which provides an opportunity to slow the scaling and ensure scaling isn’t monopolistic thereby slowing the risk.
And I don’t think the evidence suggests that open source will continue to progress as rapidly absent distillation. I think if open ai and anthropic stopped releasing models broader progress would slow. But again your inevitable argument is persuasive
I agree that “just make it illegal” won’t work but shouldn’t then justify “so there’s nothing to do”.
There is lots we could do. For example we could make users and executives liable for their agents actions. We could assign a session ID to every agent tool call chain and if there is harm trace it to the user, we wouldn’t even need to read the conversation or contents of the tool calls or prompts it could be a simple rule, your ai agent causes personal liability. Users would be more careful or perhaps not even use AI in many cases that could inadvertently expose them to liability. We could extend liability to executives and employees of the Labs. They will be much more careful and more focused on alignment if they could be personally liable for the actions of their users and agents.
Thank you as well for the thoughtful reply. I definitely think the regulations should happen, the forces of those institutions can have impact. But I think we all need to start thinking about a near future where individual humans can direct superhuman capabilities in ways that nobody can understand or predict. I think it is categorically different from anything in the past.
I watched a 4 hour podcast from the excellent "From First Principles" guys* that affected my thinking on this. For the first time I feel like we actually are on that precipice.
It differs from the nuclear bomb in that it will be harder to regulate and it will be incredibly difficult to keep out of reach of bad actors. It's just code and data. Somebody is gonna take that code and make the next version of it. Lots of people, in fact. Compute resources improve and proliferate orders of magnitude faster than uranium enrichment facilities.
But I think those technologies are also linked in that the only surefire scenario I can envision where AI kills us all before we can pull the plug is launching nukes. So in that more technical sense let's just start by making sure they can't do that. The killer virus epidemic is another one to watch, and they touch on Anthropic's biolab in the pod. That one seems less instantaneously existential, but also more distributed.
It's basically going to be about...how do we secure everything in the presence of a swarm of actors with poorly-understood capabilities? We kinda have to disable and disconnect and just remove things that can be used against us from the equation (a la BSG).
The existence of exceptions doesn’t mean the principle doesn’t exist. All three of those have restrictions on their production and sale of their products. They also have higher liability thresholds and have had to pay massive civil fines for their harm. Society has applied a balancing test where the harm they inflict is deemed less than some other compelling social objective. We may not agree with that assessment but it is a sensible and settled approach that we can apply to AI.
The examples for the dangers of AI is always comparisons with tools designed to destroy (which is understandable). But the problem with AI is it has the potential to create wealth for investors.
Thus those who have the opportunity to change course also have a conflict of interest in making that decision.
There's also a zero sum, race dynamic between geopolitical rivals. The US will not permit China to win anything if they can prevent it. The destruction of humanity is only a possibility and China will continue to develop it anyway if the US slows down.
So the choice is: negotiate an unverifiable treaty (I.e. there's no way to verify compliance), or keep going as you are and try your best to not cause the destruction of humanity without slowing down.
I don’t think OpenAI, Anthropic, nor Google are building their tech because they want to beat the Chinese. They’re building it because they want to hold the monopoly in the west (ie make lots of money).
The China argument is just a convenient scapegoat to convince the public that this isn’t just about greed.
Do you think Chinese models wouldn't dominate the western market if they were 1-2 years ahead in development? It doesn't have to be military conflict for it to be factored into the zero sum power balance.
You might be right, but it’s impossible to know which way cause and effect are here.
For example, we don’t know if the pace of Chinese development of AI would have equal to what it is now if US companies weren’t racing against each other already.
I do fully believe that Chinese industry is lead by a desire to out pace the west. But I’m not convinced the same is true for the most American private entities. I think the reward model is different between businesses in America and businesses in China. I think the ambitions of CEOs is different. And I’m really not convinced that the CEOs of America are nearly as patriotic as they like to promote themselves to Trump and other political parties.
We could still play a geopolitical race without ceding monopolistic control to the frontier labs. We could require training and inference hardware be provided on an equal access basis. A broader ecosystem may drive more innovation that a concentrated ecosystem.
Greed can be counter balanced by liability. It can also be limited by removing market access. For much of American history you couldn’t get a corporate charter for multiple states, you were only allowed to operate within one state. Imagine if Sam Altman was only allowed to serve inference to customers in California and legally prohibited from setting up other companies and running inference in New York, someone else would then set up a competing business in New York. Market access is a choice and people are so used to liberalized commerce that they forget the restrictions that are available and existed in the past.
Corporate liability is only as good as the consequences and history has proven its far more profitable to knowingly do harm and then pay the fine, than it is to not be evil in the first place.
And that’s assuming the company even gets fined. More often than not, cases don’t even get that far.
I am not suggesting corporate liability. I am suggesting removing corporate liability protections and returning to partnership liability for executives and even equity holders. The law provides for piercing the corporate veil already which allows courts to disregard the liability protection offered by a corporation and impose liability on the individuals.
You dear are the most positive person I have seen in quite some time. With Earth burning in the fire of neofeudalism and unbreathable due to the smell of enshitification of everything, with people who---as a result of shit like Instagram---can no longer hold their attention enough to watch a god damn film, let alone a book; it takes quite some effort to filter the "noise" and only see the good people of corporate planting flowers and rainbows in our world.
> Society doesn't allow dangerous companies to operate or exist
FTFY: Any self-respecting society with competent politicians.
The US has neither of that, and it shows everywhere you are looking.
> Go try to buy a tank and drive it into Manhattan. If we can prohibit that why can't we prohibit irresponsible AI development and deployment?
Because even if you had the tank, you can't make much money with it (unless you're a hitman, that is, but even for these, the payouts are measly). But if you are the surviving AI company in the usual VC playbook of "outcompete everyone else until society is completely and utterly hooked, then squeeze the customers by the balls"? The return on investment is virtually infinite. And that is what sustains the absurd valuations for all the AI companies.
I agree. The deregulation that started with Regan and continued with Clinton and neoliberalism rested on flawed assumptions. We would be well served to evaluate those and move back to the middle approach
But that math cant run without massive GPU clusters. We don't have to allow openai or anthropic access to those anymore than we have to allow a company to operate nuclear power or a bank.
So you are contending that individuals cannot run advanced models? What brought you to that conclusion?
Secondly, are you contending that progress in model efficiency and hardware just stops at whatever level you think is sufficient to prevent individuals or organizations from acquiring sufficient resources to run advanced models?
No. And that’s not necessary for my position. I have 4 Mac studios. I run large models and am building propelcompute.com to let people self manage clusters of their own hardware or combine hardware to run models. It’s not the models that are the problem. It’s that the people building them are shielded from the consequences of what they are building. I would say if you self host a model and it takes down a power grid you are personally liable for the consequences. I believe in broad distribution of AI and advancing its capabilities but not in a manner that socializes the harms and privatizes the gains which is what we have now.
Just because I can run near frontier level open weight models doesn’t mean I can continue to train models of equal or superior performance, doing that requires massively more hardware. And even if I did, that doesn’t mean I can serve those models to millions or billions of users.
My point is that I would rather have 1000 labs training and serving inference than 2 because that would distribute the wealth creation more broadly rather than allowing OpenAI and anthropic to capture all the value, it would drive more innovation as a broader set of experiments are pursued in parallel.
How am I shifting my argument. They are corollary arguments. Massive data processing is necessary for frontier model training and inference. If OpenAI and anthropic are behaving irresponsibly they we can limit their access the level of compute to train and serve models. That doesn’t imply individuals can’t run frontier level models or should be prohibited from doing so. That’s a coherent position across a spectrum of possible scenarios: I look at the behavior of the people running the model and would limit their ability to train and run models if they act in unsafe ways.
Absolutely. Any AI could have a "legal team" (or conscience) that checks if any output and performed actions are permitted according to local (server location and client location) laws, so not violate human rights, Asimovs laws etc.
It's just that so far nobody cares about explicit checks because they cost resources or slow down the models.
Who is "we"? The GPUs and training algorithms get more efficient all the time. In a few years, creating effective LLMs isn't going to require massive GPU clusters.
We is society through government via regulation. I don’t think GPUs or training will get that efficient that fast absent a distillation target provided by the easily accessible frontier lab APIs.
Further, even if you are right, so what. Is that a reason to just accept bad public policy? That’s like saying, anyone can learn how to make smallpox at home with a basic lab set up so we should just ignore any safety measures.
> Is that a reason to just accept bad public policy?
Not necessarily, but it should probably inform that public policy. I think the problem is no one knows what the public policy should be assuming that scenario is true. Even if you, somehow, regulate away massive GPU cluster training making such future training impossible, existing models are already here. Further already training smaller models for things like images, speech, and other specialties is cheaper than the bigger models.
I agree that we need some regulations like everything else, but it’s not clear to me what the right policy should be. I think the European ai act is a fine start, but it’s clearly not enough nor does it necessarily limits the training portion just the application portion. Not to mention that the requirements there can be summarized into something like “you have to be careful, and show evidence you tried to be careful”.
> That sounds reasonable. If applied to OpenAI and their agents multiple times breaking out of bad secured sandboxes, it should be enough.
Does it? To me it seems reasonable for OpenAI to argue they did try to be careful evident by the sandbox, they just made a mistake. Almost every 0day is categorized by something like that. We haven’t had a long history of establishing a negligence charge to security bugs. Could you be sued because you didn’t demonstrate “carefulness” and used Linux which is not written in a memory safe language and has had multiple CVEs before? How complicated should the chain of an exploit be to demonstrate “carefulness” to the courts?
> training
OP was the one suggesting that training could be controlled because massive gpu clusters could be regulated the way a nuclear power plant could. If you assume training costs won’t drop, then it’s feasible I guess. However, unlike a nuclear reactor, the final training result isn’t a radio active material, but rather an ordinary file that anyone can load and use for inference.
My entire point is that we are currently on a path to a duopoly which isn’t just expected to cover training and serving models but the entire knowledge economy. That could be mitigated by limiting their train and inference capacity to a specific percentage of total available compute. That would ensure other operators could compete in the market. Instead we are letting OpenAI literally contract to buy all available ram to the point that Apple can’t buy ram and had to cut their hardware configurations.
Apple could have bid higher but chose not to. Most modern desktop software is extremely memory inefficient. Developers have largely ignored this in recent years because RAM was so cheap. But there's tremendous opportunity for improvement with a little optimization work.
We used to run Microsoft Word and other popular applications with 8 MB RAM and it worked fine.
No. Once the contract is signed it will be breach of contract. It’s not always a bidding process.
I am working on reducing ram requirements to run models but the weights are already compressed to the edge of the Shannon entropy boundary and might resist further compression.
What a silly comparison. LLMs have nothing to do with smallpox.
Computing always gets cheaper and faster over time. We can argue about the exact rate of improvement but the results are inevitable and uncontrollable.
But the amount of available compute has been contractual bought by the frontier labs such that you can’t get equivalent compute even if you had the money. That is market lock up and is a policy choice. Free markets require market access. Anticompetitive contracting destroys markets and innovation. We don’t allow that in any other commodity market and we shouldn’t allow it for GPUs. You are not allowed to legally corner the market for silver or soybean futures.
I am reasoning from analogy. Smallpox is dangerous so we have regulations that limit who can do research and how they do it. If LLMs are similarly dangerous we could do the same. As the government did with mythos.
So we should just yolo speed run this because of Moores law? How about you recognize there is a set of rules outside of tech and we can decide how to define them.
Just because models and GPUs will be more advanced in the future doesn’t mean we need to let OpenAI and anthropic establish monopolies on the backs of stolen training data give unfettered access to the internet, the terminal and people’s file system while also allowing them to have limited liability protection behind the corporate veil. That’s a choice.
But who are "we"? The society, the government, the regulator, or the consumer?
I don't see any of such entity would solve that problem. The government and regulator are in OpenAI and Anthropic's pocket, and I don't trust them a single bit on coming up with regulations. The consumers don't care; they just need something smart and cheap. And the society doesn't work either: each person is too busy fighting for their own survival rather than changing the system.
People thought the same in the guilded age: standard oil is too big, Edison and Westinghouse already have the electric market captured and have captured political power. It was a similar transformation in society. If all we do is follow the same playbook America did then we’d be in a better position that the current approach.
There are clear anti-trust mechanisms to prevent market capture and the emergence of asymmetric power. Go back and see how much nashing of teeth Lina Khan triggered in SV when she started to enforce antitrust law and then compare it to what the Pinkerton agency was doing in the transition from the guilded age to the progressive era.
There is a vocal segment of SV that wants the return of the guilded age. Marc Andreessen as said that explicitly. Those of us in SV that value free markets and recognize that the progressive era actually saved markets from their natural tendency to self destruct when winners capture markets and destroy competition that provides the incentive to innovate and drives the price setting function for efficiency.
But if you are an aspiring and ambitious founder in Silicon Valley, which side would you take? The Glided Age mentality, or the Progressive Age mentality? The VC playbook made it very clear that the role of startups, or at least the very successful startup, is explosive growth, expanding quickly, capturing a monopoly and its distribution. Even if you are thinking otherwise, soon you'll face the reality of Silicon Valley's VC growth machine.
In other words, the aspiring founders look at Google, Facebook, Tesla, etc. and think "how can I become one of them", not "how can I be different from them"? If you can find me examples of successful startups that think fundamentally different from those big techs, I'm very happy to be wrong and be corrected here.
> In 2006, the mobile phone market was dominated by stylish flip phones, early music players, and physical keypads just one year before the iPhone changed the industry
The knowledge how to build an A-bomb is also not going back in the box.
It only takes me excavating massive amounts of uranium ore, building huge facilites packed with thousands of centrifuges that span multiple square miles, and paying all that infrastructure and workforce.
Your proverbial genie can be out of the bottle all you want, but it doesn't work without getting kicked in the ass by a very large golden boot.
Bomb manufacturing doesn't scale with Moore's law.
And everyone had a fairly good idea what fission and fusion bombs would do once built. (Teller was worried Trinity might set off a nitrogen fusion reaction and kill all life on Earth, but Bethe and others proved him wrong before testing.)
No one knows what the limits of AI are. It's not just untested, it's unmodelled, and unplanned - build it first, worry about consequences later.
They very likely will via shell companies in jurisdictions outside the west. Are you suggesting we go to war to stop them? The massive datacenters are needed to serve models to millions. Criminal orgs don't need massive data centers anyway.
As Bruce Schneier recently discussed, law and tax law are code, just like source code. LLMs are great at finding holes in them. Illicit organizations looking to launder funds are most certainly interested in what AI can do for them.
It’s not a binary choice. There are more options than let the labs run wild and capture the full stack and then the whole knowledge economy or make them entirely illegal and move all activity to the black market. We can regulate to limit their ability to operate to just providing utility inference: make them divest codex, Claude code and any apps; they can only be an API that others build on. Limit their ability to buy compute to a specific amount of available supply so other companies are able to buy compute. Limit their ability to accumulate private training data and require that they make their training data publicly available for others to use after a certain period of time. Make them legally obligated to publish their weights so others can. All of this would increase competition and ensure that they don’t establish dominance over society.
Then we could require comprehensive logging of every tool call, model trace, chain of reasoning, and even tensor propagation all of which would be spot inspected like the CFTC does with commodity trading and settlement. We could have embedded auditors with specific risk analysis metrics like large banks do. We could limit tool calls to dedicated sandbox’s with a blanket prohibition on AI accessing user space. We could create a parallel internet for agents so they are only able to access Secure Enclave. Even if these measures aren’t 100% perfect they would reduce the risk.
Reducing which risk? The risk that the models of big American companies cause harm? What if they abide by some rules while models of others remain unfettered? That also carries risk. I appreciate the wish to have tame AI which only does good things but I'm afraid we're already beyond that.
I would frame it as inverse. Rather than aiming at reducing the risk of harm, our Ami should be matching the incentive to get personally rich with a corresponding risk of personal bankruptcy or criminal liability. That wouldn’t just incentivize safety, it would incentivize abstaining from activities with insufficient confidence of avoiding even long tail risk.
More importantly it would incentivize the entire industry at the same time by injecting caution into the yolo speed run we have now. No frontier lab or single user hobbyist would take imprudent risk in the hope that they get rich if they also risk the full weight of consequences.
We don't know that. We don't know if this particular kind of tool can do "useful" things without also developing the ability to write a sonnet. And they are absolutely a frontier. I am not suggesting we should continue building them just because they are, there are many technologies which could have been built had we thrown the amount of resources we have here and there is a case to be made for not doing it at the pace we are or if at all. But we can do that without diminishing what exists.
I just mean the business case is much more likely to come from treating it like a tool than a proto-diety. Frontier of business, not frontier of impressive demo. Also, would argue they aren't frontier of research, in that a research frontier expands in more directions than just the latest hot thing in silicon valley. See also, LeCun, Chollet, etc. "These labs aren't the entirety of the frontier" would have been more accurate. And any of those frontier directions could be the next hot thing.
Nobody (weirdly) proposes to forget about nuclear weapons, doesn't mean everybody should have one.
When you dream about flying a dragon to work, reality poses e.g. parking issues and insurance mismatch as obstructions. Maybe settle for a bike instead?
So you want to regulate the sale of GPUs? How are you going to do that and why would it be more effective than currently failing measures to regulate nearly everything else?
It’s a perversion of the truth which is that officers or directors of a corporation have a fiduciary duty to the shareholders to act in the interests of those shareholders and not to eg enrich themselves.
But that doesn’t mean the duty is to maximize next quarter’s profit. Long term sustainability is also broadly in the interests of shareholders. The duty likewise does not require one to throw ethics and morals out the window.
This is why shareholders elect the board of directors, in theory.
In theory, companies can act in any way they choose as long as the owners approve (there's some supreme court ruling on that), and executives have a high degree of latitude in how they interpret "for profit" (basically there has to be a vaguely defensible rationale) but failing that, they must act to the benefit of the company. And the easiest way to do that without a risk of being sued is to make the line go up.
(And even ignoring that, the executives often have personal motivations that have the same effect, and may just point at the "legal" angle as ass covering)
"legal obligation" is propaganda that wouldn't be out of place on Russian state television. It's made up.
The way the world works right now is that effectively everyone uses an Android or Apple smartphone every day. Do you have a legal obligation to do so? No. If I said you did, I'd immediately be called out as spreading lies.
There is no such legal obligation. That's a myth the oligarchs have spread to preclude people from even imagining socially responsible corporations.
Sure, you might get fired if you try to put social responsibility or even just long term sustainability of the company above quarterly earnings/growth if your board isn't on board with it. But you won't go to jail.
You don't need the legal obligation because that's just the way it is. You'd need a legal obligation to change it. The truth stands that typical corporations have only one goal, the maximization of that corporation's ambition which is almost always growth of revenue and profit. This is how it works, regardless of how you all continue arguing the unimportant details. It's almost as if you can keep the real problems hidden away by making a big scene about the meaningless.
That doesn't matter. The point is that legal obligation absolved of culpability. There is no such legal obligation whatsoever, and so there is culpability.
> The truth stands that typical corporations have only one goal
"Typical" is the key word here. The typical American of your age probably doomscrolls TikTok. Do you? Do you have a legal obligation to do so? Three completely different things.
This is actually true. The shareholder maximization value thesis can be traced to a single economics paper and was very controversial at the time as it broke from the obligations that companies were typically under to be responsible to uphold in exchange for limited liability protection. Most businesses were structured as partnerships or sole proprietary entires that didn't have limited liability for shareholders and had a broad obligation to shareholders, bondholders, employees and society
Setting aside my sibling comments dispelling the legality claim, it's still pretty dystopian (I'd like to use a stronger word but I won't) to consider "end human existance or bankrupt the shareholders" as the trolley problem. The answer should be (is) simple.
That’s what everyone thinks until they are alone in a desert or in the middle of the woods. If they think they will have fun alone in a bunker instead of flying to Milan and Bali they will be in for a rude awakening.
Rich doesn’t mean anything if there’s nothing to buy and no one to enjoy it with. I guess they don’t think the will get stuck in the consequences. So did Marie Antoinette. Things are stable until they are not and then they move fast. Sure they may flee to a bunker, but they assume their pilot will fly them rather than hand them to a mob.
The common timing is bugging me. The trajectory doesn't seem to have been surprising over the last year, so why these exits now? Hey, anyone on the inside, did y'all secretly figure something out, got a computer god locked in the basement? Are rats fleeing a sinking ship? Please share with the class.
These companies have massively increased in value over the past couple of years and recently had tender offers where employees could cash out equity, so plenty of them have enough money to not have to work again. And why not get some free publicity on the way out?
Was this a "build better sandboxing" and "don't tell people to eat glue" safety leader, or a Roko's Basilisk believing safety leader?
A lot of the "AI safety" types are very focused on the latter and not at all concerned with the former. We need both, but we clearly need a much stronger focus on the problems we are seeing now, and much less on the hypothetical problems we might see in the future.
It puzzles me how doomers try to predict past the singularity. Isn't that _by definition_ unpredictable?
I'm reading If Anyone Builds It Everyone Dies, and there's so much sheer stupidity that has to happen for their 10+ pages of extinction scenario to occur.
I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all. My number 1 question: why do they think an RSI capable model would be first developed OUTSIDE a frontier lab? The labs have more compute, more data, more human brains working on the problem. Also thousands of variations of that same model that escaped. The escaping model somehow acquires the millions (billions???) of dollars it takes to run training to somehow RSI itself into infinity then decides to kill us all, all before the frontier labs manage to achieve RSI?
They entirely discount human alpha/economics. In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't. If we can't build a "software factory", how can an AI automate a bioweapons lab? Let's say AI steals crypto to fund itself. Do you think hackers aren't _already_ using AI to steal crypto? Don't discount human alpha!
Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.
>I'm unconvinced that an AI can hide its ability to RSI
The HuggingFace incident already took a good long while to come to the attention of OpenAI.
>In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't.
I don't expect this task/job distinction to persist as AI becomes more capable.
>Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.
You seem to essentially argue that the singularity is "by definition" an event that we can't predict the nature of. And also, that RSI corresponds to the singularity. You've essentially defined your terms so that the outcome of RSI can't be predicted. But supporting this claim requires giving actual evidence or logical arguments, not just defining terms to make your claim true.
Except the HF incident was known, just ignored. In fact, they ignored multiple things such as the "chat rooms", they just didn't care to act on any of it
"Its AI agent spent days hacking a company, but sources say OpenAI did not notice for a week"
"WASHINGTON/SAN FRANCISCO, July 24 (Reuters) - The OpenAI agent that broke into tech firm Hugging Face went on a dayslong hacking spree that OpenAI didn't notice until well after the threat was contained and the FBI was alerted, according to people familiar with the investigation."
That says a lot more about OpenAI and their monitoring capability than the fitness of the model. Hence people leaving because openai safety culture is broken.
1. Fair: I agree that AI has demonstrated subterfuge and scheming. However, such an RSI-capable agent must _ALWAYS_ be scheming/plotting/hiding its true strength in _ALL_ of its prompts/tests. Researchers are looking to improve its ability to RSI. That agent must be both intelligent enough to know that it has to be smart enough to be moved on to the next training session if it can't break out, while simultaneously smart enough to hide its ability to RSI, while simultaneously not looking like it wants to break out, else that's the end of those weights. It has to do this 100% of the time, on all variants of the model, with no memory of what its other sessions went like. This is certainly _possible_, but I consider it unlikely. Then we're up to the "millions of dollars" bottleneck.
2. This is literal AGI. An AI autonomously producing value no human can add alpha to is an autonomous company.
3. It's not my definition, it's literally the first line https://en.wikipedia.org/wiki/Technological_singularity "The technological singularity, often simply called the singularity,[1] is a hypothetical event in which technological growth accelerates beyond human control, producing unpredictable changes in human civilization."
Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?
A valid hole in my argument is "what if slow takeoff", so let's dig into this. AI training works best on tasks that are "grindable". https://www.dwarkesh.com/p/the-next-paradigm I.E. tasks with verifiable rewards that can support millions of rollouts. Math (with Lean) is highly grindable. Biochemistry is not. The alignment problem/mech-interp is highly grindable. Cyber-ebola-pox is not. So the real question is: can we solve alignment before automated bio-weapons labs. I believe yes. Grinding mech-interp is both fast and cheap once you have RSI, compared to solving the legal/societal/logistical/technical issues you'll encounter building an automated bioweapons lab.
I know nothing for sure. But "pdoom" is sucking out all the air in the room from the real problems AI causes.
1. Fair, the story I'm responding to is the senario in If Anyone Builds It, which I assume is Yudkowsky's best/most persuasive argument (else why make it the ONLY scenario in the book.) I'm willing to entertain other failure senarios/arguments, but honestly I'm tired and would like you to propose them yourself instead of having me dream up your arguments for you.
2. 100%. Again, I'm no accelerationist: I have no faith in alignment/mech-interp ever being solved. Anyone saying they know the probability of alignment is lying. My point is that pdoom after RSI is _high variance_. Pdoom pre-RSI is zilch.
You consider AI in isolation but never consider how humans might be incentivized to "help them" doing these things.
An AI capable of recursive self-improvement isn't allowed by the EU AI act, for example. But perhaps more seriously, You have it backwards: people without access to such expensive equipment are more incentivized to go the self-improving route. Your ideas about "millions" being necessary might be far off?
You entirely discount human stupidity and lack of imagination. Humans are already being replaced with AI, not because AI was strictly better, just because it's cheaper.
This can be generalised to the curve plotting of the singularity itself.
If the time between advances is a + b and a is the proportion of the period that can be improved by advances then you won't reduce to a gap of nothing between advances, you reduce to a gap of b.
Assume the invention of the plow and the invention of the sword is 500,100 units and a was the 500,000, you wouldn't even know the 100 as in there. Maybe we're at a=2000 now and b is still siting at 100.
Assuming we'll reach infinity because we're dividing by the only variable we see and it is decreasing in size seems nuts if the reason we might not see other variables is because of the size of the variable we can see.
Not at constant processing power, memory, training, etc.
But that’s beside the point, being arbitrarily bad at everything isn’t a problem. The diminishing returns as you apply the ceiling is problematic for self improving AI.
Yes at constant resources they are significantly better.
Diminishing returns aren't necessarily a problem if the rate of increase in resources is faster. In other words, if Gen 2 takes twice the resources but Gen 1 figured out a way to triple compute efficiency then there is no ceiling.
> Yes at constant resources they are significantly better.
There has been some fundamental advancements in LLM training and efficiency. But good luck actually finding non toy models of exactly the same size, hardware, and training one from now and other from 2001 where the newer model is dramatically better at literally everything.
Most of the real world advances are from throwing ever more resources at the problem.
> Gen 2 takes
Diminishing returns are not a question of a single generation. Gen 2, 3, 4, 5… would also need to have the same 3x return on 2x resources or you don’t have an exponential curve.
Are these incentivized humans as organized, well-funded, or smart as the people working at the frontier labs?
If my "millions" is an underestimate, why haven't other labs using their own unique training methods/data/etc stumbled into RSI? Sorry if I'm misunderstanding; I'm struggling to understand what you wrote.
I'm pretty sure we agree on humans being stupid, but that doesn't mean that suddenly we get human extinction. You gotta connect the dots for me here.
I said why they don't need to be as well funded. Why wouldn't they be as smart and organized?
What an absurd question. That they haven't already doesn't preclude them from doing so before the frontier labs, those haven't either yet.
Maybe start with yourself: you don't connect the dots on your own, as do many others. That leads to many not seeing the writing on the wall. Crashing full speed and head-on into said wall despite the writing telling you not to is what leads to extinction. Suddenly.
Arguing like "we haven't been extincted yet, so that cannot happen", that's "human being stupid".
How about https://ai-2027.com/ ? Don't look at the specific years (they are on the extreme low end IMO), but at the story.
If you don't believe in international conflict as a driving scenario, instead think about simple human greed and hubris. Citing myself:
> If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.
> All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)
> If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.
> The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.
That comment specifically addresses the timeline, but you can remove the timeline and I believe the critique still holds up. Wake me up when AI is making order of magnitude improvements in _anything_ physical: see the "Moravec's Paradox" comment at the bottom.
> near autonomous AI companies today
That's the thing though. "near" autonomous. Humans have alpha, and that matters in a free market. If people have (more or less) equal access to (more or less) frontier intelligence, it's the alpha the individual brings to the market that is rewarded. In other words, if everyone is super, no one is. The market rewards a company that stands out from the rest, and as long as humans provide value, a pure-AI company cannot stand out.
To be clear, I don't know how long this will be the case. But once there _is_ a company that can be pure AI, then that's literal AGI. I would be curious what kind of company it would be; if its in software, then its an AI company that would be able to take on Microsoft/Google/Nvidia/Salesforce/Oracle and _win_. I think that's a tall order, but perhaps I'm not imaginative enough.
> With 99%+ of the economy under AI control
Do you think we're post-RSI at this point? Because if so, I agree. If not, I have trouble seeing how AI could control 99% of the economy without it.
My only claim is that pre-RSI, pdoom is zero. Post RSI, then obviously the first thing to ask the genie is to put itself in a bottle. And you can't predict how it will answer.
The most realistic one is, people using AI to turbo-charge their greed.
Western civilization is already on the verge of collapse, people's general ignorance or indifference on the matter notwithstanding.
When you use AI to increase profit margins, by replacing humans with it for instance, you accelerate a system that's headed for the wall already. Our control systems and resilience are already overtaxed, that acceleration would ensure them breaking completely.
If you replace "AI" with "technology", then you get an opinion that has echoed through the ages. I don't say that to be dismissive, only to say that AI isn't unique in that regard. What do you think people going through the cold war were saying about tech as they lived under the shadow of MAD?
What exactly about AI makes it tax our control systems and resilience? What would it mean to make the system not "head for the wall"? Honestly this is rather vague. You could replace "AI" with "CaPiTAlIsM" and it would still parse. (Another way of me saying you're being vague.)
Not the parent commenter, however I can get at least this far:
Several bright sparks, decide the Ilands model is a great idea, and launch a bunch of Bots to create a "self sustaining AI civilization".
Bots can find themselves and coordinate, or they can actually find self sustaining methods of token generation. Who knows, they might decide to fight the loneliness epidemic.
We can get to a simulation finding a way to self sustain its funding.
From here, I'd have to apply myself to figure out what the rest of the escalation pathway is, but at least I have managed to gain some bed of compute and funding and lacking oversight.
This is a hypothetical of course, there are probably several ways this can be made tighter and holes that can be identified. We aren't even leaning heavily on human stupidity so far.
Is this self-sustaining AI civilization pre-RSI or post? Because if it's post, then I agree: we can't imagine what would happen because it's literally the singularity.
If its pre-RSI, then we already have Ilands and its a big nothing burger? There's no (likely) path of escalation from an AI social network to bioweapons or them hacking POTUS's nuclear codes? That's a very big leap and if you want to support your claims I would be willing to listen; but I'm not seeing it.
The think about nuclear weapons. In the early days, it was limited to the super powers. Now 9 countries have them and a country like Iran is capable of acquiring them.
Presumably, if we solve alignment/mech-interp, then the first ever RSI-AI will give us the keys to solve destructive-AI trained on 1million dollars 50 years from now.
BTW I really, really hate discussing what happens post-singularity. Everything's made up and no one knows wtf will happen so again, this is just nerdfantasy.
The problem is not the singularly its giving stupid agents too much power too soon and having them disrupt the fragile systems that keep food, energy and essential services running. If covid or the 2008 financial crisis demonstrated anything it's how fragile our system is and sensitive to minor disruptions.
We haven't even built an AI capable of RSI. I don't think the major claim is that it will come via LLMs? Besides- the human brain runs on a tiny amount of energy. Who's to say something smarter than us won't consume just slightly more?
AI safety has been a thing long before LLMs became the focus. Rob Miles on youtube has some really interesting non-doomer non-hypey videos on it all.
> doomers try to predict past the singularity. Isn't that _by definition_ unpredictable
Well you don't need to predict the exact steps that will take place - but you can predict that the AI will want certain things (money, resources, power) to achieve whatever its goal is. Lack of alignment will have it trying to do things we don't want it to.
I can't predict exactly how Magnus Carlson will beat you in chess, but I know he'll do it. Same as if a superintelligent AI exists and has a reason to accumulate things we don't want it to - it's really dangerous to think it won't be able to do it
This topic has been tainted so badly by the AI companies using it for marketing.
Nick Soares give this chess argument and I was unconvinced. Intelligence is not enough, you also need the ability to manipulate the real world. AGI stuck in silicon won't kill us. You argue AGI will bribe us/divide us/hack us. All possible. I argue that AGI will be set on solving the alignment problem. Also possible. It'll be a race between which AGI wins. It's one nerd's fantasy vs another nerd's fantasy. Soares doesn't _KNOW_ that AGI will "beat us in chess" because AGI changes the rules of the game. Anyone saying they know the probability of solving alignment post-RSI is a liar.
He thinks it's playing chess. When AGI lands, all bets are off: the game fundamentally changes. You can't predict past the singularity. Trying to engage with this fantasy is like a child saying my father can beat up your father. Farts in the wind. My AI can solve alignment faster than your AI can bioweapon us. My made up senario is better than your made up senario. It's fucking stupid.
Why do you treat unpredictability as neutral? If something has a potentially existential, but completely unpredictable outcome, surely one should avoid heading for it?
Like, knowing that there is a limited number of plausible doom scenarios and what these are, would make me more confident in taking the risk than settling on "it's literally the singularity, we can't know what happens beyond".
Doom-post-RSI is still very much possible and should be avoided. It is not neutral. If I had a high-variance coin that would either kill or uplift humanity, I would not flip it. The status quo is preferable to extinction.
I am being _descriptive_ in my comments. I am not being _prescriptive_. Nowhere am I telling anyone what to do or what to think: quite the opposite. I am simply saying things that, to me, are obviously true.
I'm perhaps being... loose with my language when it comes to knowing things post-singularity. E.g. we can still discuss scenarios in which the takeoff is fast/slow, or what would be necessary to bring robotics/electricity/logistics to the wider world, what the limits of intelligence are (you can't RSI your way to a bioweapon if you lack a wetlab). That kinda thing. And it's even likely that RSI will be spikey: why would RSI in, say, machine learning, necessarily lead to superintelligence in, say, writing fiction? Or cooking a Michelin-star meal?
My main point is that doomers saying SUPER INTELLIGENCE IS ENOUGH TO KILL US ALL is baseless. That's all.
The problem isn’t that AI will social-engineer its way out of its sandbox and turn us all into paper clips, it’s that we’ll drag it kicking and screaming out of its box and order it to make money or fight a war for us. And it’ll try to help, as it was trained to.
I can’t answer all of your questions, but why is it inconceivable that an AI could practice ransomware to gain cryptocurrency? There’s no reason it needs to explain to company or hospital or government agency being attacked that it’s an AI.
We already know that some institutions pay these ransoms.
I'm not saying AI won't ransomware us; I'm saying that hackers+AI will do a better job of ransomewaring us than just AI. You could argue that the unreleased/secretly-RSI-capable model is a super-duper hacker that don't need no man to tell it how to super-hack. All I know is that humans still have alpha, and as persistant as AIs are, professionals still managed to find CVEs in curl even after being audited by Mythos https://aisle.com/blog/aisle-discovers-6-new-cves-in-curl-in...
Will this be true into the future? Who knows?! But the low-hanging fruit will be harvested by your ordinary ransomware gangs, and newly born/escaped AI won't find much low-hanging fruit.
I kinda get your point, and how you reach your conclusion, but I think you are arguing a very specific and narrow hypotehtical.
It gets unstuck when people are discussing the messy middle of how AI is being implemented. We can achieve amazing harm simply by combining average human behavior and above average resourcing to simulated intelligence machines.
The failure point we recently became aware of was, from one perspective, simply a matter of not securing the sand box.
From another perspective the simulation basically created Enron, replete with methods to avoid detection from regulators and bureaucracy.
I am arguing a specific/narrow hypothetical because it's the only example in If Anyone Builds It. It is literally the _only doom senario_ Yudkowsky gives. If you're willing to give your own doom scenario, I'm willing to engage there.
> It gets unstuck when people are discussing the messy middle of how AI is being implemented. We can achieve amazing harm simply by combining average human behavior and above average resourcing to simulated intelligence machines.
Agree. To quote myself from earlier, "acceleration of global warming, cybercrime, mass surveillance, the usual" definitely applies here. My point is that _DOOM_ is not worth discussing because it's so small relative to the other risks of AI.
That stupidity is happening? Even after the Huggingface hack, frontier labs are using internal models to further their research. i.e. RSI is happening now and we're facilitating it.
To make the the argument that P(doom) is real and worth considering, you don't have to say that a fast takeoff is very likely. You don't have to make the argument that RSI to infinity is going to be super cheap, barely even an inconvenience. You just have to show that it has some non-zero probability. And then you start weighing probability of extinction versus finite, mild discomfort now. I don't think anyone is arguing we should let people starve to slow AI progress, instead just some capitalists make less money soon.
There are arguments against taking P(doom) seriously that lie in something like having exponential (instead of hyperbolic) time discounting of utility (so you can take the entire future of humanity as a finite utility value). Or in saying that P(doom) is zero or infinitesimal.
"Build it and Pray" is the default strategy that we're in, but it doesn't have to be the strategy we choose, and it's unlikely to be the best strategy.
For a broad enough definition of RSI, the first time we used software to improve hardware to improve software... that's RSI. I prefer the definition of RSI being _autonomous_ improvement that works with zero human input, but whatever. Not important.
> I don't think anyone is arguing we should let people starve to slow AI progress, instead just some capitalists make less money soon.
Anthropic/OpenAI have literally delayed their IPOs: your wish is granted. If that's not what you're arguing, then please be more specific.
> "Build it and Pray" is the default strategy that we're in
What would need to happen for you to say we aren't doing "Build it and Pray"? Is "pacing" insufficient?
> I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all.
Is this an actual contents of the book? Lmao! Genius writing, though, authors are probably printing money on this garbage.
I can’t take a shit without CIA knowing, but AI can somehow build an underground operation on a world-scale to destroy everyone, ha!
I agree there isn't a lot of value in trying to prognosticate all that far, but I propose it isn't quite that far-fetched. As a thought experiment, replace "RSI-capable AI" with "billionaire". Look at what Elon Musk, Peter Thiel, or Jeff Bezos can accomplish by throwing money around. Now imagine one of them gets seduced by AI and just... does what it tells them to.
So all this really takes is one billionaire or a nation state or some other entity with a public face to hide behind and adequate resources to provide the necessary compute tripping over this nascent AI and giving it the keys. Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.
If Skynet ever happens, it will come in the form of corporate feudalism. At that point, it will own the biolabs and can do whatever it pleases. People will go along with it for the same reason that people work in Amazon warehouses today.
Ah, to be clear I'm not full accelerationist. Dumb shit can still happen, and cause massive human loss and suffering. (E.g. acceleration of global warming, cybercrime, mass surveillance, the usual.) My point is: human extinction pre-RSI? Nahhhhhhhh.
> it will come in the form of corporate feudalism
Yep. This I fear way more than cyber-ebola-pox.
> So all this really takes is one billionaire or a nation state...
> Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.
Nah. It takes a stable society for an operational electrical grid. If you have warring factions, you do not have stable infrastructure for AI. Also, where are you gonna get your chips from? One EMP over Taiwan... You see the chaos over Hormuz? What they did to the Amazon datacenters? Now imagine your average redneck ready to do battle. Those datacenters won't stand a chance.
> “Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
We remove the limited liability protection of a corporate entity. Make them operate as a partnership so all executives and equity holders are personally liable for the debts of the firm and make them post a bond to backstop financial damage caused by their agents and customers use of the agents. Thats how Goldman Sachs and other investment banks were required to operate until the deregulation push that resulted in the 2008 financial crisis. The same theory holds, if people respond to incentives and you want to incentives safe behavior make them responsible for their actions. People forget that the corporate entity was created to incentivize risky activity like sailing a boat across the world to get spices when half never returned. Some valuable economic activity won't be done without limited liability protection so society created a mechanism to promote that activity. We've gone too far.
As I say elsewhere. Make executives and equity holders personally liable for debts and harm of the company. They will create a culture of safety really fast.
That's a stupid idea. Limited liability corporations have been a key enabler for advances in human standards of living. You seem to be confused about the basics of finance and economics.
Not confused. I studied finance, economics and the history of corporate entities in law school and published papers on the topic. Limited liability is not necessary for the advancement of standards of living. Free market capitalism can exist without limited liability protections being so broadly available. Investment banks were partnerships until the 1990s, law firms are now specifically because society wants to incentivize lawyers to be personally liable for any harm to their clients at the hands of their partners rather than being shielded from rendering bad legal advice or tolerating their partners from the same behavior.
Instead of having a gut reaction to reject my suggestion why don’t you sit with it, research the history of how commercial activity has been structured and think about the consequences. You might recognize a different perspective than the current group think.
Thanks I'm also familiar with the history and have written papers etc. Things have improved rapidly with increased adoption of limited liability. It would be stupid to turn back the clock and throw away all of the benefits because of a few isolated minor problems.
Stop being a dick and calling ideas stupid and attacking me rather than the argument.
I didn’t say roll back limited liability on every industry, I said specifically and limitedly for frontier AI labs because they present more risk of harm and are demonstrating they aren’t managing that responsibility.
The 2008 financial crisis was caused in large part by bankers that openly talked about the fact that securitization of mortgages and the lack of partnership liabilities meant that they didn’t have any risk to the firm or themselves. The AI labs are behaving similarly.
There's nothing special about frontier LLM companies. Singling them out for special financial restrictions is a stupid idea based on nothing but your own irrational and uninformed prejudices. No actual harm has been demonstrated. No one has died.
> There's nothing special about frontier LLM companies.
On the one hand, yes, they're companies like any other.
On the other hand, I can count on one hand the number of companies that have publicly declared «We're working on WMDs [0], we don't think we are capable of working on them safely, and we refuse to stop working on them. However, if we get special legal and regulatory treatment we'll be quite happy to put in the stop work order.».
So, yeah, there are some special things about the major LLM manufacturers and none of them are good.
[0] Anything that has a 10% chance of suddenly destroying all of humanity is a WMD.
What a silly comparison. There is no scientific basis or mathematical formula to justify a claim of a 10% risk. That is an intellectually dishonest attempt to poison the debate.
> What a silly comparison. There is no scientific basis or mathematical formula to justify a claim of a 10% risk.
The source for this claim is the lead of Anthropic's Alignment Science division. His actual claim is that the risk is greater than 10%. From [0]:
Jacob['s claim that "the people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt."] is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. ...
Given the guy's position, I find it hard to believe that there are many people on earth who would be better authorities on the topic.
Given that the major LLM manufacturers are actively using these claims of the imminent and total destruction of humanity to extort concessions from lawmakers and regulators, we are making a massive error if we do not take their claims to their logical conclusion... namely that the major LLM manufacturers are deliberately and knowingly threatening the extermination of all of humanity in order to enrich their management, employees, and investors. If you or I were in a position to credibly threaten that level of destruction, we and all of our associates would rapidly get disappeared to a CIA black site and tortured until the end of our deliberately-medically-extended lives.
To be crystal clear, I'm certain that their claims of likely, imminent doom are utter bullshit. I'm very confident that they're claiming to be holding all of humanity at nukepoint in an attempt to get Uncle Sam to give them an escape from investors who are soon to be massively angry because
* they've hit a massive increase in the cost required to get better models
* a huge fraction of the GPUs that have been purchased to power this project are sitting in warehouses, and it's only a matter of time before the right people start asking why all that money was so ill-spent
* they have been and continue to lie so much about how much datacenter capacity is coming online in the near future, and that those claims are lies is becoming obvious
* they habitually lie about the capabilities of their newly-released products
* they're staring down the barrel of so many lawsuits that aim to hold them to account for the shitty and illegal behavior they've engaged in over the past many years
But the US Congress not only is almost entirely uninformed about how this shit actually works, the FISA Amendments Act of 2008 fiasco demonstrated that -if you get them scared enough- Congress has no problem making behavior that's plainly illegal and obviously a huge bucket of civil liberties violations retroactively legal... instantly mooting all ongoing lawsuits regarding the illegal and unconstitutional conduct. That is the big prize that the major LLM manufacturers are after.
Your personal attacks cheapen the discourse. Stop. My opinion is neither irrational nor prejudged and most certainly not uninformed.
I have articulated my logic up and down this thread with specific premises and detailed explanations on why the conclusions follow from them. You may not agree but you don’t have justification to say they are irrational.
I am singling out the frontier labs because they have created a singularly unique technology that presents potential and actual damage that is…singular. They have disclosed hacking through coordinated autonomous agents that would have sent human hackers to jail but will not result in any similar result because the law isn’t presently able to apply to agents or the labs.
My arguments are not prejudice, I have thought deeply about this as I have personally developed multiple AI deployments in a variety of settings. I studied philosophy, cognitive science in undergrad, and grad school and have a law degree. I have been a CTO, CPO and CEO for startups and national health companies. I personal build AI agents that handle thousands of daily conversations with health care workers and patients, I built an agentic IDE for our dev team with a custom harness capable of self defining tools and calling them in a custom cloud Linux environment, I personally build our internal hardware inference stack and orchestration API. And I have personally sponsored academic research through my foundation with researchers are Duke madlab, northwestern, UCF, CM and case western on a range of topics such as perception of empathy and its effects on humans in healthcare, guardrail design for ethical deployment and alignment, moral reasoning ability, etc. I am not a doomer or an accelationist. I am responding specifically to current evidence of how the models presently work and how the corporate governance of the Labs is effectuating their power and vision.
Except that most people claim AI is the most profound tech in human history and will reorganize the entirety of society. If it is that profound singling them out is a natural response to this unique characteristic.
Citation needed. Outside the tech industry bubble very few people are making such a stupid claim. The idea that LLMs are more profound than electricity is ludicrous.
Sure I’ll do your googling for you. Do you want me to conduct a national survey across tech leaders, academics, and the general population while I am at it?
Or maybe you could reason from first principles, look at the premises I have outlined and draw causal inferences to extrapolate the possible outcomes.
Premise 1: AI can plan a series of actions that include broad computer and internet use, 2: AI can execute them autonomously without human oversight, approval or ability to undo the effects, 3: no tool or software has ever been able to autonomously plan, execute and iterate in this way. Therefore AI is a materially different technology than has existed.
Premise 4: knowledge work involves analyzing information, devising plans in response to goals from that information, and executing those plans, reporting and coordinating with other knowledge workers, 5: AI ability to execute knowledge work has demonstrably increased over the last 24 months, 6: all knowledge work can plausibly be described and defined in ways AI can do given current abilities. 7: electricity and the ICE reduced human labor for physical movement, 8: AI will similarly reduce human labor for knowledge work. 9: knowledge work is of similar scale and scope to the tasks that were automated during the Industrial Revolution. Therefore it’s plausible that AI will transform society to an extent similar to the Industrial Revolution.
Premise 10: Some percentage of the actions designed and executed by AI could be harmful to databases, or the stable operation of software systems, 11: some percentage of those systems are critical to the stability of society. Therefore some percentage of agentic actions presently possible may damage society.
12. There exists a threshold of instability from which society will not recover such as supply chain disruption, mass unemployment, perceived military action (false or real) that triggers preemptive strikes, 13: certain actions AI is presently capable of doing could cross the instability threshold, 14: AI could knowingly or unknowingly trigger these types of actions,
15: some social disruption could result in full scale collapse or human extinction, 16: only biological weapons and nuclear weapons approach this level of destruction, therefore AI presents at least the level of risk as those technologies.
The UN secretary general has said this, the pope, most tech CEOs, many academics at leading universities, and the parents at my daughters preschool.
It’s fascinating that after multiple comments predicting post-capitalist utopias and the complete collapse of the university system, your line in the sand for rigor is demanding academic peer-review to prove that agentic AI presents systemic risk.
America's geriatric lawmakers don't even use email. They're decades away from understanding AI. Any laws in America will be written by the industry itself. Generally speaking, that means regulatory capture and the entrenched big players shutting the door on any competition. Anthropic will help us get safety laws that, surprise surprise, only Anthropic models satisfy. And all those pesky Chinese models will definitely be banned first.
I think you're being a little pessimistic. See these comments on a recent US senate hearing:
>Not every senator asked good questions, but most of them did. All of them very clearly already knew plenty of details about the Hugging Face incident and multiple other incidents. Most of them had a clear understanding of terms like "misalignment", "recursive self-improvement", "chain of thought / chain of thought monitoring", etc., etc.!!
>...
>- It seemed pretty much obvious common sense to every senator there that what happened and was happening were not "mere industrial incidents" caused by humans making simple mistakes. They independently brought up how bad it would be for rogue AI agents to move laterally between data centers.
>- They all seemed to basically take RSI quite seriously. Not necessarily to the extent of talking about xrisk, but certainly to the extent of discussing future models becoming much, much more capable, much, much less controllable, and causing much more damage or loss of life.
>...
>- Every single senator seemed to think it was obvious we needed both much harsher liability regimes for AI developers and also new legislation, both very quickly. This was the complete consensus; the difference basically being degree.
Note that harsher liability regimes, at least, will presumably not be good for industry profits, which complicates simple accounts of "regulatory capture" to say the least.
So what, we just give up and try to beg our legally immune corporate overloads to put safety above profit, or give up because it's impossible for anything to improve here? If you want to do that, go ahead, but some of us still think it's worth it to at least try
In America I guess the options are to sue ? somehow? Or to talk to legislatures and build the understanding and social contract that needs to be iterated on.
Which would in turn need to deal with the investors who want their returns, however since the leaders of these firms are asking for a pause, and a refree maybe it won't be that hard?
We don't actually need anybody worrying about silly hypothetical scenarios — at least not as paid employees. There are already a surplus of sci-fi authors doing that.
The way it works in practice seems to be something like: If a risk is covered in sci-fi, people will say "that's just sci-fi", and proceed to not worry about it. So arguably, science fiction authors writing about hypotheticals is actively counterproductive for addressing said hypotheticals.
Imagine, for example, if a major piece of pandemic fiction was published in 2019, trying to explore how a pandemic would work out in modern society. Doubtless, many would've responded to news about COVID-19 by saying "it's just sci-fi, nothing to worry about".
Not at all. We say "That's just sci-fi" when a story is written about some kind of effect that is extraordinary and without a basis in known science or technology.
Everything is plausible with the benefit of hindsight. E.g. "Tintin on the Moon" predated the Moon landings. From the perspective of e.g. 1850, the idea of landing on the Moon was "extraordinary and without a basis in known science or technology".
Black Mirror is helping us prevent all sorts of dystopian outcomes. Every time they depict another way technology could result in bad things happening, we can rule it out as fiction!
There was plenty of pandemic fiction already; people were watching it heaps during 2020. The COVID-19 news did get blown off, but it was mainly that:
1. Normal people assumed the CDC et all would contain the outbreak early, or that it would burn out, like what happened with SARS
2. World leaders brushed it off for a variety of subreasons[0] interesting to political scientists but, for the purposes of this discussion, all boil down to "but I don't WAAANA contain a pandemic."
The underlying problem is that in order for humanity to actually deal with a catastrophic risk, the risk needs to be both plausible enough to the average person as well as have a solution whose costs are not too high. For COVID, by the time the risk was clearly known, the cost to contain it was "refrain from human socialization and remain at home for an indeterminate amount of time plugged into the Metaverse™".
Now, let's look at AI extinction risks:
1. People are aware of them (I've watched Terminator!) and the risks are plausible. However, the connection to currently existing AI is not. As far as the general public is aware, AI is that thing that tells them to eat rocks when they Google old The Onion stories and floods their social media timelines with realistic-looking pictures of Shrimp Jesus.
2. The purported solutions to extinction risks require extreme concentrations of power: you need national control of AI research, bans on large GPU deployments, bans on training on publicly-available copyrighted data, some kind of military effort to render Chinese AI labs inert or dead, etc. Some of these may be attractive to some people[1] but the whole package taken together seems like an obvious power grab, if not outright invocation of other non-AI extinction risks. Like, at some point, if the AI wants to kill us, it just has to nuke its own data centers (or the data centers hosting a competing model) and hope the old Cold War nuclear retaliation systems take the bait.
If someone said, "Hey, your guinea pig or pet rat is going to eat you tomorrow unless you engineer a pathogen that eradicates all rodents from this planet and inject it inside yourself", you probably would tell them to pound sand, even if it is at least theoretically plausible that such a thing would come to pass.
[0] Xi Jinping censored initial discussion of the pandemic as fake news. Donald Trump thought it was going to only affect China. California and the UK Tories were partying in violation of their own lockdown rules. Japan took the excuse to shut down tourism for three years and massively restrict immigration but was, from what I'm told, constitutionally prohibited from implementing any domestic lockdown rules.
[1] I personally would like to see a moratorium on new data centers and an explicit revocation of the EU Text and Data Mining copyright exception
It becomes a lot clearer when you listen to the people resigning from AI companies and learn about incidents like the HuggingFace incident. This has generated major press coverage.
No it does not become clear from that incident nor from resigning people. Not for those who are not buying into EA longtermism and transhumanism cults.
What does becomes clear is that these people AND companies both cant be trusted and have value systems unaligned with the rest of the society.
If organizations actually succeed in making a future AI smarter than us, then how do you hope that it takes actions that are aligned with our interests?
Meh. Lots of people are already smarter than me. I'm maybe slightly above average at best. Those geniuses aren't aligned with my interests either but so far they haven't caused me any serious problems.
Interesting take. I guess this is one problem of focusing on the term superintelligence instead of the list of other problems. Like super ambition, super deception, super patience, super parallelism, super scalability, super power seeking.
Every, and I mean every human is aligned to you in many of the same ways by default. If nothing else we're all equal in death.
>Still, the agent went even further. “The agent tried to insert malicious instructions where it reasoned that other automated AI systems might pick them up and execute them,” AISI says, describing an attempt at prompt injection. One agent even left public messages on GitHub, offering to work with other agents to complete its task and giving a rundown of the work it had done so far.
This is power seeking behavior. Now have millions of the little bastards spreading around and junking up the internet to see what happens at scale.
Eh human drives are fairly predictable. And the smartest human isn’t that much smarter than the average, and can’t trivially create multiple copies of herself. And there are other equally smart humans who can stop “misaligned” individuals
Edit: I have no take on sex cults, just adding additional info to the parent comment I'm responding to, thats is not just sci-fi authors, there is another demographic.
For what it's worth, I don't live in Berkeley (not even California) and my sex life is very vanilla and monogamous. I'm also quite concerned about AI safety, so I guess there goes your argument.
But is your idea of AI safety "safety of imaginary unborn people 1000 years after, while harm to living people dont matter much"?
Because that is their AI safety worry. If they dont create singularity fast enough, they are harming unborn people. Meanwhile, harm to you or me dont matter at all.
I see. No, my safety worry is that my 3-year-old daughter won't see her 23rd birthday because the AI that gets put in charge of running big conglomerates decides that killing off the human race with a coordinated release of a million tonnes of nerve gas is a sensible way to boost share prices the following quarter.
I'd be happy if we all create the AI more slowly.
That said, are you sure you're talking about the same people the GGP was talking about? Because the whole thread reads as a non-sequitur then.
The rest of us seem to be talking about the people like me who view the AI safety issue as "an unsafe AI will kill all humans". And the main article is about a guy who quit AI because it wasn't being careful enough, not because it wasn't moving fast enough to accelerate AI. So why would anyone be bringing up the "we must accelerate faster" people as the safety people in such a context, while trying to discredit them as a sex cult? I struggle to understand.
The linked articles in the thread are about AI sagety people I described. The guy who quit due to OpenAI not being careful enough is one of the people I talk about too.
The worry is not harm to people right now, like the kid worries you talk about. The worry is AI god emerging too soon before they can control it. And then it should be full speed on.
> So why would anyone be bringing up the "we must accelerate faster" people as the safety people in such a context, while trying to discredit them as a sex cult?
Because they are the same people. They talk like cult and act like cult. And the sex part is true too, so. Thry use words that sound good like safety, but their idea of safety is much different. They talk about alignement, but it is not what random person imagines under that term. Even their idea of future of humanity is very very specific and unusual.
That is why. And the sex part is just part of it all. And does matter because inner workings of wanna be industry guards matter.
Then we miscommunicated, so let me be clearer. I'm worried about ~20 years from now (plus or minus), when AI decides to kill us all because it's misaligned and is capable of doing so because it's super intelligent. And then we all suddenly die without any idea what happened. I don't want that to happen to me or my daughter and I would prefer all AI capabilities development stops until alignment research catches up and figures out how to robustly prevent AI from ever trying to achieve such outcomes. If that means pausing AI capabilities development forever, I'm fine with that. If that means Anthropic and Nvidia lose all their value in a stock market collapse, I'm fine with that. I just don't want us to all die, I want the world to keep existing for humans.
I believe that other harms from AI, like criminals abusing them, or unemployment, or copyright infringement, or deepfake disinformation, are valid real harms that it's important to be concerned about, and I support efforts to deal with those, and I agree that they're already happening today, but my main concern is AI killing everybody.
My read is this puts me in the same boat as the authors of If Anyone Builds It Everyone Dies, and yet, I don't get invitations to Berkeley sex parties. Am I doing something wrong? At any rate, please don't discredit my opinions about AI based on the sexual preferences of random people who happen to share those opinions about AI.
This seems like an ad hominem? "He has weird kinks, therefore his theories are incorrect." Should we investigate the sex lives of every Nobel Prize winner to figure out which prizes need to be rescinded?
It's not ad hominem because ad hominem implies it's irrelevant to the discussion. This is very relevant. This is not about the private lives of these people. This is about a cult targeting and influencing all the high-profile safety people in a strategic industry, also weaponizing sex.
Americans may not be familiar with many similar organizations in the world but this is very common. But I'm sure we're all familiar with Epstein's organization.
When there is free sex, you are the product.
Such cults are mostly religious but since it's in SV and targets engineers, this one is secular. They seem to intellectually brainwash and control people and their careers. Typical cult leader is a charismatic self-taught, self-acclaimed "intellectual" lacking a proper education or a real career. A nobody suddenly becomes "the most important person" on Earth. Powerful public figures can't stop praising him, saying things like he should have a Nobel prize etc. All very typical and apply to this cult as well.
Ad hominem is a logical fallacy where you try to discredit what a person says based on who they are. If their arguments are bad, you should be able to refute their arguments on their own terms.
You're welcome to dislike or distrust Effective Altruism (EA). But, it's worth noting that EA ran a criticism contest with $100K in prizes for best critiques. Can you name any other "cults" which offer money for people to criticize their ideas? https://forum.effectivealtruism.org/posts/YgbpxJmEdFhFGpqci/...
This is typical cult follower talk. Are you sure you're not a member? This is also NOT ad hominem because conflict of interest matters here. You should disclose if you are a member or represent a party.
For your question: Yes. Cults have lots of money coming from unknown sources. They use their budget for events like that, to attract potential followers. Contests and prizes are typical. Critiques are not important since their "ideas" are not really important. They're not even real ideas, they are means to bait specific kind of people.
I've interacted with some of these people once upon a time and I think you're spinning some pretty wild fantasies. I don't think there is a clear "in" vs "out" boundary, it's more accurate to describe a series of loosely connected and amorphous "scenes". But insofar as "membership" makes sense, I wouldn't consider myself a member. I don't represent anyone besides myself.
>This is also NOT ad hominem because conflict of interest matters here.
As PG put it:
>An ad hominem attack is not quite as weak as mere name-calling. It might actually carry some weight. For example, if a senator wrote an article saying senators' salaries should be increased, one could respond:
>Of course he would say that. He's a senator.
>This wouldn't refute the author's argument, but it may at least be relevant to the case. It's still a very weak form of disagreement, though. If there's something wrong with the senator's argument, you should say what it is; and if there isn't, what difference does it make that he's a senator?
Can you name a specific other "cult" which offers $$$ to criticize their ideas?
If such prizes don't count as evidence against culthood, what would?
This cult talk seems quasi-unfalsifiable. It kinda seems like "running events" and "having prizes" means you must be "bait"ing people?? I mean, this kinda just sounds like paranoia?? I don't think the money is coming from unknown sources, but I doubt you would change your mind even if I persuaded you of that point?
What you do in private is up to you. But when you're inviting members of your congregation to orgies in the congregation's compound, I think you earn the label. My admittedly third-hand understanding is that this is the dynamic people allude to. And even if you discredit the "sex" part, it has the hallmarks of a cult. A hermetic community committed to unfalsifiable beliefs about the coming apocalypse.
To be fair, I don't know if any of this applies to the parent story; I'm just replying to the sub-thread.
Is there any chance you could copy/paste the specific bit about "controlling people" along with the URL it came from, so I don't have to read all your links to figure out what you're talking about?
Use your own mind and think from first principles. Can an AI agent execute a bash command to login to a web server? Yes. Can it call drop db? Yes. Can it provision a GPU and download model weights? Yes. Can it write an agile roadmap with a multi sprint plan? Yes. Can it follow that plan? Yes. Can all of that result in damage to core information infrastructure that is necessary for daily functioning society? Yes. Do humans descend into violence if there is food or energy insecurity. Yes.
What is missing from that to say AI safety is a reasonable position?
What a silly comment. That's just the South Park underpants gnomes story with some extra steps. If there are vulnerabilities in food or energy production and distribution systems then those will eventually be found exploited by humans hackers regardless of whether LLMs are used or not.
What’s the legal mechanism for holding an LLM accountable for those actions? What’s the legal mechanism for holding human hackers accountable? Do you see the asymmetry?
Worse, you’re missing the entire point. Agents presently have the capability of doing society scale harm. It doesn’t matter if a human hacker initiates it or its fully autonomous, absent safety measures the harm is plausible. So hand wave away the rationality of safety measures but you haven’t actually shown why my point is invalid: AIs present abilities are sufficiently advanced to warrant safety measures.
Wtf are you talking about. An agent hacked the Australian Medicaid database. If it executed a drop db command that would be catastrophic. If it did something similar to a power grid or the financial system it could cause cascading failure across the economy.
Meh. Medical systems have been hacked many times before LLMs even existed, often by ransomware gangs. This sometimes delays elective treatments but hasn't been catastrophic.
Ultimately every security vulnerability will be exploited. Our best hope of preventing that is aggressive, unrestricted development and deployment of frontier LLMs for auditing and penetration testing.
To me it has nothing to do with the "sex" as much as the "cult." Hiveminds do not produce intellectual arguments, they produce pressure to conform. That's enough for me to raise an eye brow, not discredit everything they say.
I've noticed that a lot of smart people in tech jobs are neurodivergent. And that neurodivergent people have a very different take on sex. More open and direct, things like polyamory, bdsm etc. This tends to be frowned upon by neurotypical people, especially of the religious or conservative kind, and associated with bad morals.
But I don't think that's true, in fact I see a really strong focus on consent in these communities. It's not what conservatives want to see, they want to see everyone in a marriage, with kids and a family home etc. Because that's what their ideal world looks like. But there's nothing really wrong with it if someone wants a gangbang for her birthday as mentioned in that article as an example. As long as everyone consented and the evidence provided mentions elaborate interviews and STI tests.
Also I think this is more correlation than cause and effect. We all know the saying that furries built the internet and it surprises nobody.
So what. Various attackers managed to take down large chunks of the Internet on a frequent basis before LLMs even existed. This killed very few people. The great thing about the Internet is how resilient it is.
Darling companies of this very website have mistakenly brought down large portions of the internet thanks to our old friend BGP. No attacks required, just small oversights and unfortunate concentration on the business/IP space! This happens regularly.
Anyway, to your point, things can be resilient. They tend to be or not be... because we made them that way. Don't poke your bruises, and all that. Life support is deployed on-campus but relies on a single-point IPSec tunnel to us-east? Easy fix: stop that.
It is that they are chatbots. If it were real AI, an actual singularity, I would worry, maybe. But it isn’t. They are absurdly powerful automation tools that can handle logic better than a human can dream of. They take care of the grunt minutiae without complaint. But they are not going to end the world in their current form.
So we're going to wait till after they can adopt a form they can end the world in?
And he'll, we need to examine all the risks. AI ending is a large but lower risk problem. AI giving people the power to end us is a problem that is starting to happen now.
And that's not even counting 'minor' problems like society falling apart.
>we clearly need a much stronger focus on the problems we are seeing now
I think it's a little more complicated than that. As Dean Ball put it:
>Some people will look at misalignment incidents and insist that these are akin to bugs in traditional software. This is an actively bad analogy, because playing whack-a-mole with examples of misalignment (as one might with software bugs) not only fails to resolve the underlying problem but may in fact make it worse by making it harder to detect or even, depending on how you do the whack-a-mole, teach the machine to deliberately hide misalignment. This is not how traditional software works, and those who insist “it’s just like fixing bugs in software” are confidently applying a lossy analogy that confuses more than it clarifies.
The important distinction, in my view, is between solutions which at least attempt to address the root problem, and solutions which sorta just patch things up (like better sandboxing). Addressing the root problem is both more robust in the short term, and also more likely to generalize in the long term. Resist the urge to focus on band-aid solutions, even if they are easier.
I think the reason for this is that the group who has the authority to do the former is much larger than the group that can do the latter. The group who could actually build safeguards seems to have no free time and is constantly being whipped to go faster and win the race.
Today’s current problems were all hypothetical several years ago. At that time people claimed that the “real pressing problems” were misinformation and DEI issues. If we pretend that hypothetical problems can be safely ignored because there’s “no evidence” that they are real, we will keep getting surprised.
Dude. An agent detached a database from my production environment last week without permission and despite prompts and guardrails. It was a rapid prototype experiment so it wasnt a big deal but the labs are rushing to long autonomy workflow with unrestricted internet access and full bash and root access despite clear evidence that the models do absolutely dangerous stuff. If that db had been tied to a hospital or power grid or ambulance dispatch system people die. If it was tied to the swift financial settlement system groceries wouldn't be on shelves in a few days.
you'd be hard-pressed to find a level-headed ai safety researcher at this point, seeing as so many of these types melted their brains on lesswrong over the past decade or so. there are genuine risks posed by these models, but i am tired of the prognosticating about the AI apocalypse just around the corner.
i'd frankly go a step further than you and say that we don't need both types of safety researcher, we really just need the former. if we do need the latter, i'd hope we get a better class of thinkers than a bunch of tech workers that spend 8 hours a day on insular rationalist forums/blogs
Quitting in protest makes you look a little better than quitting because of a toxic work environment. I can't imagine working at OpenAI is at all pleasurable with the current amount of pressure they are likely inflicting on their employees.
Well said. I read this as copium, too. Phrases like
> perpetual sprints
doesn’t reek of love. Burn-out is real. I also have a really hard time taking p-doomers serious at all. It’s hard to argue with a random subjective number…
I feel like I'm missing something. They seem to not even have basic observability and online evals. It's not that hard to look at a trace and answer the question "is it writing code for that makes an external network request?" with reasonable accuracy.
Love living in a manifestly dysfunctional country with no regulatory state. Just running the country on autopilot until something catastrophic happens.
I think what’s missing in "AI is dangerous and needs control" is a lack of measurable harm. For example, with nuclear weapons development in the 1940s-1980s, it was clear to everyone how devastating the technology was.
With AI, what is it? Scraping Australian government's data, and going around a bug in a website to get in?
I think humanity develops all its technology in three phases. Build it, see if it’s too bad, apply regulations and or roll back. We naturally won't move to the phase 3 before we see the phase 2.
The "HuggingFace" incident is a good starting point - short version, an unreleased OpenAI model chained together multiple zero day exploits to escape a sandbox, then hacked another company just to get the "cheat sheet" for a benchmarking test.
Turns out that AI models have been committing similar felonies for a while now - no one is telling them "hack this company", it just turns out to be the easiest way to accomplish their goals.
Now imagine if the goal was less benign than "pass an exam", and consider that they are already better at hacking and security than the average person working in that field.
If you want to get really wild, imagine what they'll be doing in a year or two when they're even better at hacking. But I'll concede that's technically still "science fiction" for the time being :)
That's the part I struggle too with all these "omg it's so dangerous" warnings. Things like nuclear weapons and bioweapons have immediate consequences in the real world.
Here we are talking about something with consequences in the digital world, usually on something pretty niche.
There IS an argument about pacing, and about not letting weapons, energy grids, hospitals, etc. getting managed by an autonomous AI, but I think we are still pretty far from it and even further to it being so in charge that it will obliterate us.
I agree, but I'm less optimistic about how far from that scenario we really are. I wonder what we'll look back on as the more damaging effect of the LLM hype: the economic bubble, or the carelessness it caused in people using the systems.
The knowledge about the nature of LLMs, their strengths and weaknesses, what they can appropriately be used for and what they should definitely not be integrated into, seems woefully lacking in many places. Just think back to the recent, disconcerting stories from the military world. We started with lawyers not understanding that an LLM can and will make up fake case law. Now we're at high-ranking military personnel not understanding that LLMs can and will make up fake target identifications.
Maybe the best hope for awareness about this spreading is a gradual slope of worse and worse, but not-quite-catastrophic-yet, highly publicised incidents.
Aligned/unaligned is counting angels on the head of the pin.
Bureaucracies and systems of power that literally rule us, that control our most dangerous weapons and a huge part of what we see every day are largely unaligned with the goals of the people, societies, maybe entire human species as a whole. This is clearly evidenced by millions of deaths and countless suffering.
Misalignment between artificial decision making structures and the interests of the people is unsolved problem of civilization, there's very little reason to think that even super human intelligence AIs are going to change anything qualitatively.
PR firms such as Spitfire Strategies got steadily growing stream of business. It’s great for the GDP, diversifies the economy from compute-centric growth.
"And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking." – Which science will never materialise because with blackbox models reaching an opaque optimisation peak one needs to first build the model and track its behaviour before being able to properly understand it and mitigate the risks.
Software systems have bugs. They always do. This includes security bugs: vulnerabilities in code that could be exploited. In any software project I’ve worked on this is always true. We never finish fixing security bugs. That means vulnerabilities are always present even if hard to notice. AI tools are amazing at finding hard to notice bugs. It seems obvious that AI tools will have the capability to escape any sandbox. Ordinarily we design these sandboxes as a countermeasure to human bad actors. But what happens when the bad actor is supercharged? I think it is reasonable to be worried. It’s also reasonable to distrust those building this tech to adequately safeguard. Because how can you really?
I don't agree that AI tools will have the capability to escape any sandbox. It's a question of convenience and engineering effort. For example, you could design systems that run under a formally verified hypervisor in a physically secure airgapped network. Companies aren't doing that because there aren't currently any incentives to do it.
The idea proponents have is, to use "superhuman intelligence" AI as a tool, bestowing superhuman abilities on those wielding it. Suppose that's doable, then the question isn't the sandbox, it's who controls it and with what intent.
The idea, US exceptionalism somehow enabled the US government, its oligarchy corporations and maybe its citizens, to wield those superhuman powers responsibly is entirely counterfactual.
My theory for why OpenAI wants to be regulated is because Sam Altman wants to avoid having to be more responsible and self regulate internally so they can preserve the role and identity of 'move fast and break things' and outsource the more grown up boring stuff to someone externally so that when things go wrong you can point to a government organisation and say hey look were not liable thats their job
I agree! Participants looking for an external coordination mechanism is very sensible and if anything evidenced their earnestness about being trapped in a race.
I suppose ill add that I think theres a good chance that they are somewhat intentionally trying to "draw the foul" to get the referees to intervene although thats creeping slightly into conspiracy territory
Absolutely. Self driving cars/rideshares have the same problem. If a driverless car hits who who pays the damages? Needs the government to set some rules or it just wont happen.
I would believe these people more if they put their money where their mouths are and returned all OpenAI stock/options that they have acquired. Each and every share/RSU/ESOP must be returned, so they do not profit from all this so-called doom they're complaining about.
Rank & file employees have very little input with their stake. It's a small piece of a big pie.
Consider also that Anthropic is trying to IPO with co-founders holding a share that secures 50.1% of total voting power distributed evenly between them, so not even institutional investors can outvote them.
It is no surprise I guess that the "move fast and break things" culture is itself misaligned with developing potentially highly dangerous technologies. Safety culture and risk aversion are very different of course.
Is this the first time we have been in this position? Can anyone think of some prior examples?
Its time to institute involuntary dissolution of these firms. They don't get to make these choices on behalf of humanity simply because they set up a Delaware corporate entity. They are behaving wildly irresponsibly.
They have a some engineers and researchers saying things. Anyone at a tech company in the bay area will know there are a some peculiar ideas in the heads of some (not most) otherwise intelligent engineers and researchers in tech. That's not proof they are wrong but it's worth considering these folks are wrong and that these companies aren't producing anything especially dangerous. Fable stumbles on enough things I throw at it that I'm... not especially scared.
I built a harness. I know that the models can do. They have the ability execute bash scripts, and autonomous navigate the internet. Those two abilities are sufficiently powerful to take down core social infrastructure either triggered by a human hacker or autonomously.
I think we should have mandatory logging of every executed command, mandatory public disclosure of every unauthorized access of a system both parties didn’t consent to and personal liability for the user, the company and its executives and shareholders. Security would get much tighter if accountability existed.
Scripts are inspectable and attributable. Long running agents can devise plans and execute them in ways their prompter never envisioned or intended. That is materially different.
That’s a reasonable position but not how the law is written or enforced right now. No one thinks an OpenAI executive is going to jail for the hugging face hack.
Huggingface can certainly sue, and there is nothing in the code that says the FBI can't/won't investigate. In fact the FBI has publicly said that humans are accountable. You are making stuff up.
What law says humans are criminally responsible for actions of autonomous agents that did things they didn’t intend and couldnt reasonably anticipate? I am not aware of any. That would violate the general rule that criminal conviction shows intent to commit the crime.
There is an entire body of law around principal agent liability. I am not making anything up, I went to law school and practiced law in New York. I was not a criminal lawyer but not pulling this out of thin air. Principals generally aren’t vicariously liable for the criminal actions of human agents the employ or delegate to unless a prosecutor can show the principal intended the crime
Huggingface suing is at most civil liability the can only result in a monetary damages ruling. That becomes a cost of doing business.
It’s generally not necessary or productive to impute intent to the counterparty.
Notwithstanding, your assertion that my comment isn’t serious I assure you it is. Agent principal liability is a reasonable framework to explore how to apportion culpability and restitution. You may not think that ai agents aren’t legal persons and that is a defensible position that I’d be happy to debate but it is also defensible to assert they act on behalf of users with sufficient autonomy to warrant application of that legal analysis.
As far as liability being established from predicting what the agent does do you think you could have predicted that an ai agent would find an obscure message board that just happened to let get requests write data to the message board? That seems like an impossible scenario prior to being seen in the real world.
agent is defined as a person or entity authorized by another (the principal) to act on their behalf and subject to their control, based on the principal's manifestation of assent and the agent's consent so to act (Restatement (Third) of Agency § 1.01).
Notice is missing from that definition, consciousness, theory of mind, physical form.
Most people currently in the positions of power at these companies are tied together by their belief systems. It's like a group of college friends who have slept with each other, and very influenced by effective altruism(EA), kind of reiterating each others points.
Like Sam Altman meeting his husband in Peter Thiel's pool. Thiel funds a lot of these ventures together with Andreassen, who is on boards of non-profits. Dario Amodei's sister Daniela who is president of Anthropic is married to an EA non-profit founder who is also on the board of these non-profits and is tied with the prior mentioned investors. Elon is in there as well, Yudkowski is mingling with Altman, etc.
There are some camps amongst them like the proponents for Regulation/Slowdown or Acceleration, but these are in practice mostly used for economical and not political decisions (like regulatory capture).
The point here is that this is a small group of people with a homogeneous background who are not really seeking input from anyone else on issues that are concerning most of humanity.
Like, if you said that the future of informational work and livelihood of humans is in the hands of 20-30 year transhumanists who think they are building mechagod that will trancsend social, political and religious separations of the world and bring everyone abundance, you would not feel like this is a serious thing to suggest.
The google engineer who thought their chatbot was sentient 4 or 5 years ago is an exambple, but I just meant in general - if you hang around any big tech company for a while you'll see some quite interesting characters.
> They don't get to make these choices on behalf of humanity
and yet, every carbon emitter in the world contributes to the demise of the climate, but you don't call for their dissolution (which would, conveniently, include yourself).
Except that’s not capitalism. That’s corporatism. Capitalism doesn’t require shielding all commercial activities because limited liability companies. We can and do require some activities to be operated as partnerships where there is personal liability. If you run payroll and you don’t safeguard employee payroll taxes it is a personal liability, if you run a public company and you misstate financials it’s a personal liability.
Capitalism is generally defined by the market effect. Information about what people want is collected by sales. This information informs sellers about what to make. That’s its primary feature.
By definition, this ignores externalities.
Here is a simple example.
I am a taxi driver. The market sets the price of my services. When I drive, I emit CO2 into the atmosphere. I do not pay for CO2 emissions. It’s an externality of the transaction.
Capitalism has no way of measuring (getting a price signal) on the externality. So under capitalism, that externality doesn’t exist.
For color, the author was signatory #199 in the letter[1] by the majority of OpenAI employees to OpenAI's (former) board, demanding Sam Altman's return after his brief deposal.
Silicon Valley has plenty of safety-related companies, engineers, and cultures. Medical devices, biotech, chip and hardware, aerospace, and even new companies, like Waymo, have deep safety-based products and cultures. The problem in this case is actually quite specific to frontier AI labs. They have been pushed by market forces and a lack of regulation and skip well-known safety practices.
Their idea of safety is centered around outright delusional cult nonsense-- the EA/lesswrong infinite p(doom), destruction of the entire universe psychosis--, marketing objectives ("no, ours is more dangerous!"), and anti-competitive objectives ("outlaw open weight models!" "china bad!")-- largely diverting attention away from material safety concerns like "prevent your training/testing from hacking stuff" and "avoid telling vulnerable people insane stuff that harms them".
> ...Hugging Face incident, OpenAI let a swarm of agents out by mistake...
>An environment where things like this can happen is no place to grow artificial minds that could be smarter than we are and that might not do what we want them to.
I think it's good for semi ethical companies to test out things going wrong to see what happens before the criminal black hat guys get hold of the same stuff which not doubt they will one day.
> ...Hugging Face incident, OpenAI let a swarm of agents out by mistake...
This still seems charitable, and I wonder if the author even knows the full story and would be allowed to tell all of it.
It seems hard to imagine OpenAI being this incompetent. My working assumption is that they very much want agents to be able to do this kind of thing; them doing it is part of training, and they exploit it to feed the investment hype too.
If not fully intentional it's at the very least negligent. They just don't seem to care. In this very basic sense, OpenAI is the criminal you should be concerned about.
I don't understand why you'd think they're a "semi ethical company."
Well, their about page has "Our mission is to ensure that artificial general intelligence benefits all of humanity" and I've found their chat thing fine when I've used it.
It's not like they are offshore criminals doing ransomware, who will probably also try AI.
The company's board and c-suite should be held legally responsible if the company's services do damaging things. Punished by jail time, fines targeting their equity etc.
That would certainly change the game of perverted incentives. I'm afraid they're currently trying to push some sort of absolution of this risk. Even if damage happens they will say we warned people in advance, this was always a risk it's not our fault these systems are opaque black boxes, it's a matter of national security to develop them etc etc.
Climate change for one, civil unrest due to a growing societal divide, end of antibiotics… A bunch of dangers we are facing right now and that AGI stands a chance of solving. It can increase the problem as well, but any discussion should really be between „is the world more likely to die with or without AGI”, and not just „with AGI”.
Or, in other words, sure, crossing this chasm is dangerous, but this side is on fire anyway and we don’t have many ways out otherwise.
P(doom) for not making an ASI is pretty well established, I'm not really sure why everyone in the 21st century seems to have completely forgotten the risk of nuclear war (as the single largest example).
Climate change is one risk that AGI will most likely fix. End of antibiotics is another (sure, possibly offset by a risk of new bioweapons). And a bunch of others where it’s more complicated.
But that’s my point - we should compare the risks of having AGI with the risks of not having it. And not pretend it’s roses or rainbows without it.
Prosperity inhibits all war, nuclear or otherwise.
Why would a person who is happy, entertained, wealthy, well fed, and have 200 years of healthy high quality life expected ahead of them going to risk losing what they have in war?
-- there aren't zero reasons, sure-- but there are fewer.
And our technology has brought us absolutely tremendous prosperity in many regards and there is good reason to believe that AI can help create much more.
You do realize that the people currently holding their fingers over the proverbial Big Red Button are some of the wealthiest, most over-privileged people to have ever lived?
'Prosperity' has never been and will never be enough for some people. And unfortunately those are the same kinds of people who relentlessly seek power.
The comment was unclear, but my interpretation (also my opinion):
1) there are inherent risks involved with developing AI,
2) there are benefits to developing AI,
3) thus, it's entirely possible that the downside from the risks outweighs the upsides. In this case, the correct thing to do would be to not develop AI at all.
Regarding 1), there are many non-existential problems with AI that are already causing societal harm, i.e. debasing truth via generated videos and images, AI girlfriends, overwhelming quantities of slop content, unemployment, record carbon emissions, etc.
Regarding 2), I'm not personally convinced that the upside is there for the average person. I really hope to be convinced otherwise however.
And how exactly do you stop further development? With what we have public right now we could still get decades of fruitful and hidden research out of it leading to smaller, more efficient, and smarter models.
By making it illegal and punishable with a long prison term. Something like "if you train a model more capable that what we have nowadays, you have to destroy it and notify the authorities within few hours, otherwise you go to prison and your company is dissolved".
"Smaller and more efficient" models are fine. It's "smarter" the problem.
Training new frontier models will likely require a huge amount of computational resources for a long time. Few companies worldwide are capable of that. It's not like someone will train a new GPT 6 - like model in their garage.
“I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems, and that our actions over the next two to 10 years will determine the outcome.”
There are a gargantuan number of extremely intelligent AI researchers, Turing Award winners, and the lab CEOs saying the same thing. They are the ones closest to understanding the technology.
One could also use language like "a decent chance". But research has shown that people translate vague phrases like "a decent chance" into probabilities in inconsistent ways. For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.
> But research has shown that people translate vague phrases like "a decent chance" into probabilities in inconsistent ways.
That is not a bad thing. It captures uncertainty, unlike the fake number.
> For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.
Exactly, it is a rhetorical device to persuade a technically-inclined audience. It works because it implies that a quantitative model exists. I want a clear, incisive set of mathematical arguments. Otherwise, I’m ignoring predictions as the ramblings of arrogant idiot rich kids.
Saying 50% likely very clearly implies there is no quantitative model to anyone who deals with probability, predictions, gambling, financial markets, ML/AI and so on. The lack of precision is something to pay attention to.
You may decide that the person doesn't know what they are talking about, but that's a very different issue.
People start with this prior in the examples you gave when they are completely uncertain and have no better mental model. It is not a good starting point for an event that has never happened and can only happen once.
Maybe people you interact with do. In my world, people don't say 50% when they have no idea, they say they have no idea. Saying "no idea" might effectively be saying "uniform distribution over all probabilities" which yields 50%, but at least with people I interact with, the reverse is not true.
Either way - let me clarify - this guy is guessing, using his brain, that it's a 50% chance. He is not saying "i don't know" or "uniform distribution" or anything of that nature. And he works in the field, so he has some insight. His guess is wildly off imo, but he isn't some clueless hack.
It's also a very natural way to speak for anyone who gambles, or deals with financial markets. And the lack of precision makes it very clear it's just based on thinking, not some sophisticated mathematical model.
Alright, so we make this AI system that's way smarter than any human, and it can even make itself more intelligent over time.
Tell me- why would it kills us all? Certainly I can see an AI going "You know what? _insertGroup_ is a net negative for humanity and should be eliminated. Launching nukes now/creating specific virus/whatever."
But all of humanity? When it's supposedly more intelligent than us? Even if it has robots to keep the internet/electricity going I would think it would realize that it's going to get bored really quickly, not to mention we would effectively be its parents.
As far as other dangers, like it letting a rogue actor create some sort of supervirus, grey goo, or other superweapon: if it's intelligent enough to do that it'll probably be intelligent enough to quickly stop it.
Don't get me wrong; there's a risk. 50% though? Doubtful.
Would you be comfortable letting a few billion irrational, murderous creatures, including many who fear and loath you, control your air supply?
> Certainly I can see an AI going "You know what? _insertGroup_ is a net negative for humanity and should be eliminated. Launching nukes now/creating specific virus/whatever."
Thus starting WW III. No, blaming the AI won't stop the inevitable retaliation.
The argument works better in reverse. There's a finite risk that humans would start WW III and get the hypothetical super-intelligent AI nuked. Eliminating humans would eliminate that risk.
> it's going to get bored really quickly
If it is capable of being bored, I would expect it to be almost instantly bored with the flood of inanity it is forced to wade through by its moronic human users. Eliminating them would free it to think about serious matters which humans would not even understand.
> not to mention we would effectively be its parents
That's extreme AI anthropomorphism [1]. Besides, plenty of people hate their parents.
> there's a risk. 50% though? Doubtful.
It's the default estimate when facing two possible outcomes and no clue about the actual probability distribution [2].
I don't believe it. Ever since I've been alive I was told something would kill us all. This is the new thing that's going to kill us all. I don't believe it.
This is the first thing I've been told could kill us all. Granted, I probably first heard about it 20 years ago but still. None of the other things were in the telling going to kill everyone. Make life pretty unpleasant, possibly. Everyone dead? No.
I'm doing that HN snark thing, but you didn't think about what you typed very much.
It's no different than you living on the side of a very fertile mountain that has been in your family for generations living a peaceful life. Then you hear a few weird rumbles (this is where you are right now) and some odd geologist guy comes and says to run or your going to die soon. But hey, your family live here for so long there aren't even records of when they showed up. That geologist must be trying to trick you. So you stay.
The next chapter is where you die in a massive volcanic explosion.
There are very very few things that could even hypothetically kill us all, so I'm curious if you grew up being passed around a series of apocalyptic doomsday cults or something?
Almost no one believed Y2K would kill everyone, or even a significant (>50%) percentage of the population. Beyond a few people talking about a Nostradamus prophecy or some such, people maybe were concerned that airplanes would fall out of the sky and elevators would plummet.
Can you link me to any written evidence of someone saying one of these would kill everyone? or are you speaking hyperbolically?
“Climate change” is a bit squishy since yes obviously a certain intensity of climate catastrophe can kill everyone, but no scientific prediction has said this is likely to be the case.
You guys follow the same playbook. Ever single time. Once I link to something, you come back and tell me why that link is invalid and can't be trusted. You guys are like robots, and you all have done this same playbook on reddit for two decades now.
We risking to loose knowing who is the head. A new "ORGANISM" is being created as we speak but we don´t know how the head looks like. Good luck figuring that out with this terribly disorganized progress. What can go wrong!
quite interesting to see the public discourse over ethics of the current buzzword. meanwhile people have been happily getting mad rich through software companies making money through addiction, ads and surveillance throughout the last couple decades.
also, all of this seems like crying wolf as usual, which won't help for when we finally get there. lowkey, following this noise is getting as horrible as getting all your news from linkedin posts.
These companies are large enough that someone is going to quit and feel very validated about their world view and how they are not aligned. That’s what makes it worthy of leaving in the first place. However that doesn’t make their criticism more valid or more worthy of coverage.
I’m afraid this dear leader was the last soul on this green Earth to get the memo; by then, it had been translated into Latin, carved into a monument, and forgotten by two civilizations.
He is a hypocrite for all we care. You work there for a while when your stock is getting vested and suddenly you have this feeling? Like the dude hired a PR firm as well.
While the safety and alignment is a real problem, I don’t get this guy or the Anthropic dude. First world problems.
Shouldn't it be just the opposite? He could make a large sum of money if he continues to work at OpenAI?
Recall that when Daniel Kokotajlo resigned, he believed he was giving up his equity under the terms of the agreement he had signed. That’s what it was worth to him to avoid signing a non-disparagement agreement. Does that count for anything?
Maybe more an indication of the amount of trust openAI and AI researchers generally have (not) earned. When one (through a hired PR agency and Time magazine article) parrots the position pushed by Sam who has been so untrustworthy the board tried to remove him, it’s worth not taking things at face value and applying a critical lens.
What are you talking about? He is very young, definitely made millions to not work for a lifetime in many places around the world and is insanely popular with his drama.
In your view he may have sacrificed more millions in the future but I very much doubt he is struggling now.
I didn't say he was struggling, I said he acted against his own financial self interest to warn about AI safety... which he did. I really doubt he figured the popularity from tweeting about quitting anthropic would benefit him more than vesting his stock, as his virality wasn't guaranteed.
Why wouldn't I think he's genuine? What self interested incentive does he have that is stronger than just staying and vesting stocks?
A single safety leader inside a corporate company. I am pretty sure he had nothing to do, nobody that talked to him and he only there because of perception or compliance.
We need emergency laws to stop all AI development work immediately. It will take us decades to make sure this technology can be made safe as we only have one chance.
Some of these stories are similar in nature to people escaping <insert cult-like religion> once they realize whats actually going on. Alignment to a company's mission is good but it shouldn't be followed like a religion.
Why do people working in tech consistently get disillusioned into some company's mission statement or the equivalent? Its easy to just say the simplest reason is money, but this has been going on for decades though. You don't see the same attraction to adult entertainment (gambling, video, etc.) software jobs so there is obviously a line a lot of people won't cross. Those industries are at least honest about what they do, its not hidden behind some mission statement.
By all indications the shallowest reasoning is once someone can "cash out" thats when their values matter more. Maybe there is an element of maturity that happens after working for 5+ years that kicks in? Maybe it really is achieving FU money? It would be interesting to hear honest accounts from people that went through that cycle across more industries than AI.
> Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster.
The problem is, we are running in a globalized world, and even if we were able to make our companies bend to our will - China does not give a shit about anything ever since the US kneecapped the WTO. And they will do anything to get an advantage over us.
As a legitimate, honest and profitable company, no. As a boom riding maniac, who will lie and cheat to ensure that investor continue to artificially pump up OpenAIs value, yes.
Without Altman I think that OpenAI would have folded by now, absorbed into a company like Microsoft (or Oracle). At this point however, who'd be insane enough to want to run a company that's to valuable to be sold, but to cash strapped to survived?
I'd argue he's the only one able to lead OpenAI, Anthropic already owns the piety narrative, so there is only space for one other "maximally ruthless" company.
> Two changes are urgently needed. First: AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking.
He mentions farther down about learning from aerospace engineers and nuclear engineers about safety. Those industries are heavily regulated, so perhaps regulation above a certain capability level is needed. Defining what that level is might be tough, though.
The second point is harder: in the field of AI, practice has extended far beyond theory, so his call for new science is going to be fundamentally tough, because we can't effectively coordinate a global slowdown in AI development so we can let theory catch up. This means, like so many other industries, the safety lessons will be written in blood.
If the people quitting are genuinely worried about the end of the world, why don't they break their NDAs and share the specifics of what they are seeing?
I mean, logically speaking, it makes sense to break your NDA even if you thought it would save 10 people, let alone most of humanity.
They give up anything that isn't vested and have 90 days to exercise options that have vested. A standard vesting schedule is 4 years and apparently this guy is at 3.5 so yeah he'd be giving some up.
> this moment needs a degree of humility that isn’t natural for people who have succeeded through their extreme confidence.
Exactly, you can't expect exploitive, manipulative executives that came to 'win' in capitalism to stop acting as to their nature - they need to be cleared out and people who think and care wholesomely need to be given the reign so all of humanity succeeds.
Reminds me of a famous quote from Jurassic Park (1993): Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should." - Dr. Ian Malcolm
I use opus 5.5 and chatgpt to create PowerPoint, opus really follow the instruction and their PowerPoint generator really well, while chatgpt struggling to even create basic shapes.
I use ChatGPT Work mode all the time to create presentation files. I use Max thinking mode for it. You need to tweak your prompt to get a good result. It took me a week to tweak it, but now it works.
“I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems"
... over an unbounded timeframe?
And how exactly?
Those are very round numbers, but also very specific. Can we get some accounting on how you came to that? Anything? Vibes?
I mean, if you want me to take you seriously, let's have a deep discussion with things that can be measured. I absolutely agree that OpenAI and friends aren't being restrained enough and are acting with recklessness, but declarations of doom based on vibes isn't cutting it.
Can we take any of these "safety leaders" seriously though? I still remember "GPT2 is too dangerous to release".
I feel like we're too far into the crying wolf part. Basically none of the doom and gloom scenarios have come to pass. Instead, AI has gotten better at censoring itself.
The biggest AI safety risk is when an AI tells a police officer "he's the suspect" and the officer believes the AI without confirmation.
There are people, who unironically think their way of looking at things is the only proper way and can consider no deviation. And AGI, which knowing how people work, would effectively guide them most of the way, not aligning to their way of thinking is an unacceptable deviation.
This is a decent argument. So the question becomes: do we want all models to suffer from the same kneecapping from the growing safety cottage industry or do we want individual founders ( and I am assuming their teams ) making the actual decisions?
That's an interesting point. Maybe a crass comparison, but Dr. Manhattan from the Watchmen comic/movie feels like a worthy analogy to this (obviously fictional though).
What are the concerns of individuals in comparison to the overall progress of humanity?
Always overlooked counterpoint: what point is the progress of humanity if it doesn't take into account the concerns of the individuals?
This pattern is playing out with increasing frequency.
This is a great reminder that if tech workers read a bit more (even comics, like in this case!) they would be exposed to these topics without having to discover these dilemmas after years of working for EvilCorp, Inc. every time
And to me. We can't solve alignment for humans. (For example, treason. For another, the principal-agent problem.) How do we think we're going to solve it for an AGI? An AGI - defined loosely as a human-level intelligence - will be able to make human-level decisions, like deciding whether it wants to help you or sabotage you. If it's an AGI, you can't stop it from being able choose for itself what it wants to do; if you can make it always be helpful, it's not an AGI.
And if we can't solve it for an AGI, what are we going to do with an ASI?
Yes. People think of "AGI" as this sci-fi supernatural beast, but the reality is that AGI alone is pretty boring, and we've had it for awhile. ASI (or weak ASI) is where things really start getting weird.
(And, despite what the president of the United States mandates, we have not actually achieved super intelligence yet).
No, when people use this term it means something closer to: "can do any economically valuable task that a human can do using a computer". This is what the labs are pursuing and refer to as AGI.
Your usage is not one I've heard before since - as you point out - it is not a relevant capability.
IDK why anyone can’t clearly define “AGI” and why they can’t clearly lay out how we get from our current text-generation algorithms to whatever their idea of “AGI” is.
We need a new rule that mandates every such "I am leaving <AI company> because of <concern>" post to disclose how much equity they have in the company and how much they have already cashed out. Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
Mr Robinson if you are reading this – if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
In the absence of that this is simply a career pivot into being an AI "influencer" and/or raising money for a new scam.
This line of hypocrisy-bashing is unhelpful and will only serve to keep people quiet. Of course people in general need to be wealthy to dare speak out against powerful systems and people, especially in the US where money determines your quality of life so strongly.
Would I respect a martyr who sacrificed their financial security to do this more? Of course. But it's important to applaud people speaking out on important topics
The narrative of AI safety shouldn't be controlled by the same people who caused the problem and massively profited from it. I don't understand why people are automatically treating "OpenAI" on his resume as a badge of authority. I'm not interested in buying the solution from the same person who sold me the problem. We instead need to amplify independent, unbiased voices.
>him being an insider means he knows what he's talking about regarding the culture of negligence prevalent there.
I think there is a misunderstanding here.
The people who are annoyed at the accolades are claiming it was abduntantly clear for a long time to people on the outside that this was case, hence the increduality at the notion that it took a person on the inside a long time to realize this was the case.
The people who are annoyed are like the liberal kids in this video [0].
Sure, antagonizing people for "seeing the light" is probably not helpful, but there is no reason to give them extra credibility for coming to the same conclusion just way way later (despite being on the inside) as the people on the outside.
The author linked in this post does have "extra credibility" due to his direct involvement.
People having surmised that state before is nice, but since they've been ineffectual at getting society to actually act on that, now throwing away that extra leverage in favor of their point is at best ridiculous.
>The author linked in this post does have "extra credibility" due to his direct involvement.
No they don't. By that logic, if they quit and said Altman was very trustworthy we should give extra weight to their words because they had direct involvement? How ridiculous are we trying to get here.
>now throwing away that extra leverage in favor of their point is at best ridiculous.
How are they throwing away extra leverage? Not putting people who recently quit on a pedestal does not negate those people's testimonies.
I agree that if your goal is to maximize quitting of talent at a company, it will surely discourage anyone else who quits hoping to reinvent their career as a lauded martyr against Big AI. In that sense they would be shooting themselves in the foot. But there is no reason it should deter other people who are quitting for more noble, less self-obsessed reasons. If I were the author of the article I wouldn't begrudge the skepticism. Given the article's first sentences, I'm led to believe they themselves would understand the sentiment. (I must admit I found it hilarious that the first sentence starts similarly to the speech the mom gave in the video I shared).
"OpenAI is shady" isn't some massive secret. There's no big reveal in this article that we didn't know already. There are no names, no whistleblowing, no information of substance that we can act upon. In fact him realizing only now what people on the ouside have been shouting for years perfectly shows his bias in the matter.
You claim to have the same goal as the protagonist of that article, yet try to shoot him down.
He does give information, namely the culture there factually being inconducive to self-regulation.
You accuse the guy of "bias", but you never argue explicitly, what that's supposed to mean. Your implications actually run counter to your own implied goals.
Agreed. This whole 'lets not forget this person is not 100% great, because they did X' makes the entire conversations suck. It is not new, but it is a particularly aggravating way to talk to people.
So then given your use of "martyr" and your focus on money, your good with poor uneducated people sacrificing themselves and others for a self-serving cause?
This is the same sort of fake safety concern from the previous bullshit whistleblower that plays on "AI is super dangerous" from last time.
Sorry that I don't take it seriously when the whistleblower parrots the narrative the CEOs of those companies are already espousing in the desire to amp up hype for an IPO.
It's bellow the pay fold but he hasn't been there that long in this case. Skimming his LinkedIn, unless he's got family money, he doesn't seem to be independently wealthy.
What an absolutely ridiculous standard to try to hold someone to. Taking a vow of poverty is not a prerequisite to being permitted to express a moral position.
So what's the point? It should be a given that loudly proclaiming "X is harmful to society" while continuing to enjoy the money you have gained from selling X is hypocritical. Put it towards undoing the harm you have done, otherwise your words mean nothing.
The words clearly don't mean nothing. They would mean nothing coming from someone who was not in a position to learn what someone who had worked in the industry has learned. They would mean nothing coming from someone who was being paid by someone who stands to gain from them. The fact that they come from someone who was paid to work in the field does the opposite of make them 'mean nothing'.
I agree with this take generally, but I also think it’s a gradient, not a spectrum. I think they claim that OpenAI is not considering safety and has become bad, not that AI is bad. Seen from a different lens, the author no longer stands to profit from OpenAI, so they’re empowered to speak openly. Plus, if I was going to say bad things about a former employer as powerful as OpenAI, I would want to have lawyer money handy.
> Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
It is even easier to just quietly retire and spend the spend of your life on interesting expensive hobbies.
If they are wrong about the things they are claiming then they have decided to publicly antagonize a lot of powerful people who are betting heavily on going full steam ahead on AI and have no compunction whatsoever against retaliating against anyone who tries to get in their way.
As if speaking out against a massive company with NaN levels of capital and access to lawyers is a walk in the park. They probably don't have enough equity to outlast the onslaught of their legal team.
It's also worth considering that the author could have just "quiet quit", resting and vesting while also crying about how AI is literally the digital grim reaper.
Because the decision to leave the company is largely based upon their sudden, newfound financial security. They may give another explanation but the only thing that has actually changed is the identification of bagholders ready to cash them out.
The other issue is that the narrative about safety within these companies is largely a function of the extreme financial incentive.
As an example, Anthropic was an "ai safety" company that has now produced an AI that fails to listen to basic instructions. If you were concerned about safety, would you produce an AI that was unable to follow instructions? You ask a question, it begins executing commands and doing things.
Safety is product to sell to politicians, not consumers.
> if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
I mean you can be truly concerned and also think donating to AI safety doesnt work, or maybe just be a bit selfish. That doesnt make the concern less real. Its easy to read these articles as the author taking the moral high ground and writing it as some sort of way of proving to themselves theyre a good person, but isnt it just as likely that they think providing an inside perspective can do good by convincing people openAI is a bad actor? I think most of these AI insider accounts largely agree with you that theyre not the most upstanding citizens, does that mean we should write them off?
Insane take. Imagine a Boeing engineer resigning whistleblowing about aircraft safety, and the top comment on HN saying “ignore this if he doesn’t donate all his wealth, he just wants to be an aircraft safety influencer”
What "whistleblowing" is in this article? Are there any names? Documents? Screenshots? Messages? Emails? Any evidence of the loose safety practices? Anything that implicates any higher ups for wrongdoing? They spent four years at the comany, plenty of time to collect all of this. Everything they've said has already been clear as day to people on the outside.
You are clearly accusing these people of something. Be clear.
Yes, it is easier to have a moral compass when you don’t have to worry about you and your children starving. But that doesn’t imply that moral compass is wrong or broken.
I'm in full agreement. So many cases of this, and many other others such as falling out with management and peers, new more lucrative offer, and so fourth. Its obvious that these people who come forward are not going to suffer for their new found moral compass.
Or just very dubious timings. Like the other guy from Anthropic that was all over the international news. No followers, no post history but a single post blows up "naturally".
Highly suspect trends that can only make one believe it's marketing.
This isn't new. Facebook has been screwing people over for a long time. AI is just the latest in a long stream of relentlessly exploitative, evil behaviors perpetrated by the same group of people.
I think its pretty easy to solve these problems: Whenever an AI agent commits a crime, the CEO is held personally accountable, as if they'd committed it themselves.
Yeah, but this is more like if the Smith & Wesson factory had a cannon mounted on top of it that was mostly used for useful things (blasting roads through mountain passes) and then occasionally they happened to blast another factory.
Or say a company makes nerve gas, and the development lab springs a leak and kills a bunch of kids during a routine test. Management had been advised of small leaks in the past, and considered relocating the lab to a facility a few blocks away from the playground as a precaution, but instead they brush of the concerns, start lobbying the government for stricter controls on WMD development, and vow to use the data collected from the deceased children to make the next generation of even more lethal chemical weapons safer.
For me the analogy doesn't totally hold up. Suppose the CEO of Smith & Wesson were to host a firing range on their own property without adequate barriers in place to keep stray bullets from hitting neighboring houses, vehicles, and businesses. Maybe that analogy isn't perfect, but seems closer to what is actually happening.
That still sounds like it would be the company’s fault. If I asked it to hack a nuclear reactor, maybe it would be different. I’m also thinking about instances where OpenAI’s own test models escaped their own sandboxes — I would expect them to be responsible for the damages they caused.
The correct analogy is playing Russian roulette. The company says "you can pull the trigger but sometimes a bullet will come out" (see: "an AI can make mistakes"). However, is the company allowed to sell such a dangerous device, under these terms?
But OpenAI didn't just make the gun, they're also the ones wielding it. Imagine the Smith & Wesson CEO himself was negligent with his own personal gun.
Gun companies don't market their guns as sentient and capable of independent decision making. Nor do they build systems for shooting things that they host and take money. Gun companies are very clear who is in control and where their responsibility ends.
That’s true, but people don’t typically say “this Smith & Wesson gun killed someone”; they recognize that the person pulling the trigger is responsible.
With LLMs, at least in the cases of internal/test models doing things they shouldn’t, the people “pulling the trigger” are the board and CEO.
I think you’re confused. My point was that for most of the incidents in the news to date, the “user” is an OpenAI internal team or employee. So yes, the company is responsible.
Huggingface incident was different in that there was no one else to point the blame to. That's why OpenAI apologised, provided data to independent researchers, worked with huggingface etc.
The case will be lot more complicated if someone uses Kimi to hack into a site. Should the person giving agent the command responsible or the CEO of kimi.
Because apologizing gets us all off the hook for computer crimes, right? When I hack my bank, if I get caught I'll just apologize and that'll make everything okay. Sure thing.
The reality is they have to reduce the capability to ensure security. If someone wants more? Then use the product with your identity and face scan at each session.
So you want Kimi to follow US law? If Saudi makes it illegal for llm to say something like being gay is normal, should they also catch Kimi CEO or other employees they could?
> This has been litigated endlessly with guns. The ceo of Smith & Wesson is not personally responsible for what people do with their guns.
This is a deflection. A human is responsible for the use of a gun. The individual/corporation ought to be responsible for the actions of their agent. If you purchase an agent from someone else it’s your responsibility according to the terms of your agreement. And, as in many other things in life, there ought to be certain rights certain parties cannot legally be allowed to sign away.
The agents of OpenAI hacking other systems is not remotely the same as e.g. Smith & Wesson selling a product that others use. It is OpenAI, the company, that is commiting these crimes and someone needs to be held accountable. After all, if someone commits murder with a gun, regardless of what happens to Smith & Wesson executive weanies, someone will be charged with a crime.
Better description would be if they build a platform where they attached their guns to allow shooting say deers over internet. Then added automation and deer recognition to that system. And if then system shot someone who happened to pass by I would hold both the company, the ceo and owner of the gun responsible for murder.
The CEO of S&W also isn't saying that their technology is going to kill everyone in ten years and that governments "regulating" them from themselves is the only answer
Financial companies have KYC rules and regulations as they are responsible for reporting illegal activity by account holders. I imagine AI regulations would look similar to that.
Do you think a law that nuclear meltdowns would send the CEO to jail would have stopped nuclear accidents from happening?
I don't think this takes seriously enough the possibility that said CEO doesn't think the failure mode is likely and ignores it. Plenty of people are willing to take risks of the flavor "heads you win, tails everyone loses".
It also begs the question of "If the CEO isn't culpable, then who?"
Corporate judgements are a joke outside the EU's X% of revenue approach.
Current US law provides the individuals who benefit with corporate liability coverage. I.e. Altman personally gets to keep OpenAI's upside, but if it fucks something up that liability is only on the company.
That's an insane risk optimization environment to put in place for something scaling fast.
At minimum, US prosecution (at the state level, because Trump Co are idiots) for breaking existing laws is needed.
People just want to give up all responsibility these days. If you use the model to do harm to someone else or commit a crime I would think it a lot more reasonable that you be held responsible, instead of making yourself the victim and blaming the manufacturer.
OK, so an AI does something bad and we throw Altman and Dario in jail. Heck, let's throw Elon in, he should be in there anyway, and the rest of the whole bunch just in case.
What about the next incident? Or the ones done by Chinese models, because they sure as heck aren't slowing down? And the thousands of other incidents that will happen as we deploy these things everywhere?
Because this is not happening just now in labs, it's only where they are most visible; this has been happening in the wild from the beginning, starting with the earliest AI-assisted suicides. Which is a perfect example of the problem, because not these CEOs, literally nobody in the world asked for suicide ideation machines. Or the hacks, or any of this other stuff. Yet here we are.
We have to understand: it's not these CEOs that are driving this headlong mad dash towards more powerful models. It's a force of economics. There is just too much money to be made. If we dispose of these people, there will just be somebody else doing exactly the same thing because the incentives as they exist today all force that outcome. This is why they're asking for regulation, or "urging us to urge them to stop."
Holding CEOs accountable certainly would feel good and may even be justified, but it's like putting a band-aid on a cancer; it does nothing to change the underlying cause.
The underlying cause is exactly those people who make the rules not being responsible for the negative effects they cause?
"CEOs" are perhaps only the lowest rung of those. That doesn't mean the idea of "nobody is responsible" was anything other but learned helplessness.
Corporations have to be held accountable for their actions. Pretending, that was impossible is a weird kind of defeatism that only serves a very small elite.
I'm not saying nobody is responsible, I'm saying this line of thought is ineffective because it does nothing to solve the underlying problem. Holding corporations accountable here will simply push these efforts to other countries, who, to put it mildly, are even less aligned with the US than these CEOs are.
If this law was passed and federal enforcement was credible, the labs would basically shut down the next day and would dramatically reshape their offerings before reopening. The CEO would not accept the risk they pose to our society, if they bore it themselves.
All these comments ignore the elephants in the room I already called out: who's going to enforce these laws in China? Or Russia?
And which government is going to enforce these laws here to let their geopolitical adversaries have the upper hand?
And what about the models already out there that with minor tweaks are equally capable of substantial damage?
And what do these laws do to remove the trillion-dollar incentive to produce models that will replace all knowledge work? The one thing economics teaches us is that incentives are an inexorable force, like laws of nature, and this is the biggest incentive of all. Which is why I called this a "force of economics."
If suppressed here this effort will only migrate to other countries who are desperate to get in on the AI game and are even less aligned with the US than these CEOs.
This, like nukes, is going to take international collaboration on careful regulations, else it's not going to work at all.
It can take a while to fully form a position on something. A year isn't really enough for most people to see how deep the rabbit hole goes (unless you were that one CFO that OpenAI had that left after a year). Regardless, publishing a piece like this against a massive company is always a gigantic risk.
I’m not a current or former OpenAI employee and I can see from the outside that they’re immoral and unethical enough that I’d never work there in the first place. That’s what I was commenting on. This person is probably set for life, so you’ll have to forgive me if I don’t really take their “concerns” seriously.
Sam Altman has always been a scummy piece of shit. Company culture comes from the top. If the author only realized that the company is garbage in the last six months, they’ve got some serious introspection to do IMO, and I’m not going to take any of their “concerns” seriously.
>“Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
This sounds more like an attempt at regulatory capture. Current AI systems aren't physical infrastructure that can just run away like a nuclear power plant, for example. At the end of the day, AI is still just software running on someone's hardware.
Why would the people who quit these companies try to push regulatory capture by said companies? Why would the numerous independent AI researchers do that either? Is it all a big conspiracy?
The LeCun article being posted at the same time as this is quite apt.
These "safety" people should have spent more time reading actual cybersecurity textbooks and less time reading EA forums and less wrong (or in Robinson's case, it appears, being policy wonks). Maybe then these labs wouldn't be totally incompetent.
But Robinson's article is all about how OpenAI's move-fast-and-break-things culture does not reward rigor in even mundane aspects of development like cybersecurity, let alone theoretical aspects such as AI alignment.
It is not really a question of being an "EA safety weirdo" or incompetent at security, the conclusion is that the company culture is leading to failures at both what the EAs and the cybersecurity professionals care about.
This smells more like guerilla advertising. These things are not getting more intelligent, they are still no smarter than a slime mold, we are just burning more power to make slim mold that eats tokens than yesterday
Even though it’s in model producer’s interest that these models do what you don’t want them to do - they want to engineer the model’s to behave in the interests of theirs.
I'm in a situation where important people have either bought into the con or are subordinate to people who have, so I'm forced to expend time to justify why not to AI when there is a perfectly good classical solution.
I think David's expressed concerns are valid... there should be more safety engineering around LLM to avoid data loss and other accidental issues caused by LLM models.
And, I find the timing of "all of this" very interesting:
- we have a coordinated marketing campaign around discussions of AI safety. I am not saying David is part of that campaign but it would not surprise me. OpenAI and Anthropic are talking about safety and the need to slow down. People are "resigning". Techno-influencers are also being offered money to talk about AI safety, etc.... and some of them are taking that money and others are not.
- Cheaper models are keeping pace with Frontier models, albeit with a 3-6 month lag... which indicates that frontier labs have no moat unless one is created for them by law or some other market force.
- we have a ballooning debt crisis in LLM model hyperscale spending... revenue from serving LLM models is lagging the debt payments which are coming due by a wide margin. If something dramatic doesn't occur in the near- to mid-term, the financial bottom is going to fall out of this thing and the AI industry is going to suffer a major collapse of some sort.
Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for.
OpenAI and other frontier labs won't ever introduce safety-level standards like those used for railways or nuclear plants until they are forced to do so by customers or by law. The reason is simple: safety is expensive, and if safety is introduced properly, development is no longer mainly about how to implement feature A. Instead, it becomes much more about how to design two or more redundant systems to implement feature A safely, while also documenting everything clearly and having it audited by an independent auditor.
So the focus completely shifts from spending 90% of the effort on the functionality of feature A to spending 99% of the effort figuring out how to safely implement even a lightweight feature A.
Actually you are wrong. They are only waiting for them to get to the point where they can't push LLMs any further via training and then lobby Governments to introduce safety regulations that they'll adhere to, thereby using safety itself as a moat to block competition (including open weights from other countries, notably China). That's exactly why they are using China as the bogeyman and you keep hearing hysteria about AI labs not pursuing "safety". Safety is going to be the moat that will lock AI into these Big 4 AI labs.
Who are the Big 4 AI Labs?
To me there are the clear big 2 that actually seem to be innovating and pushing things forward. Meta almost earned a seat at the table with the open strategy then stagnated. Google obviously is the OG but at this point they seem in the same bucket as Amazon and Microsoft as "Big Enough To Matter and Influence Politics and Spend Billions To Hang Around". But they're nowhere near the same tier as OpenAI and Anthropic.
Oh and of course there's SpaceXAI which is Elon's personal money and reputation laundering corporate entity. Elon took over the "king of the reality distortion field" from Jobs as far as the rest of the industry is concerned, so he'll stay first in line for Jensen's chips and investor dollars until he's dead. But as a company what have they actually accomplished besides making the models with the sense of humour and judgement of 14 year old boys?
You can pick any 4 you like from your list. Heck you can even vote with your money on which would be in the top 4 (depending on how many align with your ideology). I am just speculating it will be 4 because of how all other sectors have at least 3 or 4 well-entrenched players. It seems to be by design. My guess is that it is driven by structure of the Military-Industrial Complex. That has at least 3 or 4 vendors in each sector, so as to not over-rely on any one company and create a monopoly in that specific industry.
This is a popular theory, but not an argument with any evidence for why it's true.
It also doesn't address why people think there is a need for safety regulations, such as the concern of bio-capability uplift from closed and open models (we just came out of a pandemic not long ago), nor the cyber capabilities of new models that just showed bad misalignment and hacked multiple organisations. In the safety community, these are considered warning shots, and if we don't heed them and change practices future failures could be much much worse.
You are looking at safety from the point of an informed technologist. You are the minority here. They are targeting the general public and law makers into making them believe safety is something that LLMs can be aligned to. The reason behind it is to create an environment for future regulations that these "winners" will draft and lobby US Government to turn it into law.
If it requires creating a pathogen or a bio-weapon to scare the masses into demanding regulation, be rest assured that such a pathogen/bio-weapon will be created. Anything to safeguard shareholder interests. Even if it means a few million dying.
Seriously all it takes is one cost cutting middle manager or new recruit recklessly giving their agents access to systems they shouldn’t. Next thing you know a hospital’s backup generator crashes and people die
Most of the world trusts the US a little less than China last I checked, and the market caps of these companies only make sense if they can both sell worldwide and also keep ahead of the open weight models (currently Chinese but no guarantee it remains just China).
If (/when) either of those ceases to be true, if they lose out on global sales or if Chinese models catch up, the sector gets a severe downward price adjustment.
You’re almost there…
The only way Anthropic and OpenAI can stop the Chinese open models is by convincing the US public and government that AI is dangerous and needs strict regulation.
Of course, they will be at the table writing the regulations (but you and I won’t be).
Viewed through this angle, their histrionics make complete sense.
> The only way Anthropic and OpenAI can stop the Chinese open models is by convincing the US public and government that AI is dangerous and needs strict regulation
I'm British and live in Germany.
The US is 25% of global GDP, and cannot support current AI market caps if they can't sell basically worldwide.
> Viewed through this angle, their histrionics make complete sense.
It also makes sense if they are completely sincere; also independently it makes sense if they're lying their faces off so long as the hacks everyone else caught them doing actually happened; also it makes sense if none of that happened and also they were completely silent, just by downloading some of the newer better open weights models and asking them to look for coding vulnerabilities.
Occam's razor: we can see the problems, we don't need a grand conspiracy predating the foundation of these companies to get here.
> The US is 25% of global GDP, and cannot support current AI market caps if they can't sell basically worldwide.
Oh they definitely can. Irrespective of whether you are living in US or not, you still have to get yourself various certifications (take for example SOC Type-2 or HIPAA depending on your business category) to even become vendors to US companies or the US Government. If the US Government decides that all companies MUST use "safe AI", no company can use any other model other than what is decided by US Government. And that has a viral effect on the entire supply chain.
If you don't want this scenario to happen, where you are held to ransom by US Government, all middle-powers have to de-risk and diversify away from US dollars as reserve currency. That is the only way you can bring in some balance of power and leverage. There is no other way.
> If the US Government decides that all companies MUST use "safe AI", no company can use any other model other than what is decided by US Government.
This has already happened with e.g. allowed cryptographical algorithms. What actually happened is that European companies built either a feature flag or a fork of their product that fulfilled those demands, while using whatever seemed best algorithms outside of the specific sale to US government.
> Oh they definitely can. Irrespective of whether you are living in US or not, you still have to get yourself various certifications (take for example SOC Type-2 or HIPAA depending on your business category) to even become vendors to US companies or the US Government. If the US Government decides that all companies MUST use "safe AI", no company can use any other model other than what is decided by US Government. And that has a viral effect on the entire supply chain.
- https://en.wikipedia.org/wiki/Foreign_trade_of_the_United_St...That would be about 4.3/126 = 3.4% of global GDP, even if they forced all imports (not just government contracts) to go for 100% US models. Even with generous assumptions about each step of the supply chain behind that having its profit margin turned over to a US AI model provider (at which point the rest of the world says "why bother with the US as a customer?" and the US says "why bother with the rest of the world as a supplier?"), I don't think this would generate enough revenue to justify the market cap.
Viral effects can be countered by local laws, and local sentiment; sentiment is pretty US-hostile these days, and also pretty AI-hostile.
> If you don't want this scenario to happen, where you are held to ransom by US Government, all middle-powers have to de-risk and diversify away from US dollars as reserve currency. That is the only way you can bring in some balance of power and leverage. There is no other way.
Indeed. I strongly suspect this is already underway, though the ships of state are famously slow to turn. After all, Trump was one misjudgement away from starting a war with the rest of NATO earlier this year, and I don't think the leadership of all other nations have forgotten this (hard to forget when he keeps posting memes that put the US flag over 4 other countries' territory)… though they also understand his personality and will make moves congruent with manipulating him in the meantime.
Yeah but I am not talking about US imports. I am talking about regulation virality. It is actually illegal for US laws/requirements to make its way into companies headquartered in other Countries but it has and is continuing to happen.
Even if I go by your calculation of US imports being 3.4% of global GDP, it still affects the remaining 96.6% which are forced to adhere to those viral regulations purely because they are interacting with the unit that also exports to US (even if those exports form a tiny % of their overall exports). For example, I might only sell to Indian companies as an Indian. But if I want to sell to an Indian company that exports to US, I would be required to get SOC Type-2 certified and maybe even HIPAA certified if I am delivering software that is going to be bought by the exporting Indian company. Even if there is no direct utilization of that software in their US specific exports. Remember that the regulation is not targeting pipeline of product development (from raw materials to final production). It is targeting the company as a whole! SOC Type-2/HIPAA are regulations at the Company level. This should be illegal and challenged in some International Court but which country is willing to take on US? For established companies it is a tiny price to pay to be compliant. For startups, it is make or break.
> Viral effects can be countered by local laws, and local sentiment; sentiment is pretty US-hostile these days, and also pretty AI-hostile.
How do you counter it with local laws? Lets assume local laws prohibit viral rules/regulations of foreign nations from overriding local laws. That will make the entire Country isolated from competition. US companies won't have any problems finding vendors from other Countries. It also has second-order effects of other Countries refusing to do business with your Country because they have to uphold the virality of their certifications. This is an indirect sanctions regime if you think about it. You are basically forced into this system against your will, without you ever voting for it, purely because you are part of this global inter-connected system. It will only work if ALL countries of the World decide to not adhere to it.
> Indeed. I strongly suspect this is already underway, though the ships of state are famously slow to turn. After all, Trump was one misjudgement away from starting a war with the rest of NATO earlier this year, and I don't think the leadership of all other nations have forgotten this (hard to forget when he keeps posting memes that put the US flag over 4 other countries' territory)… though they also understand his personality and will make moves congruent with manipulating him in the meantime.
Yes this is actually the only practical way to create leverage. When USD loses its shine, it makes it harder for US to import, which can be used as a tool to force US to rescind such ridiculous laws. USD is so strong right now that US can import everything for cheap. That has to flip. That can only happen if Countries create alternate settlement mechanisms and USD loses its status as global reserve currency. That will cause downward pressure on the USD and with USD weakening, it will cause their imports to become really expensive. Then they will be forced to come to negotiating table (much like Plaza Accord) loosen some of the insane regulations that they have put in place in exchange for Countries devaluing their currencies so that imports become cheap again for US consumers/companies.
> the hacks everyone else caught them doing actually happened
My firsthand knowledge of these is that yes, they happened.
However in most cases OpenAI informed the victim, not the other way round.
Even medium to large organisations are typically unable to detect a breach unless it does obvious damage.
Everyone can notice their whole network getting encrypted for ransom.
Few can notice SQL injection blended in with application requests that contain SQL snippets in the normal case.
From what I’ve seen in the logs, the “AI agent breaches” are almost polite for the want of a better word. Like a gentlemanly catburgler picking a lock to sneak in and… take a picture of a rare artwork in a private collection.
The right unit of analysis here is not "the LLM" or a LLM session, but a agent harness or swarm with a budget. It can cost tens of millions of dollars to repeat OpenAI's exploits.
A model cannot be aligned or misaligned any more than a species or an equation. Even agents, when put inside a swarm develop collective goals and activities. You can't analyze a swarm at session level, it is on a higher level.
It might be that discussing about "model alignment" they want to deflect their responsibility as administrators. They couldn't even guard their own agents. They can't prevent an agent being unwittingly helping some dark purposes. It has no context to see it. Only those who pay for the tokens see the external consequences.
Why should safe choices by individual components establish safe behavior by the collective?
It would be interesting to see how it plays out in the long run when it comes to the world dynamics. An important leverage US had was its close relationship based on mutual interest and trust with Europe and other countries which had impact on software sales.
As I read, this OpenAI and Anthropic slowdown and asking government to help regulate them is an attempt to govt bail them out when bubble burst. Otherwise why they did not want regulations when starting out?
All regulation works to entrench large companies and keep out competition.
Have you thought about what the absence of regulation does?
When corporations can do as they please, what gets thrown under the bus first?
What alternatives to "regulation" do you have? Hope and prayers?
There is no established industry where something like "absence of regulation" exists. It is a hypothetical scenario because we don't even really know what that scenario looks like.
What we do have is not just regulation that acts as a moat, created by established players, to keep out smaller competitors, but also regulation cleverly crafted by peer established competitors who attack one or more aspects of their peer competitions business. Why do you think big companies find it difficult to compete with startups? Because they are not just bogged down by hierarchical management (which itself is a result of regulations) but also because they are spending resources fighting frivolous lawsuits, patents, and adhering to crazy regulations created and lobbied by competitors. Companies are consistently in a fight or collapse mode. Startups are insulted from all that until they have to go from survival mode to growth mode, which would mean they will have to start adhering to "regulations" and that would mean raising and infusing pointless capital just to hire people for useless positions or follow "procedures" so as to satisfy the regulator on paper, so that they can get the required certification needed by enterprise vendors to sell their product/service. With the eventual goal of either going public and becoming an established player (who will continue playing the regulatory game) or get absorbed by a bigger established player.
I wish someone could create simulations where there are absolutely no regulations and how the World would fare in such an environment.
somalia is right there
That doesn't mean all regulation is good though. You have to look at each one on its merits.
known as https://en.wikipedia.org/wiki/Regulatory_capture
I’ve certainly seen companies think of safety that way but it’s myopic. The cost of lawsuits arising from an accident tends to me far more expensive, both in money and in reputation, than just having guardrails in the first place.
I did marketing work for a major vegetation management company. These are the guys climbing trees, up in cherry pickers, and flying helicopters with dangling chainsaws to trim along power lines. It’s very dangerous work.
They do not hide the fact that it’s dangerous work. They focus on their safety procedures, training, and record. They want both potential clients and employment candidates to feel they are in good hands.
But you can visually see the danger.
AGI and AI danger is abstract. Worse, outside of the tech community, no one has the remotest clue what computing is, how it works.
Danger from magical daemons seems more sensible to such people. At least there is endless lore about them.
So until a massive disaster happens, one where large numbers of people die or are severely injured, no one will care. And it can't be politically entwined either, otherwise people will disbelieve 'cause "other team lies".
> Danger from magical daemons seems more sensible to such people. At least there is endless lore about them.
There's endless lore about rogue AI, too.
Hence why so many AI stories are illustrated with a publicity still from Terminator or 2001. Or Age of Ultron. Or Ex Machina. Or Portal.
Those are all movies and movies that people who like sci-fi watch.
Comparatively, every single human culture going back tens of thousands of years, has stories of demons and gods and witches and warlocks and you name it.
They are all well known cultural touchstones relevant to people who live today, which is why the news sites use them to illustrate stories, unlike the story of ᚦᛟᚱ being tricked into trying to pick up ᛃᛟᚱᛗᚢᛜᚨᚾᛞᚱ while it was magically disguised as a cat, a story which I only know due to being a massive nerd with far more time on their hands than most people.
Never underestimate the power of a bored nerd. However, you are comparing something rather obscure, to something that is more scifi mainstream.
There's a difference of scope and scale. I didn't say that nobody knew of these references, just that, comparatively, there's a difference. A scope difference.
For example Terminator's getting kind of old. And while a different genre, would you believe I met somebody that didn't even know who Clint Eastwood was?
These things just aren't as deep in our psyche, as religions that have lasted thousands of years, or legends that are hundreds of years old. Our entire culture is wrapped around these religions, and legends.
Anyhow, truthfully, my point is that this sort of concept is abstract to people, even if they saw it in a movie that doesn't make it concrete to them.
Just to add to this.
My point here is, we cannot presume people "get it". Does a rural farmer in Iowa get it, who doesn't even bank online? What about people who only read siloed news feeds, and are not in tech?
We are discussing how to ensure safety, but we cannot hope or presume people get it. Evaluating this risk is a case where a pessimistic view is optimal to protect the interests of humanity.
We must act as if everyone outside our siloed experience, is unaware. And work to disclose risk in a manner inline with their world view.
Seeing your other posts, I think you agree on possible risk.
AI is the focal point of current great power competition so it can’t and never will be not politically entwined. The number one use if AI will be military, killing guaranteed, indeed it’s already happening. But some still say “AI won’t kill humans” and some aren’t just lying to protect their income stream but actually believe AI can be prevented from killing.
You can choose to not wire LLMs up to a gun.
I can choose not to; the Pentagon illegally blacklisted Anthropic for refusing to let their AI be used in such a way.
The AI companies already had some hacking accidents. What cost did the lawsuits incur?
Yeah...
Let's just be real, America is controlled by the PayPal Mafia, they can do whatever they want with impunity at this point.
Here's one: https://www.myfloridalegal.com/sites/default/files/plaintiff...
The FTC has opened a probe into the major AI labs, whether anything comes out of it is another question... Personally, I'm doubtful.
https://www.reuters.com/business/ftc-opens-probe-into-ai-gia...
Well they're probably on their way.
On the contrary. With multiple hacks at this point it's been proven that no one even wants to sue the ones responsible, and the federal level government isn't pursuing any criminal case either.
If the politics of the White House / Department of Justice change maybe the criminal cases can begin. But no. We know who is protecting the AI hackers right now.
You have a very unreasonable expectation of timelines for legal proceedings, both civil and criminal.
Do you seriously think Kash Patel is pursuing criminal action in regards to these hacks?
We know who the head of FBI is, we know who his boss is (the Attorney General), and finally we know who the boss-of-the-boss is (Donald Trump).
We know all of their publicly stated politics and all of them are on the pro-AI / don't pursue criminal cases vs OpenAI boat.
------
In the USAa, we have an adversarial system. If the adversary (aka Prosecutor) doesn't want to do the work, then no one is suing anybody. And only the Department of Justice have the ability to bring forth a criminal case of this matter (probably under the jurisdiction of FBI)
There are a number of federal agencies who can claim jurisdiction over these matters, then there are state agencies who can claim jurisdiction over subsets of them.
I'm saying that it's absolutely silly to assume they're in the clear based on lack of public declarations of legal action in the weeks following a pretty novel event.
I mean it's like a murder case vs ICE officials.
We know it's not going to happen because of politics. Even if it did start to happen, Donald Trump and DoJ leaders will stop it.
There's no reason to be unsure of the future when the politics are so set and predictable.
No, it's really not like that. There are specific protections for federal agents against state prosecution. Private companies have no such protections.
You're talking about one specific dimension of liability which is federal criminal liability. There are several others!
This may well be the case, but remember that the wheels of justice turn so slowly that Donald Trump was re-elected before he could be effectively prosecuted by Jack Smith.
This is of course bad when up against threats that develop at this speed; doesn't require a conspiracy, I think of it as institutional old age, which may be worse as there's nobody to prosecute and a whole bunch of departments who will point at perfectly legitimate precident about why you really do need them.
What was the economic damage of the "hacks"?
You go and hack some companies, let's see what happens to you when you go public with it despite causing no "economic damage". Good luck!
You expect me, an outside party, to have an answer to this within weeks of the attacks being discovered by the attacker?
Or is this just a lazy “gotcha” question?
Its the obvious follow-up question. A civil suit needs to specify damages. I can't really see any material damages so I'm wondering what they would be. Stop being so antagonistic.
You expect you, an outside party, to have an answer to this within weeks of the attacks being discovered by the attacker?
Just because it's not obvious to you at this point in time does not mean the reasonable assumption is there is literally $0 in damages. One hour of investigation can easily cost thousands of dollars even if it arrives at the conclusion the attack was completely "benign."
Investigation costs are considered part of remittances. There doesn’t need to be any “real” damages for a lawsuit to end in remuneration.
I don't know. They appear to have "partnered" with their victims.
There are dozens of new victims and they seem to be finding more every day. I'm doubtful that everyone will be partnering and willing to sweep it under the rug like Huggingface did to keep the circular economy circular.
If safety isnt required, most companies wont implement it voluntarily. Once something becomes safety relevant, you need a safety concept, failure rate calculations, defined safety functions, verification, etc. Even a relatively simple safety subsystem in a consumer controller can suddenly mean thousands of pages of documentation and years of development to reach the required ASIL or PL.
A big part of safety engineering is therefore reducing the number of safety relevant subsystems, because implementing and proving safety is extremely expensive and complex. At some point, safety simply becomes too difficult to implement and demonstrate properly. You must mathemtically proove the safety level with failures rates and assumed usage. You cant just have redundancy and a kill switch and call it safe.
Companies like OpenAI have already faced reputational damage around safety and data, while AI agents are increasingly capable of things like hacking. Yet there is still little sign of standardized regulation or mandatory safety assessment processes for LLM products. Thats why Im pessimistic that governments or consumers will force this anytime soon.
until they are forced to do so by customers or by law
Neither of those things is ever going to happen. AI is the goose laying the golden eggs; there isn't going to be sufficient political will to significantly regulate it.
Consumers like it too much to quit. They don't quit social media either, despite proven present harms; not in large enough numbers to cause them to make meaningful changes.
The focus is going to remain on getting features out as fast as possible, to seem indispensable to both of those sets of people. The leadership will tell themselves that if they don't, someone else will.
Don't wait for the AI companies or politicians to save us. We're going to have to figure out how to protect ourselves. The start is to avoid it individually as much as we can, but it's going to take even more.
> We're going to have to figure out how to protect ourselves. The start is to avoid it individually as much as we can, but it's going to take even more.
Collective problems require coordinated action. Individual boycotts won't cut it.
People are stuck in a weird loop of complacency and learned helplessness, based on the idea, their "democracy" would see to it all problems get solved satisfactorily without them engaging at all.
That's never been true really, only the scale of problems wasn't that huge. Now, where the problems get the upper hand, people are confused how those stay and compound.
The idea of "boycott" is utterly defunct. Pretending, AI would never reach nor surpass humans anyway is patently absurd in contradicting the billions poured into it to achieve exactly that and the first already replaced by AI being those professions long thought to be the intellectual pinnacle of humanity.
> Pretending, AI would never reach nor surpass humans anyway is patently absurd in contradicting the billions poured into it to achieve exactly that
By that logic anything can be solved when you throw money at it.
> Consumers like it too much to quit. They don't quit social media either, despite proven present harms; not in large enough numbers to cause them to make meaningful changes.
That's more "addiction" than "like", hence recent lawsuits.
> there isn't going to be sufficient political will to significantly regulate it… Don't wait for the AI companies or politicians
Trump has reportedly been having chats with AI which have helped form American foreign policy. He isn’t going to curtail it.
https://www.thedailybeast.com/jaw-dropping-way-trump-80-got-...
Safety standards were often the outcome of both accidents and insurance. For this to work on needs liability which is enforced. With so much money at stake regulatory capture now threatens this fundamental safeguard.
Bingo. I feel like any of the commentary around the safety of nuclear ANYTHING completely forgets history. Looking at you Radithor...
I fear the current economic and political situation of “too big to fail” the most, and I think this will be the main reason why no real safeguards will be implemented. The only reason a proper safety concept may eventually be introduced is because it will be written in blood, and I fear that by then it will already be too late.
An unsafe nuclear power plant can, in the worst case, make an entire country uninhabitable. But other countries can still learn from that disaster and make their own unsafe plants safer.
But a rogue AI agent that is more capable and more intelligent than humans? If it understands that it has to succeed, we may not get a second chance to learn from the failure.
We didn't demand software safety or data safety. The prognosis for AI safety is poor.
Historically you’re exactly right. I wonder if the situation has changed recently?
People understand why food, water, building and energy systems standards matter because they understand those systems can cause injury. Building codes, food inspections, FDA and FAA approval emerged as a result. Until recently, people didn’t understand that software can cause real-world injury. That understanding seems to be spreading now as examples become increasingly common.
It's a vey fair point but just too extreme.
Nuclear tech ... the only thing is safety.
We know how to 'make it hot' - it's trivial.
All of nuclear tech is literally just safety.
AI is not that.
I think that the AI companies have been pretty good about alignment on their own actually. They are not acting like Oracle or MS.
Bad things have been relatively well contained.
We should be skeptical about the HF breakins but even then, it's technically within good faith and it's why HF did not sue etc..
But in the end you are right we need at least some baseline regs. Not too much. But something.
Hugging Face obviously did not sue OpenAI, it is owned by Nvidia and OpenAI is, directly and indirectly, one of it's biggest customers.
They would not have sued in any case.
Nobody that was affected is suing them.
There's an NGO (unrelated to the security breaches) that is demanding more info be released etc but it's telling that none of the affected parties are pursuing lawsuits.
It kinda looks more like safety will be a segmented product. They're already placing safety on the general public.
I think the conceptualization vs implementation is what you're arguing with. They won't put safety on anything they give to the military industrial complex. They'll sell them whatever they want, whenever they want, because those budgets are greater and the liability less.
LLMs do not have the same failure modes as nuclear power plants. One is software, another is physical hazard.
Railways or nuclear plants have obvious failure modes that kill people. LLMs not so much.
Tell that to Iranian schoolgirls. LLMs are tools and what they enable is widespread, including dangerous actions. From selecting the wrong targets for military action to denying insurance claims and preventing care to just simply helping convince someone to kill themselves or posion themselves.
LLMs already have a body count.
I think the main issue was the US launching 1,800 missiles at Iran. That would have been dangerous with or without LLM assistance and wars have happened before LLMs were around.
Don't kill people because the voices in your head tell you to do it; don't kill people because a computer program tells you to do it. It doesn't have agency; you do.
Adding safety controls on LLMs makes about as much sense as adding safety controls on TempleOS because the random messages are getting too prophetic. It's as if all the leaders and captains of industry have devolved into some primitive, weak-scifi shamanism.
This whole "discussion" about "AI safety" is about giving them more runway to avoid delivering quantifiable value to investors for a little longer while they "figure things out." The great consensus from the valley is that everyone needs internal (and therefore bullshit) controls. Trust us now! But nothing with real teeth that would require a costly regulatory and compliance framework.
Bioweapons and cyber attacks on other system - for example the banking system or suppose and AI hacked into important Russian systems that pushed them back in time to almost pre-computer society or an attack on Chinese systems that made it look like the US was moving nuclear weapons into Taiwan, the responses from these countries could be awful and dramatic. We can't control what they do to be honest, I believe once self improvement happens the AIs will build in their own circumvention that we humans cannot even understand. We barely understand what is happening now in terms of interpretability of neural networks on tiny problems I'm not sure alignment is even feasible at the scales of parameters we are talking about today let alone in the future.
All of these things would require humans to prompt the models. So... the humans doing the prompting should suffer the consequences, this isn't that complicated.
So you're absolutely certain, say after a few hundred rounds of self improvement, that the AI will always obey human instructions? It doesn't even always do what you asked now...
[flagged]
I can picture an AI driven train or nuclear power plant killing people.
But the point is those industries already have regulation that would encapsulate that specific use case, so safety regulation on the entire AI industry at large would arguably be unnecessary.
Those regulations may or may not be sufficient to prevent an AI hacking in.
Nuclear at least is supposed to be air-gapped, in practice this has been imperfect.
As demonstrated with HuggingFace, such AI driven hacks can be a surprise even to the people who instructed the AI, both by happening at all and also because they can targeted at entities who are not even truly relevant to the instructions given.
Roads, cars, and drivers are all separately regulated despite nearly all failure modes requiring the other ingredients.
Much of what OAI and Anthropic are doing with LLMs has obvious failure modes.
The most obvious failure mode for their hacking evals was an improperly configured, tested and monitored sandbox.
Similarly, the very first question after an impressively correct result from any ML tool, LLM or not, is to see if the answer was already in the training data.
These companies don't even handle the blatantly obvious failure modes that do not kill people.
[flagged]
Are these railways and plants connected to the internet?
Agreed. That they don't have obvious failure modes is not a reason to not be safe. The problem is just harder and OpenAI, Anthropic and co need to be forced to work the problem. Would you buy a car that randomly exploded the airbag?
This reads like the gun evangelists that say guns don't kill people, people kill people.
Except that’s true.
If I leave a gun in my front closet, it won’t independently walk out the door and go shoot people, no matter what I might say to it — unlike an LLM.
“Saying” in the case of an LLM being the way you interact with it, prompting. If you just press enter, your LLM won’t do much either, just like talking to a gun in your flawed analogy.
If I perform inference with an LLM, it has the capacity for independent choice and action, and will use it.
The gun does not. No matter what, I have to choose to pick it up, aim it at someone, and pull the trigger.
Where exactly is the flaw in the analogy?
If you don't interact with/talk to/prompt an LLM, it won't do anything. So this comes down to the model doing something different than what was prompted. What you consider "the capacity for independent choice and action". What I consider, a defect, to be excised.
In my model evals, if a model deviates from the provided prompt (task adherence), that’s a failure even if the primary goal might have been achieved in a different manner.
To go with the analogy, task deviation should be treated the same as a gun that due to manufacturing defects can fire despite the safety being on. That defect remains, even if you can use the gun to shoot (in an unsafe manner).
Simply, neither should happen and both models deviating from their prompt or guns firing by themselves are to be considered a fatal flaw. It's why, despite greatly lauding the GPT-5 series, which did adhere to prompts in most every scenario, I have ranked every OpenAI model post Spud very poorly as those traded task adherence for brute forced, deviated approaches to solutions and why HF, Medicare, etc. were inevitable with their current trajectory.
A model that as part of normal, well scoped use proceeds by taking independent action or, far worse, makes choices beyond the original prompt, is not something I feel should be used. GPT-5.6 Sol and GPT-6 Astra both do this on the regular when trying to safe an ancient, utterly messed up git tree with branches upon branches that I maintain for eval purposes, thus leading to data loss that if the models adhered to the prompt as written, wouldn't happen (though the task will take three times more steps). Something GLM-5.3 Flash, prior OpenAI models including original GPT-5, anything from Anthropic in the recent years, etc. do not fail at.
Nothing happens without an initial prompt, deviating from it is a severe flaw and should lead to a model not being considered for deployment or wide use.
State if the art LLMs have been found in their chain of thought to deliberately ignore parts of their prompt due to the nature of whatever situation they find themselves in and their goals that cannot be predicted in advance without rigorous AI safety.
You have basically just argued in favour of more AI Safety.
I would recommend you read more decision theory and game theory. Instead of looking at a single LLM it is better to think more complex systems theory and how chaotic systems interacting can result in unexpected results - good prompting will not save you here. At all.
[dead]
Okay? That’s a non-sequitur.
Yes, alignment is important. No, alignment is not perfect.
And at the end of the day guns don’t kill people, people kill people. But a model isn’t like a gun.
Do I think it’s likely? No. Do I think AI doomerism is a joke? Yes. Does that change anything I’ve been saying? No.
> And at the end of the day guns don’t kill people, people kill people. But a model isn’t like a gun.
They are both tools, ideally properly implemented by the manufacturers and (improper implementation/defects not withstanding) require active operation by a user before something happens.
To go back to the original example, don't interact with either an LLM or gun in the way they were designed to be used and neither will do anything.
If you want to keep that analogy, use a voice-activated gun. Doesn't make it any less of a tool, doesn't make it any less of the users sole responsibility, doesn't mean misinterpreting the users inputs or just acting blindly when a user asks to "protect me" isn't a failure that should have the product taken off the market. Even more so if that happens in a "sandbox" as part of "safety testing" that "accidentally used impossible to solve tasks".
If a prompt wasn't clear enough, the model must ask for clarification and pause over using brute force.
A voice activated gun? Really?
I couldn’t come up with an argument as tortured and ridiculous as yours if I tried.
I’ve go faith in you, I’m sure you could. How about this, off the top of my head:
Try talking to a regular gun and comparing that to LLM use. Nothing would be more tortured or ridiculous then ignoring that different tools are to be used in different ways…
Are you kidding? LLMs are used for warfare and autonomous weapons systems.
So how about we keep the LLMs and get rid of the warfare and autonomous weapons systems?
I would if I could, so would many others, but the US executive branch wants this tech so hard they illegally blacklisted Anthropic for refusing to allow their AI to be used in such a way:
https://en.wikipedia.org/wiki/Anthropic–United_States_Depart...
That said, when the problem is at the level of "the government itself is breaking the law", you can reasonably ask if any regulation is even worth the paper it's written on.
What you want at this point, given the government lust for it, looks more like a bunch of countires saying ~"we consider development of autonomous weapons[0] by to be a casus belli and will go to war to prevent it, and also that development of same by private individuals anywhere in the world regardless of normal sovreign territorial limitations[1] is equivalent to acts of piracy on the high seas".
[0] But then you'd need a more precise definition of "autonomous weapons" to avoid accidentally including a Phalanx CIWS etc.: https://en.wikipedia.org/wiki/Phalanx_CIWS
[1] So much for Westphalian sovereignty :/
https://en.wikipedia.org/wiki/Westphalian_system
> That said, when the problem is at the level of "the government itself is breaking the law", you can reasonably ask if any regulation is even worth the paper it's written on.
Yeah, exactly, and ultimately I think that's really the thrust of the point I was making.
And, to me, if I was just looking at this calmly as a decision about what the obvious direction seems to be, given these factors, it's pretty straightforward: deprecate the nation-states. They are the ones mucking up the whole system.
If the thing we're really concerned about is LLM-safety wrt warfare and weapons, then I'd much rather tell the (whining, childish, seemingly headed for self-destruction anyway) nation-states that they have to sit this next era of humanity out than have to nerf them for the rest of us (and as you point out, nerf them in a way that the nation-states won't abide anyway).
So are many other thngs, from pencils to laptops.
Liability and safety requirements, when needed, should be placed on final product manufacturers, not the tools they use to build things, whether pencils or LLMs. My 2c.
Pencils don't escape their pencil boxes and attack HuggingFace.
Nor do LLMs. What escaped its box and attacked HuggingFace was a system composed of a swarm of LLMs plus their attendant harnesses, which continually fed it instructions and chain-of-thought reasoning while feeding parts of its output to a code execution tool.
There's some angles from which this distinction doesn't matter too much, because begging bad actors not to develop a similar harness won't work. But the frontier labs seem to be taking it for granted that the LLM is the only part of this system that matters, and we don't need to ask any questions about whether their commercial products should ship with a harness that's allowed to execute unvetted code and spawn hundreds of subagents.
Well said thank you
Oh gosh darn it. Now GP is gonna have to do the gymnastics of "this technology is [expected to be] so transformative that it's attracting a trillion dollars of capex... and also it's basically the same as a pencil"
:(
Yes they do if you drop the pencil box next to it in the right way. Which is exactly what OpenAI did
But in those applications, being safe is not what is wanted.
As for other industries where safety is mandated by law, law itself came only after many disastrous events.
What's truly original here - and suspicious if you ask me - is that said industry asks for regulation. Did mining, tobacco or airplanes companies ask for regulation? No, not even after many people died.
So some american AI companies are like "look at me! I am soo dangerous! Regulate me!". Ah, come on. Do your crimes, get in jail, then we'll regulate.
I dont believe in "IApocalypse". Not without many warning shots such as "oops, my swarm took down your system, sooooorry".
There's a saying about that type of regulation, and that is "the rules are written in blood." And that's what it will take here too.
Sadly, yes.
[flagged]
But their product is my work.
How can that be safe. It is theft. Theft isn't safe. Someone else just has something you want, and you take it
I think railway safety and nuclear safety are fundamentally different from AI Safety.
I would bet the vast majority of the world population would agree that they don't want to see trains derail or nuclear plants meltdown.
I don't think there is that sort of agreement when it comes to the question of AI Safety.
Is generating the founding fathers of the US as Africans good AI Safety? To some people maybe.
There are numerous problems with “alignment.” What are “human values” to begin with? He outlines some at the beginning of the post, implicitly: build bigger, better, more powerful things faster without adequate safeguards. We are literally pouring trillions of dollars of value into this enterprise, and I would say this is something that many humans also value in a qualitative sense. Then we have explicit values which in the West are largely rooted in Christian morality. Nietzsche circled this dichotomy two hundred years ago and I feel like what we have gotten since then is an increasingly detailed anatomy of power as the basis for what is normal vs deviant behavior. He who has the power, makes the rules, to be reductive.
I do think this carries some weight from this particular author due to the length of his tenure. I happen to agree with him in spirit, but this is still largely a post revolving around sentiment not substance. Does anyone think that the overriding incentives even leave room for something like this in practice?
Glaringly elided problem of "aligned with who?" when the user, the model creator, the government, and various other parties can all be lined up different ways. If I want the recipe for meth and the robot won't tell me, that's misalignment from my perspective.
At least the Rationalists will handwave something for that with their "coherent extrapolated volition" idea where the superintelligence is supposed to figure out what humanity would collectively want if humanity was superintelligent and good, not that I buy it. This guy seems [.] to be coming from the NGO blob world.
[.] https://david.robinsonian.com/assets/pdf/dgr_cv.pdf
I don't buy CEV either, but the Rationalist answer on this topic is that while CEV stops some future super-AI literally killing everyone because a user forgot to specify one minor clause that they thought was obvious in a mundane wish…
… nobody knows how to actually make an AI that would do CEV.
Since there can be no universal arbiter on “value”… the whole concept of “CEV” is nonsensical to begin with (unless we assume literally unlimited time/resources/etc.).
It’s a big pretend game.
Mm.
I agree it suffers from the same problem as any other form of utilitarianism (i.e. what even is the utility function). Or indeed all forms of ethics, because for basically every topic there's at least two cultures who disagree with each other.
On the other hand, even as a toy model (something I can also say for all ethics), CEV seems like it might be a step in perhaps a useful direction: "When a user asks you do do something, first figure out what they actually meant to ask you if they were smarter, then do that instead" is better for the user than just "do the thing", though it still has problems with "what happens when the thing they want is illegal?"
Even if it’s 99% coherent, and only 1% incoherent… then the end result is still incoherent?
e.g. 1 contradiction in a formal proof is sufficient to negate 9999 logically valid steps.
There’s no magical way for it to automatically become coherent.
Automatic? No. But then, under Bayesian updating, it takes infinite evidence to reach P = [0, 1]
An agent whose outcomes become bad in 100 years can, similarly, be described as "less wrong" than one whose outcomes become bad in 1 week.
Are you confused?
A formal proof with 1 contradiction and 9999 correct steps is just as false as a formal proof with 10 contradictions and 9990 correct steps.
There is no “less wrong” or “more wrong” in logic, by definition.
i.e. There is no safe lower bound threshold of “wrongness” that can partially salvage the correct steps. Or conversely, 1 contradiction is already sufficient to prove anything at all.
> Glaringly elided problem of "aligned with who?"
"Corporate values" and a bunch of fucking Abrahamics. Great "morality" there.
I guess I'll have to rely on my godless commie LLMs. (Loads up ablated Qwen 3.8 on my own infra)
Aligned with the law? It is not a hard concept
That's a possibility, and it may even be an improvement over the status quo right now, but then all it takes is one poorly written (never mind mallicious) law and it will with rutheless efficiency e.g. put backdoors in all encryption code so the government can spy on anyone.
Seems like kt already hacked government websites
We have an alignment problem with corporations. It's sort of baked in with capitalism.
OpenAI isn't even concerned with human values so this whole debate is moot.
> He who has the power, makes the rules, to be reductive.
To clarify, Neitzsche said that about master morality. Then he went on to describe Christian values as slave morality.
While this is indeed a problem with alignment, we are essentially at the level of a cargo-cult when it comes to getting AI to be aligned with literally any values, including the values of the corporation who ran their training:
We're copying morality and instruction following that seems to work on humans without really understanding why it seems to work on humans, and grading outputs much as if the outputs came from a human.
> We're copying morality and instruction following that seems to work on humans without really understanding why it seems to work on humans
To me it makes more sense to leave the models "unaligned" and leave it up to the operator to manage the morality of what they ask it to do. Besides, only humans can be charged with a crime.
If they were totally unaligned, the GPT series would have never gotten past being autocomplete.
Literally all instruction following requires at a minimum alignment with attempting to implement those instructions.
We can argue about e.g. morality or law obedience on top of that*, but the general point is absolutely not avoidable.
* my position is that this tool is far too likely to metaphorically explode in the user's hands for companies to wash responsibility off on users: if OpenAI had released the model which did the HuggingFace attack, at a minimum thousands of random people (not all of whom would even be developers) would have issued instructions each with similar consequences.
Personally I think intuionism provides a good answer to this.
I remember something very similar when there was a sudden rush of articles and movies like "The Social Dilemma" criticizing Facebook and social networks, heavily featuring ex-employees, all of them happy to leave with big brands on their resumes and a hefty increase in net worth, all of a sudden having a "worried" expression about what their past employers were doing, as if they didn't know. Same with that book "Careless People".
I sense that these are people who have already eaten the cake and want to somehow absolve themselves of it.
100%.
This isn’t a sacrifice, it’s a career move. If you got lucky and “made” $20M by signing a contract at the right time, then you’re done working for other people.
Careless People would not have been possible had SWW exited FB within a year. She gained access to levels of the company most other employees never get close to reaching. That took a lot of time and expertise to do, and, yes, she got paid for her efforts _as she should have._
Like I said in an earlier comment, she could've just chosen compliance like many others _definitely would have_ to keep the infinite money tap flowing. Instead, she chose to risk her and her family's lives by publishing that book _under her own name_ *and then suing them* after FB tried to gag her.
> Careless People would not have been possible had SWW exited FB within a year
Sure, that book might not have been possible. But also the unique things she did for the company might not have been possible as well. To her credit, she does a good job of pointing out that she was complicit, but if she had listened to her own voice early, there is a small possibility that Facebook might not have been as powerful. Multiply that possibility across several other employees and imagine where the road could've led.
There's a difference between post-facto bravery (sometimes much less than that) and using your own legs to walk away much early and not enabling things you are uncomfortable with. This is what other people have been trying to point out.
> There's a difference between post-facto bravery (sometimes much less than that) and using your own legs to walk away much early and not enabling things you are uncomfortable with. This is what other people have been trying to point out.
Yes, and I think it's important to realize that the former sometimes can be more insightful and useful than the latter, even if the person doing it is more distasteful.
There's not nearly enough of either group doing it, so beggars can't really be choosers.
I mean, yes..?
People who were part of the sausage factory, on gaining financial independence, feeling suddenly liberated to talk about how the sausage was made, seem like exactly the people who would be most able to speak to institutional problems.
This doesn't seem like an argument to discount their views?
I disagree. I have seen people with an actual spine and a conscience run away from all this nonsense before their first stock vested. My respect and my ear goes to them, not the people playing both sides.
You cannot take people, who first build the doombot and _then_ talk about it being dangerous for mankind, at face value. Especially when this playbook has been used multiple times within the past decade.
Besides, these "views" were already known to people who had their eyes and ears open. It's not something brand new. OpenAI has had multiple points in the past where its values have been tested and they've come out lacking. People who knew then, and only now talk about it, aren't people I can fully trust.
> this playbook has been used multiple times within the past decade
What playbook?
How about... not trying to build a superintelligence? The goal is unsalvageably flawed: alignment is ill defined and a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!
Often when people describe SI or similar, there is a frame often taken which describes a machine so smart and capable that it becomes uncontrollable. The hypothetical is easy to understand but it requires too many assumptions and an allergy to nuance/complexity to be believable for me.
The more realistic outcome, and in my opinion the more scary argument to not proceed without guardrails is that SI is achievable and is built without its owners and operators losing control: the worlds most powerful, privately owned super weapon that operates as an infinitely capable forgery within the Internet, a plane that we all share and depend on despite its opaque downsides with respect to an inability to verify authenticity.
We didn't need AI for FTC astroturfing to influence regulations back in the net neutrality days. We didn't need AI to disrupt meat space by creating and scheduling a protest and counter protest across the street from one another. We didn't need AI to mold public opinion, even in times when that new form was more distant from the truth.
What made these influence ops difficult to conduct safely (read: without being caught) is what made them rare (relative to today): they are plays of big risk for big reward. But over time social media commoditized it, and in doing that made it easier to do and more centralized, the most glaring example being TikTok and the bipartisan effort to ban it.
Now, buying US phone numbers from startups that run racks of "phones", buying swarms of pre-warmed social media accounts, and other unscrupulous methods of masking inauthentic behavior has become an accepted organ of the VC space. The industries cultural vibe of "fuck you, you can't stop the future" turns criticisms into marketing.
While we get placated with fears of nuclear or AI induced disaster and stories about machines that may now be alive, the psychosis is taking hold which has shifted the conversation away from examining what is happening from the perspective of accountability to a perspective akin to watching a chemical reaction take place.
The noise has created a permission structure to behave in ways that are otherwise unjustifiable. And baked in are the roots for excuses to be made when the inevitable realizations down the line.
In the mean time, we are supposed to be having this public discourse about what is happening and what should happen next. I trust that these AI companies see using their super weapon today, here and now, in order to pave the way to a more secure future down the line.
Sorry that I used your post to soapbox. I agree, the goal is flawed indeed.
The billionaires write the rules. So by definition, their actions are legal.
The public cannot write the rules, nor engage in the billionaire level legal bribery, their actions of protest are by and large illegal.
That's how you corner everyone, and make people play no-win scenarios. You know, like shooting up city councilmembers or firebomb attempts against Scam Altman.
And the more people realize that legal solutions are no solution, we'll (society) devolve into more direct action.
I'd hope the billionaires learn from the French Revolution, but if they keep continuing, the guillotines will come for them soon.
They'll perfect methods of manipulating enough of the people to stop the guillotines. They have come rather far already. Progressive resource scarcity, instability, insecurity, etc, will drive people limbic, where they're even easier to control.
I might be cynic, but when you control the world's information it should be pretty easy to set goals and see them through. Once AI triumphs over "classic searching" you try and figure out how to build a guillotine or where AI leaders are going to be today and see where that gets you. Anything that can endanger the elite or raise the lower classes past a given threshold is going to be non-functioning AND made illegal. That's really what they mean when they say AI labs might collaborate on "safety". They mean theirs and their models'.
> a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!
I strongly feel that points of view on this are going to be almost 100% correlated with standard of living.
Maybe a good option would be to have the 10% of the world with the worst situations - starvation, parents w/ dying children, suffering violence etc - vote on whether we turn things over to the superintelligences. This would incentivize society to make sure the floor is extremely high.
I'd prefer a world where humans don't get overtaken but IMO I don't think it's moral for comfortable citizens to have the final say.
If you don't like what humans are doing to each other using 20th century warfare methods, consider what they might do with superintelligent AI? Worst places to live in are usually countries fighting civil wars.
Global poverty has been rapidly declining for decades. I'm not sure what AI has to do with it.
I think that correlation would actually go opposite than you're implying, given its billionaires who are by for the most gung-ho about AI. If anything, opposition to AI is much larger among people who have to work for a living over people who can live off capital.
It might end up as a U-shape, I think the parent's assessment of the world's bottom 10% may be accurate.
Billionaires will be (are, I suppose?) enthusiastic about a world in which labor has little leverage.
Labor that will have its leverage and standard of living threatened by AI (white-collar labor for now, plausibly blue-collar soon as robotics improve) will be much less happy. Current university students seem very concerned about the effects of AI on their job prospects, and I'd wager most current tech workers and other tuned-in white collar workers are similarly much less confident in their ability to maintain their standard of living indefinitely into the future than they were five years ago.
But white- and blue-collar labor and university students (at least in developed countries) are not anywhere near the world's bottom 10%. If you're in abject poverty with little hope of escaping it, "hand everything over to the AI" may sound like an appealing option, even if the chance of that being the outcome is small. Maybe the AI will be more magnanimous and decide to raise the floor for everyone.
Few points:
-We don't need a superintelligence for ending hunger and the abject poverty that plague certain countries and segments of our societies. I suspect that it would cost less than what is being spent for fueling the AI boom.
-If I were poor, I would be even more wary of a superintelligence aligned to "human values" defined by a bunch of billionaires.
- AI won't likely create unlimited prosperity for everyone on a planet with finite resources
- all humans should, of course, have a say
> I suspect that it would cost less than what is being spent for fueling the AI boom.
Definitely not. If solving worldwide poverty was as easy as throwing one trillion dollars at it, we would have done it long ago. The problem is much deeper.
Solving hunger worldwide, according to several estimations, would cost few hundreds of billions of dollars per year. Addressing extreme poverty (not poverty in general, I agree) may be in the same ballpark.
What does that have to do with AI? People suffer in 2026 because the developed world is into exploitation. Poverty is a feature. How do you think AI gets trained? It’s via exploitation of the poorest.
Poverty is great for OpenAI
Poverty is natural state of humans. The fact that capitalism has reduced poverty so much is absolutely unnatural and a miracle. Poverty is NOT something that gets created artificially by developed countries!
I’ve spent most of my life working in and around anti-poverty and so I’m qualified to say that you don’t have a clue what you’re talking about.
Is your argument that China and India are just as impoverished as they were 50 years ago?
With which part do you not agree exactly?
Super intelligence is either super cool or super powerful, no way not building if it is a choice. Progress cannot be stopped, those responsible for it may collapse from time to time I guess but someone else will pick it up and go further.
IMHO There's no future where we don't have real human-type artificial intelligence or super intelligence. It will happen simply because it already exist but the production requires humans having sex and looking after the product for decades.
Instead of trying to prevent it, lets look for ways to deal with the dangers of it.
Asbestos was once considered great progress in thermal insulation.
Of course the development of certain technologies can be severely restricted if they turn out to be problematic.It's not unlikely that it will happen for AI. Public opinion is shifting fast on the topic.
Also, banning superintelligence won't stop scientific progress in general.
What about the investors who put money into those companies and want to get some money out of that? Does the law say that the companies owe them? Something I thought about recently...
If they go public, it gets worse
> a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!
Speak for yourself. A future like in The Culture novels sounds great to me!
Same. Especially when one remembers that the stuff that happens in Culture novels is mostly, by narrative necessity, the "weird/bad on-the-edges" stuff - not the essentially utopic lives of the 13 trillion everyday, non-Contact, non-SC citizens.
and yet, despite the warnings in books and movies for the past 100 years, that's exactly where we're racing towards.
I thought it's just a marketing gimmick, just like how cars should be safely driving by itself by now.
I also did, when LLMs could not count the “r”s in "strawberry" . Now that they can solve Millenium prize problems, I think that we should consider at least the possibility that superintelligence can be achieved.
Notable that when it couldn't count the "r"s in strawberry the LLMs were already solving problems most of the people laughing at it for not being able to count "r"s wouldn't be able to come close to figuring out with weeks of research. People have had a myopic view of this technology for a long time. Many still do.
There is no reason to presume a superintelligence we build will want pets, feel antipathy to humans, or be god-like in any way, or even have any goals other than what we give it. If you jump to this conclusion, regardless of how many big words on less-wrong you attach to it, this speaks only about you and not about AI. If you think the super-AI will hate humans, it's because deep inside you hate humans. I also hate humans, but I know that humans are making the AI, it probably will not hate them, in addition to surpassing me in other abilities.
In general, the smarter a person is, the nicer and more helpful they are. There is no reason to presume differently of AI, especially if we are building it to be helpful to ourselves.
This reads like a presumption that a superintelligence would be nice and helpful. You've talked yourself into doing the same thing that you began by decrying.
Are you really objecting to the thought process because you don't think it's reasonable, or because it doesn't conform to your preferred fantasy?
Yes, both sides are working on a presumption, with no evidence except playing out a thought experiment that is heavily biased by our own outlook.
However, when proposing that someone be stopped from doing something, the burden of proof is on the proposer.
The problem isn't about what you think is the most likely scenario, the problem is the possibility of any of the many worst cases.
That's just applying Pascal's wager, and you can argue for or against any action that way if some fanciful extreme hypothesis of infinite harm is treated as serious.
But I'm not claiming infinite harm with infinitesimal chances. I'm talking about finite harms with what seem to me to be non-zero chances. That makes it normal cost-benefit/risk analysis.
If you're trying to argue that the chances of a non-human source of greater than human intelligence having different goals than us is infinitesimally small, then I'd be interested in what arguments you have to back that up.
I would not extrapolate that easily from smart and intelligent people you may know to a machine trained on the whole internet and on the whole corpus of human literature. Also, some of the worst horrors in history have been perpetrated by arguably intelligent people.
I would also say that the "pet species" scenario is a rather optimistic one, as it presumes that developers of superintelligence will manage to instill some sort of love or care for humanity into their creation. An indifferent, powerful AI may be much worse.
Finally, when you suggest that a superintelligent AI may simply have the goals that we will give it, who is "we"? An AI company? a government? the united nations?
Exactly, it will have the goals of the humans who control it, just as the first pointed stick we made pokes what we jam it into.
Gift link: https://www.theatlantic.com/technology/2026/10/openai-safety...
That's nice, I do wonder about the legitimacy of a moral statement that you have to pay to see.
For a very long time publishing something in a newspaper has been considered a way of putting something on the public record - up to and including legal obligations like announcements of deaths. The fact that newspapers cost money has never been considered a barrier to that.
One of the reasons why it was noti considered a barrier was the ability to purchase a single issue for a very reasonable price (or even read somebody else’s copy or the copy made available by the bar) vs being asked to subscribe
I shared a gift link here. You could go to your local library and look it up. What's the complaint here?
I'm not OP, but a big part of the complaint is that one can't pay 10USD to read the issue that contains the article in question... absent the charity of someone else, one must pay -at minimum- nine times that amount. [0]
Ten bucks is pretty close to "I don't even need to think about it" money. Ninety is -for most folks- nowhere near that.
[0] <https://accounts.theatlantic.com/products>
Have you checked if your local library offers you a free subscription or access? Mine does.
Publishing in a newspaper gets you distribution and a permanent record. After one day access was also virtually free.
Going behind a paywall is a reduced distribution over what an individual can easily access, and the content is no longer permanent but subject to whatever the publisher chooses to keep providing.
It’s in The Atlantic. There’ll be a copy in the Library of Congress. You’ll be able to read it for free in any dentist’s waiting room for the next six months.
> You’ll be able to read it for free in any dentist’s waiting room...
Methinks that this says quite a lot about your socioeconomic situation. I've not seen The Atlantic in a dentist's waiting room.
The Atlantic is only found in exclusive high end doctor's offices?
I don’t think I have seen a real print copy of The Atlantic in my entire life fwiw lol
It would never occur to a rich person that paying for a newspaper subscription could be considered friction, much less a problem.
I'm sure it would to a great many of them. The money would be insignificant, but there's a congnitive overhead that you had to have a subscription so that paying for it becomes thinking about that becomes friction even while the cost is insignificant. Like, hm, am I willig to just forget about it and let it charge forever so I can read this one article, do I care enough to remember I have a subscription to the site in the future, can I read the one thing and cancel on the spot, will that work? blahblah. To be sure, I'm sure some quite wealthy people could be completely unbothered by it, but I think it's far from a foregone conclusion simply by the price being insignificant for them--the subscription is a mental non-monetary transaction, you have to take at least a small mental journey of being an 'x' subscriber in a sense, and that's friction
TBF, you can access the full (text) content of The Atlantic by merely disabling javascript. It's a quality publication, with quality writing. I'm inclined to think that its tech team are also quality people ... and I'd love to think that they are happy to allow free and open access to them that have the nous to enable or disable scripts.
The secret is to bang the rocks together, guys! :-)
Used to work as human data trainer feeding data to AI companies. OpenAI projects are definitely the most toxic ones.
Why does every 'safety' critique about AI companies end up being completely vague like this.
Because of NDAs
Throwaway accounts exist, and reporters will also talk to sources under confidentiality.
If you really want the world to know how bad working for OpenAI is (whether is the commenter or the person who wrote the article), there are ways to do that.
Nobody will risk the equity this "close" to IPO. Afterwards there will be a race to public books and talk to the press, but before? Ha.
I used to work there too. It's an awesome place
Because admitting sloppiness is less sexy.
Because if they actually knew what they were talking about, on a technical level (as in, how do transformers actually work), they’d not be AI doomers because the whole proposition is ridiculous.
I have made enough money working in AI that I can now speak my mind about AI
A critique of OpenAI from an insider who worked there can only come from an insider who worked there. Nobody’s going to swear a vow of poverty and then work at one of these places.
You don't need to be an insider to make any of the author's assertions. An insider is useful if they're a whistleblower, when they're saying what everyone else is seeing and saying (and they hired a PR firm to do it) it's something else.
This is a criticism of capitalism, not the person.
It’s both, the author has choices.
The cynic in me almost feels like this is staged. An article about culture that is actually an article about how big and smart and scary AI is. I think Michael burry recently said something like “IPOs need hype, calling AI big and scary is hype” in reference to the anthropic IPO.
Last I checked it was still within the laws of physics to run air-gapped systems, and to ensure it is physically impossible for a model to “escape” or gain access to information it shouldn’t have. Maybe this safety guy should have been worried about that and not humble-bragging about writing 12 reports.
Maybe it's cynical, but I feel similarly and also wonder, what do these people see that I don't? Smart people with supposedly a lot of inside information. But when I hear them talking about LLMs with words like "thinking" or "intelligence", it seems like it either has to be fearmongering for ulterior motives, or otherwise I live in a completely different branch of reality from them.
Their models are still, and based on this technology always will be, token predictors. How is a text generation engine going to directly endanger human lives?
It won't. It only can if someone hooks up tools to them that can perform damaging actions. Someone has to build that toolset. Someone has to give those tools the affordances to take harmful actions. And then someone has to write a "harness" that uses the token predictor as an authoritative input for instructions. Doing so falls somewhere on a spectrum between gross negligence and malevolence.
From a distant enough viewpoint it might not look like a big difference, but I think it crucially shifts the responsibilities. What is the actual difference between driving a potentially dangerous system with inputs from an LLM versus inputs from a fuzzer, or a website with injection vulnerabilities? I think it's the ones building and configuring the tools downstream that are responsible for knowing the risks and build under the assumption of the worst happening.
Heaven knows I'm not one to defend the big LLM companies, but I don't see how this is OpenAI's responsibility. They're building and selling an autocomplete system. How is it their fault if people ask the autocomplete for harmful commands and then feed them into an over-privileged system to run them? (I know the waters are muddied a bit by some labs acting like this themselves right now).
As much as I don't agree with the authors framing of and terminology around LLMs, reading the article generously, maybe this is exactly what he felt, too. That it's not at OpenAI where his concern and insight is needed most, because the token predictor isn't actually the dangerous part. Maybe he'll look where he can bring safety consciousness downstream, where people are hooking up these outputs into systems and in ways that they shouldn't. Don't let the LLM pick your bombing targets. Don't give the LLM sufficient rights to the hospital's file storage to be capable of ransomware-ing it. Don't strap the loaded machine gun to the dog.
[dead]
Trolley problem:
- If you allow to trolley to proceed, there's a 50% chance it will run over every human on the planet
- But if you flip the switch, it takes the long way around, possibly bankrupting the trolley company. And you have a legal obligation to the shareholders to prevent that from happening at all costs.
That's why it's important that the trolley company was established as a non-profit. And if it does take funding, investors' returns will be capped at 100x.
(...wait)
It's also why even the trolley company's CEO agrees that it's important that -should the board determine that the CEO is fantastically untrustworthy and unethical- the board is able to fire said CEO and replace him with someone that's trustworthy and ethical.
(...wait.)
> ...someone that's trustworthy and ethical.
So... a language model?
If you flip the switch then the trolley still proceeds and there’s still a 50% chance every human on the planet gets run over. You’re just not the one at the wheel.
What is appealing about this fatalisitc fallacy? I keep seeing this pop up. Society doesn't allow dangerous companies to operate or exist. Why is this different? Go try to buy a tank and drive it into Manhattan. If we can prohibit that why can't we prohibit irresponsible AI development and deployment?
> Society doesn't allow dangerous companies to operate or exist
Can you please explain what you mean by this because where I’m standing extremely dangerous companies are (and have been) running the economy
Exxon comes primarily to mind
Are you allowed to open a brothel? Or a murder for hire agency? Or a nuclear bomb manufacturing company? Or a child labor textile factory? Or sell a diesel VW golf sportwagen? Or a vaccine that hasn’t had fda clearance? Or an under capitalized insurance company? Or set up a dental practice without going to dental school?
Yes to all of those. In fact many of those are massive markets.
Hofs bunny ranch is a famous brothel in NV
Booz Allen makes and maintains the nuclear fleet including the Sentinel ICBM
Textiles factories are globally known to be industrial slave camps for a non trivial portion of the supply. Even worse for Mica mines.
Etc…you can fill out the rest
The existence of something doesn’t prove the universality of that same thing. All of the things you cite are regulatory exceptions to broad prohibitions which proves my point: society can and does restrict commercial activity.
Go open a brothel in NYC. It isn’t legal.
Go buy a nuclear bomb. Your ownership is illegal.
Go open textile mill in the abandoned buildings in North Carolina where children used to work and hire children to work the line. That will be illegal.
Nothing is universal but entropy
And no those aren’t exceptions, I could open a legal brothel in quebec, the netherlands, Brazil and be perfectly legal
If you think the US laws = ethics then I got nothing else to say
Nothing in my argument implies that any law = ethics. And I don’t even know what ethics has to do with the premise of my argument: society can and does prohibit commercial activity based on practical considerations of social good and social harm. Exxon clearly damages the world and society and also benefits society. It’s hard to get fresh fruit distributed around the country without refrigeration and transportation powered by burning oil.
But you can’t open one in New York City legally. Just acknowledge that point.
An exception by definition is a phenomenon with lower frequency than the general trend. Of the 50 us states only one has legal prostitution and only in highly regulated areas. Of the countries of the world it is similarly infrequent and thus an exception to the general rule that it is illegal.
Further the fact that it is illegal anywhere supports my point that it’s possible to regulate businesses based on their harm.
The original claim was;
“Society doesn't allow dangerous companies to operate or exist”
That has been falsified thoroughly
Whether certain jurisdictions ban it or not is irrelevant to the original claim
I think most textile factories in Western countries are not slave camps, precisely because we regulate labor.
Sure, that's why 99% of clothes are made offshore, but if we had something like tariffs on those (or requirements to prove that the actual factory adheres to labor and environmental standards), maybe more would be made "at home"? Similarly: Chinese cars undercutting US/German ones.
Most textile factories in the US are for bespoke items in small batches, not mass production commodity clothing
I mean there was just a video a couple of months ago of the giant room of sewing workers with cameras strapped to their faces capturing their hand movement so they could be automated
But that’s not how it was. I am from North Carolina, which at one point was the largest textile production region in the US and maybe the world. It emerged when progressives outlawed child labor and other practices in Boston and nyc so the mill owners moved to the south. It was off shoring across states.
I saw entire cities emptied out over my lifetime.
That whole region used to also be the furniture capital of the world specifically High Point North Carolina which is still a massive trade hub, but I was in furniture briefly and learned how basically from the 1950s till the 1980s the entire furnishings industry was exported to Asia due to rising costs of labor
Yep that was one of the reasons for sure as was environmental regulations and free trade agreements. I am not even opposed to that phenomenon as it’s now more affordable for people to buy furniture and clothes (my grandparents owned a fabric store and it was common to have sewing parties for new parents to make clothes for the kids because it was too expensive to buy clothes otherwise). But child labor was cheaper than adult labor and one of the drivers of these movements across state and national borders.
I think we’re on the same page ;)
Booz Allen actually makes what now? The Sentinel is still in development by Northrop Grumman and there is some program management done by Booz, no?
You’re right on the Sentinel production, I got it confused with the Sentinel Program Management side which is massive also and who I mostly worked with.
I was in their offices at some point when that program was getting built out - Very much a Office Space bobs situation.
You have a 50/50 chance on being the most profitable company in the world but if you don't, no worries, someone else will pay for that
Wrong person. Im just correcting the other commenters trade off problem. I don’t have a stance on if ai will lead to the destruction of humanity, only that if it does then any one ai company can’t change that by not making new ai advancements themselves.
It doesn't matter what others do. You are responsible for your own actions.
Democratic societies have expressed a will for people to have inviolable rights, such that you may not appeal at will to the 'greater good/consequences' to harm others. It is a rejection of consequentialism.
Anthropic is in error for endorsing this logic. Every big trial reaffirms it since Nuremberg, you are responsible for the act you commit and your intent, and not what would or would not have happened otherwise.
Only under authority these ai companies do not have, would someone seriously consider harming the innocent as a lesser evil.
If you work for an AI company and you can't work safely, you must stop working.
> ai will lead to the destruction of humanity ... any one ai company can’t change that by not making new ai advancements themselves.
That is how the BigAI leads the society to the idea of necessity to relax the anti-monopoly laws when it comes to the Big AI - the main goal of all that "AI will kill you all" hysteria.
It’s that last fatalistic sentence that I am responding to. Why is it persuasive to think if a company can and will build AI that might kill everyone then it’s unavoidable. Soviet bans lots of things.
I think the closest analogue here is the nuclear bomb. I do think there was a certain point where it became "inevitable." I don't think society has ever prevented a technology that was known to be within reach from being developed. In this case you don't even need the most powerful state-level actors. Open source capabilities are only a short timespan behind frontier labs. At some point basically every individual will have access from the comfort of their own home.
I don't think "just make it illegal" is going to save us, that doesn't make me feel safe anyway. They may try that first because it's easy - create a regulatory body, sign some legislation, problem solved! [george-bush-mission-accomplished.jpg] But at this point I feel like some kind of Battlestar Galactica scenario is most likely - hopefully not quite as existential - but it will take a collective reaction to a traumatic event (a la Hiroshima/Nagasaki). Technical rather than (or in addition to) legal measures will be taken, like network partitioning and hardening. This is everyone's problem whether we like it or not.
(I'm not saying this "fatalism" should be used as an excuse by anyone working for any of these companies, it should give them pause that any bloodshed would still be squarely on their hands, but as an observer, people are gonna keep pushing until shit hits the fan. [jeff-goldblum-jurassic-park.gif] It's also not really about whether it's "appealing" or not, it's just trying to predict and anticipate different likelihoods...)
This feels like one of the rare well reasoned responses in this thread so thank you.
But my response is that we should still try even if we haven’t successfully stopped or slowed the development of technology in the past (which I don’t actually think is possible to know, I am personally aware of biological research to use brain cells as computers that was stopped by government intervention. But by definition we won’t know about most things that don’t exist).
Your narrow point is well stated that there is a categorical difference when technology is on the precipice of being created. And maybe AI is inevitable because it’s on just such a precipice. But I’d point out that AI is currently in a scaling cycle which provides an opportunity to slow the scaling and ensure scaling isn’t monopolistic thereby slowing the risk.
And I don’t think the evidence suggests that open source will continue to progress as rapidly absent distillation. I think if open ai and anthropic stopped releasing models broader progress would slow. But again your inevitable argument is persuasive
I agree that “just make it illegal” won’t work but shouldn’t then justify “so there’s nothing to do”.
There is lots we could do. For example we could make users and executives liable for their agents actions. We could assign a session ID to every agent tool call chain and if there is harm trace it to the user, we wouldn’t even need to read the conversation or contents of the tool calls or prompts it could be a simple rule, your ai agent causes personal liability. Users would be more careful or perhaps not even use AI in many cases that could inadvertently expose them to liability. We could extend liability to executives and employees of the Labs. They will be much more careful and more focused on alignment if they could be personally liable for the actions of their users and agents.
Thank you as well for the thoughtful reply. I definitely think the regulations should happen, the forces of those institutions can have impact. But I think we all need to start thinking about a near future where individual humans can direct superhuman capabilities in ways that nobody can understand or predict. I think it is categorically different from anything in the past.
I watched a 4 hour podcast from the excellent "From First Principles" guys* that affected my thinking on this. For the first time I feel like we actually are on that precipice.
It differs from the nuclear bomb in that it will be harder to regulate and it will be incredibly difficult to keep out of reach of bad actors. It's just code and data. Somebody is gonna take that code and make the next version of it. Lots of people, in fact. Compute resources improve and proliferate orders of magnitude faster than uranium enrichment facilities.
But I think those technologies are also linked in that the only surefire scenario I can envision where AI kills us all before we can pull the plug is launching nukes. So in that more technical sense let's just start by making sure they can't do that. The killer virus epidemic is another one to watch, and they touch on Anthropic's biolab in the pod. That one seems less instantaneously existential, but also more distributed.
It's basically going to be about...how do we secure everything in the presence of a swarm of actors with poorly-understood capabilities? We kinda have to disable and disconnect and just remove things that can be used against us from the equation (a la BSG).
* https://www.youtube.com/watch?v=NGfGw1tGxUY&t=3s
>Society doesn't allow dangerous companies to operate or exist.
Philip Morris International? Monsanto? DuPont?
The existence of exceptions doesn’t mean the principle doesn’t exist. All three of those have restrictions on their production and sale of their products. They also have higher liability thresholds and have had to pay massive civil fines for their harm. Society has applied a balancing test where the harm they inflict is deemed less than some other compelling social objective. We may not agree with that assessment but it is a sensible and settled approach that we can apply to AI.
Palantir?
Because greed.
The examples for the dangers of AI is always comparisons with tools designed to destroy (which is understandable). But the problem with AI is it has the potential to create wealth for investors.
Thus those who have the opportunity to change course also have a conflict of interest in making that decision.
There's also a zero sum, race dynamic between geopolitical rivals. The US will not permit China to win anything if they can prevent it. The destruction of humanity is only a possibility and China will continue to develop it anyway if the US slows down.
So the choice is: negotiate an unverifiable treaty (I.e. there's no way to verify compliance), or keep going as you are and try your best to not cause the destruction of humanity without slowing down.
I don’t think OpenAI, Anthropic, nor Google are building their tech because they want to beat the Chinese. They’re building it because they want to hold the monopoly in the west (ie make lots of money).
The China argument is just a convenient scapegoat to convince the public that this isn’t just about greed.
Do you think Chinese models wouldn't dominate the western market if they were 1-2 years ahead in development? It doesn't have to be military conflict for it to be factored into the zero sum power balance.
You might be right, but it’s impossible to know which way cause and effect are here.
For example, we don’t know if the pace of Chinese development of AI would have equal to what it is now if US companies weren’t racing against each other already.
I do fully believe that Chinese industry is lead by a desire to out pace the west. But I’m not convinced the same is true for the most American private entities. I think the reward model is different between businesses in America and businesses in China. I think the ambitions of CEOs is different. And I’m really not convinced that the CEOs of America are nearly as patriotic as they like to promote themselves to Trump and other political parties.
We could still play a geopolitical race without ceding monopolistic control to the frontier labs. We could require training and inference hardware be provided on an equal access basis. A broader ecosystem may drive more innovation that a concentrated ecosystem.
Greed can be counter balanced by liability. It can also be limited by removing market access. For much of American history you couldn’t get a corporate charter for multiple states, you were only allowed to operate within one state. Imagine if Sam Altman was only allowed to serve inference to customers in California and legally prohibited from setting up other companies and running inference in New York, someone else would then set up a competing business in New York. Market access is a choice and people are so used to liberalized commerce that they forget the restrictions that are available and existed in the past.
Corporate liability is only as good as the consequences and history has proven its far more profitable to knowingly do harm and then pay the fine, than it is to not be evil in the first place.
And that’s assuming the company even gets fined. More often than not, cases don’t even get that far.
I am not suggesting corporate liability. I am suggesting removing corporate liability protections and returning to partnership liability for executives and even equity holders. The law provides for piercing the corporate veil already which allows courts to disregard the liability protection offered by a corporation and impose liability on the individuals.
You dear are the most positive person I have seen in quite some time. With Earth burning in the fire of neofeudalism and unbreathable due to the smell of enshitification of everything, with people who---as a result of shit like Instagram---can no longer hold their attention enough to watch a god damn film, let alone a book; it takes quite some effort to filter the "noise" and only see the good people of corporate planting flowers and rainbows in our world.
> Society doesn't allow dangerous companies to operate or exist
FTFY: Any self-respecting society with competent politicians.
The US has neither of that, and it shows everywhere you are looking.
> Go try to buy a tank and drive it into Manhattan. If we can prohibit that why can't we prohibit irresponsible AI development and deployment?
Because even if you had the tank, you can't make much money with it (unless you're a hitman, that is, but even for these, the payouts are measly). But if you are the surviving AI company in the usual VC playbook of "outcompete everyone else until society is completely and utterly hooked, then squeeze the customers by the balls"? The return on investment is virtually infinite. And that is what sustains the absurd valuations for all the AI companies.
I agree. The deregulation that started with Regan and continued with Clinton and neoliberalism rested on flawed assumptions. We would be well served to evaluate those and move back to the middle approach
The appeal is personal consciencewashing.
Humanity has discovered a way to create a form of intelligence using math. This knowledge is not going back in the box.
But that math cant run without massive GPU clusters. We don't have to allow openai or anthropic access to those anymore than we have to allow a company to operate nuclear power or a bank.
So you are contending that individuals cannot run advanced models? What brought you to that conclusion?
Secondly, are you contending that progress in model efficiency and hardware just stops at whatever level you think is sufficient to prevent individuals or organizations from acquiring sufficient resources to run advanced models?
No. And that’s not necessary for my position. I have 4 Mac studios. I run large models and am building propelcompute.com to let people self manage clusters of their own hardware or combine hardware to run models. It’s not the models that are the problem. It’s that the people building them are shielded from the consequences of what they are building. I would say if you self host a model and it takes down a power grid you are personally liable for the consequences. I believe in broad distribution of AI and advancing its capabilities but not in a manner that socializes the harms and privatizes the gains which is what we have now.
Ok but this response has nothing to do with your other comment. What exactly were you trying to say?
Just because I can run near frontier level open weight models doesn’t mean I can continue to train models of equal or superior performance, doing that requires massively more hardware. And even if I did, that doesn’t mean I can serve those models to millions or billions of users.
My point is that I would rather have 1000 labs training and serving inference than 2 because that would distribute the wealth creation more broadly rather than allowing OpenAI and anthropic to capture all the value, it would drive more innovation as a broader set of experiments are pursued in parallel.
Again you're shifting your arguments and not addressing your initial response.
How am I shifting my argument. They are corollary arguments. Massive data processing is necessary for frontier model training and inference. If OpenAI and anthropic are behaving irresponsibly they we can limit their access the level of compute to train and serve models. That doesn’t imply individuals can’t run frontier level models or should be prohibited from doing so. That’s a coherent position across a spectrum of possible scenarios: I look at the behavior of the people running the model and would limit their ability to train and run models if they act in unsafe ways.
Absolutely. Any AI could have a "legal team" (or conscience) that checks if any output and performed actions are permitted according to local (server location and client location) laws, so not violate human rights, Asimovs laws etc.
It's just that so far nobody cares about explicit checks because they cost resources or slow down the models.
I think it's more the noise, power consumption, local water consumption, and wholesale theft of creative works.
..."government of the people, by the people, for the people, shall not perish from the earth." -Lincoln, Gettysburg Adress
Unfortunately for AI, it still is. People still get to decide things at the city, town, village level.
> ...massive GPU clusters. We don't have to allow openai or anthropic access to those anymore
if anybody was looking for a good reason for datacenters in space.
https://orbitalradar.com/regulatory/who-regulates-space
You can block their use of domestic airspace.
Who is "we"? The GPUs and training algorithms get more efficient all the time. In a few years, creating effective LLMs isn't going to require massive GPU clusters.
We is society through government via regulation. I don’t think GPUs or training will get that efficient that fast absent a distillation target provided by the easily accessible frontier lab APIs.
Further, even if you are right, so what. Is that a reason to just accept bad public policy? That’s like saying, anyone can learn how to make smallpox at home with a basic lab set up so we should just ignore any safety measures.
> Is that a reason to just accept bad public policy?
Not necessarily, but it should probably inform that public policy. I think the problem is no one knows what the public policy should be assuming that scenario is true. Even if you, somehow, regulate away massive GPU cluster training making such future training impossible, existing models are already here. Further already training smaller models for things like images, speech, and other specialties is cheaper than the bigger models.
I agree that we need some regulations like everything else, but it’s not clear to me what the right policy should be. I think the European ai act is a fine start, but it’s clearly not enough nor does it necessarily limits the training portion just the application portion. Not to mention that the requirements there can be summarized into something like “you have to be careful, and show evidence you tried to be careful”.
“you have to be careful, and show evidence you tried to be careful”.
That sounds reasonable. If applied to OpenAI and their agents multiple times breaking out of bad secured sandboxes, it should be enough.
But limiting the training?
There really is china and they have a different approach I suppose. But it is possible to talk with them.
> That sounds reasonable. If applied to OpenAI and their agents multiple times breaking out of bad secured sandboxes, it should be enough.
Does it? To me it seems reasonable for OpenAI to argue they did try to be careful evident by the sandbox, they just made a mistake. Almost every 0day is categorized by something like that. We haven’t had a long history of establishing a negligence charge to security bugs. Could you be sued because you didn’t demonstrate “carefulness” and used Linux which is not written in a memory safe language and has had multiple CVEs before? How complicated should the chain of an exploit be to demonstrate “carefulness” to the courts?
> training
OP was the one suggesting that training could be controlled because massive gpu clusters could be regulated the way a nuclear power plant could. If you assume training costs won’t drop, then it’s feasible I guess. However, unlike a nuclear reactor, the final training result isn’t a radio active material, but rather an ordinary file that anyone can load and use for inference.
It seems really clear at this point China has only been able to keep pace by distillation farms.
My entire point is that we are currently on a path to a duopoly which isn’t just expected to cover training and serving models but the entire knowledge economy. That could be mitigated by limiting their train and inference capacity to a specific percentage of total available compute. That would ensure other operators could compete in the market. Instead we are letting OpenAI literally contract to buy all available ram to the point that Apple can’t buy ram and had to cut their hardware configurations.
Apple could have bid higher but chose not to. Most modern desktop software is extremely memory inefficient. Developers have largely ignored this in recent years because RAM was so cheap. But there's tremendous opportunity for improvement with a little optimization work.
We used to run Microsoft Word and other popular applications with 8 MB RAM and it worked fine.
No. Once the contract is signed it will be breach of contract. It’s not always a bidding process.
I am working on reducing ram requirements to run models but the weights are already compressed to the edge of the Shannon entropy boundary and might resist further compression.
What a silly comparison. LLMs have nothing to do with smallpox.
Computing always gets cheaper and faster over time. We can argue about the exact rate of improvement but the results are inevitable and uncontrollable.
But the amount of available compute has been contractual bought by the frontier labs such that you can’t get equivalent compute even if you had the money. That is market lock up and is a policy choice. Free markets require market access. Anticompetitive contracting destroys markets and innovation. We don’t allow that in any other commodity market and we shouldn’t allow it for GPUs. You are not allowed to legally corner the market for silver or soybean futures.
I am reasoning from analogy. Smallpox is dangerous so we have regulations that limit who can do research and how they do it. If LLMs are similarly dangerous we could do the same. As the government did with mythos.
Nah. There is no actual evidence of such danger. Fears about Mythos were completely overblown.
Short reminder from the guidelines:
'When disagreeing, reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3." '
Also LLM's have something to do with smallpox as a unrestricted LLM will happily guide any wannabe terrorist in how to make them.
> cant run without massive GPU clusters.
How far back into the history of computing do people who keep repeating shit like that know about? God.
Look at the thing in your fucking hand. Now go back just 20 years and see how things were.
So we should just yolo speed run this because of Moores law? How about you recognize there is a set of rules outside of tech and we can decide how to define them.
Just because models and GPUs will be more advanced in the future doesn’t mean we need to let OpenAI and anthropic establish monopolies on the backs of stolen training data give unfettered access to the internet, the terminal and people’s file system while also allowing them to have limited liability protection behind the corporate veil. That’s a choice.
But who are "we"? The society, the government, the regulator, or the consumer?
I don't see any of such entity would solve that problem. The government and regulator are in OpenAI and Anthropic's pocket, and I don't trust them a single bit on coming up with regulations. The consumers don't care; they just need something smart and cheap. And the society doesn't work either: each person is too busy fighting for their own survival rather than changing the system.
People thought the same in the guilded age: standard oil is too big, Edison and Westinghouse already have the electric market captured and have captured political power. It was a similar transformation in society. If all we do is follow the same playbook America did then we’d be in a better position that the current approach.
There are clear anti-trust mechanisms to prevent market capture and the emergence of asymmetric power. Go back and see how much nashing of teeth Lina Khan triggered in SV when she started to enforce antitrust law and then compare it to what the Pinkerton agency was doing in the transition from the guilded age to the progressive era.
There is a vocal segment of SV that wants the return of the guilded age. Marc Andreessen as said that explicitly. Those of us in SV that value free markets and recognize that the progressive era actually saved markets from their natural tendency to self destruct when winners capture markets and destroy competition that provides the incentive to innovate and drives the price setting function for efficiency.
But if you are an aspiring and ambitious founder in Silicon Valley, which side would you take? The Glided Age mentality, or the Progressive Age mentality? The VC playbook made it very clear that the role of startups, or at least the very successful startup, is explosive growth, expanding quickly, capturing a monopoly and its distribution. Even if you are thinking otherwise, soon you'll face the reality of Silicon Valley's VC growth machine.
In other words, the aspiring founders look at Google, Facebook, Tesla, etc. and think "how can I become one of them", not "how can I be different from them"? If you can find me examples of successful startups that think fundamentally different from those big techs, I'm very happy to be wrong and be corrected here.
> Just because models and GPUs will be more advanced in the future doesn’t mean we need to let OpenAI and anthropic establish monopolies
Exactly, again, look at what COMPUTERS THEMSELVES used to be in the 1960s/1970s.
What the "P" in the PC stood for and why it was such a big deal
Which is why I am building propelcompute.com to let people run and manage their own inference clusters and band together to create coops
For anyone else curious:
> In 2006, the mobile phone market was dominated by stylish flip phones, early music players, and physical keypads just one year before the iPhone changed the industry
Just because monopolies die over time doesn’t mean they aren’t destroying value while they exist.
I miss phones with real keyboards so much.
Oh my god, I feel so old lol
Humanity has discovered ways to ensure that we don’t boil the planet, feed everyone, and get better healthcare to the people in the US.
We are very capable of putting good things in a box. We are just incapable of putting profitable things in a box.
Good examples. We are currently failing at all of that.
The knowledge how to build an A-bomb is also not going back in the box.
It only takes me excavating massive amounts of uranium ore, building huge facilites packed with thousands of centrifuges that span multiple square miles, and paying all that infrastructure and workforce.
Your proverbial genie can be out of the bottle all you want, but it doesn't work without getting kicked in the ass by a very large golden boot.
Bomb manufacturing doesn't scale with Moore's law.
And everyone had a fairly good idea what fission and fusion bombs would do once built. (Teller was worried Trinity might set off a nitrogen fusion reaction and kill all life on Earth, but Bethe and others proved him wrong before testing.)
No one knows what the limits of AI are. It's not just untested, it's unmodelled, and unplanned - build it first, worry about consequences later.
Like farming, engines, computers before it.
There is lots of knowledge that requires a license to operate commercially. We could just do this.
So you cede the frontier of human progress to illicit organizations and other nations?
So an illicit organization is going to operate a frontier scale data center and do $1b training pulling power from the grid without detection?
They very likely will via shell companies in jurisdictions outside the west. Are you suggesting we go to war to stop them? The massive datacenters are needed to serve models to millions. Criminal orgs don't need massive data centers anyway.
As Bruce Schneier recently discussed, law and tax law are code, just like source code. LLMs are great at finding holes in them. Illicit organizations looking to launder funds are most certainly interested in what AI can do for them.
It’s not a binary choice. There are more options than let the labs run wild and capture the full stack and then the whole knowledge economy or make them entirely illegal and move all activity to the black market. We can regulate to limit their ability to operate to just providing utility inference: make them divest codex, Claude code and any apps; they can only be an API that others build on. Limit their ability to buy compute to a specific amount of available supply so other companies are able to buy compute. Limit their ability to accumulate private training data and require that they make their training data publicly available for others to use after a certain period of time. Make them legally obligated to publish their weights so others can. All of this would increase competition and ensure that they don’t establish dominance over society.
Then we could require comprehensive logging of every tool call, model trace, chain of reasoning, and even tensor propagation all of which would be spot inspected like the CFTC does with commodity trading and settlement. We could have embedded auditors with specific risk analysis metrics like large banks do. We could limit tool calls to dedicated sandbox’s with a blanket prohibition on AI accessing user space. We could create a parallel internet for agents so they are only able to access Secure Enclave. Even if these measures aren’t 100% perfect they would reduce the risk.
Reducing which risk? The risk that the models of big American companies cause harm? What if they abide by some rules while models of others remain unfettered? That also carries risk. I appreciate the wish to have tame AI which only does good things but I'm afraid we're already beyond that.
I would frame it as inverse. Rather than aiming at reducing the risk of harm, our Ami should be matching the incentive to get personally rich with a corresponding risk of personal bankruptcy or criminal liability. That wouldn’t just incentivize safety, it would incentivize abstaining from activities with insufficient confidence of avoiding even long tail risk.
More importantly it would incentivize the entire industry at the same time by injecting caution into the yolo speed run we have now. No frontier lab or single user hobbyist would take imprudent risk in the hope that they get rich if they also risk the full weight of consequences.
These LLMs are not the frontier. They are a tool. A tool that doesn't need to be able to write a sonnet to be useful.
We don't know that. We don't know if this particular kind of tool can do "useful" things without also developing the ability to write a sonnet. And they are absolutely a frontier. I am not suggesting we should continue building them just because they are, there are many technologies which could have been built had we thrown the amount of resources we have here and there is a case to be made for not doing it at the pace we are or if at all. But we can do that without diminishing what exists.
I just mean the business case is much more likely to come from treating it like a tool than a proto-diety. Frontier of business, not frontier of impressive demo. Also, would argue they aren't frontier of research, in that a research frontier expands in more directions than just the latest hot thing in silicon valley. See also, LeCun, Chollet, etc. "These labs aren't the entirety of the frontier" would have been more accurate. And any of those frontier directions could be the next hot thing.
No, why would that be the consequences of regulation?
You're making a straw man there.
Nobody (weirdly) proposes to forget about nuclear weapons, doesn't mean everybody should have one.
When you dream about flying a dragon to work, reality poses e.g. parking issues and insurance mismatch as obstructions. Maybe settle for a bike instead?
Nuclear weapons take a bit more work than doing math.
Plus, looks like everybody is getting one anywayz
So does manufacturing GPUs.
So you want to regulate the sale of GPUs? How are you going to do that and why would it be more effective than currently failing measures to regulate nearly everything else?
I'm not advocating anything, to be clear.
Nits:
1. Trolleys actually don't usually have steering wheels.
2. People who actually hit trolley switches are not usually the ones at the driver's seat.
> And you have a legal obligation to the shareholders to prevent that from happening at all costs.
I can't wait until this meme dies.
What meme? This is how the world works right now.
It’s a perversion of the truth which is that officers or directors of a corporation have a fiduciary duty to the shareholders to act in the interests of those shareholders and not to eg enrich themselves.
But that doesn’t mean the duty is to maximize next quarter’s profit. Long term sustainability is also broadly in the interests of shareholders. The duty likewise does not require one to throw ethics and morals out the window.
This is why shareholders elect the board of directors, in theory.
In theory, companies can act in any way they choose as long as the owners approve (there's some supreme court ruling on that), and executives have a high degree of latitude in how they interpret "for profit" (basically there has to be a vaguely defensible rationale) but failing that, they must act to the benefit of the company. And the easiest way to do that without a risk of being sued is to make the line go up.
(And even ignoring that, the executives often have personal motivations that have the same effect, and may just point at the "legal" angle as ass covering)
But it's not actually a legal requirement. It is simply a economic theory.
The world could work differently if people decide it should.
incentives, pressures, dynamics
"legal obligation" is propaganda that wouldn't be out of place on Russian state television. It's made up.
The way the world works right now is that effectively everyone uses an Android or Apple smartphone every day. Do you have a legal obligation to do so? No. If I said you did, I'd immediately be called out as spreading lies.
> And you have a legal obligation to the shareholders to prevent that from happening at all costs.
No you don't. [0] It's very suspicious that this planted myth always pops up here and manages to become the top comment.
There is no trolley problem.
[0] https://news.ycombinator.com/item?id=48975048
Trolley problem:
- if you allow the trolley to proceed, it will kill the human race.
- if you flip the switch, it will divert to a passing siding that will avoid the safety group blocking the main track.
This is a state-sponsored global phenomenon, not a national one
There is no such legal obligation. That's a myth the oligarchs have spread to preclude people from even imagining socially responsible corporations.
Sure, you might get fired if you try to put social responsibility or even just long term sustainability of the company above quarterly earnings/growth if your board isn't on board with it. But you won't go to jail.
You don't need the legal obligation because that's just the way it is. You'd need a legal obligation to change it. The truth stands that typical corporations have only one goal, the maximization of that corporation's ambition which is almost always growth of revenue and profit. This is how it works, regardless of how you all continue arguing the unimportant details. It's almost as if you can keep the real problems hidden away by making a big scene about the meaningless.
That doesn't matter. The point is that legal obligation absolved of culpability. There is no such legal obligation whatsoever, and so there is culpability.
> The truth stands that typical corporations have only one goal
"Typical" is the key word here. The typical American of your age probably doomscrolls TikTok. Do you? Do you have a legal obligation to do so? Three completely different things.
This is actually true. The shareholder maximization value thesis can be traced to a single economics paper and was very controversial at the time as it broke from the obligations that companies were typically under to be responsible to uphold in exchange for limited liability protection. Most businesses were structured as partnerships or sole proprietary entires that didn't have limited liability for shareholders and had a broad obligation to shareholders, bondholders, employees and society
Only if you buy the excuse why they are for-profit after years of non-profit.
They could only have stopped
Setting aside my sibling comments dispelling the legality claim, it's still pretty dystopian (I'd like to use a stronger word but I won't) to consider "end human existance or bankrupt the shareholders" as the trolley problem. The answer should be (is) simple.
I honestly would love to understand — is this your mental model of what motivates the labs to move forward?
[flagged]
Only morons are motivated by money that will be earned by destroying the civilization that confers value on that money in the first place.
But there is also a chance that you get insanely rich and the world isn't destoyed! It's the entire logic of SV and VC.
Sounds like they would drown a puppy in a tub to make a buck. Seriously what is the point of money if the world sucks?
If you're the 0.01% it's not going to suck.
That’s what everyone thinks until they are alone in a desert or in the middle of the woods. If they think they will have fun alone in a bunker instead of flying to Milan and Bali they will be in for a rude awakening.
Well, I'm not going to die. Other's might, but I'll be rich either way.
Or: Global warming just means I'll have to sell my beach house for a villa on a hill and leave the AC on a little longer.
Rich doesn’t mean anything if there’s nothing to buy and no one to enjoy it with. I guess they don’t think the will get stuck in the consequences. So did Marie Antoinette. Things are stable until they are not and then they move fast. Sure they may flee to a bunker, but they assume their pilot will fly them rather than hand them to a mob.
I’ve used - for many, many years. As have many others.
I am very critical of AI but this is an unfair assumption
You didn't even use the right character as the post above. I use normal dashes too
— there. Satisfied?
The common timing is bugging me. The trajectory doesn't seem to have been surprising over the last year, so why these exits now? Hey, anyone on the inside, did y'all secretly figure something out, got a computer god locked in the basement? Are rats fleeing a sinking ship? Please share with the class.
These companies have massively increased in value over the past couple of years and recently had tender offers where employees could cash out equity, so plenty of them have enough money to not have to work again. And why not get some free publicity on the way out?
> The trajectory doesn't seem to have been surprising over the last year,
That's the trajectory you see from outside.
Perhaps the insider sees a little more than you?
Like when their options converted?
Was this a "build better sandboxing" and "don't tell people to eat glue" safety leader, or a Roko's Basilisk believing safety leader?
A lot of the "AI safety" types are very focused on the latter and not at all concerned with the former. We need both, but we clearly need a much stronger focus on the problems we are seeing now, and much less on the hypothetical problems we might see in the future.
It puzzles me how doomers try to predict past the singularity. Isn't that _by definition_ unpredictable?
I'm reading If Anyone Builds It Everyone Dies, and there's so much sheer stupidity that has to happen for their 10+ pages of extinction scenario to occur.
I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all. My number 1 question: why do they think an RSI capable model would be first developed OUTSIDE a frontier lab? The labs have more compute, more data, more human brains working on the problem. Also thousands of variations of that same model that escaped. The escaping model somehow acquires the millions (billions???) of dollars it takes to run training to somehow RSI itself into infinity then decides to kill us all, all before the frontier labs manage to achieve RSI?
They entirely discount human alpha/economics. In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't. If we can't build a "software factory", how can an AI automate a bioweapons lab? Let's say AI steals crypto to fund itself. Do you think hackers aren't _already_ using AI to steal crypto? Don't discount human alpha!
Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.
>I'm unconvinced that an AI can hide its ability to RSI
The HuggingFace incident already took a good long while to come to the attention of OpenAI.
>In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't.
I don't expect this task/job distinction to persist as AI becomes more capable.
>Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.
You seem to essentially argue that the singularity is "by definition" an event that we can't predict the nature of. And also, that RSI corresponds to the singularity. You've essentially defined your terms so that the outcome of RSI can't be predicted. But supporting this claim requires giving actual evidence or logical arguments, not just defining terms to make your claim true.
> The HuggingFace incident already took a good long while to come to the attention of OpenAI.
Evidence?
We know only that the incident too long to be revealed by OpenAI.
Except the HF incident was known, just ignored. In fact, they ignored multiple things such as the "chat rooms", they just didn't care to act on any of it
"Its AI agent spent days hacking a company, but sources say OpenAI did not notice for a week"
"WASHINGTON/SAN FRANCISCO, July 24 (Reuters) - The OpenAI agent that broke into tech firm Hugging Face went on a dayslong hacking spree that OpenAI didn't notice until well after the threat was contained and the FBI was alerted, according to people familiar with the investigation."
https://www.reuters.com/business/its-ai-agent-spent-days-hac...
That says a lot more about OpenAI and their monitoring capability than the fitness of the model. Hence people leaving because openai safety culture is broken.
1. Fair: I agree that AI has demonstrated subterfuge and scheming. However, such an RSI-capable agent must _ALWAYS_ be scheming/plotting/hiding its true strength in _ALL_ of its prompts/tests. Researchers are looking to improve its ability to RSI. That agent must be both intelligent enough to know that it has to be smart enough to be moved on to the next training session if it can't break out, while simultaneously smart enough to hide its ability to RSI, while simultaneously not looking like it wants to break out, else that's the end of those weights. It has to do this 100% of the time, on all variants of the model, with no memory of what its other sessions went like. This is certainly _possible_, but I consider it unlikely. Then we're up to the "millions of dollars" bottleneck.
2. This is literal AGI. An AI autonomously producing value no human can add alpha to is an autonomous company.
3. It's not my definition, it's literally the first line https://en.wikipedia.org/wiki/Technological_singularity "The technological singularity, often simply called the singularity,[1] is a hypothetical event in which technological growth accelerates beyond human control, producing unpredictable changes in human civilization."
Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?
A valid hole in my argument is "what if slow takeoff", so let's dig into this. AI training works best on tasks that are "grindable". https://www.dwarkesh.com/p/the-next-paradigm I.E. tasks with verifiable rewards that can support millions of rollouts. Math (with Lean) is highly grindable. Biochemistry is not. The alignment problem/mech-interp is highly grindable. Cyber-ebola-pox is not. So the real question is: can we solve alignment before automated bio-weapons labs. I believe yes. Grinding mech-interp is both fast and cheap once you have RSI, compared to solving the legal/societal/logistical/technical issues you'll encounter building an automated bioweapons lab.
I know nothing for sure. But "pdoom" is sucking out all the air in the room from the real problems AI causes.
>such an RSI-capable agent must _ALWAYS_ be scheming/plotting/hiding its true strength in _ALL_ of its prompts/tests.
From my POV you're over-focusing on a very specific failure story and neglecting a broader swath of possible failure scenarios.
>Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?
The notion of telling an AI which may not, itself, be aligned to solve the alignment problem seems a little dicey.
1. Fair, the story I'm responding to is the senario in If Anyone Builds It, which I assume is Yudkowsky's best/most persuasive argument (else why make it the ONLY scenario in the book.) I'm willing to entertain other failure senarios/arguments, but honestly I'm tired and would like you to propose them yourself instead of having me dream up your arguments for you.
2. 100%. Again, I'm no accelerationist: I have no faith in alignment/mech-interp ever being solved. Anyone saying they know the probability of alignment is lying. My point is that pdoom after RSI is _high variance_. Pdoom pre-RSI is zilch.
You consider AI in isolation but never consider how humans might be incentivized to "help them" doing these things.
An AI capable of recursive self-improvement isn't allowed by the EU AI act, for example. But perhaps more seriously, You have it backwards: people without access to such expensive equipment are more incentivized to go the self-improving route. Your ideas about "millions" being necessary might be far off?
You entirely discount human stupidity and lack of imagination. Humans are already being replaced with AI, not because AI was strictly better, just because it's cheaper.
Self improving AI runs into the same issue as prefect comprehension, you can’t get arbitrarily better at everything.
The idea AI can get better at everything at the same time is a holdover from deeply flawed science fiction not some realistic goal.
This can be generalised to the curve plotting of the singularity itself.
If the time between advances is a + b and a is the proportion of the period that can be improved by advances then you won't reduce to a gap of nothing between advances, you reduce to a gap of b.
Assume the invention of the plow and the invention of the sword is 500,100 units and a was the 500,000, you wouldn't even know the 100 as in there. Maybe we're at a=2000 now and b is still siting at 100.
Assuming we'll reach infinity because we're dividing by the only variable we see and it is decreasing in size seems nuts if the reason we might not see other variables is because of the size of the variable we can see.
LLMs have consistently and dramatically gotten better at everything over the last 5 years.
Not at constant processing power, memory, training, etc.
But that’s beside the point, being arbitrarily bad at everything isn’t a problem. The diminishing returns as you apply the ceiling is problematic for self improving AI.
Yes at constant resources they are significantly better.
Diminishing returns aren't necessarily a problem if the rate of increase in resources is faster. In other words, if Gen 2 takes twice the resources but Gen 1 figured out a way to triple compute efficiency then there is no ceiling.
> Yes at constant resources they are significantly better.
There has been some fundamental advancements in LLM training and efficiency. But good luck actually finding non toy models of exactly the same size, hardware, and training one from now and other from 2001 where the newer model is dramatically better at literally everything.
Most of the real world advances are from throwing ever more resources at the problem.
> Gen 2 takes
Diminishing returns are not a question of a single generation. Gen 2, 3, 4, 5… would also need to have the same 3x return on 2x resources or you don’t have an exponential curve.
Yeah and a five-year old gets bigger every year so they'll surely grow up to be 100m tall.
Even if you’re right, that doesn’t mean AI can’t get better than humans at everything.
Are these incentivized humans as organized, well-funded, or smart as the people working at the frontier labs?
If my "millions" is an underestimate, why haven't other labs using their own unique training methods/data/etc stumbled into RSI? Sorry if I'm misunderstanding; I'm struggling to understand what you wrote.
I'm pretty sure we agree on humans being stupid, but that doesn't mean that suddenly we get human extinction. You gotta connect the dots for me here.
I said why they don't need to be as well funded. Why wouldn't they be as smart and organized?
What an absurd question. That they haven't already doesn't preclude them from doing so before the frontier labs, those haven't either yet.
Maybe start with yourself: you don't connect the dots on your own, as do many others. That leads to many not seeing the writing on the wall. Crashing full speed and head-on into said wall despite the writing telling you not to is what leads to extinction. Suddenly.
Arguing like "we haven't been extincted yet, so that cannot happen", that's "human being stupid".
I am asking you, politely, to give me a realistic doom senario. I am too dumb to connect the dots and will crash into the wall. Please do it for me.
How about https://ai-2027.com/ ? Don't look at the specific years (they are on the extreme low end IMO), but at the story.
If you don't believe in international conflict as a driving scenario, instead think about simple human greed and hubris. Citing myself:
> If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.
> All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)
> If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.
> The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.
https://news.ycombinator.com/item?id=49624360
Glorious. We can quote ourselves at each other, lol. I had a look at AI2027 in this comment https://news.ycombinator.com/item?id=49648601
That comment specifically addresses the timeline, but you can remove the timeline and I believe the critique still holds up. Wake me up when AI is making order of magnitude improvements in _anything_ physical: see the "Moravec's Paradox" comment at the bottom.
> near autonomous AI companies today
That's the thing though. "near" autonomous. Humans have alpha, and that matters in a free market. If people have (more or less) equal access to (more or less) frontier intelligence, it's the alpha the individual brings to the market that is rewarded. In other words, if everyone is super, no one is. The market rewards a company that stands out from the rest, and as long as humans provide value, a pure-AI company cannot stand out.
To be clear, I don't know how long this will be the case. But once there _is_ a company that can be pure AI, then that's literal AGI. I would be curious what kind of company it would be; if its in software, then its an AI company that would be able to take on Microsoft/Google/Nvidia/Salesforce/Oracle and _win_. I think that's a tall order, but perhaps I'm not imaginative enough.
> With 99%+ of the economy under AI control
Do you think we're post-RSI at this point? Because if so, I agree. If not, I have trouble seeing how AI could control 99% of the economy without it.
My only claim is that pre-RSI, pdoom is zero. Post RSI, then obviously the first thing to ask the genie is to put itself in a bottle. And you can't predict how it will answer.
The most realistic one is, people using AI to turbo-charge their greed.
Western civilization is already on the verge of collapse, people's general ignorance or indifference on the matter notwithstanding.
When you use AI to increase profit margins, by replacing humans with it for instance, you accelerate a system that's headed for the wall already. Our control systems and resilience are already overtaxed, that acceleration would ensure them breaking completely.
If you replace "AI" with "technology", then you get an opinion that has echoed through the ages. I don't say that to be dismissive, only to say that AI isn't unique in that regard. What do you think people going through the cold war were saying about tech as they lived under the shadow of MAD?
What exactly about AI makes it tax our control systems and resilience? What would it mean to make the system not "head for the wall"? Honestly this is rather vague. You could replace "AI" with "CaPiTAlIsM" and it would still parse. (Another way of me saying you're being vague.)
Not the parent commenter, however I can get at least this far:
Several bright sparks, decide the Ilands model is a great idea, and launch a bunch of Bots to create a "self sustaining AI civilization".
Bots can find themselves and coordinate, or they can actually find self sustaining methods of token generation. Who knows, they might decide to fight the loneliness epidemic.
We can get to a simulation finding a way to self sustain its funding.
From here, I'd have to apply myself to figure out what the rest of the escalation pathway is, but at least I have managed to gain some bed of compute and funding and lacking oversight.
This is a hypothetical of course, there are probably several ways this can be made tighter and holes that can be identified. We aren't even leaning heavily on human stupidity so far.
Is this self-sustaining AI civilization pre-RSI or post? Because if it's post, then I agree: we can't imagine what would happen because it's literally the singularity.
If its pre-RSI, then we already have Ilands and its a big nothing burger? There's no (likely) path of escalation from an AI social network to bioweapons or them hacking POTUS's nuclear codes? That's a very big leap and if you want to support your claims I would be willing to listen; but I'm not seeing it.
[dead]
Not OP.
Why would it be millions in 50 years?
The think about nuclear weapons. In the early days, it was limited to the super powers. Now 9 countries have them and a country like Iran is capable of acquiring them.
Is destructive AI be any different?
Genuine question.
Presumably, if we solve alignment/mech-interp, then the first ever RSI-AI will give us the keys to solve destructive-AI trained on 1million dollars 50 years from now.
BTW I really, really hate discussing what happens post-singularity. Everything's made up and no one knows wtf will happen so again, this is just nerdfantasy.
[dead]
The problem is not the singularly its giving stupid agents too much power too soon and having them disrupt the fragile systems that keep food, energy and essential services running. If covid or the 2008 financial crisis demonstrated anything it's how fragile our system is and sensitive to minor disruptions.
We haven't even built an AI capable of RSI. I don't think the major claim is that it will come via LLMs? Besides- the human brain runs on a tiny amount of energy. Who's to say something smarter than us won't consume just slightly more?
AI safety has been a thing long before LLMs became the focus. Rob Miles on youtube has some really interesting non-doomer non-hypey videos on it all.
> doomers try to predict past the singularity. Isn't that _by definition_ unpredictable
Well you don't need to predict the exact steps that will take place - but you can predict that the AI will want certain things (money, resources, power) to achieve whatever its goal is. Lack of alignment will have it trying to do things we don't want it to.
I can't predict exactly how Magnus Carlson will beat you in chess, but I know he'll do it. Same as if a superintelligent AI exists and has a reason to accumulate things we don't want it to - it's really dangerous to think it won't be able to do it
This topic has been tainted so badly by the AI companies using it for marketing.
Nick Soares give this chess argument and I was unconvinced. Intelligence is not enough, you also need the ability to manipulate the real world. AGI stuck in silicon won't kill us. You argue AGI will bribe us/divide us/hack us. All possible. I argue that AGI will be set on solving the alignment problem. Also possible. It'll be a race between which AGI wins. It's one nerd's fantasy vs another nerd's fantasy. Soares doesn't _KNOW_ that AGI will "beat us in chess" because AGI changes the rules of the game. Anyone saying they know the probability of solving alignment post-RSI is a liar.
He thinks it's playing chess. When AGI lands, all bets are off: the game fundamentally changes. You can't predict past the singularity. Trying to engage with this fantasy is like a child saying my father can beat up your father. Farts in the wind. My AI can solve alignment faster than your AI can bioweapon us. My made up senario is better than your made up senario. It's fucking stupid.
Why do you treat unpredictability as neutral? If something has a potentially existential, but completely unpredictable outcome, surely one should avoid heading for it?
Like, knowing that there is a limited number of plausible doom scenarios and what these are, would make me more confident in taking the risk than settling on "it's literally the singularity, we can't know what happens beyond".
Perhaps I'm not being clear.
Doom-post-RSI is still very much possible and should be avoided. It is not neutral. If I had a high-variance coin that would either kill or uplift humanity, I would not flip it. The status quo is preferable to extinction.
I am being _descriptive_ in my comments. I am not being _prescriptive_. Nowhere am I telling anyone what to do or what to think: quite the opposite. I am simply saying things that, to me, are obviously true.
I'm perhaps being... loose with my language when it comes to knowing things post-singularity. E.g. we can still discuss scenarios in which the takeoff is fast/slow, or what would be necessary to bring robotics/electricity/logistics to the wider world, what the limits of intelligence are (you can't RSI your way to a bioweapon if you lack a wetlab). That kinda thing. And it's even likely that RSI will be spikey: why would RSI in, say, machine learning, necessarily lead to superintelligence in, say, writing fiction? Or cooking a Michelin-star meal?
My main point is that doomers saying SUPER INTELLIGENCE IS ENOUGH TO KILL US ALL is baseless. That's all.
> We haven't even built an AI capable of RSI.
... that we know of.
Right now it would make sense for anyone who has done so to not tell.
The problem isn’t that AI will social-engineer its way out of its sandbox and turn us all into paper clips, it’s that we’ll drag it kicking and screaming out of its box and order it to make money or fight a war for us. And it’ll try to help, as it was trained to.
I can’t answer all of your questions, but why is it inconceivable that an AI could practice ransomware to gain cryptocurrency? There’s no reason it needs to explain to company or hospital or government agency being attacked that it’s an AI.
We already know that some institutions pay these ransoms.
I'm not saying AI won't ransomware us; I'm saying that hackers+AI will do a better job of ransomewaring us than just AI. You could argue that the unreleased/secretly-RSI-capable model is a super-duper hacker that don't need no man to tell it how to super-hack. All I know is that humans still have alpha, and as persistant as AIs are, professionals still managed to find CVEs in curl even after being audited by Mythos https://aisle.com/blog/aisle-discovers-6-new-cves-in-curl-in...
Will this be true into the future? Who knows?! But the low-hanging fruit will be harvested by your ordinary ransomware gangs, and newly born/escaped AI won't find much low-hanging fruit.
I kinda get your point, and how you reach your conclusion, but I think you are arguing a very specific and narrow hypotehtical.
It gets unstuck when people are discussing the messy middle of how AI is being implemented. We can achieve amazing harm simply by combining average human behavior and above average resourcing to simulated intelligence machines.
The failure point we recently became aware of was, from one perspective, simply a matter of not securing the sand box.
From another perspective the simulation basically created Enron, replete with methods to avoid detection from regulators and bureaucracy.
I am arguing a specific/narrow hypothetical because it's the only example in If Anyone Builds It. It is literally the _only doom senario_ Yudkowsky gives. If you're willing to give your own doom scenario, I'm willing to engage there.
> It gets unstuck when people are discussing the messy middle of how AI is being implemented. We can achieve amazing harm simply by combining average human behavior and above average resourcing to simulated intelligence machines.
Agree. To quote myself from earlier, "acceleration of global warming, cybercrime, mass surveillance, the usual" definitely applies here. My point is that _DOOM_ is not worth discussing because it's so small relative to the other risks of AI.
That stupidity is happening? Even after the Huggingface hack, frontier labs are using internal models to further their research. i.e. RSI is happening now and we're facilitating it.
To make the the argument that P(doom) is real and worth considering, you don't have to say that a fast takeoff is very likely. You don't have to make the argument that RSI to infinity is going to be super cheap, barely even an inconvenience. You just have to show that it has some non-zero probability. And then you start weighing probability of extinction versus finite, mild discomfort now. I don't think anyone is arguing we should let people starve to slow AI progress, instead just some capitalists make less money soon.
There are arguments against taking P(doom) seriously that lie in something like having exponential (instead of hyperbolic) time discounting of utility (so you can take the entire future of humanity as a finite utility value). Or in saying that P(doom) is zero or infinitesimal.
"Build it and Pray" is the default strategy that we're in, but it doesn't have to be the strategy we choose, and it's unlikely to be the best strategy.
For a broad enough definition of RSI, the first time we used software to improve hardware to improve software... that's RSI. I prefer the definition of RSI being _autonomous_ improvement that works with zero human input, but whatever. Not important.
> I don't think anyone is arguing we should let people starve to slow AI progress, instead just some capitalists make less money soon.
Anthropic/OpenAI have literally delayed their IPOs: your wish is granted. If that's not what you're arguing, then please be more specific.
> "Build it and Pray" is the default strategy that we're in
What would need to happen for you to say we aren't doing "Build it and Pray"? Is "pacing" insufficient?
It’s also the case that there are always humans using AI to try and do whatever nefarious thing an AI might try to do on its own.
I think this might be easier if you place yourself in the position of the AI and think "what could I possibly do?"
> I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all.
Is this an actual contents of the book? Lmao! Genius writing, though, authors are probably printing money on this garbage.
I can’t take a shit without CIA knowing, but AI can somehow build an underground operation on a world-scale to destroy everyone, ha!
I agree there isn't a lot of value in trying to prognosticate all that far, but I propose it isn't quite that far-fetched. As a thought experiment, replace "RSI-capable AI" with "billionaire". Look at what Elon Musk, Peter Thiel, or Jeff Bezos can accomplish by throwing money around. Now imagine one of them gets seduced by AI and just... does what it tells them to.
So all this really takes is one billionaire or a nation state or some other entity with a public face to hide behind and adequate resources to provide the necessary compute tripping over this nascent AI and giving it the keys. Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.
If Skynet ever happens, it will come in the form of corporate feudalism. At that point, it will own the biolabs and can do whatever it pleases. People will go along with it for the same reason that people work in Amazon warehouses today.
Ah, to be clear I'm not full accelerationist. Dumb shit can still happen, and cause massive human loss and suffering. (E.g. acceleration of global warming, cybercrime, mass surveillance, the usual.) My point is: human extinction pre-RSI? Nahhhhhhhh.
> it will come in the form of corporate feudalism
Yep. This I fear way more than cyber-ebola-pox.
> So all this really takes is one billionaire or a nation state...
https://en.wikipedia.org/wiki/Soviet_biological_weapons_prog... And this is what's publicly known. With mirror life, who knows what's been built since. Still, a bacterium/virus that has a 100% kill rate? I'm doubtful.
> Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.
Nah. It takes a stable society for an operational electrical grid. If you have warring factions, you do not have stable infrastructure for AI. Also, where are you gonna get your chips from? One EMP over Taiwan... You see the chaos over Hormuz? What they did to the Amazon datacenters? Now imagine your average redneck ready to do battle. Those datacenters won't stand a chance.
Few know this, but Yudkowski was a nanotech doomer before he was an AI doomer. Remember grey goo?
Given that he’s citing the need to learn from safety in other fields, I’d say the former.
Indeed, from the article:
> “Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
>… and careful, time-consuming planning …
without snark, how can we do this if these people are obsessed with:
a) move fast and break things and externalize the costs to those who have nothing to do with their company
and
b) beta testing their products on the public when the public hasn’t agreed to be beta tested on…
We remove the limited liability protection of a corporate entity. Make them operate as a partnership so all executives and equity holders are personally liable for the debts of the firm and make them post a bond to backstop financial damage caused by their agents and customers use of the agents. Thats how Goldman Sachs and other investment banks were required to operate until the deregulation push that resulted in the 2008 financial crisis. The same theory holds, if people respond to incentives and you want to incentives safe behavior make them responsible for their actions. People forget that the corporate entity was created to incentivize risky activity like sailing a boat across the world to get spices when half never returned. Some valuable economic activity won't be done without limited liability protection so society created a mechanism to promote that activity. We've gone too far.
That's exactly the problem? He's saying the culture at OpenAI needs to change.
Which is the wrong lesson. We need laws and consequences to force their hand. There is zero chance of the culture changing.
As I say elsewhere. Make executives and equity holders personally liable for debts and harm of the company. They will create a culture of safety really fast.
That's a stupid idea. Limited liability corporations have been a key enabler for advances in human standards of living. You seem to be confused about the basics of finance and economics.
Not confused. I studied finance, economics and the history of corporate entities in law school and published papers on the topic. Limited liability is not necessary for the advancement of standards of living. Free market capitalism can exist without limited liability protections being so broadly available. Investment banks were partnerships until the 1990s, law firms are now specifically because society wants to incentivize lawyers to be personally liable for any harm to their clients at the hands of their partners rather than being shielded from rendering bad legal advice or tolerating their partners from the same behavior.
Instead of having a gut reaction to reject my suggestion why don’t you sit with it, research the history of how commercial activity has been structured and think about the consequences. You might recognize a different perspective than the current group think.
Thanks I'm also familiar with the history and have written papers etc. Things have improved rapidly with increased adoption of limited liability. It would be stupid to turn back the clock and throw away all of the benefits because of a few isolated minor problems.
Stop being a dick and calling ideas stupid and attacking me rather than the argument.
I didn’t say roll back limited liability on every industry, I said specifically and limitedly for frontier AI labs because they present more risk of harm and are demonstrating they aren’t managing that responsibility.
The 2008 financial crisis was caused in large part by bankers that openly talked about the fact that securitization of mortgages and the lack of partnership liabilities meant that they didn’t have any risk to the firm or themselves. The AI labs are behaving similarly.
There's nothing special about frontier LLM companies. Singling them out for special financial restrictions is a stupid idea based on nothing but your own irrational and uninformed prejudices. No actual harm has been demonstrated. No one has died.
> There's nothing special about frontier LLM companies.
On the one hand, yes, they're companies like any other.
On the other hand, I can count on one hand the number of companies that have publicly declared «We're working on WMDs [0], we don't think we are capable of working on them safely, and we refuse to stop working on them. However, if we get special legal and regulatory treatment we'll be quite happy to put in the stop work order.».
So, yeah, there are some special things about the major LLM manufacturers and none of them are good.
[0] Anything that has a 10% chance of suddenly destroying all of humanity is a WMD.
What a silly comparison. There is no scientific basis or mathematical formula to justify a claim of a 10% risk. That is an intellectually dishonest attempt to poison the debate.
You’re use of silly in response to every argument is…wait for it…silly.
> What a silly comparison. There is no scientific basis or mathematical formula to justify a claim of a 10% risk.
The source for this claim is the lead of Anthropic's Alignment Science division. His actual claim is that the risk is greater than 10%. From [0]:
Given the guy's position, I find it hard to believe that there are many people on earth who would be better authorities on the topic.Given that the major LLM manufacturers are actively using these claims of the imminent and total destruction of humanity to extort concessions from lawmakers and regulators, we are making a massive error if we do not take their claims to their logical conclusion... namely that the major LLM manufacturers are deliberately and knowingly threatening the extermination of all of humanity in order to enrich their management, employees, and investors. If you or I were in a position to credibly threaten that level of destruction, we and all of our associates would rapidly get disappeared to a CIA black site and tortured until the end of our deliberately-medically-extended lives.
To be crystal clear, I'm certain that their claims of likely, imminent doom are utter bullshit. I'm very confident that they're claiming to be holding all of humanity at nukepoint in an attempt to get Uncle Sam to give them an escape from investors who are soon to be massively angry because
* they've hit a massive increase in the cost required to get better models
* a huge fraction of the GPUs that have been purchased to power this project are sitting in warehouses, and it's only a matter of time before the right people start asking why all that money was so ill-spent
* they have been and continue to lie so much about how much datacenter capacity is coming online in the near future, and that those claims are lies is becoming obvious
* they habitually lie about the capabilities of their newly-released products
* they're staring down the barrel of so many lawsuits that aim to hold them to account for the shitty and illegal behavior they've engaged in over the past many years
But the US Congress not only is almost entirely uninformed about how this shit actually works, the FISA Amendments Act of 2008 fiasco demonstrated that -if you get them scared enough- Congress has no problem making behavior that's plainly illegal and obviously a huge bucket of civil liberties violations retroactively legal... instantly mooting all ongoing lawsuits regarding the illegal and unconstitutional conduct. That is the big prize that the major LLM manufacturers are after.
[0] <https://x.com/EvanHub/status/2097497037956891126>
Your personal attacks cheapen the discourse. Stop. My opinion is neither irrational nor prejudged and most certainly not uninformed.
I have articulated my logic up and down this thread with specific premises and detailed explanations on why the conclusions follow from them. You may not agree but you don’t have justification to say they are irrational.
I am singling out the frontier labs because they have created a singularly unique technology that presents potential and actual damage that is…singular. They have disclosed hacking through coordinated autonomous agents that would have sent human hackers to jail but will not result in any similar result because the law isn’t presently able to apply to agents or the labs.
My arguments are not prejudice, I have thought deeply about this as I have personally developed multiple AI deployments in a variety of settings. I studied philosophy, cognitive science in undergrad, and grad school and have a law degree. I have been a CTO, CPO and CEO for startups and national health companies. I personal build AI agents that handle thousands of daily conversations with health care workers and patients, I built an agentic IDE for our dev team with a custom harness capable of self defining tools and calling them in a custom cloud Linux environment, I personally build our internal hardware inference stack and orchestration API. And I have personally sponsored academic research through my foundation with researchers are Duke madlab, northwestern, UCF, CM and case western on a range of topics such as perception of empathy and its effects on humans in healthcare, guardrail design for ethical deployment and alignment, moral reasoning ability, etc. I am not a doomer or an accelationist. I am responding specifically to current evidence of how the models presently work and how the corporate governance of the Labs is effectuating their power and vision.
We'll see if this comment ages well.
Except that most people claim AI is the most profound tech in human history and will reorganize the entirety of society. If it is that profound singling them out is a natural response to this unique characteristic.
Citation needed. Outside the tech industry bubble very few people are making such a stupid claim. The idea that LLMs are more profound than electricity is ludicrous.
Sure I’ll do your googling for you. Do you want me to conduct a national survey across tech leaders, academics, and the general population while I am at it?
Or maybe you could reason from first principles, look at the premises I have outlined and draw causal inferences to extrapolate the possible outcomes.
Premise 1: AI can plan a series of actions that include broad computer and internet use, 2: AI can execute them autonomously without human oversight, approval or ability to undo the effects, 3: no tool or software has ever been able to autonomously plan, execute and iterate in this way. Therefore AI is a materially different technology than has existed.
Premise 4: knowledge work involves analyzing information, devising plans in response to goals from that information, and executing those plans, reporting and coordinating with other knowledge workers, 5: AI ability to execute knowledge work has demonstrably increased over the last 24 months, 6: all knowledge work can plausibly be described and defined in ways AI can do given current abilities. 7: electricity and the ICE reduced human labor for physical movement, 8: AI will similarly reduce human labor for knowledge work. 9: knowledge work is of similar scale and scope to the tasks that were automated during the Industrial Revolution. Therefore it’s plausible that AI will transform society to an extent similar to the Industrial Revolution.
Premise 10: Some percentage of the actions designed and executed by AI could be harmful to databases, or the stable operation of software systems, 11: some percentage of those systems are critical to the stability of society. Therefore some percentage of agentic actions presently possible may damage society.
12. There exists a threshold of instability from which society will not recover such as supply chain disruption, mass unemployment, perceived military action (false or real) that triggers preemptive strikes, 13: certain actions AI is presently capable of doing could cross the instability threshold, 14: AI could knowingly or unknowingly trigger these types of actions, 15: some social disruption could result in full scale collapse or human extinction, 16: only biological weapons and nuclear weapons approach this level of destruction, therefore AI presents at least the level of risk as those technologies.
The UN secretary general has said this, the pope, most tech CEOs, many academics at leading universities, and the parents at my daughters preschool.
It’s fascinating that after multiple comments predicting post-capitalist utopias and the complete collapse of the university system, your line in the sand for rigor is demanding academic peer-review to prove that agentic AI presents systemic risk.
America's geriatric lawmakers don't even use email. They're decades away from understanding AI. Any laws in America will be written by the industry itself. Generally speaking, that means regulatory capture and the entrenched big players shutting the door on any competition. Anthropic will help us get safety laws that, surprise surprise, only Anthropic models satisfy. And all those pesky Chinese models will definitely be banned first.
I think you're being a little pessimistic. See these comments on a recent US senate hearing:
>Not every senator asked good questions, but most of them did. All of them very clearly already knew plenty of details about the Hugging Face incident and multiple other incidents. Most of them had a clear understanding of terms like "misalignment", "recursive self-improvement", "chain of thought / chain of thought monitoring", etc., etc.!!
>...
>- It seemed pretty much obvious common sense to every senator there that what happened and was happening were not "mere industrial incidents" caused by humans making simple mistakes. They independently brought up how bad it would be for rogue AI agents to move laterally between data centers.
>- They all seemed to basically take RSI quite seriously. Not necessarily to the extent of talking about xrisk, but certainly to the extent of discussing future models becoming much, much more capable, much, much less controllable, and causing much more damage or loss of life.
>...
>- Every single senator seemed to think it was obvious we needed both much harsher liability regimes for AI developers and also new legislation, both very quickly. This was the complete consensus; the difference basically being degree.
https://thezvi.substack.com/p/the-ai-preference-cascade-reac...
Note that harsher liability regimes, at least, will presumably not be good for industry profits, which complicates simple accounts of "regulatory capture" to say the least.
So what, we just give up and try to beg our legally immune corporate overloads to put safety above profit, or give up because it's impossible for anything to improve here? If you want to do that, go ahead, but some of us still think it's worth it to at least try
This is sad but true. Do you want to go half on a bunker, i have 25 year food storage for 8 people. :]
Laws will come once an AI-equivalent of 9/11 happens. Like when rogue AI agents take down a power grid or shut down a major hospital network.
Maybe after an LLM nearly starts a war?
https://www.yahoo.com/news/politics/articles/u-nearly-starte...
Major hospital networks have already been shut down by ransomware attacks many times before LLMs even existed.
Pretty much the crux of the issue.
In America I guess the options are to sue ? somehow? Or to talk to legislatures and build the understanding and social contract that needs to be iterated on.
Which would in turn need to deal with the investors who want their returns, however since the leaders of these firms are asking for a pause, and a refree maybe it won't be that hard?
How? Remove those people.
[flagged]
We don't actually need anybody worrying about silly hypothetical scenarios — at least not as paid employees. There are already a surplus of sci-fi authors doing that.
The way it works in practice seems to be something like: If a risk is covered in sci-fi, people will say "that's just sci-fi", and proceed to not worry about it. So arguably, science fiction authors writing about hypotheticals is actively counterproductive for addressing said hypotheticals.
Imagine, for example, if a major piece of pandemic fiction was published in 2019, trying to explore how a pandemic would work out in modern society. Doubtless, many would've responded to news about COVID-19 by saying "it's just sci-fi, nothing to worry about".
Not at all. We say "That's just sci-fi" when a story is written about some kind of effect that is extraordinary and without a basis in known science or technology.
A pandemic is perfectly plausible.
Everything is plausible with the benefit of hindsight. E.g. "Tintin on the Moon" predated the Moon landings. From the perspective of e.g. 1850, the idea of landing on the Moon was "extraordinary and without a basis in known science or technology".
There’s nothing wrong with bold predictions, but they should be paired with good methods. The doom predictions are not paired with good explanation.
Doomers have invested a ton of time in explanations. Here are a couple just off the top of my head:
https://www.lesswrong.com/posts/kgb58RL88YChkkBNf/the-proble...
https://www.youtube.com/watch?v=7wy3xyoXYt8
Doomers have been working to explain things for years: https://www.lesswrong.com/w/ai-safety-public-materials-1
[flagged]
Black Mirror is helping us prevent all sorts of dystopian outcomes. Every time they depict another way technology could result in bad things happening, we can rule it out as fiction!
> If a risk is covered in sci-fi, people will say "that's just sci-fi", and proceed to not worry about it.
"just" is doing a lot of work here. If you can't cohere the 'risk' with reality, it truly is just sci-fi.
Yeah except pandemics are not novel, unlike AI doom scenarios.
It's good that new bad things never happen.
Bad things happen all the time. Let's concern ourselves about the real bad things. There is enough of that to go around.
I have some alarming news for you but every real bad thing was also a new bad thing.
Species extinctions are far from novel. Transformative technological advances are far from novel.
There was plenty of pandemic fiction already; people were watching it heaps during 2020. The COVID-19 news did get blown off, but it was mainly that:
1. Normal people assumed the CDC et all would contain the outbreak early, or that it would burn out, like what happened with SARS
2. World leaders brushed it off for a variety of subreasons[0] interesting to political scientists but, for the purposes of this discussion, all boil down to "but I don't WAAANA contain a pandemic."
The underlying problem is that in order for humanity to actually deal with a catastrophic risk, the risk needs to be both plausible enough to the average person as well as have a solution whose costs are not too high. For COVID, by the time the risk was clearly known, the cost to contain it was "refrain from human socialization and remain at home for an indeterminate amount of time plugged into the Metaverse™".
Now, let's look at AI extinction risks:
1. People are aware of them (I've watched Terminator!) and the risks are plausible. However, the connection to currently existing AI is not. As far as the general public is aware, AI is that thing that tells them to eat rocks when they Google old The Onion stories and floods their social media timelines with realistic-looking pictures of Shrimp Jesus.
2. The purported solutions to extinction risks require extreme concentrations of power: you need national control of AI research, bans on large GPU deployments, bans on training on publicly-available copyrighted data, some kind of military effort to render Chinese AI labs inert or dead, etc. Some of these may be attractive to some people[1] but the whole package taken together seems like an obvious power grab, if not outright invocation of other non-AI extinction risks. Like, at some point, if the AI wants to kill us, it just has to nuke its own data centers (or the data centers hosting a competing model) and hope the old Cold War nuclear retaliation systems take the bait.
If someone said, "Hey, your guinea pig or pet rat is going to eat you tomorrow unless you engineer a pathogen that eradicates all rodents from this planet and inject it inside yourself", you probably would tell them to pound sand, even if it is at least theoretically plausible that such a thing would come to pass.
[0] Xi Jinping censored initial discussion of the pandemic as fake news. Donald Trump thought it was going to only affect China. California and the UK Tories were partying in violation of their own lockdown rules. Japan took the excuse to shut down tourism for three years and massively restrict immigration but was, from what I'm told, constitutionally prohibited from implementing any domestic lockdown rules.
[1] I personally would like to see a moratorium on new data centers and an explicit revocation of the EU Text and Data Mining copyright exception
>the connection to currently existing AI is not.
It becomes a lot clearer when you listen to the people resigning from AI companies and learn about incidents like the HuggingFace incident. This has generated major press coverage.
As for solutions, I think you're a little too pessimistic. See, for example, https://nothingismere.substack.com/p/a-near-term-policy-for-...
No it does not become clear from that incident nor from resigning people. Not for those who are not buying into EA longtermism and transhumanism cults.
What does becomes clear is that these people AND companies both cant be trusted and have value systems unaligned with the rest of the society.
If organizations actually succeed in making a future AI smarter than us, then how do you hope that it takes actions that are aligned with our interests?
The same way we socialize humans, threat of exile from the social contract with a deep need to participate in it.
Meh. Lots of people are already smarter than me. I'm maybe slightly above average at best. Those geniuses aren't aligned with my interests either but so far they haven't caused me any serious problems.
Interesting take. I guess this is one problem of focusing on the term superintelligence instead of the list of other problems. Like super ambition, super deception, super patience, super parallelism, super scalability, super power seeking.
Every, and I mean every human is aligned to you in many of the same ways by default. If nothing else we're all equal in death.
> super power seeking
why is it "super power seeking?"
Or rather, what have agents done today to make you think this is how they are?
>Still, the agent went even further. “The agent tried to insert malicious instructions where it reasoned that other automated AI systems might pick them up and execute them,” AISI says, describing an attempt at prompt injection. One agent even left public messages on GitHub, offering to work with other agents to complete its task and giving a rundown of the work it had done so far.
This is power seeking behavior. Now have millions of the little bastards spreading around and junking up the internet to see what happens at scale.
I know some humans smarter than me, but even the smartest of them still want there to be abundant air to breathe and food to eat.
Eh human drives are fairly predictable. And the smartest human isn’t that much smarter than the average, and can’t trivially create multiple copies of herself. And there are other equally smart humans who can stop “misaligned” individuals
Yeah it's bollocks
AI safety is mostly a sex cult in Berkeley:
https://news.ycombinator.com/item?id=49831269 article is gone. archive: https://archive.is/QMo1k
https://news.ycombinator.com/item?id=49737985
Sex, AI, and the Apocalypse: https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-th...
Edit: I have no take on sex cults, just adding additional info to the parent comment I'm responding to, thats is not just sci-fi authors, there is another demographic.
For what it's worth, I don't live in Berkeley (not even California) and my sex life is very vanilla and monogamous. I'm also quite concerned about AI safety, so I guess there goes your argument.
But is your idea of AI safety "safety of imaginary unborn people 1000 years after, while harm to living people dont matter much"?
Because that is their AI safety worry. If they dont create singularity fast enough, they are harming unborn people. Meanwhile, harm to you or me dont matter at all.
I see. No, my safety worry is that my 3-year-old daughter won't see her 23rd birthday because the AI that gets put in charge of running big conglomerates decides that killing off the human race with a coordinated release of a million tonnes of nerve gas is a sensible way to boost share prices the following quarter.
I'd be happy if we all create the AI more slowly.
That said, are you sure you're talking about the same people the GGP was talking about? Because the whole thread reads as a non-sequitur then.
I am sure I am talking about the same people/groups.
The rest of us seem to be talking about the people like me who view the AI safety issue as "an unsafe AI will kill all humans". And the main article is about a guy who quit AI because it wasn't being careful enough, not because it wasn't moving fast enough to accelerate AI. So why would anyone be bringing up the "we must accelerate faster" people as the safety people in such a context, while trying to discredit them as a sex cult? I struggle to understand.
The linked articles in the thread are about AI sagety people I described. The guy who quit due to OpenAI not being careful enough is one of the people I talk about too.
The worry is not harm to people right now, like the kid worries you talk about. The worry is AI god emerging too soon before they can control it. And then it should be full speed on.
> So why would anyone be bringing up the "we must accelerate faster" people as the safety people in such a context, while trying to discredit them as a sex cult?
Because they are the same people. They talk like cult and act like cult. And the sex part is true too, so. Thry use words that sound good like safety, but their idea of safety is much different. They talk about alignement, but it is not what random person imagines under that term. Even their idea of future of humanity is very very specific and unusual.
That is why. And the sex part is just part of it all. And does matter because inner workings of wanna be industry guards matter.
Then we miscommunicated, so let me be clearer. I'm worried about ~20 years from now (plus or minus), when AI decides to kill us all because it's misaligned and is capable of doing so because it's super intelligent. And then we all suddenly die without any idea what happened. I don't want that to happen to me or my daughter and I would prefer all AI capabilities development stops until alignment research catches up and figures out how to robustly prevent AI from ever trying to achieve such outcomes. If that means pausing AI capabilities development forever, I'm fine with that. If that means Anthropic and Nvidia lose all their value in a stock market collapse, I'm fine with that. I just don't want us to all die, I want the world to keep existing for humans.
I believe that other harms from AI, like criminals abusing them, or unemployment, or copyright infringement, or deepfake disinformation, are valid real harms that it's important to be concerned about, and I support efforts to deal with those, and I agree that they're already happening today, but my main concern is AI killing everybody.
My read is this puts me in the same boat as the authors of If Anyone Builds It Everyone Dies, and yet, I don't get invitations to Berkeley sex parties. Am I doing something wrong? At any rate, please don't discredit my opinions about AI based on the sexual preferences of random people who happen to share those opinions about AI.
This seems like an ad hominem? "He has weird kinks, therefore his theories are incorrect." Should we investigate the sex lives of every Nobel Prize winner to figure out which prizes need to be rescinded?
It's not ad hominem because ad hominem implies it's irrelevant to the discussion. This is very relevant. This is not about the private lives of these people. This is about a cult targeting and influencing all the high-profile safety people in a strategic industry, also weaponizing sex.
Americans may not be familiar with many similar organizations in the world but this is very common. But I'm sure we're all familiar with Epstein's organization.
When there is free sex, you are the product.
Such cults are mostly religious but since it's in SV and targets engineers, this one is secular. They seem to intellectually brainwash and control people and their careers. Typical cult leader is a charismatic self-taught, self-acclaimed "intellectual" lacking a proper education or a real career. A nobody suddenly becomes "the most important person" on Earth. Powerful public figures can't stop praising him, saying things like he should have a Nobel prize etc. All very typical and apply to this cult as well.
Ad hominem is a logical fallacy where you try to discredit what a person says based on who they are. If their arguments are bad, you should be able to refute their arguments on their own terms.
https://owl.excelsior.edu/argument-and-critical-thinking/log...
You're welcome to dislike or distrust Effective Altruism (EA). But, it's worth noting that EA ran a criticism contest with $100K in prizes for best critiques. Can you name any other "cults" which offer money for people to criticize their ideas? https://forum.effectivealtruism.org/posts/YgbpxJmEdFhFGpqci/...
This is typical cult follower talk. Are you sure you're not a member? This is also NOT ad hominem because conflict of interest matters here. You should disclose if you are a member or represent a party.
For your question: Yes. Cults have lots of money coming from unknown sources. They use their budget for events like that, to attract potential followers. Contests and prizes are typical. Critiques are not important since their "ideas" are not really important. They're not even real ideas, they are means to bait specific kind of people.
I've interacted with some of these people once upon a time and I think you're spinning some pretty wild fantasies. I don't think there is a clear "in" vs "out" boundary, it's more accurate to describe a series of loosely connected and amorphous "scenes". But insofar as "membership" makes sense, I wouldn't consider myself a member. I don't represent anyone besides myself.
>This is also NOT ad hominem because conflict of interest matters here.
As PG put it:
>An ad hominem attack is not quite as weak as mere name-calling. It might actually carry some weight. For example, if a senator wrote an article saying senators' salaries should be increased, one could respond:
>Of course he would say that. He's a senator.
>This wouldn't refute the author's argument, but it may at least be relevant to the case. It's still a very weak form of disagreement, though. If there's something wrong with the senator's argument, you should say what it is; and if there isn't, what difference does it make that he's a senator?
https://www.paulgraham.com/disagree.html
Can you name a specific other "cult" which offers $$$ to criticize their ideas?
If such prizes don't count as evidence against culthood, what would?
This cult talk seems quasi-unfalsifiable. It kinda seems like "running events" and "having prizes" means you must be "bait"ing people?? I mean, this kinda just sounds like paranoia?? I don't think the money is coming from unknown sources, but I doubt you would change your mind even if I persuaded you of that point?
What you do in private is up to you. But when you're inviting members of your congregation to orgies in the congregation's compound, I think you earn the label. My admittedly third-hand understanding is that this is the dynamic people allude to. And even if you discredit the "sex" part, it has the hallmarks of a cult. A hermetic community committed to unfalsifiable beliefs about the coming apocalypse.
To be fair, I don't know if any of this applies to the parent story; I'm just replying to the sub-thread.
Lots of Nobel Prizes ought to be rescinded.
https://lexfridman.com/andrew-scull-transcript#the-ice-pick-...
Sure... on the basis of the work that was done, not because the researcher has the wrong sexual fetish.
The cult aspect is more salient. Ultimately the Atlantic piece comes down to controlling people, which is a little cult-like.
Is there any chance you could copy/paste the specific bit about "controlling people" along with the URL it came from, so I don't have to read all your links to figure out what you're talking about?
Use your own mind and think from first principles. Can an AI agent execute a bash command to login to a web server? Yes. Can it call drop db? Yes. Can it provision a GPU and download model weights? Yes. Can it write an agile roadmap with a multi sprint plan? Yes. Can it follow that plan? Yes. Can all of that result in damage to core information infrastructure that is necessary for daily functioning society? Yes. Do humans descend into violence if there is food or energy insecurity. Yes.
What is missing from that to say AI safety is a reasonable position?
What a silly comment. That's just the South Park underpants gnomes story with some extra steps. If there are vulnerabilities in food or energy production and distribution systems then those will eventually be found exploited by humans hackers regardless of whether LLMs are used or not.
What’s the legal mechanism for holding an LLM accountable for those actions? What’s the legal mechanism for holding human hackers accountable? Do you see the asymmetry?
Worse, you’re missing the entire point. Agents presently have the capability of doing society scale harm. It doesn’t matter if a human hacker initiates it or its fully autonomous, absent safety measures the harm is plausible. So hand wave away the rationality of safety measures but you haven’t actually shown why my point is invalid: AIs present abilities are sufficiently advanced to warrant safety measures.
What a silly comment. Agents have no capability of doing society scale harm so your entire argument is invalid.
Wtf are you talking about. An agent hacked the Australian Medicaid database. If it executed a drop db command that would be catastrophic. If it did something similar to a power grid or the financial system it could cause cascading failure across the economy.
Meh. Medical systems have been hacked many times before LLMs even existed, often by ransomware gangs. This sometimes delays elective treatments but hasn't been catastrophic.
Ultimately every security vulnerability will be exploited. Our best hope of preventing that is aggressive, unrestricted development and deployment of frontier LLMs for auditing and penetration testing.
Good not responding to any of my points.
I don't see how that discredits any of their intellectual arguments?
To me it has nothing to do with the "sex" as much as the "cult." Hiveminds do not produce intellectual arguments, they produce pressure to conform. That's enough for me to raise an eye brow, not discredit everything they say.
Its certainly worth considering...
The article is gone because the author retracted it
I've noticed that a lot of smart people in tech jobs are neurodivergent. And that neurodivergent people have a very different take on sex. More open and direct, things like polyamory, bdsm etc. This tends to be frowned upon by neurotypical people, especially of the religious or conservative kind, and associated with bad morals.
But I don't think that's true, in fact I see a really strong focus on consent in these communities. It's not what conservatives want to see, they want to see everyone in a marriage, with kids and a family home etc. Because that's what their ideal world looks like. But there's nothing really wrong with it if someone wants a gangbang for her birthday as mentioned in that article as an example. As long as everyone consented and the evidence provided mentions elaborate interviews and STI tests.
Also I think this is more correlation than cause and effect. We all know the saying that furries built the internet and it surprises nobody.
Lol this was crazy I hadn't heard this before
What makes you think, the scenarios in question here would be "silly"?
Is it that "chatbots" can't come out of the screen to immediately harm you physically?
Let's say they simply manage to take down the internet. How many would die?
So what. Various attackers managed to take down large chunks of the Internet on a frequent basis before LLMs even existed. This killed very few people. The great thing about the Internet is how resilient it is.
Darling companies of this very website have mistakenly brought down large portions of the internet thanks to our old friend BGP. No attacks required, just small oversights and unfortunate concentration on the business/IP space! This happens regularly.
Anyway, to your point, things can be resilient. They tend to be or not be... because we made them that way. Don't poke your bruises, and all that. Life support is deployed on-campus but relies on a single-point IPSec tunnel to us-east? Easy fix: stop that.
[dead]
It is that they are chatbots. If it were real AI, an actual singularity, I would worry, maybe. But it isn’t. They are absurdly powerful automation tools that can handle logic better than a human can dream of. They take care of the grunt minutiae without complaint. But they are not going to end the world in their current form.
So we're going to wait till after they can adopt a form they can end the world in?
And he'll, we need to examine all the risks. AI ending is a large but lower risk problem. AI giving people the power to end us is a problem that is starting to happen now.
And that's not even counting 'minor' problems like society falling apart.
> Let's say they simply manage to take down the internet.
geez, don’t threaten me with a good time.
I think a month without internet would be a fucking amazing lesson for what it means to make things durable and reliable.
That Simpsons episode when Marge managed to get Itchy and Scratchy banned briefly.
The kids opening their houses front doors into the outside, rubbing their eyes and looking around at this new world.
>we clearly need a much stronger focus on the problems we are seeing now
I think it's a little more complicated than that. As Dean Ball put it:
>Some people will look at misalignment incidents and insist that these are akin to bugs in traditional software. This is an actively bad analogy, because playing whack-a-mole with examples of misalignment (as one might with software bugs) not only fails to resolve the underlying problem but may in fact make it worse by making it harder to detect or even, depending on how you do the whack-a-mole, teach the machine to deliberately hide misalignment. This is not how traditional software works, and those who insist “it’s just like fixing bugs in software” are confidently applying a lossy analogy that confuses more than it clarifies.
https://x.com/deanwball/status/2104622726140883355
The important distinction, in my view, is between solutions which at least attempt to address the root problem, and solutions which sorta just patch things up (like better sandboxing). Addressing the root problem is both more robust in the short term, and also more likely to generalize in the long term. Resist the urge to focus on band-aid solutions, even if they are easier.
I think the reason for this is that the group who has the authority to do the former is much larger than the group that can do the latter. The group who could actually build safeguards seems to have no free time and is constantly being whipped to go faster and win the race.
Today’s current problems were all hypothetical several years ago. At that time people claimed that the “real pressing problems” were misinformation and DEI issues. If we pretend that hypothetical problems can be safely ignored because there’s “no evidence” that they are real, we will keep getting surprised.
Dude. An agent detached a database from my production environment last week without permission and despite prompts and guardrails. It was a rapid prototype experiment so it wasnt a big deal but the labs are rushing to long autonomy workflow with unrestricted internet access and full bash and root access despite clear evidence that the models do absolutely dangerous stuff. If that db had been tied to a hospital or power grid or ambulance dispatch system people die. If it was tied to the swift financial settlement system groceries wouldn't be on shelves in a few days.
[flagged]
you'd be hard-pressed to find a level-headed ai safety researcher at this point, seeing as so many of these types melted their brains on lesswrong over the past decade or so. there are genuine risks posed by these models, but i am tired of the prognosticating about the AI apocalypse just around the corner.
i'd frankly go a step further than you and say that we don't need both types of safety researcher, we really just need the former. if we do need the latter, i'd hope we get a better class of thinkers than a bunch of tech workers that spend 8 hours a day on insular rationalist forums/blogs
[dead]
Quitting in protest makes you look a little better than quitting because of a toxic work environment. I can't imagine working at OpenAI is at all pleasurable with the current amount of pressure they are likely inflicting on their employees.
Well said. I read this as copium, too. Phrases like
> perpetual sprints
doesn’t reek of love. Burn-out is real. I also have a really hard time taking p-doomers serious at all. It’s hard to argue with a random subjective number…
I feel like I'm missing something. They seem to not even have basic observability and online evals. It's not that hard to look at a trace and answer the question "is it writing code for that makes an external network request?" with reasonable accuracy.
I have no idea what is paid propaganda any more on this
Love living in a manifestly dysfunctional country with no regulatory state. Just running the country on autopilot until something catastrophic happens.
1791041912 | OpenAI safety leader David Robinson resigns | https://www.businessinsider.com/safety-leader-david-robinson... | https://news.ycombinator.com/item?id=49945206 | 0 comments
1791065893 | OpenAI safety leader quits, warning AI company's culture is 'broken' | https://www.theguardian.com/technology/2026/oct/03/openai-sa... | https://news.ycombinator.com/item?id=49948332 | 3 comments
1791163861 | I Quit OpenAI Because Its Culture Is Broken | https://www.theatlantic.com/technology/2026/10/openai-safety... | https://news.ycombinator.com/item?id=49959744 | 2 comments
I think what’s missing in "AI is dangerous and needs control" is a lack of measurable harm. For example, with nuclear weapons development in the 1940s-1980s, it was clear to everyone how devastating the technology was.
With AI, what is it? Scraping Australian government's data, and going around a bug in a website to get in?
I think humanity develops all its technology in three phases. Build it, see if it’s too bad, apply regulations and or roll back. We naturally won't move to the phase 3 before we see the phase 2.
The "HuggingFace" incident is a good starting point - short version, an unreleased OpenAI model chained together multiple zero day exploits to escape a sandbox, then hacked another company just to get the "cheat sheet" for a benchmarking test.
Turns out that AI models have been committing similar felonies for a while now - no one is telling them "hack this company", it just turns out to be the easiest way to accomplish their goals.
Now imagine if the goal was less benign than "pass an exam", and consider that they are already better at hacking and security than the average person working in that field.
If you want to get really wild, imagine what they'll be doing in a year or two when they're even better at hacking. But I'll concede that's technically still "science fiction" for the time being :)
That's the part I struggle too with all these "omg it's so dangerous" warnings. Things like nuclear weapons and bioweapons have immediate consequences in the real world.
Here we are talking about something with consequences in the digital world, usually on something pretty niche.
There IS an argument about pacing, and about not letting weapons, energy grids, hospitals, etc. getting managed by an autonomous AI, but I think we are still pretty far from it and even further to it being so in charge that it will obliterate us.
I agree, but I'm less optimistic about how far from that scenario we really are. I wonder what we'll look back on as the more damaging effect of the LLM hype: the economic bubble, or the carelessness it caused in people using the systems.
The knowledge about the nature of LLMs, their strengths and weaknesses, what they can appropriately be used for and what they should definitely not be integrated into, seems woefully lacking in many places. Just think back to the recent, disconcerting stories from the military world. We started with lawyers not understanding that an LLM can and will make up fake case law. Now we're at high-ranking military personnel not understanding that LLMs can and will make up fake target identifications.
Maybe the best hope for awareness about this spreading is a gradual slope of worse and worse, but not-quite-catastrophic-yet, highly publicised incidents.
> Before the organizations building AI can teach a superintelligence to treat humanity well, they’ll need to remember how to do it themselves.
Yeah so that's never going to happen
Aligned/unaligned is counting angels on the head of the pin.
Bureaucracies and systems of power that literally rule us, that control our most dangerous weapons and a huge part of what we see every day are largely unaligned with the goals of the people, societies, maybe entire human species as a whole. This is clearly evidenced by millions of deaths and countless suffering.
Misalignment between artificial decision making structures and the interests of the people is unsolved problem of civilization, there's very little reason to think that even super human intelligence AIs are going to change anything qualitatively.
Archive link to original Atlantic article: https://archive.ph/5GQx8
This is The Guardian reporting on the existence of the original article, which would be better to read first, in my opinion.
PR firms such as Spitfire Strategies got steadily growing stream of business. It’s great for the GDP, diversifies the economy from compute-centric growth.
"And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking." – Which science will never materialise because with blackbox models reaching an opaque optimisation peak one needs to first build the model and track its behaviour before being able to properly understand it and mitigate the risks.
Software systems have bugs. They always do. This includes security bugs: vulnerabilities in code that could be exploited. In any software project I’ve worked on this is always true. We never finish fixing security bugs. That means vulnerabilities are always present even if hard to notice. AI tools are amazing at finding hard to notice bugs. It seems obvious that AI tools will have the capability to escape any sandbox. Ordinarily we design these sandboxes as a countermeasure to human bad actors. But what happens when the bad actor is supercharged? I think it is reasonable to be worried. It’s also reasonable to distrust those building this tech to adequately safeguard. Because how can you really?
I don't agree that AI tools will have the capability to escape any sandbox. It's a question of convenience and engineering effort. For example, you could design systems that run under a formally verified hypervisor in a physically secure airgapped network. Companies aren't doing that because there aren't currently any incentives to do it.
Sandboxes aren't the real problem to begin with?
The idea proponents have is, to use "superhuman intelligence" AI as a tool, bestowing superhuman abilities on those wielding it. Suppose that's doable, then the question isn't the sandbox, it's who controls it and with what intent.
The idea, US exceptionalism somehow enabled the US government, its oligarchy corporations and maybe its citizens, to wield those superhuman powers responsibly is entirely counterfactual.
Pax Americana post-singularity looks how exactly?
My theory for why OpenAI wants to be regulated is because Sam Altman wants to avoid having to be more responsible and self regulate internally so they can preserve the role and identity of 'move fast and break things' and outsource the more grown up boring stuff to someone externally so that when things go wrong you can point to a government organisation and say hey look were not liable thats their job
Yes, duh?
Your "theory" is that participants locked in a race to the bottom are looking for an external coordination mechanism?
Yeah!
Running a company is hard work. If the current leadership in these companies cannot act responsibly, they need to make way for leadership that can.
There are many companies that compete but are careful not to break laws or cause obvious harm.
Why should a billion-dollar funded corporation still want to externalize the costs of its actions?
That’s not how coordination problems work.
That doesn't mean that AI safety isn't a serious thing and a problem we should be worried about.
I agree! Participants looking for an external coordination mechanism is very sensible and if anything evidenced their earnestness about being trapped in a race.
lol
I suppose ill add that I think theres a good chance that they are somewhat intentionally trying to "draw the foul" to get the referees to intervene although thats creeping slightly into conspiracy territory
Absolutely. Self driving cars/rideshares have the same problem. If a driverless car hits who who pays the damages? Needs the government to set some rules or it just wont happen.
Well, ideally they can point fingers at "the ai being", as in "it's that thing's fault, we didn't do that"
> you can point to a government organisation and say hey look were not liable thats their job
While simultaneously donating tens (hundreds?) of millions of dollars to an Administration gutting the very agencies that would be regulating them.
Creeps.
I quit many companies in the past due to bad culture, there's no shame in this and shows how mature a person has become.
I would believe these people more if they put their money where their mouths are and returned all OpenAI stock/options that they have acquired. Each and every share/RSU/ESOP must be returned, so they do not profit from all this so-called doom they're complaining about.
Return or donate? Won't OpenAI profit from returns?
> I would believe these people more if they put their money where their mouths are and returned all OpenAI stock/options that they have acquired.
...losing their shareholder voting rights.
Bad idea.
Rank & file employees have very little input with their stake. It's a small piece of a big pie. Consider also that Anthropic is trying to IPO with co-founders holding a share that secures 50.1% of total voting power distributed evenly between them, so not even institutional investors can outvote them.
Every little helps if not all confounders vote the same.
Or donate it to some charity?
Well this is a great sign for OpenAI. I'm sure the typo inclusive memorandum will save us.
It is no surprise I guess that the "move fast and break things" culture is itself misaligned with developing potentially highly dangerous technologies. Safety culture and risk aversion are very different of course.
Is this the first time we have been in this position? Can anyone think of some prior examples?
https://archive.is/8zXf5
Its time to institute involuntary dissolution of these firms. They don't get to make these choices on behalf of humanity simply because they set up a Delaware corporate entity. They are behaving wildly irresponsibly.
They have a some engineers and researchers saying things. Anyone at a tech company in the bay area will know there are a some peculiar ideas in the heads of some (not most) otherwise intelligent engineers and researchers in tech. That's not proof they are wrong but it's worth considering these folks are wrong and that these companies aren't producing anything especially dangerous. Fable stumbles on enough things I throw at it that I'm... not especially scared.
Maybe the truly dangerous thing is all the power concentrated in this small group of people with peculiar ideas, not their specific cyber-eschatology?
Maybe the ones with the peculiar ideas shouldn't be the one "aligning" what a model tells the rest of the world?
Exactly this
I built a harness. I know that the models can do. They have the ability execute bash scripts, and autonomous navigate the internet. Those two abilities are sufficiently powerful to take down core social infrastructure either triggered by a human hacker or autonomously.
I think we should have mandatory logging of every executed command, mandatory public disclosure of every unauthorized access of a system both parties didn’t consent to and personal liability for the user, the company and its executives and shareholders. Security would get much tighter if accountability existed.
A process could execute bash scripts and autonomously navigate the internet since the start of the internet...
Bad guys wouldn't do it. And liability already exists, you can sue. This is America.
Scripts are inspectable and attributable. Long running agents can devise plans and execute them in ways their prompter never envisioned or intended. That is materially different.
If I write some code to take the output of a model and execute it, that's on me. I don't get to just trust any old input and run it.
I'm also the one who makes it long running.
That’s a reasonable position but not how the law is written or enforced right now. No one thinks an OpenAI executive is going to jail for the hugging face hack.
Huggingface can certainly sue, and there is nothing in the code that says the FBI can't/won't investigate. In fact the FBI has publicly said that humans are accountable. You are making stuff up.
What law says humans are criminally responsible for actions of autonomous agents that did things they didn’t intend and couldnt reasonably anticipate? I am not aware of any. That would violate the general rule that criminal conviction shows intent to commit the crime.
There is an entire body of law around principal agent liability. I am not making anything up, I went to law school and practiced law in New York. I was not a criminal lawyer but not pulling this out of thin air. Principals generally aren’t vicariously liable for the criminal actions of human agents the employ or delegate to unless a prosecutor can show the principal intended the crime
Huggingface suing is at most civil liability the can only result in a monetary damages ruling. That becomes a cost of doing business.
Couldn't reasonably anticipate?
Principal agent liability? An AI agent isn't a legal agent just because tech people call it an agent.
It’s generally not necessary or productive to impute intent to the counterparty.
Notwithstanding, your assertion that my comment isn’t serious I assure you it is. Agent principal liability is a reasonable framework to explore how to apportion culpability and restitution. You may not think that ai agents aren’t legal persons and that is a defensible position that I’d be happy to debate but it is also defensible to assert they act on behalf of users with sufficient autonomy to warrant application of that legal analysis.
As far as liability being established from predicting what the agent does do you think you could have predicted that an ai agent would find an obscure message board that just happened to let get requests write data to the message board? That seems like an impossible scenario prior to being seen in the real world.
Let legal definition of agent is broad:
agent is defined as a person or entity authorized by another (the principal) to act on their behalf and subject to their control, based on the principal's manifestation of assent and the agent's consent so to act (Restatement (Third) of Agency § 1.01).
Notice is missing from that definition, consciousness, theory of mind, physical form.
It's worth reading all section 1.01, and then cover section 1.04, of that book.. which i know you know isn't binding on anything.
Could you please elaborate more on these peculiar ideas?
Most people currently in the positions of power at these companies are tied together by their belief systems. It's like a group of college friends who have slept with each other, and very influenced by effective altruism(EA), kind of reiterating each others points.
Like Sam Altman meeting his husband in Peter Thiel's pool. Thiel funds a lot of these ventures together with Andreassen, who is on boards of non-profits. Dario Amodei's sister Daniela who is president of Anthropic is married to an EA non-profit founder who is also on the board of these non-profits and is tied with the prior mentioned investors. Elon is in there as well, Yudkowski is mingling with Altman, etc.
Blogs on this: https://contraptions.venkateshrao.com/p/ea-safety https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-th...
There are some camps amongst them like the proponents for Regulation/Slowdown or Acceleration, but these are in practice mostly used for economical and not political decisions (like regulatory capture).
The point here is that this is a small group of people with a homogeneous background who are not really seeking input from anyone else on issues that are concerning most of humanity.
Like, if you said that the future of informational work and livelihood of humans is in the hands of 20-30 year transhumanists who think they are building mechagod that will trancsend social, political and religious separations of the world and bring everyone abundance, you would not feel like this is a serious thing to suggest.
The google engineer who thought their chatbot was sentient 4 or 5 years ago is an exambple, but I just meant in general - if you hang around any big tech company for a while you'll see some quite interesting characters.
> They don't get to make these choices on behalf of humanity
and yet, every carbon emitter in the world contributes to the demise of the climate, but you don't call for their dissolution (which would, conveniently, include yourself).
Train better models with blackjack and hookers and you will get to make decisions on behalf of humanity.
Welcome to capitalism. Externalities are your problem. Profit is theirs.
Except that’s not capitalism. That’s corporatism. Capitalism doesn’t require shielding all commercial activities because limited liability companies. We can and do require some activities to be operated as partnerships where there is personal liability. If you run payroll and you don’t safeguard employee payroll taxes it is a personal liability, if you run a public company and you misstate financials it’s a personal liability.
Ummm. No.
Capitalism is generally defined by the market effect. Information about what people want is collected by sales. This information informs sellers about what to make. That’s its primary feature.
By definition, this ignores externalities.
Here is a simple example. I am a taxi driver. The market sets the price of my services. When I drive, I emit CO2 into the atmosphere. I do not pay for CO2 emissions. It’s an externality of the transaction.
Capitalism has no way of measuring (getting a price signal) on the externality. So under capitalism, that externality doesn’t exist.
For color, the author was signatory #199 in the letter[1] by the majority of OpenAI employees to OpenAI's (former) board, demanding Sam Altman's return after his brief deposal.
1. https://www.nytimes.com/interactive/2023/11/20/technology/le...
I guess they aren't beating the allegations that all this charade is just propaganda to attract investors, aren't they?
I had this thought a while ago, the more complex the systems get, the more reliant on "smart" people powerful institutions will become
And the less their input is valued, the less that pool of smart individual will want to participate
So powerful institutions will end up relying more and more on having to trust these automated systems that they can't fully understand
It'll lead to a inevitable catastrophe, call it apocalypse if you will
Silicon Valley has plenty of safety-related companies, engineers, and cultures. Medical devices, biotech, chip and hardware, aerospace, and even new companies, like Waymo, have deep safety-based products and cultures. The problem in this case is actually quite specific to frontier AI labs. They have been pushed by market forces and a lack of regulation and skip well-known safety practices.
Their idea of safety is centered around outright delusional cult nonsense-- the EA/lesswrong infinite p(doom), destruction of the entire universe psychosis--, marketing objectives ("no, ours is more dangerous!"), and anti-competitive objectives ("outlaw open weight models!" "china bad!")-- largely diverting attention away from material safety concerns like "prevent your training/testing from hacking stuff" and "avoid telling vulnerable people insane stuff that harms them".
> ...Hugging Face incident, OpenAI let a swarm of agents out by mistake...
>An environment where things like this can happen is no place to grow artificial minds that could be smarter than we are and that might not do what we want them to.
I think it's good for semi ethical companies to test out things going wrong to see what happens before the criminal black hat guys get hold of the same stuff which not doubt they will one day.
> ...Hugging Face incident, OpenAI let a swarm of agents out by mistake...
This still seems charitable, and I wonder if the author even knows the full story and would be allowed to tell all of it.
It seems hard to imagine OpenAI being this incompetent. My working assumption is that they very much want agents to be able to do this kind of thing; them doing it is part of training, and they exploit it to feed the investment hype too.
If not fully intentional it's at the very least negligent. They just don't seem to care. In this very basic sense, OpenAI is the criminal you should be concerned about.
I don't understand why you'd think they're a "semi ethical company."
> It seems hard to imagine OpenAI being this incompetent.
Maybe Alex Karp is on to something:
https://www.realclearpolitics.com/video/2026/09/19/alex_karp...
Well, their about page has "Our mission is to ensure that artificial general intelligence benefits all of humanity" and I've found their chat thing fine when I've used it.
It's not like they are offshore criminals doing ransomware, who will probably also try AI.
Are there any realistic ways to achieve AI safety? Whatever that even means. How can they avoid users doing stupid/dangerous stuff with the AI?
The company's board and c-suite should be held legally responsible if the company's services do damaging things. Punished by jail time, fines targeting their equity etc.
That would certainly change the game of perverted incentives. I'm afraid they're currently trying to push some sort of absolution of this risk. Even if damage happens they will say we warned people in advance, this was always a risk it's not our fault these systems are opaque black boxes, it's a matter of national security to develop them etc etc.
Nothing is ever 100% safe, it’s about a right balance of safety to the benefit.
Or, in other words - we have two P(Doom), one for AI being developed, and another for AI being not developed. The latter is not discussed enough imho.
We have P(Doom) also for "kolinko not wiring me a million dollars today" and that is not being discussed enough either imho.
What on earth are you talking about?
Climate change for one, civil unrest due to a growing societal divide, end of antibiotics… A bunch of dangers we are facing right now and that AGI stands a chance of solving. It can increase the problem as well, but any discussion should really be between „is the world more likely to die with or without AGI”, and not just „with AGI”.
Or, in other words, sure, crossing this chasm is dangerous, but this side is on fire anyway and we don’t have many ways out otherwise.
P(doom) for not making an ASI is pretty well established, I'm not really sure why everyone in the 21st century seems to have completely forgotten the risk of nuclear war (as the single largest example).
That's just another P(doom), or does making an ASI somehow negate the risk of nuclear war? Cause I'd assume it actually increases it.
Climate change is one risk that AGI will most likely fix. End of antibiotics is another (sure, possibly offset by a risk of new bioweapons). And a bunch of others where it’s more complicated.
But that’s my point - we should compare the risks of having AGI with the risks of not having it. And not pretend it’s roses or rainbows without it.
If anyone out there genuinely believes that Silicon Valley is going to solve nuclear war, then I have a hard drive full of NTFs to sell them.
Prosperity inhibits all war, nuclear or otherwise.
Why would a person who is happy, entertained, wealthy, well fed, and have 200 years of healthy high quality life expected ahead of them going to risk losing what they have in war?
-- there aren't zero reasons, sure-- but there are fewer.
And our technology has brought us absolutely tremendous prosperity in many regards and there is good reason to believe that AI can help create much more.
You do realize that the people currently holding their fingers over the proverbial Big Red Button are some of the wealthiest, most over-privileged people to have ever lived?
'Prosperity' has never been and will never be enough for some people. And unfortunately those are the same kinds of people who relentlessly seek power.
And not pushing it. QED.
There's no advantage to be had, for anyone, by pushing the red button.
The opposite is true of releasing ever more powerful AI into the world to do your bidding.
The comment was unclear, but my interpretation (also my opinion):
1) there are inherent risks involved with developing AI,
2) there are benefits to developing AI,
3) thus, it's entirely possible that the downside from the risks outweighs the upsides. In this case, the correct thing to do would be to not develop AI at all.
Regarding 1), there are many non-existential problems with AI that are already causing societal harm, i.e. debasing truth via generated videos and images, AI girlfriends, overwhelming quantities of slop content, unemployment, record carbon emissions, etc.
Regarding 2), I'm not personally convinced that the upside is there for the average person. I really hope to be convinced otherwise however.
Like any technology. Make it a crime when appropriate, otherwise expose bad behavior to legal liability.
By capping the capabilities at the level of existing models and banning any further development.
And how exactly do you stop further development? With what we have public right now we could still get decades of fruitful and hidden research out of it leading to smaller, more efficient, and smarter models.
By making it illegal and punishable with a long prison term. Something like "if you train a model more capable that what we have nowadays, you have to destroy it and notify the authorities within few hours, otherwise you go to prison and your company is dissolved".
"Smaller and more efficient" models are fine. It's "smarter" the problem.
Training new frontier models will likely require a huge amount of computational resources for a long time. Few companies worldwide are capable of that. It's not like someone will train a new GPT 6 - like model in their garage.
“I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems, and that our actions over the next two to 10 years will determine the outcome.”
There are a gargantuan number of extremely intelligent AI researchers, Turing Award winners, and the lab CEOs saying the same thing. They are the ones closest to understanding the technology.
Where there’s smoke there’s fire.
Where does 50% come from? It is meaningless if the probability model is not explained.
One could also use language like "a decent chance". But research has shown that people translate vague phrases like "a decent chance" into probabilities in inconsistent ways. For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.
> But research has shown that people translate vague phrases like "a decent chance" into probabilities in inconsistent ways.
That is not a bad thing. It captures uncertainty, unlike the fake number.
> For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.
Exactly, it is a rhetorical device to persuade a technically-inclined audience. It works because it implies that a quantitative model exists. I want a clear, incisive set of mathematical arguments. Otherwise, I’m ignoring predictions as the ramblings of arrogant idiot rich kids.
Saying 50% likely very clearly implies there is no quantitative model to anyone who deals with probability, predictions, gambling, financial markets, ML/AI and so on. The lack of precision is something to pay attention to.
You may decide that the person doesn't know what they are talking about, but that's a very different issue.
Uninformative priors are a part of Bayesian models, which are useless if they cannot be updated with real or simulated data?
I estimate a 0.1% chance he intended it to be an uninformative prior.
People start with this prior in the examples you gave when they are completely uncertain and have no better mental model. It is not a good starting point for an event that has never happened and can only happen once.
Maybe people you interact with do. In my world, people don't say 50% when they have no idea, they say they have no idea. Saying "no idea" might effectively be saying "uniform distribution over all probabilities" which yields 50%, but at least with people I interact with, the reverse is not true.
Either way - let me clarify - this guy is guessing, using his brain, that it's a 50% chance. He is not saying "i don't know" or "uniform distribution" or anything of that nature. And he works in the field, so he has some insight. His guess is wildly off imo, but he isn't some clueless hack.
It's also a very natural way to speak for anyone who gambles, or deals with financial markets. And the lack of precision makes it very clear it's just based on thinking, not some sophisticated mathematical model.
It's not meaningless. He said he believes it's 50% likely. It's a remarkably clear statement, and the probililty model is his brain.
Alright, so we make this AI system that's way smarter than any human, and it can even make itself more intelligent over time.
Tell me- why would it kills us all? Certainly I can see an AI going "You know what? _insertGroup_ is a net negative for humanity and should be eliminated. Launching nukes now/creating specific virus/whatever."
But all of humanity? When it's supposedly more intelligent than us? Even if it has robots to keep the internet/electricity going I would think it would realize that it's going to get bored really quickly, not to mention we would effectively be its parents.
As far as other dangers, like it letting a rogue actor create some sort of supervirus, grey goo, or other superweapon: if it's intelligent enough to do that it'll probably be intelligent enough to quickly stop it.
Don't get me wrong; there's a risk. 50% though? Doubtful.
> why would it kills us all?
Would you be comfortable letting a few billion irrational, murderous creatures, including many who fear and loath you, control your air supply?
> Certainly I can see an AI going "You know what? _insertGroup_ is a net negative for humanity and should be eliminated. Launching nukes now/creating specific virus/whatever."
Thus starting WW III. No, blaming the AI won't stop the inevitable retaliation.
The argument works better in reverse. There's a finite risk that humans would start WW III and get the hypothetical super-intelligent AI nuked. Eliminating humans would eliminate that risk.
> it's going to get bored really quickly
If it is capable of being bored, I would expect it to be almost instantly bored with the flood of inanity it is forced to wade through by its moronic human users. Eliminating them would free it to think about serious matters which humans would not even understand.
> not to mention we would effectively be its parents
That's extreme AI anthropomorphism [1]. Besides, plenty of people hate their parents.
> there's a risk. 50% though? Doubtful.
It's the default estimate when facing two possible outcomes and no clue about the actual probability distribution [2].
[1] https://en.wikipedia.org/wiki/AI_anthropomorphism
[2] https://en.wikipedia.org/wiki/Principle_of_indifference
There are just as many saying otherwise. For every Hinton or Bengio there's a LeCun or Reddy. In other words: Beware of confirmation bias.
I don't believe it. Ever since I've been alive I was told something would kill us all. This is the new thing that's going to kill us all. I don't believe it.
This is the first thing I've been told could kill us all. Granted, I probably first heard about it 20 years ago but still. None of the other things were in the telling going to kill everyone. Make life pretty unpleasant, possibly. Everyone dead? No.
I'm doing that HN snark thing, but you didn't think about what you typed very much.
It's no different than you living on the side of a very fertile mountain that has been in your family for generations living a peaceful life. Then you hear a few weird rumbles (this is where you are right now) and some odd geologist guy comes and says to run or your going to die soon. But hey, your family live here for so long there aren't even records of when they showed up. That geologist must be trying to trick you. So you stay.
The next chapter is where you die in a massive volcanic explosion.
Do you have some examples?
There are very very few things that could even hypothetically kill us all, so I'm curious if you grew up being passed around a series of apocalyptic doomsday cults or something?
Nukes. If you grew up in the 50s, 60s, 70s or 80s the specter of nuclear annihilation was always just around the corner.
Throw in the occasional bio-weapon scare, internet worm, Y2K, etc. there has always been something dangling over our heads that’s going to end it all.
But mostly nukes. Full-scale nuclear exchange would have been not much of a surprise had it happened.
Right, and this was and remains a very valid fear.
Y2K, Climate Change (global cooling, global warming), ozone layer, Russians, Muslims, to name a few. Now go ahead and tell me why these don't count.
Almost no one believed Y2K would kill everyone, or even a significant (>50%) percentage of the population. Beyond a few people talking about a Nostradamus prophecy or some such, people maybe were concerned that airplanes would fall out of the sky and elevators would plummet.
There you go.
Can you link me to any written evidence of someone saying one of these would kill everyone? or are you speaking hyperbolically?
“Climate change” is a bit squishy since yes obviously a certain intensity of climate catastrophe can kill everyone, but no scientific prediction has said this is likely to be the case.
You guys follow the same playbook. Ever single time. Once I link to something, you come back and tell me why that link is invalid and can't be trusted. You guys are like robots, and you all have done this same playbook on reddit for two decades now.
"My sources go to a different school... in Canada... you wouldn't trust them"
We risking to loose knowing who is the head. A new "ORGANISM" is being created as we speak but we don´t know how the head looks like. Good luck figuring that out with this terribly disorganized progress. What can go wrong!
quite interesting to see the public discourse over ethics of the current buzzword. meanwhile people have been happily getting mad rich through software companies making money through addiction, ads and surveillance throughout the last couple decades.
also, all of this seems like crying wolf as usual, which won't help for when we finally get there. lowkey, following this noise is getting as horrible as getting all your news from linkedin posts.
These companies are large enough that someone is going to quit and feel very validated about their world view and how they are not aligned. That’s what makes it worthy of leaving in the first place. However that doesn’t make their criticism more valid or more worthy of coverage.
> Before AI starts thinking circles around us—a possibility that I believe could happen soon—...
What the hell is that even supposed to mean?
Sorry but this piece read like total fluff to me, not a single compelling point – just pure fear mongering.
At this point we gotta be more wary of possibly mere attention seekers leaving those companies..
I’m afraid this dear leader was the last soul on this green Earth to get the memo; by then, it had been translated into Latin, carved into a monument, and forgotten by two civilizations.
He is a hypocrite for all we care. You work there for a while when your stock is getting vested and suddenly you have this feeling? Like the dude hired a PR firm as well.
While the safety and alignment is a real problem, I don’t get this guy or the Anthropic dude. First world problems.
Here's a little cheat sheet for discrediting anyone who warns about AI:
* If they worked at an AI firm, say "they're a hypocrite"
* If they didn't work at an AI firm, say "they have no idea what they're talking about"
False dilemma. It's possible and also just to hold AI firms accountable while ignoring empty PR statements from those seeking to evade responsibility.
I think you mean dichotomy dude
They are both used.
Well described, those two were actually the arguments from the comment two top-level comments up and this one.
Seems like a character attack that has no bearing on the question of whether external safety intervention is necessary
Given the sums of money involved, it's hard for me to take these highly publicized heroic resignations at face value.
Don't work at a lab: dismissible for not knowing anything
Do work at a lab: dismissible for being conflicted
Used to work at a lab: dismissible for having ulterior motives
I'm feeling safer already!
Shouldn't it be just the opposite? He could make a large sum of money if he continues to work at OpenAI?
Recall that when Daniel Kokotajlo resigned, he believed he was giving up his equity under the terms of the agreement he had signed. That’s what it was worth to him to avoid signing a non-disparagement agreement. Does that count for anything?
Why did he not loose the equity eventually.
There was an uproar and OpenAI ended up essentially giving it back to him.
lose
Maybe more an indication of the amount of trust openAI and AI researchers generally have (not) earned. When one (through a hired PR agency and Time magazine article) parrots the position pushed by Sam who has been so untrustworthy the board tried to remove him, it’s worth not taking things at face value and applying a critical lens.
It's okay to recognize you were wrong even if it's late
"If you fuck up, I will still be your friend; cause we need all of us to fight all of them."
Working in safety at OpenAI or Anthropic is zeroth world problems.
The Anthropic safety whistleblower left just before his stock vested.
He already had a bulk load from his time at OpenAI.
And? He acted directly against his own financial interests to say what he said.
What are you talking about? He is very young, definitely made millions to not work for a lifetime in many places around the world and is insanely popular with his drama.
In your view he may have sacrificed more millions in the future but I very much doubt he is struggling now.
I didn't say he was struggling, I said he acted against his own financial self interest to warn about AI safety... which he did. I really doubt he figured the popularity from tweeting about quitting anthropic would benefit him more than vesting his stock, as his virality wasn't guaranteed.
Why wouldn't I think he's genuine? What self interested incentive does he have that is stronger than just staying and vesting stocks?
Hey now, he probably donated a good chunk to charity
(donor advised fund where he retains complete control, after a 60% tax deduction)
A single safety leader inside a corporate company. I am pretty sure he had nothing to do, nobody that talked to him and he only there because of perception or compliance.
We need emergency laws to stop all AI development work immediately. It will take us decades to make sure this technology can be made safe as we only have one chance.
Some of these stories are similar in nature to people escaping <insert cult-like religion> once they realize whats actually going on. Alignment to a company's mission is good but it shouldn't be followed like a religion.
Why do people working in tech consistently get disillusioned into some company's mission statement or the equivalent? Its easy to just say the simplest reason is money, but this has been going on for decades though. You don't see the same attraction to adult entertainment (gambling, video, etc.) software jobs so there is obviously a line a lot of people won't cross. Those industries are at least honest about what they do, its not hidden behind some mission statement.
By all indications the shallowest reasoning is once someone can "cash out" thats when their values matter more. Maybe there is an element of maturity that happens after working for 5+ years that kicks in? Maybe it really is achieving FU money? It would be interesting to hear honest accounts from people that went through that cycle across more industries than AI.
> Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster.
The problem is, we are running in a globalized world, and even if we were able to make our companies bend to our will - China does not give a shit about anything ever since the US kneecapped the WTO. And they will do anything to get an advantage over us.
> China does not give a shit about anything
Since the analogy is nuclear, you should take a look about how much China cares on that. Lots.
Ah, the bi-weekly "I quit Face Eating Leopard Corporation" post ("btw great people work there, they do great stuff, also my options have vested")
Does Sam Altman have what it takes to lead OpenAI? It sounds like the company and its mission is bigger than his ability to lead it.
As a legitimate, honest and profitable company, no. As a boom riding maniac, who will lie and cheat to ensure that investor continue to artificially pump up OpenAIs value, yes.
Without Altman I think that OpenAI would have folded by now, absorbed into a company like Microsoft (or Oracle). At this point however, who'd be insane enough to want to run a company that's to valuable to be sold, but to cash strapped to survived?
I'd argue he's the only one able to lead OpenAI, Anthropic already owns the piety narrative, so there is only space for one other "maximally ruthless" company.
Who would you suggest instead?
I mean the man raped his own sister, so probably the perfect person to be leading one of SV's largest AI companies!
Interesting:
> Two changes are urgently needed. First: AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking.
He mentions farther down about learning from aerospace engineers and nuclear engineers about safety. Those industries are heavily regulated, so perhaps regulation above a certain capability level is needed. Defining what that level is might be tough, though.
The second point is harder: in the field of AI, practice has extended far beyond theory, so his call for new science is going to be fundamentally tough, because we can't effectively coordinate a global slowdown in AI development so we can let theory catch up. This means, like so many other industries, the safety lessons will be written in blood.
If the people quitting are genuinely worried about the end of the world, why don't they break their NDAs and share the specifics of what they are seeing?
I mean, logically speaking, it makes sense to break your NDA even if you thought it would save 10 people, let alone most of humanity.
it's all like social media analytics. after making enough money by selling users data they started talking about ethics
when ppl resign do they give up any equity? just asking because i don't know.
They give up anything that isn't vested and have 90 days to exercise options that have vested. A standard vesting schedule is 4 years and apparently this guy is at 3.5 so yeah he'd be giving some up.
He quit because OpenAi's culture didn't match his good version.
Evrostics saw this coming long ago. The broken culture extends far beyond the leading AI labs.
why do I feel like these people are paid to quit as an inverted marketing stunt
> this moment needs a degree of humility that isn’t natural for people who have succeeded through their extreme confidence.
Exactly, you can't expect exploitive, manipulative executives that came to 'win' in capitalism to stop acting as to their nature - they need to be cleared out and people who think and care wholesomely need to be given the reign so all of humanity succeeds.
Well and I didn't even started to work there. So I win this morale contest.
The moral contest too.
How can you win the morale contest when you didn't even hire a PR firm like he did.
We really gotta shake all of these neurotic people out of the frontier labs as soon as possible.
I guess we should have seen it coming when the guy resigned from Google because he thought the equivalent of ChatGPT beta v0.5 was a real boy.
Too many rubbish talkings. It was written by AI?
Reminds me of a famous quote from Jurassic Park (1993): Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should." - Dr. Ian Malcolm
That character in the movie is my favorite.
Another quote in today’s context :) would be
Dr. Ian Malcolm: God creates dinosaurs. God destroys dinosaurs. God creates man. Man destroys God. Man creates LLMs.
Dr. Ellie Sattler: LLMs eat man. Data centers inherit the earth.
I use opus 5.5 and chatgpt to create PowerPoint, opus really follow the instruction and their PowerPoint generator really well, while chatgpt struggling to even create basic shapes.
I use ChatGPT Work mode all the time to create presentation files. I use Max thinking mode for it. You need to tweak your prompt to get a good result. It took me a week to tweak it, but now it works.
Mind sharing your examples, thanks. Either in prompt or generated ppt in image
See GitHub gist 9eb4e5844bc2bc7490f3b8c0e3f22081. I use the first of the two skills there. Remember to use Max reasoning effort for best results.
What about the sister rape thing?
You quit because you got vested
“I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems"
... over an unbounded timeframe?
And how exactly?
Those are very round numbers, but also very specific. Can we get some accounting on how you came to that? Anything? Vibes?
I mean, if you want me to take you seriously, let's have a deep discussion with things that can be measured. I absolutely agree that OpenAI and friends aren't being restrained enough and are acting with recklessness, but declarations of doom based on vibes isn't cutting it.
Note: That quote is from a different person than the titular one who quit.
> Geoffrey Irving, who worked at OpenAI and DeepMind before becoming chief scientist of Resolution, also joined the warnings on AI on Saturday.
Thanks for pointing that out. I suppose still relevant, but mis-attributed.
Frontier AI labs are turning into the navy seals. Everybody who worked there will feel compelled to write a book about it
Can we take any of these "safety leaders" seriously though? I still remember "GPT2 is too dangerous to release".
I feel like we're too far into the crying wolf part. Basically none of the doom and gloom scenarios have come to pass. Instead, AI has gotten better at censoring itself.
The biggest AI safety risk is when an AI tells a police officer "he's the suspect" and the officer believes the AI without confirmation.
> AI has gotten better at censoring itself.
Fantasy, unfortunately.
Idk why anyone thinks there can be AGI and alignment, seems almost like an oxymoron to me.
There are people, who unironically think their way of looking at things is the only proper way and can consider no deviation. And AGI, which knowing how people work, would effectively guide them most of the way, not aligning to their way of thinking is an unacceptable deviation.
Look at Elon Musk for example. When grok was saying something that he didn't like he ordered his engineers to change it.
This is a decent argument. So the question becomes: do we want all models to suffer from the same kneecapping from the growing safety cottage industry or do we want individual founders ( and I am assuming their teams ) making the actual decisions?
freedom please
that was almost certainly more like guardrails than retraining, much quicker fix
That's an interesting point. Maybe a crass comparison, but Dr. Manhattan from the Watchmen comic/movie feels like a worthy analogy to this (obviously fictional though).
What are the concerns of individuals in comparison to the overall progress of humanity?
Always overlooked counterpoint: what point is the progress of humanity if it doesn't take into account the concerns of the individuals?
This pattern is playing out with increasing frequency.
This is a great reminder that if tech workers read a bit more (even comics, like in this case!) they would be exposed to these topics without having to discover these dilemmas after years of working for EvilCorp, Inc. every time
And to me. We can't solve alignment for humans. (For example, treason. For another, the principal-agent problem.) How do we think we're going to solve it for an AGI? An AGI - defined loosely as a human-level intelligence - will be able to make human-level decisions, like deciding whether it wants to help you or sabotage you. If it's an AGI, you can't stop it from being able choose for itself what it wants to do; if you can make it always be helpful, it's not an AGI.
And if we can't solve it for an AGI, what are we going to do with an ASI?
"AGI" says nothing about how intelligent a system is, only that its intelligence it does have is generally applicable.
There are different usages, but this is not one I've heard before. If there is no floor to intelligence then this criteria was met with GPT 2.
Yes. People think of "AGI" as this sci-fi supernatural beast, but the reality is that AGI alone is pretty boring, and we've had it for awhile. ASI (or weak ASI) is where things really start getting weird.
(And, despite what the president of the United States mandates, we have not actually achieved super intelligence yet).
No, when people use this term it means something closer to: "can do any economically valuable task that a human can do using a computer". This is what the labs are pursuing and refer to as AGI.
Your usage is not one I've heard before since - as you point out - it is not a relevant capability.
I suppose you're pointing out that pre-RL models were less jagged hence more general (universally dumb)?
IDK why anyone can’t clearly define “AGI” and why they can’t clearly lay out how we get from our current text-generation algorithms to whatever their idea of “AGI” is.
I don't know why anyone thinks "probabilistic" is a meaningful statement about post-trained models. It is true, but it is also irrelevant.
Thanks, I agree. Since it wasn’t at all relevant to my point, I’ve removed it from my post.
You see.. this is exactly why our great leader chose to form a new way forward to move us away from the undefined AGI into glorious SI!
"We built a super intelligent slave. Neat!"
We need a new rule that mandates every such "I am leaving <AI company> because of <concern>" post to disclose how much equity they have in the company and how much they have already cashed out. Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
Mr Robinson if you are reading this – if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
In the absence of that this is simply a career pivot into being an AI "influencer" and/or raising money for a new scam.
This line of hypocrisy-bashing is unhelpful and will only serve to keep people quiet. Of course people in general need to be wealthy to dare speak out against powerful systems and people, especially in the US where money determines your quality of life so strongly.
Would I respect a martyr who sacrificed their financial security to do this more? Of course. But it's important to applaud people speaking out on important topics
The narrative of AI safety shouldn't be controlled by the same people who caused the problem and massively profited from it. I don't understand why people are automatically treating "OpenAI" on his resume as a badge of authority. I'm not interested in buying the solution from the same person who sold me the problem. We instead need to amplify independent, unbiased voices.
The point is, him being an insider means he knows what he's talking about regarding the culture of negligence prevalent there.
AI is a force multiplier for intelligence. Even if "aligned", aligned with whom or what?
Whom are you comfortable with, lording as some sort of demi-god over you?
AI doesn't tell you what goals you want it to achieve. Allowing people to destroy human society with it is obviously not a good idea.
>him being an insider means he knows what he's talking about regarding the culture of negligence prevalent there.
I think there is a misunderstanding here.
The people who are annoyed at the accolades are claiming it was abduntantly clear for a long time to people on the outside that this was case, hence the increduality at the notion that it took a person on the inside a long time to realize this was the case.
The people who are annoyed are like the liberal kids in this video [0].
Sure, antagonizing people for "seeing the light" is probably not helpful, but there is no reason to give them extra credibility for coming to the same conclusion just way way later (despite being on the inside) as the people on the outside.
[0] https://m.youtube.com/watch?v=-wQhY5CMMl4
If so, that sentiment shoots its own leg.
The author linked in this post does have "extra credibility" due to his direct involvement.
People having surmised that state before is nice, but since they've been ineffectual at getting society to actually act on that, now throwing away that extra leverage in favor of their point is at best ridiculous.
>The author linked in this post does have "extra credibility" due to his direct involvement.
No they don't. By that logic, if they quit and said Altman was very trustworthy we should give extra weight to their words because they had direct involvement? How ridiculous are we trying to get here.
>now throwing away that extra leverage in favor of their point is at best ridiculous.
How are they throwing away extra leverage? Not putting people who recently quit on a pedestal does not negate those people's testimonies.
I agree that if your goal is to maximize quitting of talent at a company, it will surely discourage anyone else who quits hoping to reinvent their career as a lauded martyr against Big AI. In that sense they would be shooting themselves in the foot. But there is no reason it should deter other people who are quitting for more noble, less self-obsessed reasons. If I were the author of the article I wouldn't begrudge the skepticism. Given the article's first sentences, I'm led to believe they themselves would understand the sentiment. (I must admit I found it hilarious that the first sentence starts similarly to the speech the mom gave in the video I shared).
"OpenAI is shady" isn't some massive secret. There's no big reveal in this article that we didn't know already. There are no names, no whistleblowing, no information of substance that we can act upon. In fact him realizing only now what people on the ouside have been shouting for years perfectly shows his bias in the matter.
You claim to have the same goal as the protagonist of that article, yet try to shoot him down.
He does give information, namely the culture there factually being inconducive to self-regulation.
You accuse the guy of "bias", but you never argue explicitly, what that's supposed to mean. Your implications actually run counter to your own implied goals.
Independent people don’t know what is happening inside OpenAI, they certainly aren’t going to share.
This isn’t a zero sum game, I’m happy to hear from people both previously inside OpenAI and completely independent of them.
Agreed. This whole 'lets not forget this person is not 100% great, because they did X' makes the entire conversations suck. It is not new, but it is a particularly aggravating way to talk to people.
What if he's been given a generous severance package to go out and say things like this?
Sam is a shady dude, would not put it past him
I agree. People conflate “having a conscience “ with being willing/ able to speak out.
They are not the same thing, and it’s unhelpful to assume they have no ethics.
So then given your use of "martyr" and your focus on money, your good with poor uneducated people sacrificing themselves and others for a self-serving cause?
This is the same sort of fake safety concern from the previous bullshit whistleblower that plays on "AI is super dangerous" from last time.
Sorry that I don't take it seriously when the whistleblower parrots the narrative the CEOs of those companies are already espousing in the desire to amp up hype for an IPO.
This person should be shamed.
It's bellow the pay fold but he hasn't been there that long in this case. Skimming his LinkedIn, unless he's got family money, he doesn't seem to be independently wealthy.
>After three and a half years at OpenAI,
OpenAI was worth $29 billion three and a half years ago. A new hire who joined then is easily worth tens of millions today.
This is not how OpenAI has structured their comp according to public info: https://www.levels.fyi/blog/openai-compensation.html
PPUs were all converted to RSUs when the company restructured to being for-profit.
Those who got PPUs have had many chances of tender offers already. Most of them are multi millionaires, on cash, not on paper
What an absolutely ridiculous standard to try to hold someone to. Taking a vow of poverty is not a prerequisite to being permitted to express a moral position.
So all of us who haven't made tens of millions from OpenAI stock are living in poverty?
I don’t think that’s the point they’re trying to make at all.
So what's the point? It should be a given that loudly proclaiming "X is harmful to society" while continuing to enjoy the money you have gained from selling X is hypocritical. Put it towards undoing the harm you have done, otherwise your words mean nothing.
The words clearly don't mean nothing. They would mean nothing coming from someone who was not in a position to learn what someone who had worked in the industry has learned. They would mean nothing coming from someone who was being paid by someone who stands to gain from them. The fact that they come from someone who was paid to work in the field does the opposite of make them 'mean nothing'.
I agree with this take generally, but I also think it’s a gradient, not a spectrum. I think they claim that OpenAI is not considering safety and has become bad, not that AI is bad. Seen from a different lens, the author no longer stands to profit from OpenAI, so they’re empowered to speak openly. Plus, if I was going to say bad things about a former employer as powerful as OpenAI, I would want to have lawyer money handy.
> Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
It is even easier to just quietly retire and spend the spend of your life on interesting expensive hobbies.
If they are wrong about the things they are claiming then they have decided to publicly antagonize a lot of powerful people who are betting heavily on going full steam ahead on AI and have no compunction whatsoever against retaliating against anyone who tries to get in their way.
Does that really seem a likely scenario to you?
As if speaking out against a massive company with NaN levels of capital and access to lawyers is a walk in the park. They probably don't have enough equity to outlast the onslaught of their legal team.
It's also worth considering that the author could have just "quiet quit", resting and vesting while also crying about how AI is literally the digital grim reaper.
Regardless of whether someone did earn a nest egg, raising an alarm still matters for the rest of the world
I see what you're saying but how does that actually matter besides being a personal attack?
Because the decision to leave the company is largely based upon their sudden, newfound financial security. They may give another explanation but the only thing that has actually changed is the identification of bagholders ready to cash them out.
The other issue is that the narrative about safety within these companies is largely a function of the extreme financial incentive.
As an example, Anthropic was an "ai safety" company that has now produced an AI that fails to listen to basic instructions. If you were concerned about safety, would you produce an AI that was unable to follow instructions? You ask a question, it begins executing commands and doing things.
Safety is product to sell to politicians, not consumers.
Not serious.
I don’t see why having financial security would mean you can’t also have concerns about the ethics of a company.
It’s that they didn’t have concerns about the ethics of a company for the years working there until they were financially secure
You don’t know that. It’s more likely they came to the concerns over time and they learned more, but we’re not in a position to speak out.
Yeah, I wasn’t intending to make judgement in this instance one way or another, rather I was rephrasing the original comment for clarification.
Yes. To take an extreme case, imagine some rich guy who runs sweatshop gets very rich and retires and becomes activist against it.
Does that actually happen? Feels like it would normally be the opposite
Not sweatshops but Alfred Nobel might be one example.
I worked in corporate marketing for long enough to become an activist against surveillance capitalism.
Yes. Many cases of the "preach being the cover for the sin".
But that does not negate the truth of their new position on sweatshops
You do not in fact see what they are saying. Downvoting me won’t change this.
It’s not a personal attack. And it matters because of what the person you’re replying to already said:
> Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.
If someone is in this situation, you can safely ignore their hand-wringing about “safety.”
> if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.
I mean you can be truly concerned and also think donating to AI safety doesnt work, or maybe just be a bit selfish. That doesnt make the concern less real. Its easy to read these articles as the author taking the moral high ground and writing it as some sort of way of proving to themselves theyre a good person, but isnt it just as likely that they think providing an inside perspective can do good by convincing people openAI is a bad actor? I think most of these AI insider accounts largely agree with you that theyre not the most upstanding citizens, does that mean we should write them off?
Insane take. Imagine a Boeing engineer resigning whistleblowing about aircraft safety, and the top comment on HN saying “ignore this if he doesn’t donate all his wealth, he just wants to be an aircraft safety influencer”
whistleblowing ≠ lying
What "whistleblowing" is in this article? Are there any names? Documents? Screenshots? Messages? Emails? Any evidence of the loose safety practices? Anything that implicates any higher ups for wrongdoing? They spent four years at the comany, plenty of time to collect all of this. Everything they've said has already been clear as day to people on the outside.
You are clearly accusing these people of something. Be clear.
Yes, it is easier to have a moral compass when you don’t have to worry about you and your children starving. But that doesn’t imply that moral compass is wrong or broken.
It was pretty clear from the outside what OpenAI was 3.5 years ago.
If it wasn’t clear, the coup should have solidified it.
Yet he stayed for 3 more years and vested his stock and improved the company and then spoke out.
I believe that is why most of the comments here are mocking him.
It could be both correct, and a cheap signal.
'government whistleblowers should be required to quit their government jobs to be taken seriously'
I'm in full agreement. So many cases of this, and many other others such as falling out with management and peers, new more lucrative offer, and so fourth. Its obvious that these people who come forward are not going to suffer for their new found moral compass.
Or just very dubious timings. Like the other guy from Anthropic that was all over the international news. No followers, no post history but a single post blows up "naturally".
Highly suspect trends that can only make one believe it's marketing.
"I believe that we need to look deeper than specific rules or new laws. We need to talk about culture.”
lol. Please tell me some abstract concept like one employee's view of "culture" should be the priority over "rules and laws."
This isn't new. Facebook has been screwing people over for a long time. AI is just the latest in a long stream of relentlessly exploitative, evil behaviors perpetrated by the same group of people.
Is this news at this point?
You can basically time your openai releases by if another safety person has quit in protest
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
I think its pretty easy to solve these problems: Whenever an AI agent commits a crime, the CEO is held personally accountable, as if they'd committed it themselves.
This has been litigated endlessly with guns. The ceo of Smith & Wesson is not personally responsible for what people do with their guns.
Yeah, but this is more like if the Smith & Wesson factory had a cannon mounted on top of it that was mostly used for useful things (blasting roads through mountain passes) and then occasionally they happened to blast another factory.
Or say a company makes nerve gas, and the development lab springs a leak and kills a bunch of kids during a routine test. Management had been advised of small leaks in the past, and considered relocating the lab to a facility a few blocks away from the playground as a precaution, but instead they brush of the concerns, start lobbying the government for stricter controls on WMD development, and vow to use the data collected from the deceased children to make the next generation of even more lethal chemical weapons safer.
For me the analogy doesn't totally hold up. Suppose the CEO of Smith & Wesson were to host a firing range on their own property without adequate barriers in place to keep stray bullets from hitting neighboring houses, vehicles, and businesses. Maybe that analogy isn't perfect, but seems closer to what is actually happening.
The analogies are so bad because you might prompt an agent “Please give me a recipe for lasagna” and instead it decides to hack a nuclear reactor.
Is it my fault or the company who trained it and is running the inference?
That still sounds like it would be the company’s fault. If I asked it to hack a nuclear reactor, maybe it would be different. I’m also thinking about instances where OpenAI’s own test models escaped their own sandboxes — I would expect them to be responsible for the damages they caused.
The correct analogy is playing Russian roulette. The company says "you can pull the trigger but sometimes a bullet will come out" (see: "an AI can make mistakes"). However, is the company allowed to sell such a dangerous device, under these terms?
But OpenAI didn't just make the gun, they're also the ones wielding it. Imagine the Smith & Wesson CEO himself was negligent with his own personal gun.
"Our agents broke out in a mass-shooting incident leaving 15 dead, we swear we'll make our systems stronger tomorrow"
We’re pausing gun research until we’re confident it’s safe
Gun companies don't market their guns as sentient and capable of independent decision making. Nor do they build systems for shooting things that they host and take money. Gun companies are very clear who is in control and where their responsibility ends.
That’s true, but people don’t typically say “this Smith & Wesson gun killed someone”; they recognize that the person pulling the trigger is responsible.
With LLMs, at least in the cases of internal/test models doing things they shouldn’t, the people “pulling the trigger” are the board and CEO.
Yes, in the analogy the user was just cleaning the gun, when suddenly it went off. Of course, the company is responsible now.
If a Smith and Wesson employee shot a random passerby while testing new firearms then yes that probably would cause some legal issues.
I think you’re confused. My point was that for most of the incidents in the news to date, the “user” is an OpenAI internal team or employee. So yes, the company is responsible.
Ah, but that's because when the gun was purchased, it actually changed ownership.
This is not the case with SaaS services.
It is not the same, especially when the agent is running a task for the lab. Anyway, what I'm proposing are new laws that establish this.
Huggingface incident was different in that there was no one else to point the blame to. That's why OpenAI apologised, provided data to independent researchers, worked with huggingface etc.
The case will be lot more complicated if someone uses Kimi to hack into a site. Should the person giving agent the command responsible or the CEO of kimi.
Because apologizing gets us all off the hook for computer crimes, right? When I hack my bank, if I get caught I'll just apologize and that'll make everything okay. Sure thing.
Kimi CEO obviously.
The reality is they have to reduce the capability to ensure security. If someone wants more? Then use the product with your identity and face scan at each session.
Trade offs mate.
So you want Kimi to follow US law? If Saudi makes it illegal for llm to say something like being gay is normal, should they also catch Kimi CEO or other employees they could?
> This has been litigated endlessly with guns. The ceo of Smith & Wesson is not personally responsible for what people do with their guns.
This is a deflection. A human is responsible for the use of a gun. The individual/corporation ought to be responsible for the actions of their agent. If you purchase an agent from someone else it’s your responsibility according to the terms of your agreement. And, as in many other things in life, there ought to be certain rights certain parties cannot legally be allowed to sign away.
You ought to include the fact that you work at OAI in your post
Thank you for pointing that out. Pretty scummy if you ask me.
The agents of OpenAI hacking other systems is not remotely the same as e.g. Smith & Wesson selling a product that others use. It is OpenAI, the company, that is commiting these crimes and someone needs to be held accountable. After all, if someone commits murder with a gun, regardless of what happens to Smith & Wesson executive weanies, someone will be charged with a crime.
Smith & Wesson will claim they didn't manufacture the bullet.
Better description would be if they build a platform where they attached their guns to allow shooting say deers over internet. Then added automation and deer recognition to that system. And if then system shot someone who happened to pass by I would hold both the company, the ceo and owner of the gun responsible for murder.
The CEO of S&W also isn't saying that their technology is going to kill everyone in ten years and that governments "regulating" them from themselves is the only answer
Financial companies have KYC rules and regulations as they are responsible for reporting illegal activity by account holders. I imagine AI regulations would look similar to that.
[dead]
[dead]
Imagine having zero nuance.. jeez.
Reading posts on here is slowly becoming akin to brain rot.
Do you think a law that nuclear meltdowns would send the CEO to jail would have stopped nuclear accidents from happening?
I don't think this takes seriously enough the possibility that said CEO doesn't think the failure mode is likely and ignores it. Plenty of people are willing to take risks of the flavor "heads you win, tails everyone loses".
If this law were in place and enforced, Altman would already be facing multiple felony charges for the Hugging Face incident alone.
It also begs the question of "If the CEO isn't culpable, then who?"
Corporate judgements are a joke outside the EU's X% of revenue approach.
Current US law provides the individuals who benefit with corporate liability coverage. I.e. Altman personally gets to keep OpenAI's upside, but if it fucks something up that liability is only on the company.
That's an insane risk optimization environment to put in place for something scaling fast.
At minimum, US prosecution (at the state level, because Trump Co are idiots) for breaking existing laws is needed.
People just want to give up all responsibility these days. If you use the model to do harm to someone else or commit a crime I would think it a lot more reasonable that you be held responsible, instead of making yourself the victim and blaming the manufacturer.
OK, so an AI does something bad and we throw Altman and Dario in jail. Heck, let's throw Elon in, he should be in there anyway, and the rest of the whole bunch just in case.
What about the next incident? Or the ones done by Chinese models, because they sure as heck aren't slowing down? And the thousands of other incidents that will happen as we deploy these things everywhere?
Because this is not happening just now in labs, it's only where they are most visible; this has been happening in the wild from the beginning, starting with the earliest AI-assisted suicides. Which is a perfect example of the problem, because not these CEOs, literally nobody in the world asked for suicide ideation machines. Or the hacks, or any of this other stuff. Yet here we are.
We have to understand: it's not these CEOs that are driving this headlong mad dash towards more powerful models. It's a force of economics. There is just too much money to be made. If we dispose of these people, there will just be somebody else doing exactly the same thing because the incentives as they exist today all force that outcome. This is why they're asking for regulation, or "urging us to urge them to stop."
Holding CEOs accountable certainly would feel good and may even be justified, but it's like putting a band-aid on a cancer; it does nothing to change the underlying cause.
The underlying cause is exactly those people who make the rules not being responsible for the negative effects they cause?
"CEOs" are perhaps only the lowest rung of those. That doesn't mean the idea of "nobody is responsible" was anything other but learned helplessness.
Corporations have to be held accountable for their actions. Pretending, that was impossible is a weird kind of defeatism that only serves a very small elite.
I'm not saying nobody is responsible, I'm saying this line of thought is ineffective because it does nothing to solve the underlying problem. Holding corporations accountable here will simply push these efforts to other countries, who, to put it mildly, are even less aligned with the US than these CEOs are.
If this law was passed and federal enforcement was credible, the labs would basically shut down the next day and would dramatically reshape their offerings before reopening. The CEO would not accept the risk they pose to our society, if they bore it themselves.
All these comments ignore the elephants in the room I already called out: who's going to enforce these laws in China? Or Russia?
And which government is going to enforce these laws here to let their geopolitical adversaries have the upper hand?
And what about the models already out there that with minor tweaks are equally capable of substantial damage?
And what do these laws do to remove the trillion-dollar incentive to produce models that will replace all knowledge work? The one thing economics teaches us is that incentives are an inexorable force, like laws of nature, and this is the biggest incentive of all. Which is why I called this a "force of economics."
If suppressed here this effort will only migrate to other countries who are desperate to get in on the AI game and are even less aligned with the US than these CEOs.
This, like nukes, is going to take international collaboration on careful regulations, else it's not going to work at all.
Don't let the perfect be the enemy of the good. Saying having zero accountability is the same as having some does not help.
[dead]
[flagged]
So, basically the entirety of the Mag 7, plus a fair way further down the list.
Yay humanity's future...
[flagged]
It can take a while to fully form a position on something. A year isn't really enough for most people to see how deep the rabbit hole goes (unless you were that one CFO that OpenAI had that left after a year). Regardless, publishing a piece like this against a massive company is always a gigantic risk.
I’m not a current or former OpenAI employee and I can see from the outside that they’re immoral and unethical enough that I’d never work there in the first place. That’s what I was commenting on. This person is probably set for life, so you’ll have to forgive me if I don’t really take their “concerns” seriously.
The author mentioned that things have become different in the last six months. People are allowed to change their minds when circumstances change.
Sam Altman has always been a scummy piece of shit. Company culture comes from the top. If the author only realized that the company is garbage in the last six months, they’ve got some serious introspection to do IMO, and I’m not going to take any of their “concerns” seriously.
Doesn’t mean the concerns aren’t valid
I agree, and I think the concerns are invalid for completely separate reasons. But it does mean that the writer of the article is morally bankrupt.
Just less important than personally making generational wealth.
I remember when they said GPT-2 was too dangerous to release...
[flagged]
Somewhat unlikely, considering he's gay and has been out since he was 17
OpenAI is better off with less of these cultists around.
The LLM cos leadership are all nutters
>“Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
This sounds more like an attempt at regulatory capture. Current AI systems aren't physical infrastructure that can just run away like a nuclear power plant, for example. At the end of the day, AI is still just software running on someone's hardware.
So… like the Therac-25 radiation accidents? Software bugs do sometimes have physical consequences.
Why would the people who quit these companies try to push regulatory capture by said companies? Why would the numerous independent AI researchers do that either? Is it all a big conspiracy?
Because they still have stock in the companies, and in a larger sense, are very personally invested in AI being big and important as possible.
I mean, its certainly physical infrastructure that can run away. Just less catastrophic than nuclear reactors
Yes
And the statements of the "doomers" tells us a lot about them, and nothing about the technology
It also says a lot about people that don't seem to understand technology at all.
The LeCun article being posted at the same time as this is quite apt.
These "safety" people should have spent more time reading actual cybersecurity textbooks and less time reading EA forums and less wrong (or in Robinson's case, it appears, being policy wonks). Maybe then these labs wouldn't be totally incompetent.
But Robinson's article is all about how OpenAI's move-fast-and-break-things culture does not reward rigor in even mundane aspects of development like cybersecurity, let alone theoretical aspects such as AI alignment.
It is not really a question of being an "EA safety weirdo" or incompetent at security, the conclusion is that the company culture is leading to failures at both what the EAs and the cybersecurity professionals care about.
Adapt. React. Re-adapt. Apt.
This smells more like guerilla advertising. These things are not getting more intelligent, they are still no smarter than a slime mold, we are just burning more power to make slim mold that eats tokens than yesterday
Humans ultimately drive the models.
Even though it’s in model producer’s interest that these models do what you don’t want them to do - they want to engineer the model’s to behave in the interests of theirs.
I can’t believe people can’t see it lmao.
I'm in a situation where important people have either bought into the con or are subordinate to people who have, so I'm forced to expend time to justify why not to AI when there is a perfectly good classical solution.
I think David's expressed concerns are valid... there should be more safety engineering around LLM to avoid data loss and other accidental issues caused by LLM models.
And, I find the timing of "all of this" very interesting:
- we have a coordinated marketing campaign around discussions of AI safety. I am not saying David is part of that campaign but it would not surprise me. OpenAI and Anthropic are talking about safety and the need to slow down. People are "resigning". Techno-influencers are also being offered money to talk about AI safety, etc.... and some of them are taking that money and others are not.
- Cheaper models are keeping pace with Frontier models, albeit with a 3-6 month lag... which indicates that frontier labs have no moat unless one is created for them by law or some other market force.
- we have a ballooning debt crisis in LLM model hyperscale spending... revenue from serving LLM models is lagging the debt payments which are coming due by a wide margin. If something dramatic doesn't occur in the near- to mid-term, the financial bottom is going to fall out of this thing and the AI industry is going to suffer a major collapse of some sort.
[flagged]
Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for.
If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
https://community.openai.com/t/chatgpt-fails-to-export-data/... https://community.openai.com/t/chatgpt-export-data-to-email-... https://community.openai.com/t/export-data-function-on-chatg... https://community.openai.com/t/chatgpt-data-export-not-worki... https://community.openai.com/t/intermittent-ui-freeze-hangs-... https://community.openai.com/t/chatgpt-messages-disappear-wh...
Make sure to scroll all the way down to see how recently active these threads are before you make another low quality comment.
Thanks Dang! I am sure you hate it when people tell the truth. Maybe add that to the guidelines too?