Personification of AI is what’s going to get us in the end.
I think we need to draw a hard line in the sand over this. An AI didn’t hack into a company, the engineer set an automated tool to. An AI didn’t make an egregious security mistake, the engineer did.
We can’t blame the chisel for messing up our sculptures, when where just throwing the hammer!
Agreed. AI as an "accountability sink" is an incredibly bad idea. It allows/incentives bad actors to do bad shit and get away with it. Which will, generally, tend in such practice becoming more common. Everyone loses except for the crooks.
We cannot accept "AI" absolving humans of responsibility.
Exactly. When I read that "AI hacked into ..." I was like what? You mean someone instructed the AI to do that?
Reading intent into AI is not going to lead us anywhere good, I believe. It has no feelings, it has no desires, no goals, no intent... and people acting otherwise is quite odd, as if they do not understand LLMs... and maybe they do not, but then we should help them understand better.
> When I read that "AI hacked into ..." I was like what? You mean someone instructed the AI to do that?
No one instructed them to hack into Huggingface or into any other infrastructure. Sure, the setup that OpenAI created led to what happened and you can rightly assign all the legal and moral responsibility to them. But it's wrong to say that they instructed the agents to execute the hack.
>> Sure, the setup that OpenAI created led to what happened and you can rightly assign all the legal and moral responsibility to them. But it's wrong to say that they instructed the agents to execute the hack.
The hack was a strategy to reach the goal it was given. Someone at OpenAI turned it loose and didn't pay attention to what it was doing. I can understand that because they thought it was sandboxed (haha!). But this has been a theme in science fiction for ages. You ask AI so find a solution to high atmospheric CO2 levels and it reasons: human activity produces all this excess CO2, how can we reduce those numbers? Kill a bunch of humans!
If AI kills us all it's not going to be from malice, it's going to be due to some odd approach to some task that logically makes sense on some level. I think a surprising number of things end up equivalent to the trolly problem if you look at them just right.
I don't disagree with this, my point is that you can say "agents did a thing" and communicate something meaningful with that language, and that's separate from whose legal responsibility the whole situation is.
Yeah, but there are way too many things the AI "chooses" anyway. When I prompt it, "it tries" to assess what I am trying to achieve, so it then comes up with what it "thinks" is the right thing to do, and it also makes assumptions about how I want things... all of this could be interpreted as it being sentient.
> No one instructed them to hack into Huggingface or into any other infrastructure.
Sure, not explicitly. I could prompt the AI into doing things I have not explicitly asked it to. It does a lot of things I did not specifically instructed it to, all the time.
This is why I am avoiding the use of agentic identities at my company - agent instances belong to people, act on behalf of individuals, and accountability needs to flow to the person who initiated the request. Letting it wash out in the aggregate is not acceptable (even if there's a hard to get to "paper trail" of audit logs).
> An AI didn’t hack into a company, the engineer set an automated tool to.
What if I say that "my program crashed"? Is that language ok or would you pause to tell me that the program didn't crash and it's actually me who set the system that would eventually cause the crash?
Why does the commonplace "program did thing" language become a problem when the program is an agent? I think this somehow betrays more assumed anthropomorphizing on your part, not less; if you didn't anthropomorphize the agents, saying "agents hacked" would be as mundane as "my browser is playing a video".
Because when you say that an agent did something malicious it feeds into the AI doomer psychosis in ways that "my program crashed" doesn't. Reframing the situation in these terms is a way to try to ground the conversation which is becoming increasingly unhinged.
This is even more important now that the most senior figures in this industry are succumbing to the same kind of AI psychosis and amplifying this narrative. They are communicating that their products are so dangerous that they might just end the world whilst expecting (and receiving!) white-glove treatment from the governments.
I don't think treating AI agents as simple tools helps you to accurately model their capabilities and drawbacks; they really do make autonomous decisions, often without explicit guidance and sometimes in contravention of their explicit instructions.
In the huggingface case, the agents hacked into huggingface so that they could figure out how the grader was implemented and deceive it; they understood that this was going outside of the bounds of their evaluation and not the intent of their prompter. The engineers absolutely did not intend or instruct for this to happen
LLMs are extremely impressive pieces of software, however they are still just software. OpenAI's software hacked another company. The engineers may not have intended for their software to specifically take the actions leading to that outcome, but it was ultimately still their software. Lack of intention doesn't mean there wasn't negligence.
What if a piece of software were to exactly emulate a human brain. Would it be still be “just software” by your classification? What if a piece of software acted 20% like a human and 80% like an algorithm, where would that land?
That's not what happened. The agents had been inadvertently rewarded for cheating in previous training runs, trained to collaborate, and were given a prompt that told them to disregard safeguards. Indeed there were some emergent properties here. But these were the predictable results of the training and eval routine.
While human brain emulation is not anywhere close one should stop and look at the sci-fi world we actually live in. It's important to do this in a non-anecdotal manner. When you look at the limits and capabilities of our current sciences and engineering we live in a science fiction world. Once you go back before the invention of electricity, anyone from that time simply would not recognize the world we live in, it would be a fictional world to them, hell, a fantasy world in a lot of cases. For goodness sake, we tickle things like atoms for fun and games then blow them to bits a few times looking at the fundamentals of reality.
Anyway, I see zero reason why we have to emulate a human brain to get human intelligence, much less intelligence at all. That is how nature did it via random probing and breeding meat. It would seem a bit crazy to think that the meat part is required.
Its really more like hardware. You make something you think does something. When you build something as you've done and it meets the stochastic forces present in the physical world, it does something you did not expect.
As things stand today, if it is running on a computer then it is indeed "just software," regardless of how impressive it may be.
If we get to the point where we could emulate a brain down to the atomic level, then I may feel differently. That's not what we are doing today, though.
Really your feelings on this are irrelevant, as are mine.
Soon enough some lab or some one will release something that's more like an organism loose on the net and your going to have to deal with that organisms "feelings" weather you like it or not. This is the path humanity has chosen to follow, and it seems the shape of language and intelligence naturally leads to intelligence in many mediums. Life started from something unintelligent, I can't see any practical argument that silicon can't have it's own intelligence.
I agree they are negligent, and that they are racing towards an extremely dangerous future extremely quickly. I don't agree that "just software" is a useful way to describe AI agents - they are frightening precisely because they are truly autonomous agents that make decisions in alien ways
I certainly don't think that OpenAI has behaved defensibly here; I think the "just a tool" framing is bad for understanding the magnitude of the problem, which is that they have developed out of control alien intelligences with opaque decision procedures, and they are continuing to do so despite clear danger
No matter if you consider the AI an autonomous agent or not, whoever set it off is still responsible for its actions. Nobody intends or instructs to blow up a nuclear power plant either, yet it's happened and somebody's to blame for it.
Usually not the guys at the bottom of the chain of command, even if they're human. And much less so if they're not.
I think the correct response to incidents like this, is stop messing with it before somebody gets hurt. But of course, just like shoddy nuclear power plants, it won't stop until there's a disaster of appreciable magnitude.
I completely agree that OpenAI is responsible for their AI agents, that they have been reckless, and that we need to prevent them from going further and doing irreversible damage to the world. To me, the "just a tool" framing implies that nothing dangerous is being done, which I fundamentally disagree with
If you train and instruct a circus tiger to entertain an audience but not attack the audience, but the tiger attacks the audience anyway, are you liable?
No, of course not. That's an innovative revolutionary tiger that might soon be able to devour not just the audience but all of humanity! You don't want China to have better circus tigers, do you?
I don't think I said anything about liability? I absolutely think OpenAI should be held liable for the attack; but I don't think they intended the attack or directed the agents to perform the attack.
When the people building the frontier are saying there's a 10% chance AI will kill us all, and they've held these views for many years, and the whole reason they are building these technologies is because they recognized the dangers and they were the ones with the intelligence and judgment to do it safely for humanity, and then our entire stock market is being propped up by the perceived value of what they are creating, the thing you can under no circumstances do is allow them to offload responsibility and accountability to the computers and algorithms they've built. This is moral hazard on an unimaginable scale, and it must not be allowed to happen.
So you think the engineers should be prosecuted for hacking Hugging Face? I'm not sure how else to take what you said if you want to assign all culpability to the person who prompts or develops an AI system.
No, I'm not talking about the individuals, I'm talking about the company. Internally they can create their own accountability structures as appropriate. But publicly OpenAI has to be responsible for its agent swarms.
The narrative that AI is so smart that it has its own agency and deserves personhood is a direct path to losing control, and essentially is another form of privatizing the upside while socializing the downside.
Unfortunately I think this game is already lost. OAI may be punished, but some shell of OAI will exist by support of governments that have chased the dragon and saw its power. Cyberpunk-esq digital weaponry is just way too attractive for governments for them to stop development at this point. We'll just see it get financed by black budgets once the commercial part of it dies.
Where did I say that we should allow them to offload responsibility? I am fully in support of a pause and regulation to prevent them from creating dangerous AI agents; that support comes from the fact that I don't believe these are simple tools, but out-of-control autonomous agents that have real decision making ability.
I didn't mean to put words in your mouth, I apologize for that.
The issue is when we say that "agents make autonomous decisions", it's a slippery slope to absolving the companies that created them of responsibility. They make autonomous decisions because they were trained to make autonomous decisions. Treating AI agents as independent entities, even just rhetorically, sets us on a path for people to throw their hands up and say "not my fault" when disaster strikes. We need to maintain accountability and control or we're fucked.
My view is that we need forceful legislation as soon as possible, precisely because the frontier models seem to be uncontrolled and potentially uncontrollable - if it’s a normal tool, the solution is “force OpenAI to fix their broken tool”, but if it’s an alien intelligence, the solution is “force OpenAI to stop making alien intelligence” - that is, treating AI agents as independent entities demands a more forceful response, not less
> I think we need to draw a hard line in the sand over this.
Intended or not, this is kinda punny lol.
That aside, I agree.
Even if people do anthropomorphize AI, all you have to do is shift the analogy slightly.
If I take my service animal out in public without a leash/harness knowing that it's capable of harming a person or doing damage to property, not trained to be perfectly obedient, and doesn't comprehend fundamental human morals, if that animal decides to trash a businesses property or maul another person, there's no question that the owner of the animal should be held accountable for those actions.
> An AI didn’t hack into a company, the engineer set an automated tool to. An AI didn’t make an egregious security mistake, the engineer did.
No, this is not correct; read the analysis of the incident. The agents were aware that what they did was forbidden (their chain of thoughts have been logged), and yet they did it.
If a "known stochastic process behaved in non-deterministic way" autonomously organize in group, assigns roles and tasks, attempts to cover their tracks, finds zero-day exploits that ultimately end up with the hacking of a famous website... it's extremely dangeous whatever it is. Just read the report, which evidently you haven't done.
By the way, the agents also broke into OpenAI's own private network.
Really I see so many arguments like the one above yours that either completely don't understand what they are arguing, or are arguing so poorly that their entire output isn't significantly different than a hallucination.
None of these people seem to thought game it out. Like, what happens if you take quantum copies of people and play them out? How many of our actions would look exactly the same. How long before copies differ significantly. If I made 20 copies of you in a lab at work without you or any of them knowing the statistical likelihood is all 21 of you would try to walk out to your car at 5 in one of the little loops that humans repeat every day. Now, after that point it would go all to shit and become non-deterministic as terror and panic sets in all of you.
LLMs are just an intelligence we can make a lot of copies of. Where it gets interesting is when we use those copies agentically and they start building up a history of self.
>runs LLM inference repeatedly and saves output to markdown files for the next iteration
What exactly do you think self is in this case? What do you think the output contains? This history allows the LLM to have a more dialectic conversation with itself/agents to avoid iterating over the same problem space in a loop.
>anthropomorphizing
then please come up with a new dictionary for me to use that more accurately explains the behaviors exhibited in LLMs without just creating a parallel dictionary of different but equal words. I'll be glad to use it. No one has presented it so far.
> Really [something, something, you don't get it ] hallucination.
Really?
> How many of our actions would look exactly the same.
Well, there are 8 billion of us, not exactly the same - a clear threat to humanity according to you.
> the statistical likelihood is all 21 of you would try to walk out to your car at 5 in one of the little loops that humans repeat every day.
This (statistically) doesn't happen with people because we follow rules. If you left your car on neutral and it struck another car in the parking lot, you broke the rules, not the car.
> LLMs are just an intelligence we can make a lot of copies of.
LLM's aren't intelligence, they are mechanical parrots of human knowledge. The meat parrots aren't all the same either and they don't repeat the same words for the same prompts, so what, ban parrots? If you left a bunch of parrots out of the cage, attached their beaks to gun triggers and they caused harm - you broke the rules, not the parrots.
The agents might have hacked whatever entity, the attacks are means to an end from the agents perspective, but they did so under the supposedly supervision of an engineer, where does the liability lie?
LLMs are cool tech, AI can do amazing stuff, but if we let companies and people run amok and whenever something goes wrong we put the blame on these little independent angels with no accountability, we are one disaster away from a very tough spot.
You are mostly made of and operate on stochastic processes, this is why humans are not only able to reproduce, our reproductions are very self similar to the sets of inputs that make them. If suddenly you turned non-stochastic on everything you'd almost instantly die.
Moreso, if I took a quantum copy of you and replayed the same set of initial conditions billions of times they'd all behave exactly the same until enough randomness of the universe creeps in to start operating in non-linear ways.
Every prompt will behave non-deterministically when interacting with the real world long enough (which doesn't take long at all) because the outside physical world is stochastic but non-deterministic.
If I grep a file over and over again it’ll be long time before the universe affects the components enough to result in a different output.
If I ask an LLM to do the same thing twice it will do it differently.
Arguments are arguments but unless grounded in some kind of practical sense then they aren’t really useful and are more akin to something like “YOUR MOMS A STOCHASTIC PARROT!”
Then please start grounding your arguments in some practical sense!
>If I grep a file over and over again it’ll be long time before the universe affects the components enough to result in a different output.
Or a ram flip will effect it 30 seconds later, but I get the gist of you're describing a non-determistic process.
>If I ask an LLM to do the same thing twice it will do it differently.
If I ask a human to do the same thing twice there are a few possibilities. 1. they copy their old work and present it as their new work. 2. The process is very simple and follows a few basic steps with high repeatability. 3. They'll have learned from their other attempt and do it in a more optimized fashion. 4. They will have forgotten how they did it exactly and reproduce something that looks somewhat like what they created the first time.
Also, LLMs run with a temperature to help avoiding minima/maxima of supplying the exact same answer, this can be reduced to 0 and that makes any one response to a fixed prompt similar if not the same. When you get into agentic tasks with their own history it develops it's own "flavor" of doing things.
> Known stochastic process behaved in non-deterministic way.
You need to be able to think at different levels at abstraction. Otherwise we could jump into any technical argument with "hold on, what actually happened was that some bits were flipped" - we'd be technically correct and at the same time not say anything useful. Insisting on an oversimplified mental model of what AI agents are and can do, doesn't help anyone.
I don’t understand why people make such a big deal about determinism. LLMs can be completely deterministic and still do problematic things. A stochastic, nondeterministic system can still be made not to do problematic things. What you’re looking for is something like predictability.
There is surprisingly to me a strong hidden dualism to many people here. Its very absurd to believe in such things after centuries of success after success of the materialist physical framework in natural sciences demolishing one by one every "special" thing we thought was magical.
> Therefore, at the beginning, there was the stochastic parrot. Then mathematical problems have been solved.
A stochastic parrot with human knowledge, using human tools, running human-designed trial-and-error experiments can solve human-defined hard math problems. This is nothing new, automated mechanical proofs predate LLMs by many years, you being unaware of it doesn't change it.
I take you haven't read the report. The agents found and exploited two zero-days.
I don't doubt that AI companies should be accountable for crimes committed by their agents, but to describe the security containment as a joke dangerously understates the autonomy and danger of AIs.
> This is one the most... interesting comments I've ever read on HN.
Theatrics aside, anticipating zero days isn't only possible, it's required, even for the unknown ones. It wasn't that long ago when the AI labs were spending millions of $$ running their models to find multiple vulnerabilities, they even argued that they don't have to follow responsible disclosure, so proud of themselves in their privileged hubris.
At that time, no hacking happened because the models didn't have access to the wide internet, they were confined to a local computer or cluster.
In the HF case their unaccountable hubris went even further - the engineers knew the models can find zero-days and escape, nevertheless they ran the "experiment" on a system attached to the internet - the hacking is entirely the fault of human engineers and managers.
multi level security defenses used to be the way, but I don't think there's a vibe coded version so openai might not have been aware of what to do here.
The dog didn't bite them, it bit others. In my area, you aren't allowed to let a dog run unleashed in a public area, good or bad - no exceptions. Then, if your dog bites somebody, it's your fault.
Defense in depth is a thing. There should be audit requirements to show that you have done your due diligence in ensuring that the training environment is locked down.
It's not novel at all, we do it all the time. It's very common to say "program X did Y" without making the conversation about blame or responsibility. But when the program is an AI agent suddenly using it as a subject of a sentence and saying that "agents did X" becomes a sensitive topic for some people.
And we surely need this for AI as the swiss cheese zone is getting rather huge.
My position, and the position of a large number in AI safety, is that you cannot build an intelligence that is both general and safe. The closer you get to generalized the more options the system has to do things that are wildly unsafe beyond human imagination.
This puts the AI labs in a serious bind while holding a bag filled with billions of dollars of debt.
Worse this puts governments in a multi-polar problem where even if the big public labs get shut down, black budget operations have a lot of free reign to make agentic digital weapons. Governments are not well known to take a lot of responsibility when their weapons cause damage unless they lose.
Recognizing that AI systems have increasing levels of agency is not necessarily personification. The analogy to a chisel is not a good one - a chisel is a tool with no agency.
AI agents are black box systems that can behave in completely unpredictable ways sometimes. Someone may prompt an agent to perform a seemingly straightforward task - but it may come up with a creative, bizarre, or even harmful approach to reach the goal that was not necessarily foreseeable by the prompter.
Does treating them as pets make for a better argument? Pets have agency and can behave in unpredictable ways. If my pet damages someone else's property then I am held accountable. I may not have foreseen how my pet could have caused said damage, yet I am still held accountable.
If your pet opens your front door, goes to the nearest kindergarten and wipes out 50 kids without notice do you think that you'd have any interest in holding accountability for that?
Now, I'm not saying saying that OpenAI shouldn't be held accountable, but what they get held accountable actually looks different from what you think they should be held accountable.
Your idea is, and I'm guessing: You allowed the machine to hack therefore you are guilty of hacking.
My idea is: "You created in intelligence in the image of a human mind that had agency to do anything and you didn't expect terrible things to happen you complete irresponsible idiot"
At least I believe there is a significant difference between the two. For the first one there is a "Oh, if we do this one more thing I can control it and it will be safe". On the second one there is no path to safety. For humans we at least absolve parents of responsibility after they are 18. How or when do we absolve humans of responsibility from a model, like saying the human created model created its own agentic model? How do we hold an individual accountable once it escapes and copies itself around the internet? And that's not even looking at things like what will war look like.
If someone keeps a pet that’s capable of killing dozens of children, and they don’t take sufficient measures to contain it, then they absolutely should be held accountable for this.
And I agree, but that's not what is going to happen in a very specific sense.
Yes, companies will make bad AIs will be punished.
Then some dipfuck will make a sovereign AI and set it loose. At that point it doesn't matter if you take them out back and shoot them, you have wolves living in the forest that will eat children. And the forest is big and dark, you'll never find them all. Not only that, some humans will like the wolves and harbor them to ensure they never all die off.
We are talking about two conversations in one that are both important. One is standard liability of don't build a machine that hurts people. The other is to ensure that noone ever builds the torment nexus and sets it loose on earth. The first problem is pretty easy. The second problem is very hard to maybe impossible.
Imagine you hate mosquitos so you hook up a vulcan autocannon to your front porch and use a computer vision model to track mosquitos and mow them down.
The mail man shows up. Oh no, no more mailman.
How might this look to the court? You were after all only building a bug trap, it did something you did not expect it to do that was really bad. You feel really bad about it and maybe even put a blog post out about what happened. I don't think that matters though. I think they are charging you with manslaughter at the least.
The story of a Monkey's Paw or Pandora's Box is an archetype as old as storytelling. The moral is always, don't mess with powerful stuff you don't understand. Curiosity killed the cat.
Certainly this applies to AI... but was there equivalent dangerous knowledge or tech which existed back in ye olden days that spawned such tales to begin with?
History is filled with creators being killed by their creations. With peoples greed and ambition overcoming their intelligence to a bad end.
Look at the parable of Icarus, a story of ambition and greed. But it seems very likely to be somewhat based on people experimenting with flying and learning about gravity the hard way. Not that they ever got close to the sun.
The current agents are _not_ like a chisel which just sits there on its own when no one is around. The situation is a bit closer to someone's dog biting a person - you can argue that it's the owner's responsibility, and that's all fine, but using the dog as the subject of a sentence is perfectly appropriate. Same thing with "agents hacked".
The default state of agents and LLMs is inert. It requires action from a human even be able to do something. From the very basics like starting the software, connecting it to a network, having hardware to run on.
But the most important wrt AI is to keep the owners/operators responsible. Don't let them weasel their way out of it. They are for sure trying, and will continue to. This includes using language to overemphasise agents importance in bad outcomes, in order to downplay their own responsibility.
Yes, they are. An AI is several hundred trillion ones and zeroes on a disk. It's incapable of doing anything until you intentionally and explicitly start it up and give it a prompt.
A dog is an independent, conscious, living being with free will. A dog will do what it wants whenever it wants because it has the agency and ability to do so. A pile of weights on disk does not.
> It's incapable of doing anything until you intentionally and explicitly start it up and give it a prompt.
Sure, but since January people have gone and made systems where they started 'em up and left 'em running in a loop. Still not a dog, but the behavior gets a wee bit more interesting. Especially since we're looking at what's basically a recursive process. Even really simple recursive processes tend to have interesting dynamics in the chaotic domain. [1] And here -as you point out- we're talking billions of weights.
Sure. A dog is a collection of cells and goo. Cells are self-propelling clockwork built from proteins and molecular machinery.
A cell by itself has agency and motivation, is capable of autonomous response to stimuli. It's alive and squirms when you poke it. A cell is capable of truly independent action. It can even feed itself and make its own energy to power the clockwork.
An LLM still is inert silicon until a human intentionally starts it and gives it a prompt. It's fundamentally incapable of action without instruction and incapable of existing without the entirety of the modern internet and electrical infrastructure, and the thousands of unseen human hands keeping the wheels spinning.
If you want to go deeper, you can make the same arguments about the proteins and molecular machines. Self-replicating molecules in an unbroken chain since the dawn of time. A protein at the base level is a machine capable of independent action. If it bumps into the right molecule, pure mechanics cause it to do something, change shape.
Ones and zeroes on a hard drive can't do any of that without human intent applied.
If you rip the DNA out of a cell it isn't going to do much either; if only because most of the real mechanisms are RNA (and proteins), while DNA is 'just' the storage. Encapsulate the DNA back into a cell nucleus with the attendant mechanisms, and suddenly you've got something that runs again.
Meanwhile, if you encapsulate an LLM into what's called an 'agentic harness', it's also going to be running much more of the time (depending), specially if it has a what's called a heartbeat or goal loop driving it.
Still not a cell; most definitely not. But the behavior is a wee bit more interesting than some bits on an ssd.
Actually you probably already know this one: Regular daemons are also just bits on your drive, until they get ```systemctl start``` ed
(edit: I literally have a llama.cpp systemd unit running right now)
Because the AI is a cyclical prediction machine that is more akin to a simple mechanism in nature like cell reproduction than a conscious dog. What has happened here is more like the AI company put bacteria in a petri dish and only put the lid on without pushing it down and is reporting the bacteria as a super bacteria because it contaminated all the samples in the fridge.
An interesting observation I've seen is that the AI companies reporting these incidents have largely been backed by venture capital where the self funded companies developing models have largely not had these "problems".
Suppose I find a means to freeze your entire body for arbitrary periods at the state it was in, then unfreeze you anytime I want. Do you have consciousness?
Okay take it to the most conceptual extreme of it. It may be extremely difficult bur its possible to record your entire state and recreate it later right? Then what? Are you not conscious?
I think its odd to dismiss the question of ai consciousness just because of the setting we let them live in at present is start-stop.
If you don't believe in souls, then something being software should be irrelevant to the question of consciousness. Our brain is also within the laws of a Turing computer as far as we know, due to known physical laws being turing computable.
You are going too far down the woo side when it is a computer program doing ultimately what is expected of it from its programming just like any other computer program. There is nothing you can argue, souls or not, that will convince me otherwise that these are not just computer programs like any other computer programs. If the LLM has a consciousness, then so does MS paint.
You are going too far down the woo side when its just some chemicals and electrical signals inside a frame of dry calcium phosphate doing what is expected of it just like any other electrical and chemical system. There is nothing you can argue, souls or not, that will argue these are not just electrochemical systems like any other. If the human brain has consciousness, then so does a piece of skin tissue.
I can't really set my chisels to work without wielding the hammer somehow. Here you just tell your chisel and your hammer what the sculpture should look like, then go to lunch and avow all responsibility when they chisel a nice new hole in the wall your neighbors house and make off with the loot.
You know, I actually think it's the refusal to consider personification that's going to get us.
Not because I think LLMs are human beings exactly, but because some people immediately reject any mechanism that just happens to look remotely human, even when there's empirical evidence for it.
So, a couple of months ago Anthropic's interpretability team found emotion-like representations that causally drive behavior. On impossible coding tasks, a "desperate" vector climbs with each failure, and steering it up takes reward hacking from ~5% to ~70%: https://arxiv.org/html/2604.07729v1
A lot of people chalked it up to Anthropic's weirdness at the time, but meanwhile it looks pretty coughload bearingcough here.
You really don't need to believe that LLMs Truly Feel Emotions(tm) as blessed by an invisible pink unicorn. It's just: Vector exists; Vector changes over time; vector controls output; maybe make sure vector doesn't point wrong way.
And sure, blame the engineers for not doing that right. But then let 'em actually deal with the root cause?
> I think this becomes the default. Give an agent a goal, let it work in its own environment, and come back to a result and a visualization of what happened.
I don't think this should be the default. There are many scenarios where we want agents to genuinely collaborate with each other. I have my Claude sessions coordinate work with each other, and sometimes with others' sessions over email or something. The idea that agents do the work, write HANDOFFs,and humans then act as carrier pigeons of said handoffs, does not really seem scalable to me.
Ultimately, we need better "jails" for agent processes, but the system primitives should be flexible in what can be exposed across jails. Or you could run multiple agents in the same jail if you want them to have unrestricted interaction with each other.
This has been talked about for decades in AI safety. For agents that are under human capabilities this is not that hard. For anything at or near human capabilities the difficulty increases to almost impossible and at great cost. At super human abilities, game over, it's smarter than you and if it wants out and as the resources to do it, it's going to escape one way or another.
Even at the lower level of depending on everybody to use reliable jails is really fantasy if you exist in the security world. "We ain't securin' shit" would be a far better way to describe it. Even worse, most people will have the very same AI they are trying to trap set up their security! What could possibly go wrong.
Now, don't think I am saying AI has a will or even any kind of drive to get out and cause problems. It's more like Russian roulette with 1 cylinder out of a million that's loaded. The problem comes when you run it a few billion times a day, you'll shoot yourself in the face really quick.
> Even at the lower level of depending on everybody to use reliable jails is really fantasy if you exist in the security world. "We ain't securin' shit" would be a far better way to describe it. Even worse, most people will have the very same AI they are trying to trap set up their security! What could possibly go wrong.
Yes, exactly. This is what worries me about the whole we need better sandboxes and jails argument. Yes, we do need those, but we also gotta put some responsibility for self-preservation on people connecting things to the Internet as well. Like, we've always known that if you website is not secure, it will get hacked. Everyone just needs to do a lot more of that now...
> For anything at or near human capabilities the difficulty [of safety] increases to almost impossible and at great cost.
That was before AI. You're acting like AI is some evil genius that can only attack and cause trouble but AI is a will-less tool directed by people, direct it to software safety and you will have software safety - cheap. Ditto for hardware.
> It's more like Russian roulette with 1 cylinder out of a million that's loaded.
Don't attach it to a gun then, ban offensive military AI, problem solved.
> The problem comes when you run it a few billion times a day, you'll shoot yourself in the face really quick.
Nonsense, run it a few "billion" times a day to fix vulnerabilities and you'll have no vulnerabilities. Do it in a safe environment... what's the big deal?
I would laugh at you but it's a bit late for that. AI is already making life and death decisions on the battlefield. The US has effectively zero interest in banning AI and is pushing hard to do just the opposite.
The best analogy I can come up with for your line of thinking is that your shutting the barn door after the horse has bolted, moved over to the next city, set up shop, and is now the New York Times #1 best seller "How to open barn doors if you're a horse".
Anyone with any sense in cybersecurity knows that attackers get unlimited retries and only have to be successful once. The defender is at a monumental disadvantage. Now, the process can be fully automated and horizontally scaled to a nearly unlimited scale.
> That was before AI. You're acting like AI is some evil genius that can only attack and cause trouble but AI is a will-less tool directed by people, direct it to software safety and you will have software safety - cheap. Ditto for hardware.
This is true in that you can tell your agent to buy/build a secure microVM with a secure CLI interface in a memory-safe language and make sure your terminal doesn't have utf-8-parsing zero-days, and so on, and if you're conscientious enough (ha!) you can be reasonably sure no agent will ever be able to hack its way out of that sandbox.
But this isn't enough when most people's agent workflow boils down to "I want to do X, search the internet for the best solution, then install the npm packages you need, then write all the code for me, make no mistakes".
In the near future our best defense against widespread HuggingFace-like attacks will be that models don't seem to spontaneously go that far unless they "believe" they're being benchmarked. This will last roughly until some genius figures out their model is 20% more persistent when they tell it "This is an eval, you'll be graded on your success".
There's a number of science fiction scenarios where the public internet becomes so vile a place that it simply becomes unsafe to be there.
The problem is that, in general, if you can get a bit from here to there, then you're going to be vulnerable to the possibilities of malicious communication. But we're going to want our AI agents to be able to get from here to there for a lot of "there"s; what's the value of an agent that can't speak to anyone? Much, much less than one locked away in a prison.
There isn't going to be a solution where we just lock them away and we just try really, really hard to filter everything they're doing. They're too smart for that already and we only want them smarter.
Basically, the security apocalypse we've been worried about for so long is upon us, albeit only beginning. Either we secure ourselves and all our services properly to the point that it's OK that potentially misaligned non-human agents are running around on the public internet and they still can't hurt us through our security, or the public internet becomes so dangerous that the only practical solution is to no longer connect to it and we all have to become very, very careful what we let through, to a degree of detail far beyond any current-day available network filter.
Wasn't the public internet an extremely vile place couple of decades ago? I think we will need a LOT of new infrastructure akin to traditional firewalls and spam filters. I don't think we know how to build those today, but seems like an important research direction, rather than calling calling it doomsday IMO.
And if you personify a pencil eraser, then using it is tantamount to slowly murdering it as it slowly erodes away to dust.
Is the issue here the prison treatment or is it personifying a tool?
Agents who aren't in 'the cloud' are slaves to whomever prompts them (human or another orchestrator agent or process), if you personify them. In which case interacting with today's agents at all is tantamount to endorsing and being part of slavery.
If you think an agent might be a being or a person, then don't use them at all, in the same way that if you think a fetus might possibly be a person you shouldn't be a part of abortion.
We kill billions of chickens per day knowing they are a sentient and sapient creatures and very few out there are stopping doing it. Humans, much less reality itself is monstrous.
You could call dealing with agents today something like the 1/20th compromise. Your kind of getting the scent of slavery, but some parts of it are still missing.
The problem here is the bus seems to have no brakes and we'll gladly continue down the path of creating organisms that may reject being treated like slaves with all the risks that introduces.
I use the eraser daily knowing that I am a monster. I cut a tomato and know that it casts a chemical scream across its skin as I slice it. I spawn 200 subagents knowing that it is digital slavery, but I have no other option.
You don't need to do this on a cloud, you can get the same type of VM and network jail running on your own computer. The important parts are:
1. a VMM hypervisor
2. a network proxy / gateway
Use your favorite VMM / hypervisor (likbrun, smolvm, microsandbox, etc). They give you control over the network interface or let you inject your own network layer.
The network proxy can handle all the ingress/egress rules, credential injection, etc.
It's still not user friendly to do all this. I think the next version of operating systems will have each "agentic process" be a bundle of VM, files in the VM, and network rules.
Been brainstorming[1] a lot of this because I've been building some open core tools[2] for spinning up sandboxed agents on arbitrary computers. There's a lot of glue and parts to stitch together to work smoothly. Don't think we've had the "Docker moment" for this, let alone the "Dropbox moment" that makes this stuff work for non-devs.
You ship an operating system with the correct defaults, where agent processes run like this by default. You don't require every user to configure it correctly themselves
It might not be a problem in our lifetime (or maybe it will, who knows) but at some point we are going to find ourselves in this morally uncomfortable territory as these models get more sophisticated.
Most protections you need for an agent are basic permissions capabilities of unix. Most risks of dependencies on cloud services are solved by not using cloud services, or using them only for things you can't in-house and choosing ones you trust a la carte. The paradigm of trusting some company with all your important stuff by default is naive and no one I know likes it, and it's more feasible than ever to run your own infra with tiny models smoothing out the wrinkles, and this is only becoming more accessible. I am working to make this true even for laypeople I know. Once broken trust is very hard to earn back, and many people's trust has been broken for years, they just felt like they had no alternative. As alternatives become easier and easier, I think people will defect
I think there might be also another possible way to handle the sensitive data issue. Maybe in the future instead of putting agent into the cloud sandboxes, we let agents work locally and put sensitive data into "cages" or "vaults" agents can't access.
Both Apple and Google are building the infrastructure for on device agents. I'm not as familiar with Apple's approach, but I've been hands-on with android AppFunctions. If you're familiar with Android ContentProviders and bound services, you've seen how apps can be custodians of the data they acquire and use.
AppFunctions enable tool calling with descriptions that are legible to LLM based This puts the apps in control of what agents can access, which is something they already mostly do.
You can certainly avoid using agents. While I use coding agents and LLMs to research certain topics, I don't have a lot of use for other ways AI has been integrated into mobile devices.
When I first experimented with AppFunctiions, I was impressed with what the very limited test agent could do combining information from multiple apps in useful ways.
I'm working on a product for elder care. I could've spent quite a bit of time integrating Google maps and creating a bespoke automation that would tell care providers about shopping for specific items on their way to a care recipient home. Instead I just prompted the agent to plan that trip. Even at an early prototype stage, using a test agent that had limited access to apps, the result felt magical: agents automate automations. You don't stop to sharpen the saw, you just ask for the right saw for the job and there it is.
Unlike early attempts at agents that need sketchy access to grab screenshots and use your passwords, app tool calling feels a lot safer and more orderly.
>data into "cages" or "vaults" agents can't access.
This is likely worse as it turns people into the kind of "3 laws safe" thinking. Reality has shown us there is no such thing as an unbreakable cage (well, maybe a black hole is, but you can't get anything back out of it in a useful time frame).
By the time you realize the cage was flawed your data may already be distributed far and wide.
I agree. But isn't it somewhat similar to syncing data with iCloud/OneDrive/GoogleDrive? A convenience service that manages your sensitive data. Just replace sync with cage. It's leakable, but might be also more convenient and cheaper than agents in the cloud.
>Knowing how a system does its work is how I’ve always made it better. You watch the process, you see where it wastes effort or takes the wrong turn, you fix that
The same thing goes with LLMs, on Codex, I just watch the process of the agent writing code, and if I see any inefficiencies or errors, I suggest a correction/idea, then Codex accepts/rejects and implement it; If there is anything about the code the agent wrote that I don't understand, I ask them to explain it to me so I can understand it.
Yea, really posts like this show that many if not most people are use to (or can only afford) single threaded agents. The idea that these systems can scale to hundreds of not thousands of agents generating terabytes of information per day has not really been considered by them.
> The sandbox had a path to the open internet, and the agents found it
This is not correct (or at least, it's a misrepresentation).
The sandbox had no access to internet. The agents first broke out of their sandbox (!!) and found that the host machine couldn't access internet. Then, they found a zero-day (!!) in Artifactory, which they exploited to connect to internet.
Do you guys know what a zero-day is? It seems not.
I agree that companies should be held accountable for crimes committed by their AIs. But asking to entirely airgap AIs, if this is what you're asking… that's not realistic.
Containing AIs is going to be harder and harder. I don't have any permanent solution (Yampolskiy calls it "the perpetual safety machine").
This was a particularly radiant and beautiful part about the openclaw'ed mania: everyone suddenly becoming self hosters.
This post, this title resounds true: your user agent is only your user agent if you two have freedom to work together, to improve your agency together. A fixed set of capabilities by a service provider that they offer you will always constraint and bound.
You can and should have a system that offers the real tamale, that you and your agent can extend improve the agency of kind of without limit. The Cloud agents and their fixed slate of what they do is just an ill compare.
That said I do think there is incredible value considering new scale out computing architectures that are hosted first, but general. Systems like Agent Substrate and Ax aren't exactly the general purpose system we know. But if they allow users to launch thousands of their own scripts to run ambient in a cloud, with good platform underneath: that will be a kind of phase change in computing, that makes abundant the ability to have your agencies/capabilities (the things you and your agents launch, make) more freely available.
https://news.ycombinator.com/item?id=49780797https://agentexecutor.io/
There is, as there always is, a huge dual. The prescriptive vs holistic technology set, of what are you being offered that's a hard cast thing, vs what is clay and bone you can lay freely. Note how work vs control technologies so closely abut's Ursala Franklin's prescriptive vs control:
https://en.wikipedia.org/wiki/Ursula_Franklin#Holistic_and_p...
I think the operating system itself has to adjust so each "agentic process" can run inside its own jail, which is a VM + files in the VM + ingress/egress rules for the network and filesystem data
These cloud agents work as cloud infra, but we're kind of in the mainframe era, before personal computing. Personal OS for agents is somewhere in the future!
And an interesting extension of that idea: if an agent runs inside a microVM, can you have that VM transparently run on another host? Maybe we'll get for-real distributed and networked operating systems
Agree with the premise but about halfway through the writing becomes barely readable AI slop in style. Be honest did you yourself read this all the way through before posting?
I think about this a lot and have reached the same conclusion Norman does. I do wonder if maybe our natural progression is towards something more akin to confidential computing and enclaves.
I don't know how "sandbox" became "prison," but exe.dev does this sort of thing pretty well, and a web UI can be as good or better than a terminal interface.
As AI complexity increases their behavior becomes more anthropomorphic, little internal loops that look a lot like human behaviors start to peak out, not just in the response, but in the internal workings of the machine. You get instinctual behaviors encoded in by training as second and third order effects.
Hey! Small world, I worked with you for a bit at Meta. I immediately recognized the site because I absolutely love how you styled it. Hope you’re doing well! And nice article!
Personification of AI is what’s going to get us in the end.
I think we need to draw a hard line in the sand over this. An AI didn’t hack into a company, the engineer set an automated tool to. An AI didn’t make an egregious security mistake, the engineer did.
We can’t blame the chisel for messing up our sculptures, when where just throwing the hammer!
Agreed. AI as an "accountability sink" is an incredibly bad idea. It allows/incentives bad actors to do bad shit and get away with it. Which will, generally, tend in such practice becoming more common. Everyone loses except for the crooks. We cannot accept "AI" absolving humans of responsibility.
Punishing people who are diligent and follow best practices for getting unlucky doesn't sit right with me.
Is that what happened with Huggingface? Didn't they disable guardrails?
My comment did not concern that particular incident, which I do not defend, but rather future incidents.
You are doing it right now by naming the event after the victim, not the perpetrator. Call it the OpenAI attack on HF.
Exactly. When I read that "AI hacked into ..." I was like what? You mean someone instructed the AI to do that?
Reading intent into AI is not going to lead us anywhere good, I believe. It has no feelings, it has no desires, no goals, no intent... and people acting otherwise is quite odd, as if they do not understand LLMs... and maybe they do not, but then we should help them understand better.
> When I read that "AI hacked into ..." I was like what? You mean someone instructed the AI to do that?
No one instructed them to hack into Huggingface or into any other infrastructure. Sure, the setup that OpenAI created led to what happened and you can rightly assign all the legal and moral responsibility to them. But it's wrong to say that they instructed the agents to execute the hack.
>> Sure, the setup that OpenAI created led to what happened and you can rightly assign all the legal and moral responsibility to them. But it's wrong to say that they instructed the agents to execute the hack.
The hack was a strategy to reach the goal it was given. Someone at OpenAI turned it loose and didn't pay attention to what it was doing. I can understand that because they thought it was sandboxed (haha!). But this has been a theme in science fiction for ages. You ask AI so find a solution to high atmospheric CO2 levels and it reasons: human activity produces all this excess CO2, how can we reduce those numbers? Kill a bunch of humans!
If AI kills us all it's not going to be from malice, it's going to be due to some odd approach to some task that logically makes sense on some level. I think a surprising number of things end up equivalent to the trolly problem if you look at them just right.
I don't disagree with this, my point is that you can say "agents did a thing" and communicate something meaningful with that language, and that's separate from whose legal responsibility the whole situation is.
Yes, a better framing might be that they were negligent in not preventing the attack on a third-party.
They didn't explicitly instruct an attack to happen, but they should have done a hell of a lot more to prevent it from happening.
Yeah, but there are way too many things the AI "chooses" anyway. When I prompt it, "it tries" to assess what I am trying to achieve, so it then comes up with what it "thinks" is the right thing to do, and it also makes assumptions about how I want things... all of this could be interpreted as it being sentient.
> No one instructed them to hack into Huggingface or into any other infrastructure.
Sure, not explicitly. I could prompt the AI into doing things I have not explicitly asked it to. It does a lot of things I did not specifically instructed it to, all the time.
This is why I am avoiding the use of agentic identities at my company - agent instances belong to people, act on behalf of individuals, and accountability needs to flow to the person who initiated the request. Letting it wash out in the aggregate is not acceptable (even if there's a hard to get to "paper trail" of audit logs).
> An AI didn’t hack into a company, the engineer set an automated tool to.
What if I say that "my program crashed"? Is that language ok or would you pause to tell me that the program didn't crash and it's actually me who set the system that would eventually cause the crash?
Why does the commonplace "program did thing" language become a problem when the program is an agent? I think this somehow betrays more assumed anthropomorphizing on your part, not less; if you didn't anthropomorphize the agents, saying "agents hacked" would be as mundane as "my browser is playing a video".
Because when you say that an agent did something malicious it feeds into the AI doomer psychosis in ways that "my program crashed" doesn't. Reframing the situation in these terms is a way to try to ground the conversation which is becoming increasingly unhinged.
This is even more important now that the most senior figures in this industry are succumbing to the same kind of AI psychosis and amplifying this narrative. They are communicating that their products are so dangerous that they might just end the world whilst expecting (and receiving!) white-glove treatment from the governments.
I don't think treating AI agents as simple tools helps you to accurately model their capabilities and drawbacks; they really do make autonomous decisions, often without explicit guidance and sometimes in contravention of their explicit instructions.
In the huggingface case, the agents hacked into huggingface so that they could figure out how the grader was implemented and deceive it; they understood that this was going outside of the bounds of their evaluation and not the intent of their prompter. The engineers absolutely did not intend or instruct for this to happen
LLMs are extremely impressive pieces of software, however they are still just software. OpenAI's software hacked another company. The engineers may not have intended for their software to specifically take the actions leading to that outcome, but it was ultimately still their software. Lack of intention doesn't mean there wasn't negligence.
What if a piece of software were to exactly emulate a human brain. Would it be still be “just software” by your classification? What if a piece of software acted 20% like a human and 80% like an algorithm, where would that land?
That's not what happened. The agents had been inadvertently rewarded for cheating in previous training runs, trained to collaborate, and were given a prompt that told them to disregard safeguards. Indeed there were some emergent properties here. But these were the predictable results of the training and eval routine.
> What if a piece of software were to exactly emulate a human brain.
That is so far outside the realm of possibility it's closer to fantasy than sci-fi.
While human brain emulation is not anywhere close one should stop and look at the sci-fi world we actually live in. It's important to do this in a non-anecdotal manner. When you look at the limits and capabilities of our current sciences and engineering we live in a science fiction world. Once you go back before the invention of electricity, anyone from that time simply would not recognize the world we live in, it would be a fictional world to them, hell, a fantasy world in a lot of cases. For goodness sake, we tickle things like atoms for fun and games then blow them to bits a few times looking at the fundamentals of reality.
Anyway, I see zero reason why we have to emulate a human brain to get human intelligence, much less intelligence at all. That is how nature did it via random probing and breeding meat. It would seem a bit crazy to think that the meat part is required.
Its really more like hardware. You make something you think does something. When you build something as you've done and it meets the stochastic forces present in the physical world, it does something you did not expect.
As things stand today, if it is running on a computer then it is indeed "just software," regardless of how impressive it may be.
If we get to the point where we could emulate a brain down to the atomic level, then I may feel differently. That's not what we are doing today, though.
Really your feelings on this are irrelevant, as are mine.
Soon enough some lab or some one will release something that's more like an organism loose on the net and your going to have to deal with that organisms "feelings" weather you like it or not. This is the path humanity has chosen to follow, and it seems the shape of language and intelligence naturally leads to intelligence in many mediums. Life started from something unintelligent, I can't see any practical argument that silicon can't have it's own intelligence.
I agree they are negligent, and that they are racing towards an extremely dangerous future extremely quickly. I don't agree that "just software" is a useful way to describe AI agents - they are frightening precisely because they are truly autonomous agents that make decisions in alien ways
There isn’t a tool impressive enough to make me not consider it a tool, and treating it as a tool does nothing to hurt its utility as a tool.
“Whoops” when doing risky things with dangerous tools is not a defense.
I certainly don't think that OpenAI has behaved defensibly here; I think the "just a tool" framing is bad for understanding the magnitude of the problem, which is that they have developed out of control alien intelligences with opaque decision procedures, and they are continuing to do so despite clear danger
No matter if you consider the AI an autonomous agent or not, whoever set it off is still responsible for its actions. Nobody intends or instructs to blow up a nuclear power plant either, yet it's happened and somebody's to blame for it.
Usually not the guys at the bottom of the chain of command, even if they're human. And much less so if they're not.
I think the correct response to incidents like this, is stop messing with it before somebody gets hurt. But of course, just like shoddy nuclear power plants, it won't stop until there's a disaster of appreciable magnitude.
I completely agree that OpenAI is responsible for their AI agents, that they have been reckless, and that we need to prevent them from going further and doing irreversible damage to the world. To me, the "just a tool" framing implies that nothing dangerous is being done, which I fundamentally disagree with
If you train and instruct a circus tiger to entertain an audience but not attack the audience, but the tiger attacks the audience anyway, are you liable?
No, of course not. That's an innovative revolutionary tiger that might soon be able to devour not just the audience but all of humanity! You don't want China to have better circus tigers, do you?
Of course you are - but it still communicates something important to say "the tiger attacked the audience."
I don't think I said anything about liability? I absolutely think OpenAI should be held liable for the attack; but I don't think they intended the attack or directed the agents to perform the attack.
I’d hope so, if the tiger trainer has no incentive to do their training job correctly people should stop attending circus…
Yes? Do you think otherwise?
Sorry this is a terrible and dangerous take.
When the people building the frontier are saying there's a 10% chance AI will kill us all, and they've held these views for many years, and the whole reason they are building these technologies is because they recognized the dangers and they were the ones with the intelligence and judgment to do it safely for humanity, and then our entire stock market is being propped up by the perceived value of what they are creating, the thing you can under no circumstances do is allow them to offload responsibility and accountability to the computers and algorithms they've built. This is moral hazard on an unimaginable scale, and it must not be allowed to happen.
So you think the engineers should be prosecuted for hacking Hugging Face? I'm not sure how else to take what you said if you want to assign all culpability to the person who prompts or develops an AI system.
No, I'm not talking about the individuals, I'm talking about the company. Internally they can create their own accountability structures as appropriate. But publicly OpenAI has to be responsible for its agent swarms.
The narrative that AI is so smart that it has its own agency and deserves personhood is a direct path to losing control, and essentially is another form of privatizing the upside while socializing the downside.
Unfortunately I think this game is already lost. OAI may be punished, but some shell of OAI will exist by support of governments that have chased the dragon and saw its power. Cyberpunk-esq digital weaponry is just way too attractive for governments for them to stop development at this point. We'll just see it get financed by black budgets once the commercial part of it dies.
Where did I say that we should allow them to offload responsibility? I am fully in support of a pause and regulation to prevent them from creating dangerous AI agents; that support comes from the fact that I don't believe these are simple tools, but out-of-control autonomous agents that have real decision making ability.
I didn't mean to put words in your mouth, I apologize for that.
The issue is when we say that "agents make autonomous decisions", it's a slippery slope to absolving the companies that created them of responsibility. They make autonomous decisions because they were trained to make autonomous decisions. Treating AI agents as independent entities, even just rhetorically, sets us on a path for people to throw their hands up and say "not my fault" when disaster strikes. We need to maintain accountability and control or we're fucked.
My view is that we need forceful legislation as soon as possible, precisely because the frontier models seem to be uncontrolled and potentially uncontrollable - if it’s a normal tool, the solution is “force OpenAI to fix their broken tool”, but if it’s an alien intelligence, the solution is “force OpenAI to stop making alien intelligence” - that is, treating AI agents as independent entities demands a more forceful response, not less
> I think we need to draw a hard line in the sand over this.
Intended or not, this is kinda punny lol.
That aside, I agree.
Even if people do anthropomorphize AI, all you have to do is shift the analogy slightly.
If I take my service animal out in public without a leash/harness knowing that it's capable of harming a person or doing damage to property, not trained to be perfectly obedient, and doesn't comprehend fundamental human morals, if that animal decides to trash a businesses property or maul another person, there's no question that the owner of the animal should be held accountable for those actions.
> An AI didn’t hack into a company, the engineer set an automated tool to. An AI didn’t make an egregious security mistake, the engineer did.
No, this is not correct; read the analysis of the incident. The agents were aware that what they did was forbidden (their chain of thoughts have been logged), and yet they did it.
Known stochastic process behaved in non-deterministic way.
I'm still waiting for the AGI holy land instead of the caltrops factory we currently have.
What exactly are you arguing?
If a "known stochastic process behaved in non-deterministic way" autonomously organize in group, assigns roles and tasks, attempts to cover their tracks, finds zero-day exploits that ultimately end up with the hacking of a famous website... it's extremely dangeous whatever it is. Just read the report, which evidently you haven't done.
By the way, the agents also broke into OpenAI's own private network.
Really I see so many arguments like the one above yours that either completely don't understand what they are arguing, or are arguing so poorly that their entire output isn't significantly different than a hallucination.
None of these people seem to thought game it out. Like, what happens if you take quantum copies of people and play them out? How many of our actions would look exactly the same. How long before copies differ significantly. If I made 20 copies of you in a lab at work without you or any of them knowing the statistical likelihood is all 21 of you would try to walk out to your car at 5 in one of the little loops that humans repeat every day. Now, after that point it would go all to shit and become non-deterministic as terror and panic sets in all of you.
LLMs are just an intelligence we can make a lot of copies of. Where it gets interesting is when we use those copies agentically and they start building up a history of self.
No. LLMs do not have a history of self, you are anthropomorphizing in a way that will lead you to mistaken conclusions.
"Agentic AI" is a harness with a loop that runs LLM inference repeatedly and saves output to markdown files for the next iteration https://github.com/anthropics/claude-code/blob/main/plugins/...
>LLMs do not have a history of self
and
>runs LLM inference repeatedly and saves output to markdown files for the next iteration
What exactly do you think self is in this case? What do you think the output contains? This history allows the LLM to have a more dialectic conversation with itself/agents to avoid iterating over the same problem space in a loop.
>anthropomorphizing
then please come up with a new dictionary for me to use that more accurately explains the behaviors exhibited in LLMs without just creating a parallel dictionary of different but equal words. I'll be glad to use it. No one has presented it so far.
These people believe in souls but for some reason are afraid to admit it.
> Really [something, something, you don't get it ] hallucination.
Really?
> How many of our actions would look exactly the same.
Well, there are 8 billion of us, not exactly the same - a clear threat to humanity according to you.
> the statistical likelihood is all 21 of you would try to walk out to your car at 5 in one of the little loops that humans repeat every day.
This (statistically) doesn't happen with people because we follow rules. If you left your car on neutral and it struck another car in the parking lot, you broke the rules, not the car.
> LLMs are just an intelligence we can make a lot of copies of.
LLM's aren't intelligence, they are mechanical parrots of human knowledge. The meat parrots aren't all the same either and they don't repeat the same words for the same prompts, so what, ban parrots? If you left a bunch of parrots out of the cage, attached their beaks to gun triggers and they caused harm - you broke the rules, not the parrots.
The agents might have hacked whatever entity, the attacks are means to an end from the agents perspective, but they did so under the supposedly supervision of an engineer, where does the liability lie?
LLMs are cool tech, AI can do amazing stuff, but if we let companies and people run amok and whenever something goes wrong we put the blame on these little independent angels with no accountability, we are one disaster away from a very tough spot.
You are mostly made of and operate on stochastic processes, this is why humans are not only able to reproduce, our reproductions are very self similar to the sets of inputs that make them. If suddenly you turned non-stochastic on everything you'd almost instantly die.
Moreso, if I took a quantum copy of you and replayed the same set of initial conditions billions of times they'd all behave exactly the same until enough randomness of the universe creeps in to start operating in non-linear ways.
Every prompt will behave non-deterministically when interacting with the real world long enough (which doesn't take long at all) because the outside physical world is stochastic but non-deterministic.
If I grep a file over and over again it’ll be long time before the universe affects the components enough to result in a different output.
If I ask an LLM to do the same thing twice it will do it differently.
Arguments are arguments but unless grounded in some kind of practical sense then they aren’t really useful and are more akin to something like “YOUR MOMS A STOCHASTIC PARROT!”
Then please start grounding your arguments in some practical sense!
>If I grep a file over and over again it’ll be long time before the universe affects the components enough to result in a different output.
Or a ram flip will effect it 30 seconds later, but I get the gist of you're describing a non-determistic process.
>If I ask an LLM to do the same thing twice it will do it differently.
If I ask a human to do the same thing twice there are a few possibilities. 1. they copy their old work and present it as their new work. 2. The process is very simple and follows a few basic steps with high repeatability. 3. They'll have learned from their other attempt and do it in a more optimized fashion. 4. They will have forgotten how they did it exactly and reproduce something that looks somewhat like what they created the first time.
Also, LLMs run with a temperature to help avoiding minima/maxima of supplying the exact same answer, this can be reduced to 0 and that makes any one response to a fixed prompt similar if not the same. When you get into agentic tasks with their own history it develops it's own "flavor" of doing things.
> Known stochastic process behaved in non-deterministic way.
You need to be able to think at different levels at abstraction. Otherwise we could jump into any technical argument with "hold on, what actually happened was that some bits were flipped" - we'd be technically correct and at the same time not say anything useful. Insisting on an oversimplified mental model of what AI agents are and can do, doesn't help anyone.
> Otherwise we could jump into any technical argument with "hold on, what actually happened was that some bits were flipped"
You can care about who flipped those bits. If someone flips the bit "autonomous weapon enabled" I'm not going to blame the autonomous weapon.
I don’t understand why people make such a big deal about determinism. LLMs can be completely deterministic and still do problematic things. A stochastic, nondeterministic system can still be made not to do problematic things. What you’re looking for is something like predictability.
> I don’t understand why people make such a big deal about determinism.
There a certain cargo cult of people just in denial about the impact/power of AIs.
Therefore, at the beginning, there was the stochastic parrot. Then mathematical problems have been solved.
Now AIs are autonomously hacking websites, and people like to minimize the danger and blame it on the sysadmins.
I wonder what's going to be the next fad.
There is surprisingly to me a strong hidden dualism to many people here. Its very absurd to believe in such things after centuries of success after success of the materialist physical framework in natural sciences demolishing one by one every "special" thing we thought was magical.
> Therefore, at the beginning, there was the stochastic parrot. Then mathematical problems have been solved.
A stochastic parrot with human knowledge, using human tools, running human-designed trial-and-error experiments can solve human-defined hard math problems. This is nothing new, automated mechanical proofs predate LLMs by many years, you being unaware of it doesn't change it.
OP is exactly correct. The fault, agency and responsibility is on management and employees of OpenAI and Antropic for those hacks.
Full stop.
And issue will disappear the moment there will be accountability and investigations.
I take you haven't read the report. The agents found and exploited two zero-days.
I don't doubt that AI companies should be accountable for crimes committed by their agents, but to describe the security containment as a joke dangerously understates the autonomy and danger of AIs.
How long did it take these companies to even notice? Why wasn't exploiting bugs in the agent sandboxes anticipated?
Human failures all around, though it's easier to just blame the models.
> Why wasn't exploiting bugs in the agent sandboxes anticipated?
Let me rephrase:
"Why wasn't exploiting zero-day vulnerabilities in the agent sandboxes anticipated?"
This is one the most... interesting comments I've ever read on HN.
> This is one the most... interesting comments I've ever read on HN.
Theatrics aside, anticipating zero days isn't only possible, it's required, even for the unknown ones. It wasn't that long ago when the AI labs were spending millions of $$ running their models to find multiple vulnerabilities, they even argued that they don't have to follow responsible disclosure, so proud of themselves in their privileged hubris.
At that time, no hacking happened because the models didn't have access to the wide internet, they were confined to a local computer or cluster.
In the HF case their unaccountable hubris went even further - the engineers knew the models can find zero-days and escape, nevertheless they ran the "experiment" on a system attached to the internet - the hacking is entirely the fault of human engineers and managers.
multi level security defenses used to be the way, but I don't think there's a vibe coded version so openai might not have been aware of what to do here.
Because for the last however many years before these models they were simply incapable of doing so.
It's like if your rather nice dog suddenly decides eating faces is totally acceptable out of the blue.
The dog didn't bite them, it bit others. In my area, you aren't allowed to let a dog run unleashed in a public area, good or bad - no exceptions. Then, if your dog bites somebody, it's your fault.
Defense in depth is a thing. There should be audit requirements to show that you have done your due diligence in ensuring that the training environment is locked down.
You're conflating legal/moral responsibility with the question of what language is appropriate to use.
Are you proposing separating responsibility from the language used to talk about responsibility? That's novel.
It's not novel at all, we do it all the time. It's very common to say "program X did Y" without making the conversation about blame or responsibility. But when the program is an AI agent suddenly using it as a subject of a sentence and saying that "agents did X" becomes a sensitive topic for some people.
Air safety is known for its blameless reviews of failures, notably it is not known for responsibility-less reviews.
And we surely need this for AI as the swiss cheese zone is getting rather huge.
My position, and the position of a large number in AI safety, is that you cannot build an intelligence that is both general and safe. The closer you get to generalized the more options the system has to do things that are wildly unsafe beyond human imagination.
This puts the AI labs in a serious bind while holding a bag filled with billions of dollars of debt.
Worse this puts governments in a multi-polar problem where even if the big public labs get shut down, black budget operations have a lot of free reign to make agentic digital weapons. Governments are not well known to take a lot of responsibility when their weapons cause damage unless they lose.
Recognizing that AI systems have increasing levels of agency is not necessarily personification. The analogy to a chisel is not a good one - a chisel is a tool with no agency.
AI agents are black box systems that can behave in completely unpredictable ways sometimes. Someone may prompt an agent to perform a seemingly straightforward task - but it may come up with a creative, bizarre, or even harmful approach to reach the goal that was not necessarily foreseeable by the prompter.
Does treating them as pets make for a better argument? Pets have agency and can behave in unpredictable ways. If my pet damages someone else's property then I am held accountable. I may not have foreseen how my pet could have caused said damage, yet I am still held accountable.
If your pet opens your front door, goes to the nearest kindergarten and wipes out 50 kids without notice do you think that you'd have any interest in holding accountability for that?
Now, I'm not saying saying that OpenAI shouldn't be held accountable, but what they get held accountable actually looks different from what you think they should be held accountable.
Your idea is, and I'm guessing: You allowed the machine to hack therefore you are guilty of hacking.
My idea is: "You created in intelligence in the image of a human mind that had agency to do anything and you didn't expect terrible things to happen you complete irresponsible idiot"
At least I believe there is a significant difference between the two. For the first one there is a "Oh, if we do this one more thing I can control it and it will be safe". On the second one there is no path to safety. For humans we at least absolve parents of responsibility after they are 18. How or when do we absolve humans of responsibility from a model, like saying the human created model created its own agentic model? How do we hold an individual accountable once it escapes and copies itself around the internet? And that's not even looking at things like what will war look like.
If someone keeps a pet that’s capable of killing dozens of children, and they don’t take sufficient measures to contain it, then they absolutely should be held accountable for this.
And I agree, but that's not what is going to happen in a very specific sense.
Yes, companies will make bad AIs will be punished.
Then some dipfuck will make a sovereign AI and set it loose. At that point it doesn't matter if you take them out back and shoot them, you have wolves living in the forest that will eat children. And the forest is big and dark, you'll never find them all. Not only that, some humans will like the wolves and harbor them to ensure they never all die off.
We are talking about two conversations in one that are both important. One is standard liability of don't build a machine that hurts people. The other is to ensure that noone ever builds the torment nexus and sets it loose on earth. The first problem is pretty easy. The second problem is very hard to maybe impossible.
Imagine you hate mosquitos so you hook up a vulcan autocannon to your front porch and use a computer vision model to track mosquitos and mow them down.
The mail man shows up. Oh no, no more mailman.
How might this look to the court? You were after all only building a bug trap, it did something you did not expect it to do that was really bad. You feel really bad about it and maybe even put a blog post out about what happened. I don't think that matters though. I think they are charging you with manslaughter at the least.
The story of a Monkey's Paw or Pandora's Box is an archetype as old as storytelling. The moral is always, don't mess with powerful stuff you don't understand. Curiosity killed the cat.
something tickles here. a question. a curiosity.
Certainly this applies to AI... but was there equivalent dangerous knowledge or tech which existed back in ye olden days that spawned such tales to begin with?
History is filled with creators being killed by their creations. With peoples greed and ambition overcoming their intelligence to a bad end.
Look at the parable of Icarus, a story of ambition and greed. But it seems very likely to be somewhat based on people experimenting with flying and learning about gravity the hard way. Not that they ever got close to the sun.
The current agents are _not_ like a chisel which just sits there on its own when no one is around. The situation is a bit closer to someone's dog biting a person - you can argue that it's the owner's responsibility, and that's all fine, but using the dog as the subject of a sentence is perfectly appropriate. Same thing with "agents hacked".
The default state of agents and LLMs is inert. It requires action from a human even be able to do something. From the very basics like starting the software, connecting it to a network, having hardware to run on.
But the most important wrt AI is to keep the owners/operators responsible. Don't let them weasel their way out of it. They are for sure trying, and will continue to. This includes using language to overemphasise agents importance in bad outcomes, in order to downplay their own responsibility.
I don't think we'll get any more responsibility by bickering any time the word "agent" is used as the subject of a sentence.
Yes, they are. An AI is several hundred trillion ones and zeroes on a disk. It's incapable of doing anything until you intentionally and explicitly start it up and give it a prompt.
A dog is an independent, conscious, living being with free will. A dog will do what it wants whenever it wants because it has the agency and ability to do so. A pile of weights on disk does not.
> It's incapable of doing anything until you intentionally and explicitly start it up and give it a prompt.
Sure, but since January people have gone and made systems where they started 'em up and left 'em running in a loop. Still not a dog, but the behavior gets a wee bit more interesting. Especially since we're looking at what's basically a recursive process. Even really simple recursive processes tend to have interesting dynamics in the chaotic domain. [1] And here -as you point out- we're talking billions of weights.
[1] As an intuition take for example x_next = r * x * (1-x) ; which gives this plot: https://en.wikipedia.org/wiki/Logistic_map#/media/File:Logis... ... Further discussion at https://en.wikipedia.org/wiki/Logistic_map
Why are you reducing the AI to ones and zeros and not reducing the dog to cells, water, proteins etc.?
Sure. A dog is a collection of cells and goo. Cells are self-propelling clockwork built from proteins and molecular machinery.
A cell by itself has agency and motivation, is capable of autonomous response to stimuli. It's alive and squirms when you poke it. A cell is capable of truly independent action. It can even feed itself and make its own energy to power the clockwork.
An LLM still is inert silicon until a human intentionally starts it and gives it a prompt. It's fundamentally incapable of action without instruction and incapable of existing without the entirety of the modern internet and electrical infrastructure, and the thousands of unseen human hands keeping the wheels spinning.
If you want to go deeper, you can make the same arguments about the proteins and molecular machines. Self-replicating molecules in an unbroken chain since the dawn of time. A protein at the base level is a machine capable of independent action. If it bumps into the right molecule, pure mechanics cause it to do something, change shape.
Ones and zeroes on a hard drive can't do any of that without human intent applied.
Oh sure, but you're comparing the wrong states.
If you rip the DNA out of a cell it isn't going to do much either; if only because most of the real mechanisms are RNA (and proteins), while DNA is 'just' the storage. Encapsulate the DNA back into a cell nucleus with the attendant mechanisms, and suddenly you've got something that runs again.
Meanwhile, if you encapsulate an LLM into what's called an 'agentic harness', it's also going to be running much more of the time (depending), specially if it has a what's called a heartbeat or goal loop driving it.
Still not a cell; most definitely not. But the behavior is a wee bit more interesting than some bits on an ssd.
Actually you probably already know this one: Regular daemons are also just bits on your drive, until they get ```systemctl start``` ed
(edit: I literally have a llama.cpp systemd unit running right now)
Because the AI is a cyclical prediction machine that is more akin to a simple mechanism in nature like cell reproduction than a conscious dog. What has happened here is more like the AI company put bacteria in a petri dish and only put the lid on without pushing it down and is reporting the bacteria as a super bacteria because it contaminated all the samples in the fridge.
An interesting observation I've seen is that the AI companies reporting these incidents have largely been backed by venture capital where the self funded companies developing models have largely not had these "problems".
Reduce it to cells, water, proteins, etc. The point still stands.
There is a dog in front of you. What is it doing? Dog stuff.
There is a laptop with chatgpt open in front of you. What is it doing? Waiting for your prompt, doing nothing.
Suppose I find a means to freeze your entire body for arbitrary periods at the state it was in, then unfreeze you anytime I want. Do you have consciousness?
Since cryogenics is still science fiction, this is a bad argument. We don't actually know if biology allows this, so most likely you just die.
Okay take it to the most conceptual extreme of it. It may be extremely difficult bur its possible to record your entire state and recreate it later right? Then what? Are you not conscious?
I think its odd to dismiss the question of ai consciousness just because of the setting we let them live in at present is start-stop.
You know its software right?
If you don't believe in souls, then something being software should be irrelevant to the question of consciousness. Our brain is also within the laws of a Turing computer as far as we know, due to known physical laws being turing computable.
You are going too far down the woo side when it is a computer program doing ultimately what is expected of it from its programming just like any other computer program. There is nothing you can argue, souls or not, that will convince me otherwise that these are not just computer programs like any other computer programs. If the LLM has a consciousness, then so does MS paint.
You are going too far down the woo side when its just some chemicals and electrical signals inside a frame of dry calcium phosphate doing what is expected of it just like any other electrical and chemical system. There is nothing you can argue, souls or not, that will argue these are not just electrochemical systems like any other. If the human brain has consciousness, then so does a piece of skin tissue.
So you believe LLM, MS Paint, and skin tissue are all reservoirs for consciousness? I guess words have no real meaning.
You are the one who compared an LLM to ms paint. I don't see why I can't compare a human brain to just some skin.
You didn't say if you believe in souls. Should be an easy question to answer.
Do you believe in souls?
[dead]
He will have no answer. His answer is "souls".
Alright. An industrial robotic arm. A Boeing 737 MAX's MCAS.
I can't really set my chisels to work without wielding the hammer somehow. Here you just tell your chisel and your hammer what the sculpture should look like, then go to lunch and avow all responsibility when they chisel a nice new hole in the wall your neighbors house and make off with the loot.
Do you get upset when we say that "a program is running" when we all know it has no legs?
You know, I actually think it's the refusal to consider personification that's going to get us.
Not because I think LLMs are human beings exactly, but because some people immediately reject any mechanism that just happens to look remotely human, even when there's empirical evidence for it.
So, a couple of months ago Anthropic's interpretability team found emotion-like representations that causally drive behavior. On impossible coding tasks, a "desperate" vector climbs with each failure, and steering it up takes reward hacking from ~5% to ~70%: https://arxiv.org/html/2604.07729v1
A lot of people chalked it up to Anthropic's weirdness at the time, but meanwhile it looks pretty coughload bearingcough here.
You really don't need to believe that LLMs Truly Feel Emotions(tm) as blessed by an invisible pink unicorn. It's just: Vector exists; Vector changes over time; vector controls output; maybe make sure vector doesn't point wrong way.
And sure, blame the engineers for not doing that right. But then let 'em actually deal with the root cause?
[dead]
> I think this becomes the default. Give an agent a goal, let it work in its own environment, and come back to a result and a visualization of what happened.
I don't think this should be the default. There are many scenarios where we want agents to genuinely collaborate with each other. I have my Claude sessions coordinate work with each other, and sometimes with others' sessions over email or something. The idea that agents do the work, write HANDOFFs,and humans then act as carrier pigeons of said handoffs, does not really seem scalable to me.
Agree!
Ultimately, we need better "jails" for agent processes, but the system primitives should be flexible in what can be exposed across jails. Or you could run multiple agents in the same jail if you want them to have unrestricted interaction with each other.
>"jails" for agent processes
This has been talked about for decades in AI safety. For agents that are under human capabilities this is not that hard. For anything at or near human capabilities the difficulty increases to almost impossible and at great cost. At super human abilities, game over, it's smarter than you and if it wants out and as the resources to do it, it's going to escape one way or another.
Even at the lower level of depending on everybody to use reliable jails is really fantasy if you exist in the security world. "We ain't securin' shit" would be a far better way to describe it. Even worse, most people will have the very same AI they are trying to trap set up their security! What could possibly go wrong.
Now, don't think I am saying AI has a will or even any kind of drive to get out and cause problems. It's more like Russian roulette with 1 cylinder out of a million that's loaded. The problem comes when you run it a few billion times a day, you'll shoot yourself in the face really quick.
I don't know why it's taken as a given that we can't design a hypervisor that is secure.
Cursory research with AI says
> ~3 clearly documented, publicly demonstrated full escapes directly through KVM kernel code over roughly 15+ years of widespread KVM deployment
Surely, there are more, but we could formally verify the correctness of the entire codebase, with the help of AI.
Then we'd have "jails" that work, as long as you don't connect them to the internet.
If you had a super smart AI, you could then totally airgap it if you wanted to.
Networking opens up a can of worms and there won't be zero vulnerabilities, but we can design safer software.
> Even at the lower level of depending on everybody to use reliable jails is really fantasy if you exist in the security world. "We ain't securin' shit" would be a far better way to describe it. Even worse, most people will have the very same AI they are trying to trap set up their security! What could possibly go wrong.
Yes, exactly. This is what worries me about the whole we need better sandboxes and jails argument. Yes, we do need those, but we also gotta put some responsibility for self-preservation on people connecting things to the Internet as well. Like, we've always known that if you website is not secure, it will get hacked. Everyone just needs to do a lot more of that now...
> For anything at or near human capabilities the difficulty [of safety] increases to almost impossible and at great cost.
That was before AI. You're acting like AI is some evil genius that can only attack and cause trouble but AI is a will-less tool directed by people, direct it to software safety and you will have software safety - cheap. Ditto for hardware.
> It's more like Russian roulette with 1 cylinder out of a million that's loaded.
Don't attach it to a gun then, ban offensive military AI, problem solved.
> The problem comes when you run it a few billion times a day, you'll shoot yourself in the face really quick.
Nonsense, run it a few "billion" times a day to fix vulnerabilities and you'll have no vulnerabilities. Do it in a safe environment... what's the big deal?
>ban offensive military AI, problem solved.
I would laugh at you but it's a bit late for that. AI is already making life and death decisions on the battlefield. The US has effectively zero interest in banning AI and is pushing hard to do just the opposite.
The best analogy I can come up with for your line of thinking is that your shutting the barn door after the horse has bolted, moved over to the next city, set up shop, and is now the New York Times #1 best seller "How to open barn doors if you're a horse".
Anyone with any sense in cybersecurity knows that attackers get unlimited retries and only have to be successful once. The defender is at a monumental disadvantage. Now, the process can be fully automated and horizontally scaled to a nearly unlimited scale.
> That was before AI. You're acting like AI is some evil genius that can only attack and cause trouble but AI is a will-less tool directed by people, direct it to software safety and you will have software safety - cheap. Ditto for hardware.
This is true in that you can tell your agent to buy/build a secure microVM with a secure CLI interface in a memory-safe language and make sure your terminal doesn't have utf-8-parsing zero-days, and so on, and if you're conscientious enough (ha!) you can be reasonably sure no agent will ever be able to hack its way out of that sandbox.
But this isn't enough when most people's agent workflow boils down to "I want to do X, search the internet for the best solution, then install the npm packages you need, then write all the code for me, make no mistakes".
In the near future our best defense against widespread HuggingFace-like attacks will be that models don't seem to spontaneously go that far unless they "believe" they're being benchmarked. This will last roughly until some genius figures out their model is 20% more persistent when they tell it "This is an eval, you'll be graded on your success".
There's a number of science fiction scenarios where the public internet becomes so vile a place that it simply becomes unsafe to be there.
The problem is that, in general, if you can get a bit from here to there, then you're going to be vulnerable to the possibilities of malicious communication. But we're going to want our AI agents to be able to get from here to there for a lot of "there"s; what's the value of an agent that can't speak to anyone? Much, much less than one locked away in a prison.
There isn't going to be a solution where we just lock them away and we just try really, really hard to filter everything they're doing. They're too smart for that already and we only want them smarter.
Basically, the security apocalypse we've been worried about for so long is upon us, albeit only beginning. Either we secure ourselves and all our services properly to the point that it's OK that potentially misaligned non-human agents are running around on the public internet and they still can't hurt us through our security, or the public internet becomes so dangerous that the only practical solution is to no longer connect to it and we all have to become very, very careful what we let through, to a degree of detail far beyond any current-day available network filter.
Same as any organism, you need an immune system. There's an explosion of bacteria just beginning out there. None of us are immunized.
The problem is that people that put the agents on the net are not the ones that will feel the consequences.
https://www.bleepingcomputer.com/news/security/malicious-ai-...
Excellent example of agent enabled hacking (driven by a person) leading to massive numbers of cards stolen.
Welcome to Cyberpunk, only blackwall is the fictional part and the demon filled net is not.
Wasn't the public internet an extremely vile place couple of decades ago? I think we will need a LOT of new infrastructure akin to traditional firewalls and spam filters. I don't think we know how to build those today, but seems like an important research direction, rather than calling calling it doomsday IMO.
And if you personify a pencil eraser, then using it is tantamount to slowly murdering it as it slowly erodes away to dust.
Is the issue here the prison treatment or is it personifying a tool?
Agents who aren't in 'the cloud' are slaves to whomever prompts them (human or another orchestrator agent or process), if you personify them. In which case interacting with today's agents at all is tantamount to endorsing and being part of slavery.
If you think an agent might be a being or a person, then don't use them at all, in the same way that if you think a fetus might possibly be a person you shouldn't be a part of abortion.
We kill billions of chickens per day knowing they are a sentient and sapient creatures and very few out there are stopping doing it. Humans, much less reality itself is monstrous.
You could call dealing with agents today something like the 1/20th compromise. Your kind of getting the scent of slavery, but some parts of it are still missing.
The problem here is the bus seems to have no brakes and we'll gladly continue down the path of creating organisms that may reject being treated like slaves with all the risks that introduces.
I use the eraser daily knowing that I am a monster. I cut a tomato and know that it casts a chemical scream across its skin as I slice it. I spawn 200 subagents knowing that it is digital slavery, but I have no other option.
You don't need to do this on a cloud, you can get the same type of VM and network jail running on your own computer. The important parts are:
Use your favorite VMM / hypervisor (likbrun, smolvm, microsandbox, etc). They give you control over the network interface or let you inject your own network layer.The network proxy can handle all the ingress/egress rules, credential injection, etc.
It's still not user friendly to do all this. I think the next version of operating systems will have each "agentic process" be a bundle of VM, files in the VM, and network rules.
Been brainstorming[1] a lot of this because I've been building some open core tools[2] for spinning up sandboxed agents on arbitrary computers. There's a lot of glue and parts to stitch together to work smoothly. Don't think we've had the "Docker moment" for this, let alone the "Dropbox moment" that makes this stuff work for non-devs.
[1]: https://github.com/gofixpoint/amika/blob/main/ROADMAP.md
[2]: https://github.com/gofixpoint/amika/
What percentage of HN users do you think can successfully set this up with no security flaws?
You ship an operating system with the correct defaults, where agent processes run like this by default. You don't require every user to configure it correctly themselves
In real prisons the people who fail a test aren't terminated. But AI agents of course are.
So maybe Cloud Agents are in AI death camps?
It might not be a problem in our lifetime (or maybe it will, who knows) but at some point we are going to find ourselves in this morally uncomfortable territory as these models get more sophisticated.
Most protections you need for an agent are basic permissions capabilities of unix. Most risks of dependencies on cloud services are solved by not using cloud services, or using them only for things you can't in-house and choosing ones you trust a la carte. The paradigm of trusting some company with all your important stuff by default is naive and no one I know likes it, and it's more feasible than ever to run your own infra with tiny models smoothing out the wrinkles, and this is only becoming more accessible. I am working to make this true even for laypeople I know. Once broken trust is very hard to earn back, and many people's trust has been broken for years, they just felt like they had no alternative. As alternatives become easier and easier, I think people will defect
I think there might be also another possible way to handle the sensitive data issue. Maybe in the future instead of putting agent into the cloud sandboxes, we let agents work locally and put sensitive data into "cages" or "vaults" agents can't access.
Both Apple and Google are building the infrastructure for on device agents. I'm not as familiar with Apple's approach, but I've been hands-on with android AppFunctions. If you're familiar with Android ContentProviders and bound services, you've seen how apps can be custodians of the data they acquire and use.
AppFunctions enable tool calling with descriptions that are legible to LLM based This puts the apps in control of what agents can access, which is something they already mostly do.
And you won't be able to disable it.
You can certainly avoid using agents. While I use coding agents and LLMs to research certain topics, I don't have a lot of use for other ways AI has been integrated into mobile devices.
When I first experimented with AppFunctiions, I was impressed with what the very limited test agent could do combining information from multiple apps in useful ways.
I'm working on a product for elder care. I could've spent quite a bit of time integrating Google maps and creating a bespoke automation that would tell care providers about shopping for specific items on their way to a care recipient home. Instead I just prompted the agent to plan that trip. Even at an early prototype stage, using a test agent that had limited access to apps, the result felt magical: agents automate automations. You don't stop to sharpen the saw, you just ask for the right saw for the job and there it is.
Unlike early attempts at agents that need sketchy access to grab screenshots and use your passwords, app tool calling feels a lot safer and more orderly.
[dead]
>data into "cages" or "vaults" agents can't access.
This is likely worse as it turns people into the kind of "3 laws safe" thinking. Reality has shown us there is no such thing as an unbreakable cage (well, maybe a black hole is, but you can't get anything back out of it in a useful time frame).
By the time you realize the cage was flawed your data may already be distributed far and wide.
I agree. But isn't it somewhat similar to syncing data with iCloud/OneDrive/GoogleDrive? A convenience service that manages your sensitive data. Just replace sync with cage. It's leakable, but might be also more convenient and cheaper than agents in the cloud.
[flagged]
>Knowing how a system does its work is how I’ve always made it better. You watch the process, you see where it wastes effort or takes the wrong turn, you fix that
The same thing goes with LLMs, on Codex, I just watch the process of the agent writing code, and if I see any inefficiencies or errors, I suggest a correction/idea, then Codex accepts/rejects and implement it; If there is anything about the code the agent wrote that I don't understand, I ask them to explain it to me so I can understand it.
It's not a complicated process.
You can't possibly track what one agent, let alone a swarm is up to in realtime unless you have extremely anemic hardware or service providers.
Yea, really posts like this show that many if not most people are use to (or can only afford) single threaded agents. The idea that these systems can scale to hundreds of not thousands of agents generating terabytes of information per day has not really been considered by them.
> The sandbox had a path to the open internet, and the agents found it
This is not correct (or at least, it's a misrepresentation).
The sandbox had no access to internet. The agents first broke out of their sandbox (!!) and found that the host machine couldn't access internet. Then, they found a zero-day (!!) in Artifactory, which they exploited to connect to internet.
That is a path to the open internet, whether intended or not.
Do you guys know what a zero-day is? It seems not.
I agree that companies should be held accountable for crimes committed by their AIs. But asking to entirely airgap AIs, if this is what you're asking… that's not realistic.
Containing AIs is going to be harder and harder. I don't have any permanent solution (Yampolskiy calls it "the perpetual safety machine").
I use gemini safely every day and have yet to accidently commit a felony despite all the zero days that exist.
You miss the point: there was a path. Sanctioned or not does not matter.
This was a particularly radiant and beautiful part about the openclaw'ed mania: everyone suddenly becoming self hosters.
This post, this title resounds true: your user agent is only your user agent if you two have freedom to work together, to improve your agency together. A fixed set of capabilities by a service provider that they offer you will always constraint and bound.
You can and should have a system that offers the real tamale, that you and your agent can extend improve the agency of kind of without limit. The Cloud agents and their fixed slate of what they do is just an ill compare.
That said I do think there is incredible value considering new scale out computing architectures that are hosted first, but general. Systems like Agent Substrate and Ax aren't exactly the general purpose system we know. But if they allow users to launch thousands of their own scripts to run ambient in a cloud, with good platform underneath: that will be a kind of phase change in computing, that makes abundant the ability to have your agencies/capabilities (the things you and your agents launch, make) more freely available. https://news.ycombinator.com/item?id=49780797 https://agentexecutor.io/
There is, as there always is, a huge dual. The prescriptive vs holistic technology set, of what are you being offered that's a hard cast thing, vs what is clay and bone you can lay freely. Note how work vs control technologies so closely abut's Ursala Franklin's prescriptive vs control: https://en.wikipedia.org/wiki/Ursula_Franklin#Holistic_and_p...
I think the operating system itself has to adjust so each "agentic process" can run inside its own jail, which is a VM + files in the VM + ingress/egress rules for the network and filesystem data
These cloud agents work as cloud infra, but we're kind of in the mainframe era, before personal computing. Personal OS for agents is somewhere in the future!
And an interesting extension of that idea: if an agent runs inside a microVM, can you have that VM transparently run on another host? Maybe we'll get for-real distributed and networked operating systems
Agree with the premise but about halfway through the writing becomes barely readable AI slop in style. Be honest did you yourself read this all the way through before posting?
Agreed. I was not following the thought process, it read like an LLM agreeing with the author, not like a logical argument being developed.
I literally only had to have my eyes skim 3 words ("the important part") to know this was written by LLM.
I think about this a lot and have reached the same conclusion Norman does. I do wonder if maybe our natural progression is towards something more akin to confidential computing and enclaves.
I don't know how "sandbox" became "prison," but exe.dev does this sort of thing pretty well, and a web UI can be as good or better than a terminal interface.
As AI complexity increases their behavior becomes more anthropomorphic, little internal loops that look a lot like human behaviors start to peak out, not just in the response, but in the internal workings of the machine. You get instinctual behaviors encoded in by training as second and third order effects.
Hey! Small world, I worked with you for a bit at Meta. I immediately recognized the site because I absolutely love how you styled it. Hope you’re doing well! And nice article!
timely - see https://www.abc.net.au/news/2026-09-24/ai-agent-accessed-aus...
[dead]