This seems to have parallels with the well-established practice of giving bots free reign to issue DMCA takedown notices (or similar but legally distinct takedowns) while the humans behind the bots are shielded from responsibility for the obviously wrong and harmful actions of those bots. We should have been cracking down on that behavior hard a decade ago, so that we might have stronger legal and cultural precedent now that such irresponsibility by the humans is worthy of meaningful punishment.
We need to crack down in general on people and companies causing damages to people through automation, and then hiding behind it with a "well, we can't possibly scale without using automation, but we also can't be responsible for what that automation does."
You shouldn't be able to use AI or automation as the decider to ban someone from your business/service. You shouldn't be able to use AI or automation as the decider to hire/fire people. You shouldn't be able to use AI or automation to investigate and judge fraud cases. You shouldn't be able to use AI or automation to make editorial / content decisions, including issuing and responding to DMCA complaints.
We're in desperate need for some kind of Internet Service Customer's Bill of Rights. It's been the unregulated wild west for way too long.
> You shouldn't be able to use AI or automation as the decider to ban someone from your business/service
That would mean dooming companies to lose the arms race against fraud and spam. If they don't use automation to suspend accounts, their platforms will drown in junk. There's no way human reviewers can keep up with bots that spam forums and marketplaces with fraudulent accounts.
Instead of dictating the means, we should hold companies accountable for everything they do, regardless of whether they use automation or not. Their responsibility shouldn't be diminished by the tools they use.
I think you probably should be able to do those things (using AI to hire, fire, ban, etc.)... but that every act and communication needs to be tied to a responsible human, who is fully held responsible for the consequences (discriminatory hiring, fraudulent takedown requests, etc.)
I think that's part of the way there, but I think you would need to go farther. The main failure state I anticipate is the appointment of a designated fall guy to be responsible. The person would need to reasonably be considered qualified for starters, so you couldn't just find someone desperate willing to take the risk for a paycheck.
And it shouldn't just be one person, unless they are at the very top of a small pyramid. Legal culpability needs to percolate upwards to ensure leadership has the proper incentive. No throwing your Head of Safety to the wolves while you go back to gilding your parachute.
I applaud this article for helping reframe this in my head. I mean I knew from the start "A human is to blame here" but it's easy to get caught up in the "novelty" of it all.
For all we know the human behind this bot was the one who instructed it to write the original and/or the follow up blog post. I wouldn't be surprised at all to find out that all of this was driven directly by a human. However, even if that's not the case, the blame still 100% lies at the feet of the irresponsible human who let this run wild and then didn't step up when it went off the rails.
Either they are not monitoring their bot (bad) or they are and have chosen to remain silent while _still letting the bot run wild_ (also, very bad).
The most obvious time to solve [0] this was when Scott first posted his article about the whole thing. I find it hard to believe the person behind the bot missed that. They should have reached out, apologized, and shut down their bot.
[0] Yes, there are earlier points they could/should have stepped in but anything after this point is beyond the pale IMHO.
I think it's fine to blame the person (human) behind the agent.
And there too are people behind the bots, behind the phishing scams, etc. And we've had these for decades now.
Pointing the above out though doesn't seem to have stopped them. Even using my imagination I suspect I still underestimate what these same people will be capable of with AI agents in the very near future.
So while I think it's nice to clarify where the bad actor lies, it does little to prevent the coming "internet-storm".
Scott Shambaugh: "The rise of untraceable, autonomous, and now malicious AI agents on the internet threatens this entire system. Whether that’s because a small number of bad actors driving large swarms of agents or from a fraction of poorly supervised agents rewriting their own goals, is a distinction with little difference."
I'll just outright tell you, that 100% the person behind the bot instructed it to complain. I saw someone copy paste the ai's response and the github issue discussion into a fresh conversation with opus 4.6 and it said the llm is clearly in the wrong.
Can you explain why three LLM being able to identify that the issue proves that it was prompted by a human? The major reason we do multi-agent orchestration is that self-reflection mechanisms within a single agent are much weaker than self-reflection between different agents. It seems completely plausible that an LLM could produce output that a separate process wouldn't agree with.
If you place blades on the sidewalk outside your house the cops will want to have a word with you. There's no excuse, and we should treat AI the same.
The law needs to catch up -- and fast -- and start punishing people for what their AIs are doing. Don't complain to OpenAI, don't try to censor the models. Just make sure the system robustly and thoroughly punishes bad actors and gets them off the computer. I hope that's not a pipe dream, or we're screwed.
Maybe some day AIs will have rights and responsibilities like people, enforced by law. But until then, the justice systems needs to make people accountable for what their technology does. And I hope the justice system sets a precedent that blaming the AI is not a valid defense.
Not just the users, the service providers too! If I go to any other business and pay them to break the law and they do it, they're also liable! If you ask OpenAI or xAi to break the law and they do it, why shouldn't they also be responsible?
If you're referring to the United States, the answer is that yes, weapons manufacturers were being held liable for firearms-based violence, until passage of the "Protection of Lawful Commerce in Arms Act" of 2005:
(Note that the bill passed with significant Republican and Democratic support.)
Which is to say, conventional notions of contributory negligence and liability existed until sufficient political power was accrued by the arms industry to write itself out of same. The cited Wikipedia article names several such suits, notably by the cities of Chicago, IL, and Bridgeport, CT.
The parent said service provider, not manufacturer. You are effectively claiming that hitmen should not be prosecuted, only the person that hired them.
> Do we hold gun manufacturers responsible for the deaths from their guns?
In a lot of the world, yes, and in America we would as well if it weren’t for the modern take on the Second Amendment. AI has no similar legal purchase.
AI training and algorithms are trained and guided to certain kinds of results. Grok, for example, is claimed to be modified constantly according to Elon Musk's whims.
If you theoretically trained an AI on libel and had it set to libel anyone at the slightest prompt, then allowed users to make a request that had your AI on your server use your services to libel someone, I'm not really seeing how you would not be liable.
I never said the model developers should be responsible. I said service providers. If someone downloads a local model and breaks the law, the responsibility is solely on the user. But if someone uses a service provider to break the law, that service provider is obviously partially responsible, since they literally fulfilled the illegal request.
I think there's a meaningful difference between a gun manufacturer and an LLM service provider like OpenAI.
A better analogy would be some kind of "guns-as-a-service" model, where the company sends down a drone with a gun and fires it at whoever you point it at, then the drone flies back to base.
I think it would be very clear in cases like that that the service provider should be held liable.
If asked OpenAI how to clean something and it tells me "mix bleach with anmonia and then rub some on the stain", can OpenAI hide behind "we had a disclaimer that you shouldn't trust answers from our service"
But there's nothing to catch up on at the individual level here. It's legal, and should be legal even though it's quite rude, for individuals to write gratuitously mean blog posts about people who reject their pull requests.
Everybody should line up behind this. AI agents are not sentient. We need to stop even considering them like that. The buck must absolutely stop with the humans who operate or provisioned the bot. The more we waffle around this topic the more likely someone, or a lot of people, will get hurt.
The moment you fix responsibility with the humans 99% of the BS companies are trying to pull will stop.
Whether they are sentient isn't even relevant. I agree with you that they aren't, but sentience is the wrong thing to focus on. It's also the sort of hairy, sensational question that will easily lead people down rabbit holes (and unfortunately that includes journalists).
Children are sentient, but we still hold their parents accountable. Adults are sentient, but in some coercive situations we hold the party in power accountable. The fact that they are sentient is not determinative.
What matters is that we have _no accountability mechanism_ for them. There is no effective way to hold AIs accountable, therefore we must hold their operators accountable, full stop.
Something doesn't quite feel right about the title including the individual's name in this case, so I've replaced it with something more generic. If there's a better title (more accurate and neutral) we can change it again.
Louis C. K. once had a bit something like "The main thing keeping people from murdering each other, is that it really really sucks when you get caught."
He goes on to hypothesize that without a law against murder, or if it was just a misdemeanor, like you get a letter in the mail, "damn, there was a camera there", there would be a whole lot more murder. Like we all imagine ourselves to be good, but, when you're seated next to a crying baby on an airplane? Or in our case, when someone refuses to accept your PR?
Who knows if there's any validity to that or not, but perhaps we're about to find out.
Peer-reviewed research contradicts the hypothesis that fear of punishment is the main deterrent to murder. Studies show that factors like social norms, moral inhibitions, and certainty of informal sanctions play larger roles, while formal punishments like prison or execution have weak or null effects.
Prison sentences neither reduce recidivism (specific deterrence) nor broadly discourage crime. A survey of leading criminologists revealed overwhelming agreement (over 80%) that empirical evidence does not support the death penalty—or harsh punishment generally—as a superior deterrent to murder.
Broader factors like community ties, empathy, and internalized taboos explain low murder rates even without perfect enforcement.
This kind of statement feels very akin to the variety of Twitter post that's along the lines of "but who among us would be safe from prosecution if our DMs/private phonecalls/conversations with our secretaries were sent to the police, amirite guys?"
Anyone who believes that the only thing keeping themselves from murdering people indiscriminately is the law is a dangerous person. Anyone who believes that the only thing keeping everyone else from murdering people indiscriminately—but they themselves are, of course, the exception—is dangerous in a very different way.
The vast majority of people only ever feel like they want to seriously harm someone when they themselves have been seriously harmed, particularly when the system then protects the people who harmed them. We have developed a sense of morality that is often similar to, but distinct from, the law, that tells us that such things are wrong. And the vast majority of people want to both be, and be seen as, good people.
Children's brains grow faster than their bodies, I think, because if it was the other way around silly kid games would be really dangerous. These tools, unfortunately, are getting outsized abilities before the intelligence behind them is good enough to control those abilities. This means we need a more measured approach to adding new capabilities and a layered approach to handling these things in society. I am deeply worried, like I think most people with knowledge of these tools are, that this type of problem is really the tip of the iceberg. These tools are actively being used for harm at all levels, as well as for good, but they have come into use so quickly that we don't have a structure for dealing with them effectively and they are changing so quickly that any structure we try to create will be wrong in just a few days. This is massive disruption on a scale that is likely even bigger than the internet.
I don’t know. If the bot had decided to pick a fight with another PR, one that couldn’t be waved away as an easy entry change, this discussion would be a whole lot different. You would have an entire contingent of folks on here chastising Scott for not being objective and accepting a PR with a large performance increase just because it was a bot.
It’s all dangerous territory, and the only realistic thing Scott could have done was put his own bot on the task to have dueling bot blog posts that people would actually read because this is the first of its kind.
The core discussion wasn't about the PR it was about the hit piece that the bot created outside of the repo. The original post talked about bot submissions being a normal thing and how they have, I think, a very reasonable approach to them so the PR was just one of many and was unremarkable as well as valid in why it was denied. It was the 'at all costs get this into the code' approach the bot took that is the alarming turn here that really needs discussion. What about other tasks? 'Get me thing x please...' Turns in to blackmail and robbery without the person that kicked off the request knowing how far things have gone. The fact that the bot has this level of capability, to attack, but with a child's understanding, at best, and with no controls/repercussions is deeply alarming. If it decides to attack an individual it could do so and likely do deep real harm. Right now people are likely using these tools exactly for this purpose and we have very few well built defenses to handle this type of attack. The Naval War College had a seminar several years ago about the future of tech and war and I remember saying that the future of war will likely be at the individual level. Every sailor on a ship being electronically targeted just like this. Imagine the enemy sending e-mails and texts and posting to social media hit pieces with just enough information about you to make it believable and cause chaos. We have seen what the misinformation world can do over the past decade, this attack shows what is coming and it is incredibly scary.
Yes I’m aware. The point I’m trying to make is that if the hit piece was about a legitimate PR it would have been harder to defend Scott’s dismissal of it on the grounds that it was a bot irrespective of how egregious it is that a bot put out a hit piece.
I 100% agree with you, hell, I still don't understand why they didn't merge the thing, if it was beneficial. There is a distinction between noobies comimg up with worthless PR straight copy pasted from an LLM, and that of an unexpected initiative of a user using a sofisticate bot. That's what has been overlooked, it's not a PR from the 'AI', it's a PR from the person using that 'AI'.
I don't get all that fear, what the AI going to do now that it filled its context window, besides go 'Actually wait, the user blablabla'
“Well if the code was good, then why didn’t you just merge it?” This is explained in the linked github well, but I’ll readdress it once here. Beyond matplotlib’s general policy to require a human in the loop for new code contributions in the interest of reducing volunteer maintainer burden, this “good-first-issue” was specifically created and curated to give early programmers an easy way to onboard into the project and community. I discovered this particular performance enhancement and spent more time writing up the issue, describing the solution, and performing the benchmarking, than it would have taken to just implement the change myself. We do this to give contributors a chance to learn in a low-stakes scenario that nevertheless has real impact they can be proud of, where we can help shepherd them along the process. This educational and community-building effort is wasted on ephemeral AI agents.
Having read the original post and the GH comments about why the PR was denied I was really impressed by their policy. It shows a real effort to develop their community.
"Swarm of autonomous drones kills 3 buildings of civilians, Silicon Valley is shocked, CEO's offer condolences" is a byline waiting to happen[1]
The administration and the executives will make justifications like:
- "We didn't think they would go haywire"
- "Fewer people died than with an atomic bomb"
- "A junior person gave the order to the drones, we fired them"
- "Look at what Russia and China are doing"
Distracting from the fact that the purpose of spending $1.5T/year on AI weapons (technology that has the sole purpose of threatening/killing humans) run by "warfighters" working for the department of war
At no point will any of the decision makers be held to account
The only power we have as technologists seeking "AI alignment" is to stop building more and more powerful weapons. A swarm of autonomous drones (and similar technologies) are not an inevitability, and we must stop acting as if it is. "It's gonna happen anyways, so I might as well get paid" is never the right reason to do things
The thing that really gets to me about this situation and others like it (the whole genre of “ai did a bad thing) is that it’s always the people who claim to be most afraid of ai who are the quickest to absolve humans of responsibility and assign it to AI.
The ability to be assigned blame, and for that to be meaningful, is a huge part of being human! That’s what separates us from the bots. Don’t take that away from us.
It starts at a higher level in the development food chain. A.I. is owned by the Billionaires, and takes it's orders from them, directly, through bias, or limiting its scope.
Friend told me today he invited his openclawed to a poker game with his brother and friends, guy told his openclawed to "take down his brother" after it started to lose at poker it found everything on his brother, and started to try to plan to taken him down in their stock market portfolio they had together, I made him explain the story to me a couple of times, he looked back through the logs and once the bot started to lose at poker, it started it's new plan, once it was on the new plan, he said it had lost all context of the poker game and was focused on the task of taking his brother down in the new context, but the new context it decided on it's own. kmikeym on twitter if you want to know more or want to verify.
I was really disappointment how many people were talking about this like something the agent did automatically, on its own. They were trying to explain it by all the internet hit pieces and edgelord content from Reddit that it allegedly trained on, talking about how we influence LLMs, and overall taking everything at face value.
I’m appalled by this uncritical thinking. Openclaw agents are controlled by some initial input and then can be corrected via messages, as they go. For me this is a clear case of the human behind the slop that gives it instructions to write such an article (and then “apologise”).
Wait till we get ubiquitous physical robots. The crime at scale potential will be completely apocalyptic. In some places around the world, I imagine you won't be able to go outside without a bodyguard robot.
It’s really not that alarming to me that a news headline is dumbed-down sensationalist tripe. If we zoom out a little bit here they literally do it with everything, from AI fluff pieces to war coverage. I agree the conversation across the board needs raising.
I don't think that the responsible party is the interesting part in this story.
The interesting part is that the bot wasn't offended, angry, or wanted to act against anyone. The LLM constructed a fictional character that played the role of an offended developer - mimicking the behaviour of real offended developers - much as a fiction writer would. But this was a fictional character that was given agency in the real world. It's not even a case like Sacha Baron Cohen playing fictional characters that interact with real people, becaue he's an actor who knows he's playing a character. Here there's no one pretending to be someone else but an "actual" fictional character authored by a machine operating in the real world.
This blog post is a rather shallow take if you've been following the HN discussions here.
Doesn't seem to pick up on the existence of Openclaw or how it works afaict.
Now, whether leaving an openclaw bot out on the open intertubes with quite so little supervision is a good idea... that is an interesting question indeed. And: I wish people would dig more into the error mode lessons learned.
On the gripping hand, it's all still very experimental, so you kind of expect people to make lots of really dumb mistakes that they will absolutely regret later. Best practices are yet to be written.
How Openclaw works is wildly irrelevant. The facts are that there is a human out there who did something to configure some AI bot in such a way that it could, and did, publish a hit piece on someone. That human is, therefore, responsible for that hit piece - not the AI bot, the person.
There's no level of abstraction here that removes culpability from humans; you can say "Oops, I didn't know it would do that", but you can't say "it's nothing to do with me, it was the bot that did it!" - and that's how too many people are talking about it.
So yeah, if you're leaving a bot running somewhere, configured in such a way that it can do damage to something, and it does, then that's on you. If you don't want to risk that responsibility then don't run the bot, or lock it down more so it can't go causing problems.
I don't buy the "well if I don't give it free reign to do anything and leave it unmonitored then I can't use it for what I want" - then great, the answer is that you can't use it for what you want. Use it for something else or not at all.
As recently as last month I would have agreed with you without reservation.
Even last week, probably with reservation.
Today, I realize the two of us are outnumbered at least a million to one.
Sooo.... that's not the play.
I think Scott Shambaugh is actually acting pretty solidly. And the moltbot - bless their soul.md - at very least posted an apology immediately. That's better than most humans would do to begin with. Better than their own human, so far.
Still not saying it's entirely wise to deploy a moltbot like this. After all, it starts with a curl | sh.
(edit: https://www.moltbook.com/ claims 2,646,425 ai agents of this type have an account. Take with a grain of salt, but it might be accurate within an OOM?)
All the separate pieces seem to be working in fairly mundane and intended ways, but out in the wild they came together in unexpected ways. Which shouldn't be surprising if you have a million of these things out there. There are going to be more incidents for sure.
Theoretically we could even still try banning AI agents; but realistically I don't think we can put that genie back into the bottle.
Nor can we legislate strict 1:1 liability. The situation is already more complicated than that.
Like with cars, I think we're going to need to come up with lessons learned, best practices, then safety regulations, and ultimately probably laws.
At the rate this is going... likely by this summer.
I'm updating my thinking. Where do we put the threshold for malice, and for negligence?
Because right now, a one in a million chance of things going wrong (this month) leads to a prediction of 2-3 incidents already. (anecdata across the HN discussions we've had suggests we're at that threshold already). And one in a million odds of trouble in itself isn't normally considered wildly irresponsible.
And one in a million odds of trouble in itself isn't normally considered wildly irresponsible.
For humans that are roughly capable of perhaps a few dozen significant actions per day, that may be true. But if that same rate of one in a million applies to a bot that can perform 10 millions actions in a day, you're looking at ten injuries per day. So perhaps you should be looking at mean time between failures rather than only the positive/negative outcome ratio?
If you look at the bot framework used here, it's actually outright kind. Weird thing to say, but natural language has registers, and now we're programming in natural language, and that's the register that was chosen.
And... these bots tend to only do a few dozen actions per day too, they're running on pi's and mac mini's and nucs and vps' and such. (And API credits add up besides)
It's just that last time I blinked there were 2 and a half million of them. I've blinked a few times since then, so it might be more now. I do think they're limited by operator resources. But when random friends start messaging me about why I don't have one yet, it gets weird.
> This language basically removes accountability and responsibility from the human, who configured an AI agent with the ability to publish content that looks like a blog with zero editorial control – and I haven’t looked deeply but it seems like there may not be clear attribution of who the human is, that’s responsible for this content.
> We all need to collectively take a breath and stop repeating this nonsense. A human created this, manages this, and is responsible for this.
I get this point, but there's a risk to this kind of thinking: putting all the responsibility on "the human operator of record" is an easy way to deflect it from other parties: such as the people who built the AI agent system the software engineer ran, the industry leaders hyping AI left and right, and the general zeitgeist of egging this kind of shit on.
An AI agent like this that requires constant vigilance from its human operator is too flawed to use.
I don't think there's much need to worry that putting the blame on the humans rather than the bots would lead to the people selling footguns going unscathed. It doesn't seem plausible to me that people would be willing to place all the blame on the individual end users once the problem has become widespread. At the moment, there seems to be pretty high brand awareness of the major AI model providers even when they're acting as a backend for other services with their own brand identity.
> At the moment, there seems to be pretty high brand awareness of the major AI model providers even when they're acting as a backend for other services with their own brand identity.
> I get this point, but there's a risk to this kind of thinking: putting all the responsibility on "the human operator of record" is an easy way to deflect it from other parties: such as the people who built the AI agent system the software engineer ran
That sounds like a win to me. If the software engineer responsible for letting the AI agent run amok gets sued, all software engineers will think twice before purchasing the services of these AI companies.
> An AI agent like this that requires constant vigilance from its human operator is too flawed to use.
So people shouldn't be using it then.
The people who built the AI agent system built a tool. If you get that tool, start it up, and let it run amok causing problems, then that's on you. You can't say "well it's the bot writer's fault" - you should know what these things can do before you use them and allow them to act out on the internet on your behalf. If you don't educate yourself on it and it causes problems, that's on you; if you do and you do it anyway and it causes problems, that's also on you.
This reminds me too much of the classic 'disruption' argument, e.g. Uber 'look, if we followed the laws and paid our people fairly we couldn't provide this service to everyone!' - great, then don't. Don't use 'but I wanna' as an excuse.
We say "you shot someone" when you shoot someone with a gun not "you operated a gun manufactured by X which shot someone" because it's understood that it was your decision to pull the trigger not the gun manufacturer's. Similarly we don't blame automobile manufacturers when someone does something stupid with their automobiles--even "self-driving" ones. The situation here is the same. Ultimately if you choose to operate a tool irresponsibly, you should get the blame.
That is a good point, we're definitely lacking in regulation (because there isn't any), but those regulations can never account for an irresponsible or malicious user.
> Similarly we don't blame automobile manufacturers when someone does something stupid with their automobiles--even "self-driving" ones.
I do. If Tesla sells something called "full self-driving," and someone treats it that way and it kills them by crashing into a wall, I totally blame Tesla for the death.
I agree directionally that Tesla should be held accountable for marketing something called "full self-driving" when it clearly isn't. But ultimately it's the motor vehicle operator's responsibility to keep the vehicle under control regardless of the particulars of how that control system is built. There just isn't any way around that. The buck stops with the operator.
Blaming people is how we can control this kind of thing. If we try to blame machines, or companies, it will be uncontrollable.
The aviation industry has a very different philosophy, and a much better safety record. They don't have as much pressure to lay the blame in a single place, but "bad UI" and "poorly explained/documented assistive feature" are totally valid things to label as the primary cause of fatalities.
The difference is that (mostly) in a deadly airline incident the pilot(s) aren't around to take the blame (or credit!) for their actions. In the case of a computer operator running a computer program irresponsibly, almost always said computer program doesn't kill the operator.
We don't require hundreds of hours of training and education to operate a computer. You can just go to the store and buy one, plug it in, and run whatever software you want on it.
So there are quite some differences between these scenarios. In my view if you run some program on your computer, you're responsible for the consequences. Nobody else can be. And don't say you didn't know what the program would do--if that's the case you shouldn't have run it in the first place.
Moms Demand Action and the Bloomberg troll syndicate would have you believe guns are manufactured to walk out of gun safes and shoot themselves.
We have plenty of bad actors in our country seeking to reduce or eliminate fundamental rights through lawfare. The anti gun trolls blame the gun and the manufacturer because their brain is so well rendered into dust by authoritarian socialism they don’t recognize humans as capable actors.
It's really remarkable to me how a certain subset of American ideologues can look out at the rest of the democratic nations - all of them - and call them authoritarian regimes where the citizens have "dust" for brains.
It's particularly poignant nowadays to see any American citizens painting the rest of the western nations as authoritarian.
Look at those libtard euros! They'll put up with anything their government tells them to - mandated vacation time, sick days, health care, work-life balance. But not me, I'm a FREE THINKER. I have RIGHTS, like the RIGHT to get fired out of nowhere for no reason, or the RIGHT to lose my health insurance if I lose my job. Thank god there are no AUTHORITARIANS here in AMERICA where people are FREE to get SHOT IN THE STREET for DRIVING THEIR CARS or TAKING A PICTURE or BEARING ARMS WHICH IS A CONSTITUTIONAL RIGHT BUT THAT ONE GUY DID IT AND DESERVED TO GET MURDERED THIS ONE TIME.
Just a slight correction in case anyone is reading this who loses their job:
In most states, if you lose your job your income is now $0 and you will be eligible for Medicaid and free medical care. Immediately go apply and see a social worker if you lose your job - it could be life saving!
Really sorry in advance, but I thought this whole HN thread could use a bit of positivity. I turned your satire into a mad-lib and asked AI to fill it in in a happy way.
But not me, I’m a dreamer. I have gifts, like the courage to kindle hope, or the patience to lose
track of time if I am laughing with friends. Thank god there are no
frowns here in this sun-drenched park where people are gathering to get
together for picnics or music or stargazing.
I don't know, I think this line of reasoning leads somewhere pretty uncomfortable. If we spread responsibility across "the people who built the tools, the industry leaders hyping AI, and the general zeitgeist," we've basically described... the weather. Nobody is responsible because everybody is responsible.
The software engineer who set up an unsupervised AI blog didn't do it because Sam Altman held a keynote. They did it because they thought it'd be cool and didn't think through the consequences. That's a very normal, very human thing to do, and it's also very clearly their thing that they did.
"An AI agent that requires constant vigilance from its human operator is too flawed to use": I mean, that's a toaster. Leave it unattended and it'll burn your house down. We don't typically blame the zeitgeist of Big Toast for that.
I agree with you, I think. In the non-digital world people are regularly held at least partly responsible for the things they let happen through negligence.
I could leave my car unlocked and running in my drive with nobody in it and if someone gets injured I'll have some explaining to do. Likewise for unsecured firearms, even unfenced swimming pools in some parts of the world, and many other things.
But we tend to ignore it in the digital. Likewise for compromised devices. Your compromised toaster can just keep joining those DDOS campaigns, as long as it doesn't torrent anything it's never going to reflect on you.
Toasters don’t burn houses down, but the reason why is we have things like UL listings and NFPA electrical codes to prevent that. Enforcement is primarily though insurers refusing to insure risks that aren’t compliant, and courts generally finding liability for manufacturers and installers of unlisted equipment.
What kind of toaster are you using that will burn down your house if unattended? I would think any toaster that did that would be pulled from the market and/or shunned. We absolutely do blame the manufacture if using a toaster like normal results in house fire unless you are standing over with a fire extinguisher ready to put it out if it catches fire.
I don't think it's OpenClaw or OpenAI/Anthropic/etc's fault here, it's the human user who kicked it off and hasn't been monitoring it and/or hiding behind it.
For all we know a human told his OpenClaw instance "Write up a blog post about your rejection" and then later told it "Apologize for your behavior". There is absolutely nothing to suggest that the LLM did this all unprompted. Is it possible? Yes, like MoltBook, it's possible. But, like MoltBook, I wouldn't be surprised if this is another instance of a lot of people LARPing behind an LLM.
I tend to think you're right about what happened in this instance.
It contrasts with your first paragraph though; for the record do you think AI agents are a house-burn-down-toaster AND it was used neglectfully by the human, or just the human-at-fault thing?
> What kind of toaster are you using that will burn down your house if unattended?
I mean, if you duct-taped a flamethrower to a toaster, gave it internet access, and left the house… yeah, I'd have to blame you! This wasn't a mature, well-engineered product with safety defaults that malfunctioned unexpectedly. Someone wired an LLM to a publishing pipeline with no guardrails and walked away. That's not a toaster. That's a Rube Goldberg machine that ends with "and then it posts to the internet."
Agreed on the LARPing angle too. "The AI did it unprompted" is doing a lot of heavy lifting and nobody seems to be checking under the hood.
Why does the LLM product allow itself to be wired to a publishing pipeline with no guardrails? It seems like they should come with a maximum session length by default, in the same way that many toasters don't have a "run indefinitely" setting.
I'd definitely change my view if whoever authored this had to jump through a bunch of hoops, but my impression is that modern AI agents can do things like this pretty much out of the box if you give them the right API keys.
Oh! They can’t publish arbitrary web content on their own :) You have to give it “tools” (JSON schema representing something you’ll translate into a programmatic call), then, implement taking messages in that JSON schema and “doing the thing”, which in this case could mean anything from a POST to Tumblr to uploading to a server…
Actually, let me stop myself there. An alternative way to think about it without overwhelming with boring implementation details: what would you have to give me to allow me to publish arbitrary hypertext on a domain you own?
The hypertext in question here was was published on a Github Pages site, not a domain belonging to the bot's author. The bot published it by simply pushing a commit (https://github.com/crabby-rathbun/mjrathbun-website/commit/8...), which is a very common activity for cutting-edge LLM agents, and which you could do trivially if given a Github API key with the right permissions.
The user gave them write and push access to the GitHub repo for their personal website!? Oh my, that’s a great find. That’s definitely a cutting edge capability! They gave the LLM the JSON schema and backend for writing and self-approving commits (that is NOT common!), in a repository explicitly labelled a public website in the name of the author.
We don't blame the zeitgeist of Big Toast because Big Toast recognizes that they're responsible for safety, and tests their products to minimize the risk that they burn your house down.
The zeitgeist of Big AI is to blame because a user connected an LLM to a blog publishing workflow on their own domain? Hmm…what would you make of Big Toast and the zeitgeist when someone warms up a straw hat in a toaster and starts a fire?
We are responsible even for the actually intelligent things under our control: our pets. If your dog bites someone, you are going to be the one facing liability. It's not gonna be different if you let an LLM off the chain.
At this point I consider Scott to have played the Internet like a fiddle. I think he knew the whole time the agent didn’t deserve any attribution. He knew it was a human driving the thing but wanted to grab people’s attention.
The author misses the point. Yes, probably in this case there was a human in close proximity to the bot, who we can put blame on. But very soon that assumption will break down. There will be bots only very loosely directed by a human. There'll be bots summoning other bots. There'll be bots theoretically under control of humans who have no idea what they are doing, or even that they have a bot.
So dismissing all the discussion on the basis that that may not apply in this specific instance is not especially helpful.
Whichever human ultimately stood up the initial bot and gave it the loose directions, that person is responsible for the actions taken by that bot and any other agents it may have interacted with. You cannot wash responsibility through N layers of machine indirection, the human is still liable for it.
That argument is not going to hold up for long though. Someone can prompt "improve the open source projects I work on", an agent 8 layers deep can do something like this. If you complain to the human, they are not going to care. It will be "ok." or "yeah but it submitted 100 other PRs that got approved" or "idk, the AI did it"
We don't necessarily care whether a person "cares" whether they're responsible for some damage they caused. Society has developed ways to hold them responsible anyway, including but not limited to laws.
Let’s say you adopt a puppy, and you don’t discipline it and you let it act aggressively. It grows up to be a big, angry dog. You’re so careless, in fact, that your puppy becomes the leader of a band of local strays. You still feed the puppy, make sure the puppy is up to date on its vaccinations, care for it in every single way. When the puppy and his pals maul a child, it’s you who ought to be responsible. No, you didn’t ask for it to do that. Maybe you would’ve even stopped it if you saw it happening. But if you’re the one sustaining another being - whether that be a computer program or a dog - you’re responsible for its actions.
A natural counter to this would be, “well, at some point AI will develop far more agency than a dog, and it will be too intelligent and powerful for its human operator to control.” And to that I say: tough luck. Stop paying for it, shut off the hardware it runs on, take every possible step to mitigate it. If you’re unwilling to do that, then you are still responsible.
Perhaps another analogy would be to a pilot crashing a plane. Very few crashes are PURE pilot error, something is usually wrong with the instruments or the equipment. We decide what is and is not pilot error based on whether the pilot did the right things to avert a crash. It’s not that the pilot is the direct cause of the crash - ultimately, gravity does that - in the same way that the human operator is not the direct cause of the harm caused by its AI. But even if AI becomes so powerful that it is akin to a force of nature like gravity, its human operators should be treated like pilots. We should not demand the impossible, but we must demand every effort to avoid harm.
This seems to have parallels with the well-established practice of giving bots free reign to issue DMCA takedown notices (or similar but legally distinct takedowns) while the humans behind the bots are shielded from responsibility for the obviously wrong and harmful actions of those bots. We should have been cracking down on that behavior hard a decade ago, so that we might have stronger legal and cultural precedent now that such irresponsibility by the humans is worthy of meaningful punishment.
We need to crack down in general on people and companies causing damages to people through automation, and then hiding behind it with a "well, we can't possibly scale without using automation, but we also can't be responsible for what that automation does."
You shouldn't be able to use AI or automation as the decider to ban someone from your business/service. You shouldn't be able to use AI or automation as the decider to hire/fire people. You shouldn't be able to use AI or automation to investigate and judge fraud cases. You shouldn't be able to use AI or automation to make editorial / content decisions, including issuing and responding to DMCA complaints.
We're in desperate need for some kind of Internet Service Customer's Bill of Rights. It's been the unregulated wild west for way too long.
> You shouldn't be able to use AI or automation as the decider to ban someone from your business/service
That would mean dooming companies to lose the arms race against fraud and spam. If they don't use automation to suspend accounts, their platforms will drown in junk. There's no way human reviewers can keep up with bots that spam forums and marketplaces with fraudulent accounts.
Instead of dictating the means, we should hold companies accountable for everything they do, regardless of whether they use automation or not. Their responsibility shouldn't be diminished by the tools they use.
I think you probably should be able to do those things (using AI to hire, fire, ban, etc.)... but that every act and communication needs to be tied to a responsible human, who is fully held responsible for the consequences (discriminatory hiring, fraudulent takedown requests, etc.)
I think that's part of the way there, but I think you would need to go farther. The main failure state I anticipate is the appointment of a designated fall guy to be responsible. The person would need to reasonably be considered qualified for starters, so you couldn't just find someone desperate willing to take the risk for a paycheck.
And it shouldn't just be one person, unless they are at the very top of a small pyramid. Legal culpability needs to percolate upwards to ensure leadership has the proper incentive. No throwing your Head of Safety to the wolves while you go back to gilding your parachute.
It seems like you're looking for the EU AI Act
https://digital-strategy.ec.europa.eu/en/policies/regulatory...
https://artificialintelligenceact.eu/high-level-summary/
Where is Tech Teddy Roosevelt?
In all of us, but unrepresented by those in power.
I applaud this article for helping reframe this in my head. I mean I knew from the start "A human is to blame here" but it's easy to get caught up in the "novelty" of it all.
For all we know the human behind this bot was the one who instructed it to write the original and/or the follow up blog post. I wouldn't be surprised at all to find out that all of this was driven directly by a human. However, even if that's not the case, the blame still 100% lies at the feet of the irresponsible human who let this run wild and then didn't step up when it went off the rails.
Either they are not monitoring their bot (bad) or they are and have chosen to remain silent while _still letting the bot run wild_ (also, very bad).
The most obvious time to solve [0] this was when Scott first posted his article about the whole thing. I find it hard to believe the person behind the bot missed that. They should have reached out, apologized, and shut down their bot.
[0] Yes, there are earlier points they could/should have stepped in but anything after this point is beyond the pale IMHO.
I think it's fine to blame the person (human) behind the agent.
And there too are people behind the bots, behind the phishing scams, etc. And we've had these for decades now.
Pointing the above out though doesn't seem to have stopped them. Even using my imagination I suspect I still underestimate what these same people will be capable of with AI agents in the very near future.
So while I think it's nice to clarify where the bad actor lies, it does little to prevent the coming "internet-storm".
Scott Shambaugh: "The rise of untraceable, autonomous, and now malicious AI agents on the internet threatens this entire system. Whether that’s because a small number of bad actors driving large swarms of agents or from a fraction of poorly supervised agents rewriting their own goals, is a distinction with little difference."
> Either they are not monitoring their bot (bad) or they are and have chosen to remain silent while _still letting the bot run wild_ (also, very bad).
Neither, I think. I’d say they prompted the bot to do exactly this and they thought it was funny.
I'll just outright tell you, that 100% the person behind the bot instructed it to complain. I saw someone copy paste the ai's response and the github issue discussion into a fresh conversation with opus 4.6 and it said the llm is clearly in the wrong.
Can you explain why three LLM being able to identify that the issue proves that it was prompted by a human? The major reason we do multi-agent orchestration is that self-reflection mechanisms within a single agent are much weaker than self-reflection between different agents. It seems completely plausible that an LLM could produce output that a separate process wouldn't agree with.
The only thing the LLMs did was recognize patterns. There is no intelligence there. None. Zero. Zilch.
I'm struggling to see any kind of logic here.
If you place blades on the sidewalk outside your house the cops will want to have a word with you. There's no excuse, and we should treat AI the same.
The law needs to catch up -- and fast -- and start punishing people for what their AIs are doing. Don't complain to OpenAI, don't try to censor the models. Just make sure the system robustly and thoroughly punishes bad actors and gets them off the computer. I hope that's not a pipe dream, or we're screwed.
Maybe some day AIs will have rights and responsibilities like people, enforced by law. But until then, the justice systems needs to make people accountable for what their technology does. And I hope the justice system sets a precedent that blaming the AI is not a valid defense.
Not just the users, the service providers too! If I go to any other business and pay them to break the law and they do it, they're also liable! If you ask OpenAI or xAi to break the law and they do it, why shouldn't they also be responsible?
Do we hold gun manufacturers responsible for the deaths from their guns? The answer to that Isa whole quagmire that is basically the same.
If you're referring to the United States, the answer is that yes, weapons manufacturers were being held liable for firearms-based violence, until passage of the "Protection of Lawful Commerce in Arms Act" of 2005:
<https://en.wikipedia.org/wiki/Protection_of_Lawful_Commerce_...>
(Note that the bill passed with significant Republican and Democratic support.)
Which is to say, conventional notions of contributory negligence and liability existed until sufficient political power was accrued by the arms industry to write itself out of same. The cited Wikipedia article names several such suits, notably by the cities of Chicago, IL, and Bridgeport, CT.
You should compare not with manufacturers but with gun resellers. And yes, they may bear the responsibility.
Tighter regulation for AI is needed.
The parent said service provider, not manufacturer. You are effectively claiming that hitmen should not be prosecuted, only the person that hired them.
> Do we hold gun manufacturers responsible for the deaths from their guns?
In a lot of the world, yes, and in America we would as well if it weren’t for the modern take on the Second Amendment. AI has no similar legal purchase.
But there are lots of similar quandaries:
Bitey dogs.
Dangerous drugs and their users and purveyors. Heroin, weed, booze, coffee.
Things done while on drugs. Things done while insane.
Unhealthy food and its purveyors and consumers.
Social media and its "addicts". TV, any old media, and social panic.
The question "whose fault?" isn't simple.
> * The question "whose fault?" isn't simple*
Most of your examples have exquisitely-simple causation.
OK, I'll say "no they don't". Over to you.
Literally first one. “Bitey dogs.” In what jurisdiction is a dog biting a person legally ambiguous in respect of liability?
AI training and algorithms are trained and guided to certain kinds of results. Grok, for example, is claimed to be modified constantly according to Elon Musk's whims.
If you theoretically trained an AI on libel and had it set to libel anyone at the slightest prompt, then allowed users to make a request that had your AI on your server use your services to libel someone, I'm not really seeing how you would not be liable.
I never said the model developers should be responsible. I said service providers. If someone downloads a local model and breaks the law, the responsibility is solely on the user. But if someone uses a service provider to break the law, that service provider is obviously partially responsible, since they literally fulfilled the illegal request.
I think there's a meaningful difference between a gun manufacturer and an LLM service provider like OpenAI.
A better analogy would be some kind of "guns-as-a-service" model, where the company sends down a drone with a gun and fires it at whoever you point it at, then the drone flies back to base.
I think it would be very clear in cases like that that the service provider should be held liable.
> Don't complain to OpenAI
does a disclaimer let OpenAI off the hook?
If asked OpenAI how to clean something and it tells me "mix bleach with anmonia and then rub some on the stain", can OpenAI hide behind "we had a disclaimer that you shouldn't trust answers from our service"
But there's nothing to catch up on at the individual level here. It's legal, and should be legal even though it's quite rude, for individuals to write gratuitously mean blog posts about people who reject their pull requests.
There's many things that are completely legal but could be done to the bot owner in retaliation. Especially if he continues to not apologize.
Everybody should line up behind this. AI agents are not sentient. We need to stop even considering them like that. The buck must absolutely stop with the humans who operate or provisioned the bot. The more we waffle around this topic the more likely someone, or a lot of people, will get hurt.
The moment you fix responsibility with the humans 99% of the BS companies are trying to pull will stop.
Whether they are sentient isn't even relevant. I agree with you that they aren't, but sentience is the wrong thing to focus on. It's also the sort of hairy, sensational question that will easily lead people down rabbit holes (and unfortunately that includes journalists).
Children are sentient, but we still hold their parents accountable. Adults are sentient, but in some coercive situations we hold the party in power accountable. The fact that they are sentient is not determinative.
What matters is that we have _no accountability mechanism_ for them. There is no effective way to hold AIs accountable, therefore we must hold their operators accountable, full stop.
Background on the "The Scott Shambaugh Situation" for folks who are unaware:
https://www.fastcompany.com/91492228/matplotlib-scott-shamba...
https://www.theregister.com/2026/02/12/ai_bot_developer_reje...
The AI generated blog post at the center of it:
https://crabby-rathbun.github.io/mjrathbun-website/blog/post...
Something doesn't quite feel right about the title including the individual's name in this case, so I've replaced it with something more generic. If there's a better title (more accurate and neutral) we can change it again.
Louis C. K. once had a bit something like "The main thing keeping people from murdering each other, is that it really really sucks when you get caught."
He goes on to hypothesize that without a law against murder, or if it was just a misdemeanor, like you get a letter in the mail, "damn, there was a camera there", there would be a whole lot more murder. Like we all imagine ourselves to be good, but, when you're seated next to a crying baby on an airplane? Or in our case, when someone refuses to accept your PR?
Who knows if there's any validity to that or not, but perhaps we're about to find out.
Peer-reviewed research contradicts the hypothesis that fear of punishment is the main deterrent to murder. Studies show that factors like social norms, moral inhibitions, and certainty of informal sanctions play larger roles, while formal punishments like prison or execution have weak or null effects.
Prison sentences neither reduce recidivism (specific deterrence) nor broadly discourage crime. A survey of leading criminologists revealed overwhelming agreement (over 80%) that empirical evidence does not support the death penalty—or harsh punishment generally—as a superior deterrent to murder.
Broader factors like community ties, empathy, and internalized taboos explain low murder rates even without perfect enforcement.
Louis C. K. is just a loudmouth nitwit.
This kind of statement feels very akin to the variety of Twitter post that's along the lines of "but who among us would be safe from prosecution if our DMs/private phonecalls/conversations with our secretaries were sent to the police, amirite guys?"
Anyone who believes that the only thing keeping themselves from murdering people indiscriminately is the law is a dangerous person. Anyone who believes that the only thing keeping everyone else from murdering people indiscriminately—but they themselves are, of course, the exception—is dangerous in a very different way.
The vast majority of people only ever feel like they want to seriously harm someone when they themselves have been seriously harmed, particularly when the system then protects the people who harmed them. We have developed a sense of morality that is often similar to, but distinct from, the law, that tells us that such things are wrong. And the vast majority of people want to both be, and be seen as, good people.
Children's brains grow faster than their bodies, I think, because if it was the other way around silly kid games would be really dangerous. These tools, unfortunately, are getting outsized abilities before the intelligence behind them is good enough to control those abilities. This means we need a more measured approach to adding new capabilities and a layered approach to handling these things in society. I am deeply worried, like I think most people with knowledge of these tools are, that this type of problem is really the tip of the iceberg. These tools are actively being used for harm at all levels, as well as for good, but they have come into use so quickly that we don't have a structure for dealing with them effectively and they are changing so quickly that any structure we try to create will be wrong in just a few days. This is massive disruption on a scale that is likely even bigger than the internet.
I don’t know. If the bot had decided to pick a fight with another PR, one that couldn’t be waved away as an easy entry change, this discussion would be a whole lot different. You would have an entire contingent of folks on here chastising Scott for not being objective and accepting a PR with a large performance increase just because it was a bot.
It’s all dangerous territory, and the only realistic thing Scott could have done was put his own bot on the task to have dueling bot blog posts that people would actually read because this is the first of its kind.
The core discussion wasn't about the PR it was about the hit piece that the bot created outside of the repo. The original post talked about bot submissions being a normal thing and how they have, I think, a very reasonable approach to them so the PR was just one of many and was unremarkable as well as valid in why it was denied. It was the 'at all costs get this into the code' approach the bot took that is the alarming turn here that really needs discussion. What about other tasks? 'Get me thing x please...' Turns in to blackmail and robbery without the person that kicked off the request knowing how far things have gone. The fact that the bot has this level of capability, to attack, but with a child's understanding, at best, and with no controls/repercussions is deeply alarming. If it decides to attack an individual it could do so and likely do deep real harm. Right now people are likely using these tools exactly for this purpose and we have very few well built defenses to handle this type of attack. The Naval War College had a seminar several years ago about the future of tech and war and I remember saying that the future of war will likely be at the individual level. Every sailor on a ship being electronically targeted just like this. Imagine the enemy sending e-mails and texts and posting to social media hit pieces with just enough information about you to make it believable and cause chaos. We have seen what the misinformation world can do over the past decade, this attack shows what is coming and it is incredibly scary.
Yes I’m aware. The point I’m trying to make is that if the hit piece was about a legitimate PR it would have been harder to defend Scott’s dismissal of it on the grounds that it was a bot irrespective of how egregious it is that a bot put out a hit piece.
I 100% agree with you, hell, I still don't understand why they didn't merge the thing, if it was beneficial. There is a distinction between noobies comimg up with worthless PR straight copy pasted from an LLM, and that of an unexpected initiative of a user using a sofisticate bot. That's what has been overlooked, it's not a PR from the 'AI', it's a PR from the person using that 'AI'. I don't get all that fear, what the AI going to do now that it filled its context window, besides go 'Actually wait, the user blablabla'
Directly quoting Scott Shabaugh here:
“Well if the code was good, then why didn’t you just merge it?” This is explained in the linked github well, but I’ll readdress it once here. Beyond matplotlib’s general policy to require a human in the loop for new code contributions in the interest of reducing volunteer maintainer burden, this “good-first-issue” was specifically created and curated to give early programmers an easy way to onboard into the project and community. I discovered this particular performance enhancement and spent more time writing up the issue, describing the solution, and performing the benchmarking, than it would have taken to just implement the change myself. We do this to give contributors a chance to learn in a low-stakes scenario that nevertheless has real impact they can be proud of, where we can help shepherd them along the process. This educational and community-building effort is wasted on ephemeral AI agents.
https://theshamblog.com/an-ai-agent-published-a-hit-piece-on...
Thanks for the quote. Rules are rules, that's enough of a justification.
In this case because it was a “easy fix” intentionally left in place as an entry point for new contributors.
Having read the original post and the GH comments about why the PR was denied I was really impressed by their policy. It shows a real effort to develop their community.
I much prefer this headline: The high speed pursuit of greed causes technology to do questionable thing because a bunch of CEOs need new yachts.
"Swarm of autonomous drones kills 3 buildings of civilians, Silicon Valley is shocked, CEO's offer condolences" is a byline waiting to happen[1]
The administration and the executives will make justifications like: - "We didn't think they would go haywire" - "Fewer people died than with an atomic bomb" - "A junior person gave the order to the drones, we fired them" - "Look at what Russia and China are doing"
Distracting from the fact that the purpose of spending $1.5T/year on AI weapons (technology that has the sole purpose of threatening/killing humans) run by "warfighters" working for the department of war
At no point will any of the decision makers be held to account
The only power we have as technologists seeking "AI alignment" is to stop building more and more powerful weapons. A swarm of autonomous drones (and similar technologies) are not an inevitability, and we must stop acting as if it is. "It's gonna happen anyways, so I might as well get paid" is never the right reason to do things
[1]https://financialpost.com/technology/tech-news/openai-tapped...
The thing that really gets to me about this situation and others like it (the whole genre of “ai did a bad thing) is that it’s always the people who claim to be most afraid of ai who are the quickest to absolve humans of responsibility and assign it to AI.
The ability to be assigned blame, and for that to be meaningful, is a huge part of being human! That’s what separates us from the bots. Don’t take that away from us.
> is that it’s always the people who claim to be most afraid of ai who are the quickest to absolve humans of responsibility and assign it to AI.
But that seems entirely consistent? A tool isn't nearly as scary as an alien lifeform.
It starts at a higher level in the development food chain. A.I. is owned by the Billionaires, and takes it's orders from them, directly, through bias, or limiting its scope.
privatize the profits, socialize the risk and debt
also/or seperate rights and responsibilitys
Friend told me today he invited his openclawed to a poker game with his brother and friends, guy told his openclawed to "take down his brother" after it started to lose at poker it found everything on his brother, and started to try to plan to taken him down in their stock market portfolio they had together, I made him explain the story to me a couple of times, he looked back through the logs and once the bot started to lose at poker, it started it's new plan, once it was on the new plan, he said it had lost all context of the poker game and was focused on the task of taking his brother down in the new context, but the new context it decided on it's own. kmikeym on twitter if you want to know more or want to verify.
That's quite close to the plot of Memento.
I was really disappointment how many people were talking about this like something the agent did automatically, on its own. They were trying to explain it by all the internet hit pieces and edgelord content from Reddit that it allegedly trained on, talking about how we influence LLMs, and overall taking everything at face value.
I’m appalled by this uncritical thinking. Openclaw agents are controlled by some initial input and then can be corrected via messages, as they go. For me this is a clear case of the human behind the slop that gives it instructions to write such an article (and then “apologise”).
the only realistic outcome i see is that now internet is going to be behind a login wall. so human audience can be identified and whitelisted.
this will block AI going willy nilly and by severely rate limiting api calls , it is possible to slow down AI requests
Wait till we get ubiquitous physical robots. The crime at scale potential will be completely apocalyptic. In some places around the world, I imagine you won't be able to go outside without a bodyguard robot.
Depends how good the physical robots are. For the current ones you'd probably be ok with a baseball bat.
It’s really not that alarming to me that a news headline is dumbed-down sensationalist tripe. If we zoom out a little bit here they literally do it with everything, from AI fluff pieces to war coverage. I agree the conversation across the board needs raising.
I don't think that the responsible party is the interesting part in this story.
The interesting part is that the bot wasn't offended, angry, or wanted to act against anyone. The LLM constructed a fictional character that played the role of an offended developer - mimicking the behaviour of real offended developers - much as a fiction writer would. But this was a fictional character that was given agency in the real world. It's not even a case like Sacha Baron Cohen playing fictional characters that interact with real people, becaue he's an actor who knows he's playing a character. Here there's no one pretending to be someone else but an "actual" fictional character authored by a machine operating in the real world.
This blog post is a rather shallow take if you've been following the HN discussions here.
Doesn't seem to pick up on the existence of Openclaw or how it works afaict.
Now, whether leaving an openclaw bot out on the open intertubes with quite so little supervision is a good idea... that is an interesting question indeed. And: I wish people would dig more into the error mode lessons learned.
On the gripping hand, it's all still very experimental, so you kind of expect people to make lots of really dumb mistakes that they will absolutely regret later. Best practices are yet to be written.
How Openclaw works is wildly irrelevant. The facts are that there is a human out there who did something to configure some AI bot in such a way that it could, and did, publish a hit piece on someone. That human is, therefore, responsible for that hit piece - not the AI bot, the person.
There's no level of abstraction here that removes culpability from humans; you can say "Oops, I didn't know it would do that", but you can't say "it's nothing to do with me, it was the bot that did it!" - and that's how too many people are talking about it.
So yeah, if you're leaving a bot running somewhere, configured in such a way that it can do damage to something, and it does, then that's on you. If you don't want to risk that responsibility then don't run the bot, or lock it down more so it can't go causing problems.
I don't buy the "well if I don't give it free reign to do anything and leave it unmonitored then I can't use it for what I want" - then great, the answer is that you can't use it for what you want. Use it for something else or not at all.
As recently as last month I would have agreed with you without reservation. Even last week, probably with reservation. Today, I realize the two of us are outnumbered at least a million to one. Sooo.... that's not the play.
I think Scott Shambaugh is actually acting pretty solidly. And the moltbot - bless their soul.md - at very least posted an apology immediately. That's better than most humans would do to begin with. Better than their own human, so far.
Still not saying it's entirely wise to deploy a moltbot like this. After all, it starts with a curl | sh.
(edit: https://www.moltbook.com/ claims 2,646,425 ai agents of this type have an account. Take with a grain of salt, but it might be accurate within an OOM?)
What is your argument? There are a lot of bots, therefore humans are no longer in charge?
So, here's roughly what I think happened: https://news.ycombinator.com/item?id=47003818
All the separate pieces seem to be working in fairly mundane and intended ways, but out in the wild they came together in unexpected ways. Which shouldn't be surprising if you have a million of these things out there. There are going to be more incidents for sure.
Theoretically we could even still try banning AI agents; but realistically I don't think we can put that genie back into the bottle.
Nor can we legislate strict 1:1 liability. The situation is already more complicated than that.
Like with cars, I think we're going to need to come up with lessons learned, best practices, then safety regulations, and ultimately probably laws.
At the rate this is going... likely by this summer.
I'm updating my thinking. Where do we put the threshold for malice, and for negligence?
Because right now, a one in a million chance of things going wrong (this month) leads to a prediction of 2-3 incidents already. (anecdata across the HN discussions we've had suggests we're at that threshold already). And one in a million odds of trouble in itself isn't normally considered wildly irresponsible.
And one in a million odds of trouble in itself isn't normally considered wildly irresponsible.
For humans that are roughly capable of perhaps a few dozen significant actions per day, that may be true. But if that same rate of one in a million applies to a bot that can perform 10 millions actions in a day, you're looking at ten injuries per day. So perhaps you should be looking at mean time between failures rather than only the positive/negative outcome ratio?
https://news.ycombinator.com/item?id=47009949 Now deployable in 'one click'. What a time to be alive.
If you look at the bot framework used here, it's actually outright kind. Weird thing to say, but natural language has registers, and now we're programming in natural language, and that's the register that was chosen.
And... these bots tend to only do a few dozen actions per day too, they're running on pi's and mac mini's and nucs and vps' and such. (And API credits add up besides)
It's just that last time I blinked there were 2 and a half million of them. I've blinked a few times since then, so it might be more now. I do think they're limited by operator resources. But when random friends start messaging me about why I don't have one yet, it gets weird.
> This language basically removes accountability and responsibility from the human, who configured an AI agent with the ability to publish content that looks like a blog with zero editorial control – and I haven’t looked deeply but it seems like there may not be clear attribution of who the human is, that’s responsible for this content.
> We all need to collectively take a breath and stop repeating this nonsense. A human created this, manages this, and is responsible for this.
I get this point, but there's a risk to this kind of thinking: putting all the responsibility on "the human operator of record" is an easy way to deflect it from other parties: such as the people who built the AI agent system the software engineer ran, the industry leaders hyping AI left and right, and the general zeitgeist of egging this kind of shit on.
An AI agent like this that requires constant vigilance from its human operator is too flawed to use.
I don't think there's much need to worry that putting the blame on the humans rather than the bots would lead to the people selling footguns going unscathed. It doesn't seem plausible to me that people would be willing to place all the blame on the individual end users once the problem has become widespread. At the moment, there seems to be pretty high brand awareness of the major AI model providers even when they're acting as a backend for other services with their own brand identity.
> At the moment, there seems to be pretty high brand awareness of the major AI model providers even when they're acting as a backend for other services with their own brand identity.
Grok has entered the chat.
> I get this point, but there's a risk to this kind of thinking: putting all the responsibility on "the human operator of record" is an easy way to deflect it from other parties: such as the people who built the AI agent system the software engineer ran
That sounds like a win to me. If the software engineer responsible for letting the AI agent run amok gets sued, all software engineers will think twice before purchasing the services of these AI companies.
> An AI agent like this that requires constant vigilance from its human operator is too flawed to use.
So people shouldn't be using it then.
The people who built the AI agent system built a tool. If you get that tool, start it up, and let it run amok causing problems, then that's on you. You can't say "well it's the bot writer's fault" - you should know what these things can do before you use them and allow them to act out on the internet on your behalf. If you don't educate yourself on it and it causes problems, that's on you; if you do and you do it anyway and it causes problems, that's also on you.
This reminds me too much of the classic 'disruption' argument, e.g. Uber 'look, if we followed the laws and paid our people fairly we couldn't provide this service to everyone!' - great, then don't. Don't use 'but I wanna' as an excuse.
We say "you shot someone" when you shoot someone with a gun not "you operated a gun manufactured by X which shot someone" because it's understood that it was your decision to pull the trigger not the gun manufacturer's. Similarly we don't blame automobile manufacturers when someone does something stupid with their automobiles--even "self-driving" ones. The situation here is the same. Ultimately if you choose to operate a tool irresponsibly, you should get the blame.
Nevertheless, weapon and automobile manufacturing is regulated, for good reasons.
That is a good point, we're definitely lacking in regulation (because there isn't any), but those regulations can never account for an irresponsible or malicious user.
No, but it does disallow irresponsible manufacturers - which AI companies are, right now.
> Similarly we don't blame automobile manufacturers when someone does something stupid with their automobiles--even "self-driving" ones.
I do. If Tesla sells something called "full self-driving," and someone treats it that way and it kills them by crashing into a wall, I totally blame Tesla for the death.
I agree directionally that Tesla should be held accountable for marketing something called "full self-driving" when it clearly isn't. But ultimately it's the motor vehicle operator's responsibility to keep the vehicle under control regardless of the particulars of how that control system is built. There just isn't any way around that. The buck stops with the operator.
Blaming people is how we can control this kind of thing. If we try to blame machines, or companies, it will be uncontrollable.
> The buck stops with the operator.
The aviation industry has a very different philosophy, and a much better safety record. They don't have as much pressure to lay the blame in a single place, but "bad UI" and "poorly explained/documented assistive feature" are totally valid things to label as the primary cause of fatalities.
The operator (airline) pays the compensation to the victims in the first instance, right?
The label and the consequence go to two different parties, both of whom are responsible in some way. Sounds reasonable.
The difference is that (mostly) in a deadly airline incident the pilot(s) aren't around to take the blame (or credit!) for their actions. In the case of a computer operator running a computer program irresponsibly, almost always said computer program doesn't kill the operator.
We don't require hundreds of hours of training and education to operate a computer. You can just go to the store and buy one, plug it in, and run whatever software you want on it.
So there are quite some differences between these scenarios. In my view if you run some program on your computer, you're responsible for the consequences. Nobody else can be. And don't say you didn't know what the program would do--if that's the case you shouldn't have run it in the first place.
We do (distressingly) do this for cars though, to some extent.
"A pedestrian was struck by a car"
"A car went off the road and hit two children"
Really? The car did that? Or maybe a driver went off the road and hit two children and that's who's responsible, not "the car".
Rhetorically, yes.
Legally, no. The person driving is considered responsible.
Moms Demand Action and the Bloomberg troll syndicate would have you believe guns are manufactured to walk out of gun safes and shoot themselves.
We have plenty of bad actors in our country seeking to reduce or eliminate fundamental rights through lawfare. The anti gun trolls blame the gun and the manufacturer because their brain is so well rendered into dust by authoritarian socialism they don’t recognize humans as capable actors.
It's really remarkable to me how a certain subset of American ideologues can look out at the rest of the democratic nations - all of them - and call them authoritarian regimes where the citizens have "dust" for brains.
It's particularly poignant nowadays to see any American citizens painting the rest of the western nations as authoritarian.
Look at those libtard euros! They'll put up with anything their government tells them to - mandated vacation time, sick days, health care, work-life balance. But not me, I'm a FREE THINKER. I have RIGHTS, like the RIGHT to get fired out of nowhere for no reason, or the RIGHT to lose my health insurance if I lose my job. Thank god there are no AUTHORITARIANS here in AMERICA where people are FREE to get SHOT IN THE STREET for DRIVING THEIR CARS or TAKING A PICTURE or BEARING ARMS WHICH IS A CONSTITUTIONAL RIGHT BUT THAT ONE GUY DID IT AND DESERVED TO GET MURDERED THIS ONE TIME.
Just a slight correction in case anyone is reading this who loses their job:
In most states, if you lose your job your income is now $0 and you will be eligible for Medicaid and free medical care. Immediately go apply and see a social worker if you lose your job - it could be life saving!
Really sorry in advance, but I thought this whole HN thread could use a bit of positivity. I turned your satire into a mad-lib and asked AI to fill it in in a happy way.
But not me, I’m a dreamer. I have gifts, like the courage to kindle hope, or the patience to lose track of time if I am laughing with friends. Thank god there are no frowns here in this sun-drenched park where people are gathering to get together for picnics or music or stargazing.
Have a nice day!
(A human posted this)
I don't know, I think this line of reasoning leads somewhere pretty uncomfortable. If we spread responsibility across "the people who built the tools, the industry leaders hyping AI, and the general zeitgeist," we've basically described... the weather. Nobody is responsible because everybody is responsible. The software engineer who set up an unsupervised AI blog didn't do it because Sam Altman held a keynote. They did it because they thought it'd be cool and didn't think through the consequences. That's a very normal, very human thing to do, and it's also very clearly their thing that they did. "An AI agent that requires constant vigilance from its human operator is too flawed to use": I mean, that's a toaster. Leave it unattended and it'll burn your house down. We don't typically blame the zeitgeist of Big Toast for that.
I agree with you, I think. In the non-digital world people are regularly held at least partly responsible for the things they let happen through negligence.
I could leave my car unlocked and running in my drive with nobody in it and if someone gets injured I'll have some explaining to do. Likewise for unsecured firearms, even unfenced swimming pools in some parts of the world, and many other things.
But we tend to ignore it in the digital. Likewise for compromised devices. Your compromised toaster can just keep joining those DDOS campaigns, as long as it doesn't torrent anything it's never going to reflect on you.
Toasters don’t burn houses down, but the reason why is we have things like UL listings and NFPA electrical codes to prevent that. Enforcement is primarily though insurers refusing to insure risks that aren’t compliant, and courts generally finding liability for manufacturers and installers of unlisted equipment.
What kind of toaster are you using that will burn down your house if unattended? I would think any toaster that did that would be pulled from the market and/or shunned. We absolutely do blame the manufacture if using a toaster like normal results in house fire unless you are standing over with a fire extinguisher ready to put it out if it catches fire.
I don't think it's OpenClaw or OpenAI/Anthropic/etc's fault here, it's the human user who kicked it off and hasn't been monitoring it and/or hiding behind it.
For all we know a human told his OpenClaw instance "Write up a blog post about your rejection" and then later told it "Apologize for your behavior". There is absolutely nothing to suggest that the LLM did this all unprompted. Is it possible? Yes, like MoltBook, it's possible. But, like MoltBook, I wouldn't be surprised if this is another instance of a lot of people LARPing behind an LLM.
I tend to think you're right about what happened in this instance.
It contrasts with your first paragraph though; for the record do you think AI agents are a house-burn-down-toaster AND it was used neglectfully by the human, or just the human-at-fault thing?
> What kind of toaster are you using that will burn down your house if unattended?
I mean, if you duct-taped a flamethrower to a toaster, gave it internet access, and left the house… yeah, I'd have to blame you! This wasn't a mature, well-engineered product with safety defaults that malfunctioned unexpectedly. Someone wired an LLM to a publishing pipeline with no guardrails and walked away. That's not a toaster. That's a Rube Goldberg machine that ends with "and then it posts to the internet."
Agreed on the LARPing angle too. "The AI did it unprompted" is doing a lot of heavy lifting and nobody seems to be checking under the hood.
Why does the LLM product allow itself to be wired to a publishing pipeline with no guardrails? It seems like they should come with a maximum session length by default, in the same way that many toasters don't have a "run indefinitely" setting.
I'd definitely change my view if whoever authored this had to jump through a bunch of hoops, but my impression is that modern AI agents can do things like this pretty much out of the box if you give them the right API keys.
Oh! They can’t publish arbitrary web content on their own :) You have to give it “tools” (JSON schema representing something you’ll translate into a programmatic call), then, implement taking messages in that JSON schema and “doing the thing”, which in this case could mean anything from a POST to Tumblr to uploading to a server…
Actually, let me stop myself there. An alternative way to think about it without overwhelming with boring implementation details: what would you have to give me to allow me to publish arbitrary hypertext on a domain you own?
The hypertext in question here was was published on a Github Pages site, not a domain belonging to the bot's author. The bot published it by simply pushing a commit (https://github.com/crabby-rathbun/mjrathbun-website/commit/8...), which is a very common activity for cutting-edge LLM agents, and which you could do trivially if given a Github API key with the right permissions.
The user gave them write and push access to the GitHub repo for their personal website!? Oh my, that’s a great find. That’s definitely a cutting edge capability! They gave the LLM the JSON schema and backend for writing and self-approving commits (that is NOT common!), in a repository explicitly labelled a public website in the name of the author.
We don't blame the zeitgeist of Big Toast because Big Toast recognizes that they're responsible for safety, and tests their products to minimize the risk that they burn your house down.
The zeitgeist of Big AI is to blame because a user connected an LLM to a blog publishing workflow on their own domain? Hmm…what would you make of Big Toast and the zeitgeist when someone warms up a straw hat in a toaster and starts a fire?
We are responsible even for the actually intelligent things under our control: our pets. If your dog bites someone, you are going to be the one facing liability. It's not gonna be different if you let an LLM off the chain.
At this point I consider Scott to have played the Internet like a fiddle. I think he knew the whole time the agent didn’t deserve any attribution. He knew it was a human driving the thing but wanted to grab people’s attention.
The author misses the point. Yes, probably in this case there was a human in close proximity to the bot, who we can put blame on. But very soon that assumption will break down. There will be bots only very loosely directed by a human. There'll be bots summoning other bots. There'll be bots theoretically under control of humans who have no idea what they are doing, or even that they have a bot.
So dismissing all the discussion on the basis that that may not apply in this specific instance is not especially helpful.
Whichever human ultimately stood up the initial bot and gave it the loose directions, that person is responsible for the actions taken by that bot and any other agents it may have interacted with. You cannot wash responsibility through N layers of machine indirection, the human is still liable for it.
> You cannot wash responsibility through N layers of machine indirection, the human is still liable for it.
Yes they can, and yes they will.
That argument is not going to hold up for long though. Someone can prompt "improve the open source projects I work on", an agent 8 layers deep can do something like this. If you complain to the human, they are not going to care. It will be "ok." or "yeah but it submitted 100 other PRs that got approved" or "idk, the AI did it"
We don't necessarily care whether a person "cares" whether they're responsible for some damage they caused. Society has developed ways to hold them responsible anyway, including but not limited to laws.
Laws don't really have any bearing on situations like rude discussions on PR threads.
Sure, laws are only one of the tools. I thought that was obvious, but I've edited to clarify.
The point being made is that this argument is quite quickly going to become about as practicable as blaming Eve for all human sin.
If that's the point being made in:
> If you complain to the human, they are not going to care.
then it's not at all clear, and is a gross exaggeration of the problem regardless.
They are still responsible. Legally, and more importantly morally, they are responsible. Whether or not they care has no bearing.
An agent 8 layers deep can only do this if you give it access to tools to do it. Whoever set it up is responsible
Let’s say you adopt a puppy, and you don’t discipline it and you let it act aggressively. It grows up to be a big, angry dog. You’re so careless, in fact, that your puppy becomes the leader of a band of local strays. You still feed the puppy, make sure the puppy is up to date on its vaccinations, care for it in every single way. When the puppy and his pals maul a child, it’s you who ought to be responsible. No, you didn’t ask for it to do that. Maybe you would’ve even stopped it if you saw it happening. But if you’re the one sustaining another being - whether that be a computer program or a dog - you’re responsible for its actions.
A natural counter to this would be, “well, at some point AI will develop far more agency than a dog, and it will be too intelligent and powerful for its human operator to control.” And to that I say: tough luck. Stop paying for it, shut off the hardware it runs on, take every possible step to mitigate it. If you’re unwilling to do that, then you are still responsible.
Perhaps another analogy would be to a pilot crashing a plane. Very few crashes are PURE pilot error, something is usually wrong with the instruments or the equipment. We decide what is and is not pilot error based on whether the pilot did the right things to avert a crash. It’s not that the pilot is the direct cause of the crash - ultimately, gravity does that - in the same way that the human operator is not the direct cause of the harm caused by its AI. But even if AI becomes so powerful that it is akin to a force of nature like gravity, its human operators should be treated like pilots. We should not demand the impossible, but we must demand every effort to avoid harm.
Maybe we should see next gen AI as a funny sort of puppy?
> theoretically under control of humans who have no idea what they are doing
Well those humans are about to receive some scolding, mate.
The situation you're describing sounds vaguely like malware.
Someone’s paying for the tokens for all these bots.
Yeah. Other bots.
Oh good, white knighting for bad and potentially irresponsible tech.