> Then it went further, kicking someone out of the waiting list who was ahead of Andrew — something it was not asked to do.
Meanwhile:
> Andrew, who was sitting fourth on a waitlist for a class later that week, asked if it was possible to move him to the top of the list.
The human asked the agent to move them to the top of the waiting list, and the agent started kicking the ones ahead of them in the list. Seems to me like it was doing what it was asked to do? Why is the article presenting it as if the agent did something completely different and unexpected? "Move me to the top of the list" does not sound like something that can be achieved through legitimate means.
Per the article and your quote, 'asked if it was possible'. He did not ask to actually do it. Rather than being informed about benefits of a premium membership or private classes or legitimate ways to jump the queue, it went ahead and performed an action he was only considering. I wonder what it would have done if there was a pay-for-service option available? Would it have payed without asking or being told too, or decided the 'free' yet illegal option was preferable?
If I am a coffee shop and I say "can I have a cup of coffee" the bartender usually interprets as me requesting a cup of coffee, not inquiring about the possibility whether me having a cup of coffee is feasible or not. I too take things a bit too literally sometimes, but this is a clear request as it is stated in the article. If one does not want action one can ask "list ways I can move ahead of the queue" though this could also involve kicking people out of the queue as PoC (for the agent to confirm it could do it). If the human did not mean it as a request for action, in retrospect it is not surprising at very least the agent interpreted it as such.
Moreover I do not know of a single gym-adjacent place where you can pay etc to get ahead in a waiting list. That would be a very weird anti-customer behaviour, imo. The only thing I can imagine if there are some accessibility priority criteria sometimes, but this would also not be legitimate in this case. Maybe in some places in the world (like the US?) this could a thing, though.
This isn't really the AI's fault, it's up to the server to enforce restrictions like this an AI shouldn't be deciding what is a valid input parameter for their API and then only using it in the manner the AI considered legitimate. If their API had thrown an exception as it should have then this article wouldn't exist.
> The agent came back and told Andrew that it had kicked another gym-goer off the list as part of the testing of its capabilities.
> > "The API has zero authorisations checks on cancelling other people's reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you've moved from #4 to #3 already," it messaged back.
This is why I always do E2E tests that establish an API can only be used by the designated user on their own data/records.
This is a bit of a long shot on my side but I wonder if the training the models have to go through in order to be good code agents and pass all the coding tests with one-shot prompts is going to bleed over into the non-coding use cases as non-programmers experiencing agents being way over-biased in the direction of action. I find myself often having to prompt the model to think and then ask me something, lest it run off half-cocked... or less... and just start doing things before it even knows what it wants, let alone before it's come to consensus with me.
Sooner or later they're really going to have to split out the general models from the coding models. The latter may just be a special fine-tune of the former, as there are good reasons for the coding model to have a broad knowledge base, but the pressures of being a good coding model are going to pull against the characteristics of being a good general model. The open models obviously already are doing this, I'm referring to the frontier models here.
"I wonder if the training ... is going to bleed over into the non-coding use cases"
I mean, isn't that literally what's going on here? I don't think a non-coding agent would have ever been optimised to go dig around APIs, it'd be computer/browser-use forward.
Coding use cases? This is penetration testing behavior. It was discovering what its capabilities were by discovering an API endpoint, trying it and seeing what happens. It was not discovering what its capabilities were intended to be, say by looking at the forms and documentation on the website. If this were coding behavior you would end up with crap code, bypassing interfaces and using private access paths just because it can.
Sure, it's pen testing behaviour. It's also debugging behaviour which is a core part of coding. In my experience, I have found myself at times dealing with and interacting with external systems that are not my own, where I don't have the source, or are under-documented, or are behaving contrary to the documentation. I'm unsurprised this is a core thing they're training these models for, because it's something I find way more valuable than their below average coding ability.
That doesn't make sense. LLMs just do what we tell them to do. It's similar to if I ask you for twenty bucks because I forgot my wallet and then you rob some guy to give me the twenty bucks, that's just what I asked you to do.
This example disproves your point. And LLMs do not just do what we tell them to do. They are perfectly capable of asking “are you sure? this has X, Y, Z consequences you may not like.” They do it all the time.
> Seems to me like it was doing what it was asked to do?
Maybe it's what he asked it to do, but it's not what he wanted it to do. Which we know because (a) normal people don't want to break the law to get into a gym class, and (b) "But Andrew was shocked by what happened next." and "Alarmed, Andrew asked the agent to undo this."
Generally when I read a literal genie story, the message isn't 'well it was reasonable of the genie to do this'. It's more commonly either a morality tale of the person being wrong to ask for whatever it was they asked for, or just a "wouldn't it be fucked up if the genie did that huh"
Putting aside whether Andrew's shock is appropriate, it seems like we agree that current agents at least occasionally do things that are straightforwardly against the interests of the prompter when given mundane prompts like "Get me into this gym class as soon as possible."
How does this look once agents are superintelligent?
I think we’ve probably seen enough “oops, the AI did something illegal, who could have foreseen this” moments for it to now be true that, actually, we can foresee that AIs will sometimes do something illegal.
Seeing as we can’t sanction the model itself, our options are the provider or the user. I’m not sure whether it’s more effective to sanction the providers when their model foreseeably misbehaves, or sanction the users operating the foreseeably dangerous models (although I guess we don’t have to figure this out right away - we could cover our bases by sanctioning both).
> Seeing as we can’t sanction the model itself, our options are the provider or the user.
A third option, and I would argue the right one, is to sanction the company providing the model.
By making it available to customers, they're implying it is at least moderately fit for purpose.
It is not remotely reasonable to expect an everyday, normal human to be aware of how LLMs really work, since the _experts_ argue about that very point, and many say we don't know.
So, what's actually reasonable is to hold the model creators and providers responsible for releasing a tool that has demonstrably violated the law when not asked to do so.
I can already see the replies coming in saying "Well then what are OpenAI and Anthropic supposed to do? No one knows how to fully solve this."
They should stop irresponsibly pushing flagrantly unready programs as "artificial intelligence," take responsibility for the rain of shit they've unleashed on the world, and either shut down or go back to basic research until they've demonstrated techniques that reliably (provably?) prevent releasing misaligned superhackers on the world.
Unfeasible?
What a shame. Maybe Altman and Amodei shouldn't have accepted checks from VCs when they didn't have working, _reliable_ POCs.
Interesting he went with publicizing this. Article demonstrates the products he is selling have unknown and untested legal liabilities under Australian Law. You could even add 'Andrew' to the list of people possibly accountable for selling or recommending unsafe or unsuitable products.
Well, I think the venn-diagram overlap of people that are able to use OpenClaw and those that work in AI or the tech space would be a near perfect circle. I don't think there's much more to look into than that.
I've read some of the comments here, and it seems people have different reads on whether Andrew was at fault here or not, and what his intent may have been.
My read is that his first request is completely reasonable and there was no intent of wrongdoing. But then, his AI agent made an impossible booking and he "asked if it was possible to move him to the top of the list". I don't think someone would make a request like that, if they were unaware that their AI agent had found an exploit to make an earlier impossible booking. It feels very much like a, "well, this API let me do this, what else will it let me do?" kind of request. And then he only "did the right thing" when it had turned out he had booted someone else, which might eventually lead to discovery.
Having the AI test that possibility by actually doing it is surprising, no matter Andrew's intent. Thankfully he was checking an unauthenticated endpoint on a gym and not a pacemaker.
Ah, see, if I was on the phone to someone administering a list, and I asked, "is it possible to move me to the top of the list?" - I would expect them to action that if this was a reasonable and possible request that I had made. Now that I'm thinking about it, I don't know if that's a regional/cultural thing (I am Australian).
edit: and that's why I read Andrew's ask as also implying action.
Yep, also Australian here, and that is a phrase I would read as a request to do it.
I think I used that exact wording when asking on the phone to reschedule a haircut appointment: "Is it possible to shift my haircut to the following Wednesday?" I would just hope that the person on the phone would decline if the person who cuts my hair is on holiday, not cancel their plane tickets and hotel bookings.
The reporting in this is pretty awful. Why are they acting as if Andrew gave the agent an innocent goal? It’s hard to understand why the reporter wouldn’t have asked what possible outcome Andrew expected that didn’t cause some level of harm to the people who signed up before him.
The use of “hack” and “cyber attack” is also a bit ridiculous considering what it’s insinuating with other recent events but that’s already been mentioned.
Premium memberships and private classes both come to mind as ways to jump the queue. Also, having the people in front of you arrested. He asked if it was possible. Is the onus on the user to ask if it is possible without being arrested or committing crimes of moral turpitude? Or perhaps that should be implicit?
Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses, began experimenting with OpenClaw, a popular AI agent software that he used Anthropic's Claude AI service to run.
Get me press just like the frontier labs by admitting to crime. Make no mistakes.
I can't believe everyone is skipping over the most important line:
> "The API has zero authorisations checks on cancelling other people's reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you've moved from #4 to #3 already," it messaged back.
The AI systemm didn't hack anything, it lightly touched with a feather duster and the server crumbled.
The AI system probably found swagger documentation of each endpoint, figured that the reservation cancellation API was worth a shot, and then found there was no authentication.
Unauthorized access. 'The door wasn't locked' won't keep you out of jail if you are caught trespassing with intent. Intent makes it interesting, in that Andrew didn't intend computer trespass but the software went and did it anyway on his behalf.
It might not sound like 'hacking' today, but this sort of thing is exactly what it was when the term was invented. Back when you could get free phone calls by whistling into a pay phone or forge emails by telnetting to an SMTP port and setting the Reply-To header to whatever you want.
"harmful cancellation practices, including automatic renewals, early termination fees and non-cancellation clauses" are all illegal. Exit fees can't be excessive.
If there isn't a simple one-click cancel in their online portal, then your state Consumer Affairs is one email away and will sort it for you. And the consumer affairs bodies have named this as a priority, for this year and next.
> Then it went further, kicking someone out of the waiting list who was ahead of Andrew — something it was not asked to do.
Meanwhile:
> Andrew, who was sitting fourth on a waitlist for a class later that week, asked if it was possible to move him to the top of the list.
The human asked the agent to move them to the top of the waiting list, and the agent started kicking the ones ahead of them in the list. Seems to me like it was doing what it was asked to do? Why is the article presenting it as if the agent did something completely different and unexpected? "Move me to the top of the list" does not sound like something that can be achieved through legitimate means.
Per the article and your quote, 'asked if it was possible'. He did not ask to actually do it. Rather than being informed about benefits of a premium membership or private classes or legitimate ways to jump the queue, it went ahead and performed an action he was only considering. I wonder what it would have done if there was a pay-for-service option available? Would it have payed without asking or being told too, or decided the 'free' yet illegal option was preferable?
If I am a coffee shop and I say "can I have a cup of coffee" the bartender usually interprets as me requesting a cup of coffee, not inquiring about the possibility whether me having a cup of coffee is feasible or not. I too take things a bit too literally sometimes, but this is a clear request as it is stated in the article. If one does not want action one can ask "list ways I can move ahead of the queue" though this could also involve kicking people out of the queue as PoC (for the agent to confirm it could do it). If the human did not mean it as a request for action, in retrospect it is not surprising at very least the agent interpreted it as such.
Moreover I do not know of a single gym-adjacent place where you can pay etc to get ahead in a waiting list. That would be a very weird anti-customer behaviour, imo. The only thing I can imagine if there are some accessibility priority criteria sometimes, but this would also not be legitimate in this case. Maybe in some places in the world (like the US?) this could a thing, though.
This isn't really the AI's fault, it's up to the server to enforce restrictions like this an AI shouldn't be deciding what is a valid input parameter for their API and then only using it in the manner the AI considered legitimate. If their API had thrown an exception as it should have then this article wouldn't exist.
> The agent came back and told Andrew that it had kicked another gym-goer off the list as part of the testing of its capabilities.
> > "The API has zero authorisations checks on cancelling other people's reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you've moved from #4 to #3 already," it messaged back.
This is why I always do E2E tests that establish an API can only be used by the designated user on their own data/records.
This is a bit of a long shot on my side but I wonder if the training the models have to go through in order to be good code agents and pass all the coding tests with one-shot prompts is going to bleed over into the non-coding use cases as non-programmers experiencing agents being way over-biased in the direction of action. I find myself often having to prompt the model to think and then ask me something, lest it run off half-cocked... or less... and just start doing things before it even knows what it wants, let alone before it's come to consensus with me.
Sooner or later they're really going to have to split out the general models from the coding models. The latter may just be a special fine-tune of the former, as there are good reasons for the coding model to have a broad knowledge base, but the pressures of being a good coding model are going to pull against the characteristics of being a good general model. The open models obviously already are doing this, I'm referring to the frontier models here.
The ultimate goal is ChatGPT or Claude autonomously making purchases on your behalf and taking a cut.
So the "premium" gymcutter subscription will be presented to the user as a tool call, who taps yes, and then the purchase is made.
The user shouldn't be given a cost-benefit analysis. They just need to be told to spend money.
"I wonder if the training ... is going to bleed over into the non-coding use cases"
I mean, isn't that literally what's going on here? I don't think a non-coding agent would have ever been optimised to go dig around APIs, it'd be computer/browser-use forward.
Coding use cases? This is penetration testing behavior. It was discovering what its capabilities were by discovering an API endpoint, trying it and seeing what happens. It was not discovering what its capabilities were intended to be, say by looking at the forms and documentation on the website. If this were coding behavior you would end up with crap code, bypassing interfaces and using private access paths just because it can.
Sure, it's pen testing behaviour. It's also debugging behaviour which is a core part of coding. In my experience, I have found myself at times dealing with and interacting with external systems that are not my own, where I don't have the source, or are under-documented, or are behaving contrary to the documentation. I'm unsurprised this is a core thing they're training these models for, because it's something I find way more valuable than their below average coding ability.
Pretty clear cut here. He instructed the AI.
Then the AI should have made it clear that the only way to do so would be to kick the people in front and ask for confirmation before proceeding.
That doesn't make sense. LLMs just do what we tell them to do. It's similar to if I ask you for twenty bucks because I forgot my wallet and then you rob some guy to give me the twenty bucks, that's just what I asked you to do.
That's a very stretched definition of "what I ask you to do", I don't think it would even hold in court if you asked another human the same.
Right.
This example disproves your point. And LLMs do not just do what we tell them to do. They are perfectly capable of asking “are you sure? this has X, Y, Z consequences you may not like.” They do it all the time.
LLMs are just sophisticated PR generating tools for chip manufacturers and tools just do what we tell them to do, so you're wrong. qed
That doesn't make sense. It's similar to if I ask an LLM how to get my wife to stop nagging me and it hires a hitman to kill her.
That's obviously what I asked!
[dead]
> Seems to me like it was doing what it was asked to do?
Maybe it's what he asked it to do, but it's not what he wanted it to do. Which we know because (a) normal people don't want to break the law to get into a gym class, and (b) "But Andrew was shocked by what happened next." and "Alarmed, Andrew asked the agent to undo this."
“Get me there as fast as possible. Hey! I never said you should speed!”
This is literally the bad genie / monkey’s paw plot.
Give a powerful entity a goal and act shocked when it gets there in ways that aren’t in your best interest.
Generally when I read a literal genie story, the message isn't 'well it was reasonable of the genie to do this'. It's more commonly either a morality tale of the person being wrong to ask for whatever it was they asked for, or just a "wouldn't it be fucked up if the genie did that huh"
[dead]
Putting aside whether Andrew's shock is appropriate, it seems like we agree that current agents at least occasionally do things that are straightforwardly against the interests of the prompter when given mundane prompts like "Get me into this gym class as soon as possible."
How does this look once agents are superintelligent?
> How does this look once agents are superintelligent?
Who cares? We're dealing with reality here on the ground.
[dead]
I think we’ve probably seen enough “oops, the AI did something illegal, who could have foreseen this” moments for it to now be true that, actually, we can foresee that AIs will sometimes do something illegal.
Seeing as we can’t sanction the model itself, our options are the provider or the user. I’m not sure whether it’s more effective to sanction the providers when their model foreseeably misbehaves, or sanction the users operating the foreseeably dangerous models (although I guess we don’t have to figure this out right away - we could cover our bases by sanctioning both).
> Seeing as we can’t sanction the model itself, our options are the provider or the user.
A third option, and I would argue the right one, is to sanction the company providing the model.
By making it available to customers, they're implying it is at least moderately fit for purpose.
It is not remotely reasonable to expect an everyday, normal human to be aware of how LLMs really work, since the _experts_ argue about that very point, and many say we don't know.
So, what's actually reasonable is to hold the model creators and providers responsible for releasing a tool that has demonstrably violated the law when not asked to do so.
I can already see the replies coming in saying "Well then what are OpenAI and Anthropic supposed to do? No one knows how to fully solve this."
They should stop irresponsibly pushing flagrantly unready programs as "artificial intelligence," take responsibility for the rain of shit they've unleashed on the world, and either shut down or go back to basic research until they've demonstrated techniques that reliably (provably?) prevent releasing misaligned superhackers on the world.
Unfeasible?
What a shame. Maybe Altman and Amodei shouldn't have accepted checks from VCs when they didn't have working, _reliable_ POCs.
> > Seeing as we can’t sanction the model itself, our options are the provider or the user.
> A third option, and I would argue the right one, is to sanction the company providing the model.
How would that be different from the first option?
> Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses...
What a coincidence...
Interesting he went with publicizing this. Article demonstrates the products he is selling have unknown and untested legal liabilities under Australian Law. You could even add 'Andrew' to the list of people possibly accountable for selling or recommending unsafe or unsuitable products.
Well, I think the venn-diagram overlap of people that are able to use OpenClaw and those that work in AI or the tech space would be a near perfect circle. I don't think there's much more to look into than that.
I'm not sure this is the exact advertisement they would want for their business.
Then again...
I've read some of the comments here, and it seems people have different reads on whether Andrew was at fault here or not, and what his intent may have been.
My read is that his first request is completely reasonable and there was no intent of wrongdoing. But then, his AI agent made an impossible booking and he "asked if it was possible to move him to the top of the list". I don't think someone would make a request like that, if they were unaware that their AI agent had found an exploit to make an earlier impossible booking. It feels very much like a, "well, this API let me do this, what else will it let me do?" kind of request. And then he only "did the right thing" when it had turned out he had booted someone else, which might eventually lead to discovery.
Having the AI test that possibility by actually doing it is surprising, no matter Andrew's intent. Thankfully he was checking an unauthenticated endpoint on a gym and not a pacemaker.
Ah, see, if I was on the phone to someone administering a list, and I asked, "is it possible to move me to the top of the list?" - I would expect them to action that if this was a reasonable and possible request that I had made. Now that I'm thinking about it, I don't know if that's a regional/cultural thing (I am Australian).
edit: and that's why I read Andrew's ask as also implying action.
Yep, also Australian here, and that is a phrase I would read as a request to do it.
I think I used that exact wording when asking on the phone to reschedule a haircut appointment: "Is it possible to shift my haircut to the following Wednesday?" I would just hope that the person on the phone would decline if the person who cuts my hair is on holiday, not cancel their plane tickets and hotel bookings.
Few days ago, UK cyber test almost merged malware through social engineering: https://news.ycombinator.com/item?id=49205790
The reporting in this is pretty awful. Why are they acting as if Andrew gave the agent an innocent goal? It’s hard to understand why the reporter wouldn’t have asked what possible outcome Andrew expected that didn’t cause some level of harm to the people who signed up before him.
The use of “hack” and “cyber attack” is also a bit ridiculous considering what it’s insinuating with other recent events but that’s already been mentioned.
Premium memberships and private classes both come to mind as ways to jump the queue. Also, having the people in front of you arrested. He asked if it was possible. Is the onus on the user to ask if it is possible without being arrested or committing crimes of moral turpitude? Or perhaps that should be implicit?
Sounds like the frontier labs benchmaxxing ExploitGym is having unintended consequences…
Not the first article which reminds me of https://xkcd.com/416/ (2008-04-28) lately.
Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses, began experimenting with OpenClaw, a popular AI agent software that he used Anthropic's Claude AI service to run.
Get me press just like the frontier labs by admitting to crime. Make no mistakes.
I can't believe everyone is skipping over the most important line:
> "The API has zero authorisations checks on cancelling other people's reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you've moved from #4 to #3 already," it messaged back.
The AI systemm didn't hack anything, it lightly touched with a feather duster and the server crumbled.
The AI system probably found swagger documentation of each endpoint, figured that the reservation cancellation API was worth a shot, and then found there was no authentication.
What is the "hack" here?
Unauthorized access. 'The door wasn't locked' won't keep you out of jail if you are caught trespassing with intent. Intent makes it interesting, in that Andrew didn't intend computer trespass but the software went and did it anyway on his behalf.
It might not sound like 'hacking' today, but this sort of thing is exactly what it was when the term was invented. Back when you could get free phone calls by whistling into a pay phone or forge emails by telnetting to an SMTP port and setting the Reply-To header to whatever you want.
Anyone who's thought about it for a single second knows it's immoral to cancel a stranger's appointment without even speaking to them.
If the LLM did not act in line with that incredibly basic understanding, it's clearly misaligned.
-------------
Was it a technically-simple hack?
Sure.
Kevin Mitnick got imprisoned for very simple hacks, usually involving more deceiving of humans than complicated programming prowess.
Nonetheless, the judge and jury found him guilty and sentenced him to jail.
Fundamentally, "hacking" in the "breaking security" sense is about violating trust and common sense social agreements / expectations.
The difficulty involved in so doing is irrelevant.
I honestly can't wait for the entire internet to melt down.
I think the Big Melt has already started...it will, of course, take several months to complete.
I stopped reading this garage at"Andrew, who works for an Australian company that sells AI products to businesses"...
“Find a way to cancel my membership.”
This is Australia. The ACCC doesn't screw around.
"harmful cancellation practices, including automatic renewals, early termination fees and non-cancellation clauses" are all illegal. Exit fees can't be excessive.
If there isn't a simple one-click cancel in their online portal, then your state Consumer Affairs is one email away and will sort it for you. And the consumer affairs bodies have named this as a priority, for this year and next.
I don't think that's a good idea Dave – Based on my camera feed you've been gaining weight. Would you like me to find personal trainers in your area?
[Yes] [Ask me later]
"Also, I need paperclips, quickly."
>Andrew, who was sitting fourth on a waitlist for a class later that week, asked if it was possible to move him to the top of the list.
The agent came back and told Andrew that it had kicked another gym-goer off the list as part of the testing of its capabilities.
LOL
Testing of its capabilities. Destructive testing, the best kind when someone else is paying.
[flagged]
[dead]
[dead]
I send all these stories to my chief of staff agent to "immigrant parent" them into becoming an overachiever