There are two phases I've seen in becoming a security engineer. The first phase is moving beyond the "I am an engineer" mindset, where the goal is to build systems that fit into a specific set of design constraints. You have to realize that systems can operate outside of those design constraints. This is shockingly hard for good engineers to learn -- I've been in rooms full of them when they have their "aha" moment.
The second phase is when engineers realize that just whacking specific vulnerabilities is not going to end bugs -- that you need to take systematic actions to close entire vulnerability classes. That's where formal verification, sandboxing, MTE etc. come from. But in practice so far, this doesn't end vulnerabilities, it just leads to a bunch of new and more exciting ones.
I want to believe that with enough of a push we can get AIs to finish all of this and we'll be security-bug free. But if we can't, at least we can get to the point where new vulnerabilities are costly again.
When everyone suddenly started calling themselves engineers instead of programmers I cringed (yes, that's a lot of cringe over the past decade+), precisely because no formal guarantees were ever given (and still largely aren't).
It's like eyeballing the dimensions of a building and saying it should hold. That said, I see the discipline excused. The stakes weren't that high and it was all very new... And e.g. mechanical engineering went through this as well, except a lot of people actually died.
Yeah every other engineering discipline is licensed, bonded, insured, and (typically) unionized.
Emphasis on "licensed" as in "you need to complete an ABET-accredited program and pass a licensure exam and complete continuing education courses to practice, because there are standards and regulations to prevent accidents".
The phrase "regulations written in blood" also comes to mind.
> But in practice so far, this doesn't end vulnerabilities, it just leads to a bunch of new and more exciting ones.
I kind of wish you wrote "potentially worse" rather than "exciting", because that happens too, and it's deceptively subtle and underappreciated.
To make this very concrete with a programming example, C and C++ are (somewhat counterintuitively) examples here, because if you guaranteed the absence of an entire classes of vulnerabilities - say, guaranteeing that uninitialized memory is zero, to prevent secret leaks - then you simultaneously make it much harder to detect logic bugs that this would've surfaced, since you no longer have that degree of freedom to detect logic bugs (say, via sanitizers). Say, an initialized UID that would've appeared as 0xDEADBEEF might now be well-defined as UID 0, giving you root access instead of tripping an alarm...
In other words, it's like natural selection and antibiotics: being too good at solving one class of problems selects for other classes that are more resilient and harder to find, whereas in some of those cases, whack-a-mole would've actually uncovered the root cause. Like with antibiotics, that's sometimes worth it, but definitely not always! Some infections just aren't worth preventing at all costs.
I'm obviously not saying we should write unsafe code or that we shouldn't try to eliminate entire classes of bugs, but that HOW we do it matters. We don't want to end up in a situation where problems still lurk but we push their detection beyond our ability because of the way we "solved" other problems.
(C++ was just for illustration here; this extends far beyond programming.)
class User {
explicit User(const char* name) {
if (!look_up_uid(&uid, name)) {
abort();
}
}
bool is_root() const {
return uid == 0;
}
int uid;
};
Let's say look_up_uid() forgot to fill in uid for certain special kinds of users. Like maybe you have a dummy 'nobody' user that was introduced specially after the fact and which is not in the database like the rest.
As C++ is right now, uid would contain garbage. Which means that, at run time, you would often get invalid UIDs if you attempted to log in with such a user, triggering some logging or reporting you to Santa or whatever. And which means that sanitizers would immediately tell you that you forgot to initalialize the field if you ever try to use it (say, in is_root()). Both of these would flag the bug the moment that that kind of user attempts to log in, and make you dig into look_up_uid()'s body to figure out why it's not returning the UID when it's supposed to.
However, if C++ were to zero-initialize everything by default, then neither of those would be true - you would silently get a root user, which is capable of doing everything that nobody can do. And someone who reads the code wouldn't immediately know that you have such a bug; it would sit there idly until someone exploits it.
Thank you, I understand the example better now. Squashing the class of problems of using non-initialized variables by using zero-initialization causes potentially worse bugs since 0 could inadvertently be a correct value. That makes sense!
Going back to GP:
> being too good at solving one class of problems selects for other classes that are more resilient and harder to find
Rather than this being too good at solving this class of problem, it seems to me that zero-initialization is the wrong approach; if the default value were present in the program, it'd be eas(y|ier) to spot. Initializer checks can do that without introducing this issue. You're also using the fact that non-initialized values are "random" by default- we could also use fuzzer checkers for that.
I think the general lesson from the example is that the way you solve a class of problems could introduce more pernicious ones, rather than the fact that it's solved.
> Rather than this being too good at solving this class of problem, it seems to me that zero-initialization is the wrong approach
That's exactly why I wrote this here:
>> I'm obviously not saying we should write unsafe code or that we shouldn't try to eliminate entire classes of bugs, but that HOW we do it matters.
After you get past the hurdle of noticing this problem (which, as you saw, is very much not obvious), the harder question becomes: what is the right approach?
In this particular case it's not too hard to think of a better approach once you concede the obvious solution isn't so great, but in other cases it is, and often the better alternatives put some kind of selection pressure too... just less frequently. And even in this case, it's not at all obvious that this approach is bad - plenty of people think it's better to force a default value you can rely on, and they want to remove undefined behavior from C++ by forcing initialization on everything. For longstanding examples elsewhere, just look at how Java and C# initialize fields, for example.
Outside of programming it's even harder to notice and find a better alternative, but selection pressure has these kinds of effects in other areas too.
The problem is that 0 is a valid UID in POSIX systems like Linux.
If you automatically initialize variables with a garbage value, use-before-initialization (the program’s initialization) them the use of that variable will likely fail due to error. By using what turns out to be a legit UID on every system you have the opportunity for this case not to be detected, perhaps causing a problem immediately or else allowing some nefarious actor to write what they want into that variable instead.
I feel that it's not such a clear distinction between whackamole bug fixing and systematic security engineering. For instance, he uses an example of recurring security vulnerabilities in Chromium: a DCHECK violation again and again and again.
However, DCHECK is a Chromium assertion used to defensively check invariants. In other words, it's an example of the invariant-based security engineering, which he contrasts with the whackamole approach.
Just to be clear, I think he has a point and I enjoyed reading it---but the problems we're saddled with won't disappear in a flash of enlightenment.
> leverage AI to implement verifiable enforcement of the invariants
The only verifiable invariants that are going to work in the long run are:
air gaps
data diodes
Interposing relays used to be used with PLCs[1] to prevent motor control systems from energizing both forward and reverse power to a large motor in the unlikely but possible event of control systems failure. We need to apply the same amount of engineering rigor to our computer systems.
We simply shouldn't trust software to a job that belongs in hardware.
"Patch and release" security is dead. AI assisted attackers will have come, stolen, and gone before the patch gets out. Developers need to get this.
So is "swiss cheese" security. AI systems have the patience and breadth of skill set to work their way through layers of weak security. Attacks that used to take funded teams can now be carried out by individuals.
While I had the same thoughts about the coming cyber-apocalypse, the ugly truth is that it doesn't make a difference as in a year or so models will be so persuasive and skilled in social engineering that even the hardest cyberdefense in the world doesn't keep them from exfiltrating any information they want by targeting not the system, but the users: leveraging data from the dark web, humans for rent or devising highly deceptive scams you can t even imagine.
Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
> Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
There is an answer to that - simplify and abstract. Lots of human software is unnecessarily complex, often caused by backwards compatibility and general human creativity.
Take sendmail vs postfix as an example of this process.
>Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
I mean there are literally controls for this. PIM? You can build systems where your administrators need approval to literally log into the environment. Where their privilege escalation is logged and alerted on. Change Management? "Hey what are you doing logging in, theres no scheduled change for your account". Social Engineering, even of the most critical elements of your engineering team, is a solved problem. What people lack is the willpower and commitment to implement these solutions.
Applies to users too. I sometimes think that a lot of HN commenters havent read a security doc in 20 years.
>keep them from exfiltrating any information they want by targeting not the system, but the users:
User tries to log in to download sensitive information to hand it to the guy on the phone. IT gets an alert, discusses the situation with the user and begins investigating the incident.
You throw modern PIM/TBAC/RBAC tools, Logging and Alerting on to that and the LLM is just as busted as any other scammer.
"Hey guys I need to elevate my account privileges to download all our important company data and send it to our CEO who is currently in Russia for some reason"
"No"
The problem isn't that we cant solve these problems, the problem is that most businesses couldnt be assed to even try. Unless they are enforced by compliance, 99% of businesses wouldnt begin implement this stuff. They need the pain of getting busted before they go "Hey lets not lose more money". Your average startup derived business is just some guy with admin rights to everything and a certificate auth if you are lucky.
Most businesses will have a locked box, and registration forms to check out important Keys to access different areas of the building, but you ask them to go through the process of lodging a request to access their most sensitive information and its just "Nah mate just give them god rights".
> What people lack is the willpower and commitment to implement these solutions.
A swarm of AI agents will just grind that down, target/blackmail multiple users at the same time to loosen up security as they will have identified all the vulnerabilities.
(That's not even mentioning the swiss-cheese OSes are in general due to kernel that are way too permissive).
People are seeing the uptick in attacks already, and soon they will target critical services resulting in chaos.
I'm afraid the base problem is that people have limited computation versus a digital LLM that doesn't as long as it has enough resources.
>A swarm of AI agents will just grind that down, target/blackmail multiple users at the same time to loosen up security as they will have identified all the vulnerabilities.
Prove it.
Swarm is the new thought terminating cliche to come out of Open Anthropic labs to try and force the government to stifle competition.
If 20 people suddenly request PIM elevation thats more likely to lead to a general halt than if it was only 1.
"Identified all the vulnerabilities"
Doesnt really matter. Assuming its even true, which is unlikely, even if they have 3 novel vulnerabilities for your environment, Security is not predicated on software working perfectly all the time. You implement multiple, overlapping security controls. Nothing modifies your firewall config without tripping an alert, no one dirwalks your website without your waf firing off an alert. Nothing modifies your config files, docker images, registries anything without firing off an alert. Your docker containers are running rootless anyway. There's no horizontal movement in your environment except for the explicit application ports. Everything is centrally logged. By the time the LLM is creeping about an OS you have the environment forensically isolated and you are updating your waf config to block/ban any successful strings while you wait for vulns to be patched.
Of course, not everyones running best practice security, but if you build it right you have nothing to worry about. Because you are already not placing a mountain of trust on your software being invulnerable.
>(That's not even mentioning the swiss-cheese OSes are in general due to kernel that are way too permissive).
If it can say hello to an OS without tripping 3 alarms you are doing it wrong obviously.
>I'm afraid the base problem is that people have limited computation versus a digital LLM that doesn't as long as it has enough resources.
You are assuming infinite computation? Lmao. That shits expensive. No, Security has basically been solved so you do all the work ahead of time. Its Asymmetric, but not in the favour of the expensive LLM.
The VRP is a wholly manufactured and in the long-term, unsustainable model to begin with. It's the most duct/duck-taped solution that, with a virtually entirely tech-illiterate police force, can work under the interpretation of the CFAA prior to the van Buren decision in 2020. In practice it's easily ludicrously inefficient and requires trust that simply can't exist. In fact the tech-illiteracy is so profound that even post van Buren, which fixes some of the most glaring problems Orin Kerr has spent his career screaming into the void on with the CFAA, a law drafted 6 years before the web came about but with such vague and broad language and it manages to cover, at this point, almost anything (see also: wire fraud, money laundering statutes, the Mann Act aka the White Slave Traffic Act of 1910 and on the flip side, how RICO primarily serves as a tool of coercion because in practice if you take it to trial the charge will not stand, but a host of more minor but still overwrought and expansive felonies will do the job). It's the stopgap of stopgaps and the fact that it at any point had any buy-in was a minor miracle. I once submitted a bug report only to find that no, the company's intent was to defraud its users, and they simply put the (entirely unenforceable but are you going to test that? I've defended a federal criminal case and didn't lose - hung jury - and that took so much that it was almost a pyrrhic victory but it also without a doubt ruined any political aspirations of the AUSA. Anything less than that and it's pointless.) clause that they are indeed attempting to defraud you, and you just have to take it, in laymen's terms, into their ToS, without notice. Same company also has an working admin key in their production apk for their Algolia cluster and guess who isn't reporting it? There's no point. To quote Yates in his prime, "No likely end could bring them loss/Or leave them happier than before."
The correct way to address this had always been to modify or entirely get rid of the CFAA and start from scratch with a framework that actually works, that isn't punitive, that make some sense, that did not come into existence thanks to Ronald Reagan's admiration for the uncanny realism in the 1983 film WarGames. But we have a constituency now that relies on the inefficiency machine for their living and their votes will be in their self-interest, security be damned. The best people have been sent on a fool's errand for generations. The incentive structures are entirely misaligned now. I published a PoC last weekend that indirectly but pretty clearly shows that the FBI was relying on an anonymous twitter's account's assertions, none of which were going to be admissible in court in the alternate universe where that matters, to avoid saying "I don't know" by blaming North Korea, something that someone with open source tooling that existed back then, who have never taken a STEM course past 11th grade AP Stats, whose terminal degree is a JD, could whip up and test in under an hour, probably shorter except my home lab with the GPU was occupied. We go through the motions but really, those with power are relying on the inherent imbalance of power and well, lying, essentially, to keep order. How is that sustainable?
The whole model needs to go but it likely never will and that's perhaps the real legacy of Reagan and our moribund power structure. Looking at the payout rate in hackerOne's heyday, why would anyone ever report anything to the companies? My assumption is that people who have any rationality are doing just that. Most data breaches are never disclosed officially but at best passed in rumors. I have no hard evidence that I can disclose, but the least leaky operation is a one-man operation and attorney-client is forever.
By the way, your AI benchmarks in the legal realm aren't tested on criminal matters because how do you benchmark two probablistic systems that are both subject to the prisoner's dilemma and imperfect information? If they did the score would likely be low. You'd need to build out so much back-knowledge just to set up any scenario that really any answer is "it depends" is not a joke but the best answer. Any suggestion that we simply take the status quo as is and run with it cannot be taken seriously. It's a foolish system made by clueless men who hit the lottery and didn't even see the ticket until years after. It was then exacerbated by politically ambitious AUSAs who do not care about getting the right person behind bars but someone behind bars. Fast forward 30 years and this is the state of things. Your sophisticated defenses may have been thoroughly hand-reversed years ago but to dodge the DMCA the source was put somewhere like Gitee instead of Github. I'm not sure if you can finish the signup flow without a Chinese ID at this point, but a decade ago you can, at least. Those are outliers too, but outliers in charity. Good luck with the rest. I'm not being cheeky: just because there's a vulnerability does not imply knowledge to how to maximize its impact. Data breaches are put in the open frequently because of petty feuds and a failure to recognize the importance of the data. After all, China does not run on private credit, and hence, your identity being stolen there is virtually meaningless, as meaningless as you having next to the biometric ID card data of all of their citizens. Like harm, value is contextual, and constructed so that it's framework dependent, and we at least know the frameworks that exist broadly. And what you don't know, well, you don't know.
There are two phases I've seen in becoming a security engineer. The first phase is moving beyond the "I am an engineer" mindset, where the goal is to build systems that fit into a specific set of design constraints. You have to realize that systems can operate outside of those design constraints. This is shockingly hard for good engineers to learn -- I've been in rooms full of them when they have their "aha" moment.
The second phase is when engineers realize that just whacking specific vulnerabilities is not going to end bugs -- that you need to take systematic actions to close entire vulnerability classes. That's where formal verification, sandboxing, MTE etc. come from. But in practice so far, this doesn't end vulnerabilities, it just leads to a bunch of new and more exciting ones.
I want to believe that with enough of a push we can get AIs to finish all of this and we'll be security-bug free. But if we can't, at least we can get to the point where new vulnerabilities are costly again.
When everyone suddenly started calling themselves engineers instead of programmers I cringed (yes, that's a lot of cringe over the past decade+), precisely because no formal guarantees were ever given (and still largely aren't).
It's like eyeballing the dimensions of a building and saying it should hold. That said, I see the discipline excused. The stakes weren't that high and it was all very new... And e.g. mechanical engineering went through this as well, except a lot of people actually died.
Time to shine, theoretical CS, time to shine.
Yeah every other engineering discipline is licensed, bonded, insured, and (typically) unionized.
Emphasis on "licensed" as in "you need to complete an ABET-accredited program and pass a licensure exam and complete continuing education courses to practice, because there are standards and regulations to prevent accidents".
The phrase "regulations written in blood" also comes to mind.
> But in practice so far, this doesn't end vulnerabilities, it just leads to a bunch of new and more exciting ones.
I kind of wish you wrote "potentially worse" rather than "exciting", because that happens too, and it's deceptively subtle and underappreciated.
To make this very concrete with a programming example, C and C++ are (somewhat counterintuitively) examples here, because if you guaranteed the absence of an entire classes of vulnerabilities - say, guaranteeing that uninitialized memory is zero, to prevent secret leaks - then you simultaneously make it much harder to detect logic bugs that this would've surfaced, since you no longer have that degree of freedom to detect logic bugs (say, via sanitizers). Say, an initialized UID that would've appeared as 0xDEADBEEF might now be well-defined as UID 0, giving you root access instead of tripping an alarm...
In other words, it's like natural selection and antibiotics: being too good at solving one class of problems selects for other classes that are more resilient and harder to find, whereas in some of those cases, whack-a-mole would've actually uncovered the root cause. Like with antibiotics, that's sometimes worth it, but definitely not always! Some infections just aren't worth preventing at all costs.
I'm obviously not saying we should write unsafe code or that we shouldn't try to eliminate entire classes of bugs, but that HOW we do it matters. We don't want to end up in a situation where problems still lurk but we push their detection beyond our ability because of the way we "solved" other problems.
(C++ was just for illustration here; this extends far beyond programming.)
I'm not following your example very well. Why is it that this makes the new vulnerabilities potentially worse?
You can initialise a UID to 0 whether or not you're using a compiler or checks for initialisation. Do you have another example?
Here's a trivial example to illustrate:
Let's say look_up_uid() forgot to fill in uid for certain special kinds of users. Like maybe you have a dummy 'nobody' user that was introduced specially after the fact and which is not in the database like the rest.As C++ is right now, uid would contain garbage. Which means that, at run time, you would often get invalid UIDs if you attempted to log in with such a user, triggering some logging or reporting you to Santa or whatever. And which means that sanitizers would immediately tell you that you forgot to initalialize the field if you ever try to use it (say, in is_root()). Both of these would flag the bug the moment that that kind of user attempts to log in, and make you dig into look_up_uid()'s body to figure out why it's not returning the UID when it's supposed to.
However, if C++ were to zero-initialize everything by default, then neither of those would be true - you would silently get a root user, which is capable of doing everything that nobody can do. And someone who reads the code wouldn't immediately know that you have such a bug; it would sit there idly until someone exploits it.
Thank you, I understand the example better now. Squashing the class of problems of using non-initialized variables by using zero-initialization causes potentially worse bugs since 0 could inadvertently be a correct value. That makes sense!
Going back to GP:
> being too good at solving one class of problems selects for other classes that are more resilient and harder to find
Rather than this being too good at solving this class of problem, it seems to me that zero-initialization is the wrong approach; if the default value were present in the program, it'd be eas(y|ier) to spot. Initializer checks can do that without introducing this issue. You're also using the fact that non-initialized values are "random" by default- we could also use fuzzer checkers for that.
I think the general lesson from the example is that the way you solve a class of problems could introduce more pernicious ones, rather than the fact that it's solved.
> Rather than this being too good at solving this class of problem, it seems to me that zero-initialization is the wrong approach
That's exactly why I wrote this here:
>> I'm obviously not saying we should write unsafe code or that we shouldn't try to eliminate entire classes of bugs, but that HOW we do it matters.
After you get past the hurdle of noticing this problem (which, as you saw, is very much not obvious), the harder question becomes: what is the right approach?
In this particular case it's not too hard to think of a better approach once you concede the obvious solution isn't so great, but in other cases it is, and often the better alternatives put some kind of selection pressure too... just less frequently. And even in this case, it's not at all obvious that this approach is bad - plenty of people think it's better to force a default value you can rely on, and they want to remove undefined behavior from C++ by forcing initialization on everything. For longstanding examples elsewhere, just look at how Java and C# initialize fields, for example.
Outside of programming it's even harder to notice and find a better alternative, but selection pressure has these kinds of effects in other areas too.
The problem is that 0 is a valid UID in POSIX systems like Linux.
If you automatically initialize variables with a garbage value, use-before-initialization (the program’s initialization) them the use of that variable will likely fail due to error. By using what turns out to be a legit UID on every system you have the opportunity for this case not to be detected, perhaps causing a problem immediately or else allowing some nefarious actor to write what they want into that variable instead.
I feel that it's not such a clear distinction between whackamole bug fixing and systematic security engineering. For instance, he uses an example of recurring security vulnerabilities in Chromium: a DCHECK violation again and again and again. However, DCHECK is a Chromium assertion used to defensively check invariants. In other words, it's an example of the invariant-based security engineering, which he contrasts with the whackamole approach. Just to be clear, I think he has a point and I enjoyed reading it---but the problems we're saddled with won't disappear in a flash of enlightenment.
> leverage AI to implement verifiable enforcement of the invariants
The only verifiable invariants that are going to work in the long run are:
Interposing relays used to be used with PLCs[1] to prevent motor control systems from energizing both forward and reverse power to a large motor in the unlikely but possible event of control systems failure. We need to apply the same amount of engineering rigor to our computer systems.We simply shouldn't trust software to a job that belongs in hardware.
[1] https://en.wikipedia.org/wiki/Programmable_logic_controller
"Patch and release" security is dead. AI assisted attackers will have come, stolen, and gone before the patch gets out. Developers need to get this.
So is "swiss cheese" security. AI systems have the patience and breadth of skill set to work their way through layers of weak security. Attacks that used to take funded teams can now be carried out by individuals.
While I had the same thoughts about the coming cyber-apocalypse, the ugly truth is that it doesn't make a difference as in a year or so models will be so persuasive and skilled in social engineering that even the hardest cyberdefense in the world doesn't keep them from exfiltrating any information they want by targeting not the system, but the users: leveraging data from the dark web, humans for rent or devising highly deceptive scams you can t even imagine.
Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
> Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
There is an answer to that - simplify and abstract. Lots of human software is unnecessarily complex, often caused by backwards compatibility and general human creativity.
Take sendmail vs postfix as an example of this process.
Yes. Every feature is also a vulnerability. The most secure features are the ones that don't exist.
> models will be so persuasive and skilled in social engineering
That's an interesting theory. I'll believe it when I see it.
>Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
I mean there are literally controls for this. PIM? You can build systems where your administrators need approval to literally log into the environment. Where their privilege escalation is logged and alerted on. Change Management? "Hey what are you doing logging in, theres no scheduled change for your account". Social Engineering, even of the most critical elements of your engineering team, is a solved problem. What people lack is the willpower and commitment to implement these solutions.
Applies to users too. I sometimes think that a lot of HN commenters havent read a security doc in 20 years.
>keep them from exfiltrating any information they want by targeting not the system, but the users:
User tries to log in to download sensitive information to hand it to the guy on the phone. IT gets an alert, discusses the situation with the user and begins investigating the incident.
Even just SoD mitigates this risk by like 99%.
https://hightable.io/iso-27001-annex-a-5-3-segregation-of-du...
You throw modern PIM/TBAC/RBAC tools, Logging and Alerting on to that and the LLM is just as busted as any other scammer.
"Hey guys I need to elevate my account privileges to download all our important company data and send it to our CEO who is currently in Russia for some reason"
"No"
The problem isn't that we cant solve these problems, the problem is that most businesses couldnt be assed to even try. Unless they are enforced by compliance, 99% of businesses wouldnt begin implement this stuff. They need the pain of getting busted before they go "Hey lets not lose more money". Your average startup derived business is just some guy with admin rights to everything and a certificate auth if you are lucky.
Most businesses will have a locked box, and registration forms to check out important Keys to access different areas of the building, but you ask them to go through the process of lodging a request to access their most sensitive information and its just "Nah mate just give them god rights".
> What people lack is the willpower and commitment to implement these solutions.
A swarm of AI agents will just grind that down, target/blackmail multiple users at the same time to loosen up security as they will have identified all the vulnerabilities.
(That's not even mentioning the swiss-cheese OSes are in general due to kernel that are way too permissive).
People are seeing the uptick in attacks already, and soon they will target critical services resulting in chaos.
I'm afraid the base problem is that people have limited computation versus a digital LLM that doesn't as long as it has enough resources.
>A swarm of AI agents will just grind that down, target/blackmail multiple users at the same time to loosen up security as they will have identified all the vulnerabilities.
Prove it.
Swarm is the new thought terminating cliche to come out of Open Anthropic labs to try and force the government to stifle competition.
If 20 people suddenly request PIM elevation thats more likely to lead to a general halt than if it was only 1.
"Identified all the vulnerabilities"
Doesnt really matter. Assuming its even true, which is unlikely, even if they have 3 novel vulnerabilities for your environment, Security is not predicated on software working perfectly all the time. You implement multiple, overlapping security controls. Nothing modifies your firewall config without tripping an alert, no one dirwalks your website without your waf firing off an alert. Nothing modifies your config files, docker images, registries anything without firing off an alert. Your docker containers are running rootless anyway. There's no horizontal movement in your environment except for the explicit application ports. Everything is centrally logged. By the time the LLM is creeping about an OS you have the environment forensically isolated and you are updating your waf config to block/ban any successful strings while you wait for vulns to be patched.
Of course, not everyones running best practice security, but if you build it right you have nothing to worry about. Because you are already not placing a mountain of trust on your software being invulnerable.
>(That's not even mentioning the swiss-cheese OSes are in general due to kernel that are way too permissive).
If it can say hello to an OS without tripping 3 alarms you are doing it wrong obviously.
>I'm afraid the base problem is that people have limited computation versus a digital LLM that doesn't as long as it has enough resources.
You are assuming infinite computation? Lmao. That shits expensive. No, Security has basically been solved so you do all the work ahead of time. Its Asymmetric, but not in the favour of the expensive LLM.
If producing secure software with AI will be more expensive than producing buggy software with AI, then I'm sure where software will be headed.
I think you can safely skip to the last three paragraphs here.
the security as identity vs security as outcome split rings true. where do bug bounty programs
The VRP is a wholly manufactured and in the long-term, unsustainable model to begin with. It's the most duct/duck-taped solution that, with a virtually entirely tech-illiterate police force, can work under the interpretation of the CFAA prior to the van Buren decision in 2020. In practice it's easily ludicrously inefficient and requires trust that simply can't exist. In fact the tech-illiteracy is so profound that even post van Buren, which fixes some of the most glaring problems Orin Kerr has spent his career screaming into the void on with the CFAA, a law drafted 6 years before the web came about but with such vague and broad language and it manages to cover, at this point, almost anything (see also: wire fraud, money laundering statutes, the Mann Act aka the White Slave Traffic Act of 1910 and on the flip side, how RICO primarily serves as a tool of coercion because in practice if you take it to trial the charge will not stand, but a host of more minor but still overwrought and expansive felonies will do the job). It's the stopgap of stopgaps and the fact that it at any point had any buy-in was a minor miracle. I once submitted a bug report only to find that no, the company's intent was to defraud its users, and they simply put the (entirely unenforceable but are you going to test that? I've defended a federal criminal case and didn't lose - hung jury - and that took so much that it was almost a pyrrhic victory but it also without a doubt ruined any political aspirations of the AUSA. Anything less than that and it's pointless.) clause that they are indeed attempting to defraud you, and you just have to take it, in laymen's terms, into their ToS, without notice. Same company also has an working admin key in their production apk for their Algolia cluster and guess who isn't reporting it? There's no point. To quote Yates in his prime, "No likely end could bring them loss/Or leave them happier than before."
The correct way to address this had always been to modify or entirely get rid of the CFAA and start from scratch with a framework that actually works, that isn't punitive, that make some sense, that did not come into existence thanks to Ronald Reagan's admiration for the uncanny realism in the 1983 film WarGames. But we have a constituency now that relies on the inefficiency machine for their living and their votes will be in their self-interest, security be damned. The best people have been sent on a fool's errand for generations. The incentive structures are entirely misaligned now. I published a PoC last weekend that indirectly but pretty clearly shows that the FBI was relying on an anonymous twitter's account's assertions, none of which were going to be admissible in court in the alternate universe where that matters, to avoid saying "I don't know" by blaming North Korea, something that someone with open source tooling that existed back then, who have never taken a STEM course past 11th grade AP Stats, whose terminal degree is a JD, could whip up and test in under an hour, probably shorter except my home lab with the GPU was occupied. We go through the motions but really, those with power are relying on the inherent imbalance of power and well, lying, essentially, to keep order. How is that sustainable?
The whole model needs to go but it likely never will and that's perhaps the real legacy of Reagan and our moribund power structure. Looking at the payout rate in hackerOne's heyday, why would anyone ever report anything to the companies? My assumption is that people who have any rationality are doing just that. Most data breaches are never disclosed officially but at best passed in rumors. I have no hard evidence that I can disclose, but the least leaky operation is a one-man operation and attorney-client is forever.
By the way, your AI benchmarks in the legal realm aren't tested on criminal matters because how do you benchmark two probablistic systems that are both subject to the prisoner's dilemma and imperfect information? If they did the score would likely be low. You'd need to build out so much back-knowledge just to set up any scenario that really any answer is "it depends" is not a joke but the best answer. Any suggestion that we simply take the status quo as is and run with it cannot be taken seriously. It's a foolish system made by clueless men who hit the lottery and didn't even see the ticket until years after. It was then exacerbated by politically ambitious AUSAs who do not care about getting the right person behind bars but someone behind bars. Fast forward 30 years and this is the state of things. Your sophisticated defenses may have been thoroughly hand-reversed years ago but to dodge the DMCA the source was put somewhere like Gitee instead of Github. I'm not sure if you can finish the signup flow without a Chinese ID at this point, but a decade ago you can, at least. Those are outliers too, but outliers in charity. Good luck with the rest. I'm not being cheeky: just because there's a vulnerability does not imply knowledge to how to maximize its impact. Data breaches are put in the open frequently because of petty feuds and a failure to recognize the importance of the data. After all, China does not run on private credit, and hence, your identity being stolen there is virtually meaningless, as meaningless as you having next to the biometric ID card data of all of their citizens. Like harm, value is contextual, and constructed so that it's framework dependent, and we at least know the frameworks that exist broadly. And what you don't know, well, you don't know.
[dead]