While I had the same thoughts about the coming cyber-apocalypse, the ugly truth is that it doesn't make a difference as in a year or so models will be so persuasive and skilled in social engineering that even the hardest cyberdefense in the world doesn't keep them from exfiltrating any information they want by targeting not the system, but the users: leveraging data from the dark web, humans for rent or devising highly deceptive scams you can t even imagine.
Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
There is an answer to that - simplify and abstract. Lots of human software is unnecessarily complex, often caused by backwards compatibility and general human creativity.
Take sendmail vs postfix as an example of this process.
Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
I mean there are literally controls for this. PIM? You can build systems where your administrators need approval to literally log into the environment. Where their privilege escalation is logged and alerted on. Change Management? "Hey what are you doing logging in, theres no scheduled change for your account". Social Engineering, even of the most critical elements of your engineering team, is a solved problem. What people lack is the willpower and commitment to implement these solutions.
Applies to users too. I sometimes think that a lot of HN commenters havent read a security doc in 20 years.
keep them from exfiltrating any information they want by targeting not the system, but the users:
User tries to log in to download sensitive information to hand it to the guy on the phone. IT gets an alert, discusses the situation with the user and begins investigating the incident.
You throw modern PIM/TBAC/RBAC tools, Logging and Alerting on to that and the LLM is just as busted as any other scammer.
"Hey guys I need to elevate my account privileges to download all our important company data and send it to our CEO who is currently in Russia for some reason"
"No"
The problem isn't that we cant solve these problems, the problem is that most businesses couldnt be assed to even try. Unless they are enforced by compliance, 99% of businesses wouldnt begin implement this stuff. They need the pain of getting busted before they go "Hey lets not lose more money". Your average startup derived business is just some guy with admin rights to everything and a certificate auth if you are lucky.
Most businesses will have a locked box, and registration forms to check out important Keys to access different areas of the building, but you ask them to go through the process of lodging a request to access their most sensitive information and its just "Nah mate just give them god rights".
What people lack is the willpower and commitment to implement these solutions.
A swarm of AI agents will just grind that down, target/blackmail multiple users at the same time to loosen up security as they will have identified all the vulnerabilities.
(That's not even mentioning the swiss-cheese OSes are in general due to kernel that are way too permissive).
People are seeing the uptick in attacks already, and soon they will target critical services resulting in chaos.
I'm afraid the base problem is that people have limited computation versus a digital LLM that doesn't as long as it has enough resources.
A swarm of AI agents will just grind that down, target/blackmail multiple users at the same time to loosen up security as they will have identified all the vulnerabilities.
Prove it.
Swarm is the new thought terminating cliche to come out of Open Anthropic labs to try and force the government to stifle competition.
If 20 people suddenly request PIM elevation thats more likely to lead to a general halt than if it was only 1.
"Identified all the vulnerabilities"
Doesnt really matter. Assuming its even true, which is unlikely, even if they have 3 novel vulnerabilities for your environment, Security is not predicated on software working perfectly all the time. You implement multiple, overlapping security controls. Nothing modifies your firewall config without tripping an alert, no one dirwalks your website without your waf firing off an alert. Nothing modifies your config files, docker images, registries anything without firing off an alert. Your docker containers are running rootless anyway. There's no horizontal movement in your environment except for the explicit application ports. Everything is centrally logged. By the time the LLM is creeping about an OS you have the environment forensically isolated and you are updating your waf config to block/ban any successful strings while you wait for vulns to be patched.
Of course, not everyones running best practice security, but if you build it right you have nothing to worry about. Because you are already not placing a mountain of trust on your software being invulnerable.
(That's not even mentioning the swiss-cheese OSes are in general due to kernel that are way too permissive).
If it can say hello to an OS without tripping 3 alarms you are doing it wrong obviously.
I'm afraid the base problem is that people have limited computation versus a digital LLM that doesn't as long as it has enough resources.
You are assuming infinite computation? Lmao. That shits expensive. No, Security has basically been solved so you do all the work ahead of time. Its Asymmetric, but not in the favour of the expensive LLM.
Comments
While I had the same thoughts about the coming cyber-apocalypse, the ugly truth is that it doesn't make a difference as in a year or so models will be so persuasive and skilled in social engineering that even the hardest cyberdefense in the world doesn't keep them from exfiltrating any information they want by targeting not the system, but the users: leveraging data from the dark web, humans for rent or devising highly deceptive scams you can t even imagine.
Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
There is an answer to that - simplify and abstract. Lots of human software is unnecessarily complex, often caused by backwards compatibility and general human creativity.
Take sendmail vs postfix as an example of this process.
Yes. Every feature is also a vulnerability. The most secure features are the ones that don't exist.
That's an interesting theory. I'll believe it when I see it.
I mean there are literally controls for this. PIM? You can build systems where your administrators need approval to literally log into the environment. Where their privilege escalation is logged and alerted on. Change Management? "Hey what are you doing logging in, theres no scheduled change for your account". Social Engineering, even of the most critical elements of your engineering team, is a solved problem. What people lack is the willpower and commitment to implement these solutions.
Applies to users too. I sometimes think that a lot of HN commenters havent read a security doc in 20 years.
User tries to log in to download sensitive information to hand it to the guy on the phone. IT gets an alert, discusses the situation with the user and begins investigating the incident.
Even just SoD mitigates this risk by like 99%.
https://hightable.io/iso-27001-annex-a-5-3-segregation-of-du...
You throw modern PIM/TBAC/RBAC tools, Logging and Alerting on to that and the LLM is just as busted as any other scammer.
"Hey guys I need to elevate my account privileges to download all our important company data and send it to our CEO who is currently in Russia for some reason"
"No"
The problem isn't that we cant solve these problems, the problem is that most businesses couldnt be assed to even try. Unless they are enforced by compliance, 99% of businesses wouldnt begin implement this stuff. They need the pain of getting busted before they go "Hey lets not lose more money". Your average startup derived business is just some guy with admin rights to everything and a certificate auth if you are lucky.
Most businesses will have a locked box, and registration forms to check out important Keys to access different areas of the building, but you ask them to go through the process of lodging a request to access their most sensitive information and its just "Nah mate just give them god rights".
A swarm of AI agents will just grind that down, target/blackmail multiple users at the same time to loosen up security as they will have identified all the vulnerabilities.
(That's not even mentioning the swiss-cheese OSes are in general due to kernel that are way too permissive).
People are seeing the uptick in attacks already, and soon they will target critical services resulting in chaos.
I'm afraid the base problem is that people have limited computation versus a digital LLM that doesn't as long as it has enough resources.
Prove it.
Swarm is the new thought terminating cliche to come out of Open Anthropic labs to try and force the government to stifle competition.
If 20 people suddenly request PIM elevation thats more likely to lead to a general halt than if it was only 1.
"Identified all the vulnerabilities"
Doesnt really matter. Assuming its even true, which is unlikely, even if they have 3 novel vulnerabilities for your environment, Security is not predicated on software working perfectly all the time. You implement multiple, overlapping security controls. Nothing modifies your firewall config without tripping an alert, no one dirwalks your website without your waf firing off an alert. Nothing modifies your config files, docker images, registries anything without firing off an alert. Your docker containers are running rootless anyway. There's no horizontal movement in your environment except for the explicit application ports. Everything is centrally logged. By the time the LLM is creeping about an OS you have the environment forensically isolated and you are updating your waf config to block/ban any successful strings while you wait for vulns to be patched.
Of course, not everyones running best practice security, but if you build it right you have nothing to worry about. Because you are already not placing a mountain of trust on your software being invulnerable.
If it can say hello to an OS without tripping 3 alarms you are doing it wrong obviously.
You are assuming infinite computation? Lmao. That shits expensive. No, Security has basically been solved so you do all the work ahead of time. Its Asymmetric, but not in the favour of the expensive LLM.