> There was no unauthorised access and no compromised systems. This was an overload attack, not an intrusion.
Hopefully your logging infra is rock solid and nothing has been dropped in the flood. It wouldn't be the first time a DOS was used to mask the actual attack by overwhelming the monitoring infra.
> Use a CNAME or ALIAS record instead of an A record. An A record ties your domain to one specific IP address on our platform. That fixed binding was exactly the problem during the attack: wherever we could change the address on short notice, availability could be restored, wherever we could not, only the blunt measure remained.
I don't understand how this helps. CNAMES have TTLs like A records and they eventually have to terminate at an A record somewhere so why pay for an extra hop?
> I don't understand how this helps. CNAMES have TTLs like A records and they eventually have to terminate at an A record somewhere so why pay for an extra hop?
I assume the customer controls the domain DNS records here rather than the hosting provider.
CNAME record: hosting provider can change underlying IP freely.
A record: hosting provider must get in touch with customer to change DNS.
Naive question: services that can be used for amplification attacks, are they constantly getting patched to prevent the latest iteration of attack type?
In other words, if there are a bunch of services prone to amplification attacks, can traffic from these services be upstream-blackholed for the duration of the attack?
If it's not traffic coming directly from an IoT botnet, which is probably where the source of the spoofed traffic that initiates the amplification, then isn't there likely a smaller, more manageable number of services responsible for the attack traffic?
Or are we talking services that form the substrate of the internet that have inherently exploitable protocols that it would take a large herd of organized cats in order to update in a way that doesn't break the internet, and will still take ~10 years?
I still think in IPv4, so this may be a stupid question, but it's it known how many unique IP addresses were attempting to connect in the space of that time, and then it's there logging to identify those with unusually large amounts of individual traffic?
I sometimes work on a system so byzantine and poorly-documented with so many horribly obsolete systems - they have Windows XP machines all over the place - that there's no way it hasn't been popped.
I think it's safe though.
I think various groups have popped it, taken a look around with growing expressions of horror on their faces, and gotten back out as quickly as they could. Nah man, no way, not touching that, don't want my prints on this particular gun, there's no way this is legitimately how they've left it, this has to be some kind of a setup.
Yes, that's bunny.net, it's named in the post. Their Zurich PoP appears to sit on CDN77's network, which is on their side of the setup, not something we configured.
Yes. The blog version was drafted with AI assistance and then edited and reviewed by us. The facts, numbers and timeline are ours and match the technical postmortem PDF linked in the post, which is the drier, more complete version. Happy to go into any of the technical details here.
> There was no unauthorised access and no compromised systems. This was an overload attack, not an intrusion.
Hopefully your logging infra is rock solid and nothing has been dropped in the flood. It wouldn't be the first time a DOS was used to mask the actual attack by overwhelming the monitoring infra.
> Use a CNAME or ALIAS record instead of an A record. An A record ties your domain to one specific IP address on our platform. That fixed binding was exactly the problem during the attack: wherever we could change the address on short notice, availability could be restored, wherever we could not, only the blunt measure remained.
I don't understand how this helps. CNAMES have TTLs like A records and they eventually have to terminate at an A record somewhere so why pay for an extra hop?
> I don't understand how this helps. CNAMES have TTLs like A records and they eventually have to terminate at an A record somewhere so why pay for an extra hop?
I assume the customer controls the domain DNS records here rather than the hosting provider.
CNAME record: hosting provider can change underlying IP freely.
A record: hosting provider must get in touch with customer to change DNS.
I probably incorrectly read
> we provide that customer’s connection to the internet
as providing the DNS infra for their customers.
Naive question: services that can be used for amplification attacks, are they constantly getting patched to prevent the latest iteration of attack type?
In other words, if there are a bunch of services prone to amplification attacks, can traffic from these services be upstream-blackholed for the duration of the attack?
If it's not traffic coming directly from an IoT botnet, which is probably where the source of the spoofed traffic that initiates the amplification, then isn't there likely a smaller, more manageable number of services responsible for the attack traffic?
Or are we talking services that form the substrate of the internet that have inherently exploitable protocols that it would take a large herd of organized cats in order to update in a way that doesn't break the internet, and will still take ~10 years?
I still think in IPv4, so this may be a stupid question, but it's it known how many unique IP addresses were attempting to connect in the space of that time, and then it's there logging to identify those with unusually large amounts of individual traffic?
"We found three concrete gaps during this incident, and we would rather be upfront about them than gloss over them." claudism?
Sticks out like a sore thumb, doesn't it?
> There was no unauthorised access and no compromised systems. This was an overload attack, not an intrusion.
"AI said it's all good. There are no attackers within our walls."
I sometimes work on a system so byzantine and poorly-documented with so many horribly obsolete systems - they have Windows XP machines all over the place - that there's no way it hasn't been popped.
I think it's safe though.
I think various groups have popped it, taken a look around with growing expressions of horror on their faces, and gotten back out as quickly as they could. Nah man, no way, not touching that, don't want my prints on this particular gun, there's no way this is legitimately how they've left it, this has to be some kind of a setup.
Security by obsolescence layered with security by complexity.
it's not enough to salt your hash; you must mold your Swiss Cheese
Yes, that's bunny.net, it's named in the post. Their Zurich PoP appears to sit on CDN77's network, which is on their side of the setup, not something we configured.
Did AI write this?
Yes. The blog version was drafted with AI assistance and then edited and reviewed by us. The facts, numbers and timeline are ours and match the technical postmortem PDF linked in the post, which is the drier, more complete version. Happy to go into any of the technical details here.