Looking at our SOC 2 report (we don't use Delve, our auditor isn't on their list) I don't think this is quite the smoking gun it might look like if you're not reading SOC 2 reports for a living.
There's a fair amount of boiler plate language in these reports, and a bunch of re-stating the SOC 2 controls. I'd expect two reports (same auditors, same platforms) to be nearly identical. If they're both using AWS, Github, Stripe, Vetty, they're subbing a lot of the exact same thing out to the same companies, referencing the same set of internal controls.
Reading ours. There's a section titled $Company's Controls, followed by 20 pages listing the various SOC 2 controls. e.g.
---
CC9.0 Common Criteria Related to Risk Mitigation
CC9.1 The entity identifies, selects, and develops risk mitigation activities for risks arising from potential
business disruptions.
IR-01 A Security Incident Response Plan that outlines the process of identifying, prioritizing, communicating, assigning, and tracking confirmed incidents through to resolution is accessible to all relevant employees and contractors and is reviewed annually.
---
Then there's another 20 pages of those same controls being listed, some language about how they tested the controls, and hopefully "No Exceptions Noted".
That's not going to change much between companies.
This mirrors my thoughts. A page of boiler play text with some check boxes, with some checked vs unchecked is going to be 99.8% similar between companies as well. A lot of audits are very much forms with boiler plate and fill in the blank. There is no point rewriting everything from scratch.
it does highlight the efficiency value of boilerplate. you only have to proof it once, really well of course. all downstream instances get the benefit of that one very good review.
boilerplate overlap is expected, no one rewrites these from scratch
though when everything lines up the same way across hundreds of reports, it gets weird...
I mean look at those reports; same pagination, same auditor showing up almost everywhere, no exceptions across all clients.. not even efficient templates should be like that
you still expect variation in scope, findings, structure, even if the base language is reused
I mean it’s a template, but in theory someone went and checked stuff. Did you actually have a quarterly security team meeting? Was there minutes? Was there an invite?
Did someone actually go and confirm your role based access control matrix is up to date and user accounts have the right access? Were all of those screenshots watermarked with timestamps?
There is work to do, whether or not auditors are doing it is another question.
yes. I think some overlap is normal, but this is not that, eg. seen:
• same pagination across hundreds of reports → 100% template output
• same auditor license everywhere → either extreme concentration or just rubber stamping
• zero exceptions across all clients → unrealistic, real audits always find something.. right?
• system descriptions pulled from marketing sites → .. copy paste
at one point you’re really looking at reports that were never really produced per each company
> "We may receive compensation from vendors listed below. All recommendations are based on independent research."
this + new HN account? couldn't be more obviously a competitor. not to defend delve, but can’t be pushing this like some noble effort with the goal of transparency
also lol @ the fake realtime "just searched for" toasts on a setInterval in the bottom left.
> lol @ the fake realtime "just searched for" toasts on a setInterval in the bottom left.
There is intermittent XHR traffic, but it's most definitely not returning any such information and it's posting what looks (in FF's dev tools) like garbage binary data.
hello, this isn’t a competitor. I run a consulting company in the cybersecurity space and saw a chance to make this whole process more transparent.
I agree it came off a bit clickbaity, I'm sorry, Claude probably pushed it too far. but I don’t have an audience anywhere, no following on social, so I needed to ship something fast and make it engaging. the intent wasn’t just this Delve thing, the goal is to move away from it and turn it into a proper hub for compliance transparency over time. But i need a way to marketing this intially.
it’s been less than 24h, I built and pushed everything pretty quickly, so yeah there are rough edges. I’m already working through them and fixing things.
on the account being new, I get how that looks. I mostly use X and reddit, this is actually my first time posting on HN so I had to create an account.
Not clickbaity, but downright misleading and, for all we know about the XHR traffic, potentially malicious.
> But i need a way to marketing this intially.
Posting a site which blatantly fakes statistics, such as the ones mentioned two posts up from here, is not doing any favors.
Sending XHR requests with opaque binary data every few seconds is not doing you any favors. No information-only site has _any_ business XHR-posting opaque state every several seconds.
"Post it quickly at all costs, accuracy be damned," is not a viable strategy for a legitimate and believable site.
" let r = ["Acme Corp", "CloudVault", "DataSync Pro", "NexGen AI", "SecureStack", "TrustLayer", "Vanta", "ComplianceIQ", "InfraSec", "ByteShield", "PipelineOps", "CyberNova", "TokenGuard", "ZeroTrust Labs", "Aether Security", "PrismData", "CloudArmor", "RiskLens", "AuditTrail", "ShieldIO"]
, n = ["just checked", "searched for", "ran a scan on", "verified"]
, a = ["San Francisco, CA", "New York, NY", "Austin, TX", "London, UK", "Berlin, DE", "Toronto, CA", "Seattle, WA", "Chicago, IL", "Denver, CO", "Boston, MA", "Singapore", "Sydney, AU"];
"
fake popups, xyz domain, recent zeitgeist, 100% straight vibecoded. good hustle I have to say. a domain that'll now get ranked on google for SOC 2 compliance which likely has a high CPC and good DR to piggyback off.
fair call on the popups, they’re not real-time. I added them quickly to make the page feel less empty while testing engagement, probably not the best call in hindsight and I’ll remove or replace them with something real.
on the "vibecoded" part, yeah I moved fast. this was built in under a day to get something out and see if people even care about this angle. that doesn’t mean the underlying data or direction is fake though.
the domain choice is just speed and availability, not some SEO master plan. if this turns into something real I’ll move it to a proper brand/domain.
and yeah I get why it looks like a growth/SEO play, but the actual goal is to push more transparency around these audits. if I just wanted traffic there are easier angles than going after something this niche and messy.
either way, appreciate you calling it out, some of it is fair and already being fixed.
Cybersecurity compliance, but .xyz site built under a day to get something out fast to drum up engagement and "test the vibes on the idea". Makes one wonder what became of this industry.
Something bad happens because of lack of regulation -> People strive for regulation -> Govt's actually regulates and sets some norms/procedures -> system works for a while -> Then someone takes the same idea and molds it into something else to bypass the regulation -> they get promoted because they are "clever" and get rewarded -> Then something bad happens as the tool is used by public.
From Prediction markets to Buy now, pay later to Delve to so many other things.
Is there a name to this particular phenomenon, because this just keeps on repeating in multiple industries.
Does SOC2 in general have a particularly high reputation?
The only security compliance frameworks that have any particular reputation with me are the ones associated with the department of defense where the consequences range between a slap on the wrist warning or a small 5 figure fine to execution for espionage (which only ever happened for Julius and Ethel Rosenberg, though one could imagine there may have been more, uh, unofficial consequences that nobody ever heard about). In other words, people actually care about the enforcement of security standards in meaningful ways and there are meaningful consequences.
Everything else... well they're all at least a little better than a participation trophy and the process proving you're trying isn't meaningless. It's just not been my experience with these things that they're particularly good guarantees that the spirit embodying the compliance program is actually being done particularly well.
It's the universal de facto standard at least in North America, and nobody takes it especially seriously. About the best thing you could say for it is that it verifies that you're an actual company and not 3 raccoons in a trench coat. But if you're savvy about how you manage your auditors, you can get an attestation for 3 raccoons as well.
Most of this isn't that damning. SOC IIs are already highly templatized, so pages matching up really isn't meaningful. In fact, an overly detailed or overly verbose template is more likely to have matching pages since you'd never have to add additional content to it.
System descriptions don't necessarily hold much weight. They're often more about giving a general shape of the system to help orient the reader, rather than providing a technically complete picture.
Most of the meat in these is about the controls being tested (which are semi-standardized within an auditor) and the results. Many of these controls are really basic and easy to get "no exceptions noted".
That being said, nearly everyone has at least one exception, even if it's minor. The fact that they didn't find any across all of their clients is a strong indicator they're not diving deeply enough.
Probably not, in fact your auditors not being terribly thorough might be a selling point. But your clients, who are the ones asking for the box to be checked, might.
In my experience, clients don't dig deeply into the report or the auditor, they just want to see that you 1) have the report 2) it doesn’t have any egregious exceptions. Perhaps if this makes big enough news, that’ll change.
"You" probably don't, but it's not just "you". There's also the counterparty who's asking to see that report. Maybe they're doing it for paper-pushing purposes of their own, but ultimately, somewhere up the chain, there's someone thinking "I can't personally audit all my suppliers, and I can't be sure they're doing the right thing, so I'm going to ask them to get an independent audit".
Of course, this shows that the entire system is a bit of a charade, but the point is that someone cares and they're gonna be annoyed when they find out that the audit appears to be a sham.
Whether they have a good alternative is a separate question. But here's another way to look at it: if we show blatant disregard for self-regulation, the government is eventually going to show up and come up with more onerous rules.
Is it true, though? Or has everyone just been psyched into asking for that certification out of a vague fear of "consequences" or of being left behind?
It's not either-or. Companies care about security because of the consequences. If you're a big company contracting a small one, you don't want to get owned through that vendor because you know you'll be the one holding the bag (data loss, reputational damage, regulatory scrutiny, lawsuits).
Small vendors will tell you what you want to hear because they're desperate for your business. Independent auditing is, in theory, a way to get closer to the ground truth. Well, in theory.
As the company? No. In fact, it's likely better for you if they do a bad job. You potentially get shielded from blame, but don't actually have to put in the work.
As a user/customer/potential victim? Yeah, you do.
The no exceptions noted piece is kinda funny. Most SOC2 auditors at least put in the minimal effort of finding one person who didn’t do their cybersecurity training, so the report’s not total boiler plate.
I'd find this more compelling if you looked at a few thousand Vanta or Drata reports grouped by auditor. You're going to find the same commonalities with only trivial language differences.
SOC2 reports are private between you and the auditor (that way if you "fail" you can just find another auditor or have a re-do, and no one is the wiser), and basically always gated behind a sales touchpoint (another hint about what utility they provide). I guess the Delve ones leaked which is why they can all be compared.
220 out of 494 "no exceptions" seems quite high to me. Nobody I've ever dealt with allows an exception to make its way into the report.
what do you expect? if you’re “automating” an audit, it already means you don’t care. the LLM is there to blur the calculus of responsibility, take the blame if someone cares enough to look. happy customers, until someone “delves” a little too deep (like you did) and ruins the slumber party.
This was my general reaction when it was determined that the "log review" portion of the PCI checklist (can't remember what level) could be satisfied by computer "review", and that newer PCI versions were moving towards preferring automated "review"
I've worked with SOC2-certified companies where employees would email each other plaintext credentials, publish them in Notion pages, etc. You cannot cure stupidity by "complying".
That's a difficult question to answer. It shouldn't be, but it is. The reality is, SOC2 is a sales-enablement tool. You should:
* Run a SOC2/compliance program that is entirely disjoint from your security practice.
* Defer SOC2 until the work required to sell into customers demanding it (phone calls, questionnaires) exceeds the cost of obtaining SOC2.
* Prepare for SOC2 by making simple best-practices engineering decisions, in particular single-signon for virtually everything and protected branches for all your repositories.
* Do not allow SOC2 to force any engineering decisions that you would not have intuitively made yourself (this is a big risk with the evidence-gathering platforms like Drata, Delve, and Vanta).
* Assume your SOC2 Type I report will suffice as a first attestation (ie: buy you 1 year of time) with all your customers, and understand that you cannot fail to obtain a Type I; your Type I is guaranteed.
Over 5-6 years of discussing SOC2 with other security practitioners pretty intensively, the overwhelming weight of the evidence is that ~practically nobody actually reads SOC2 reports; they just check the box for each vendor and move on. Plan accordingly.
Since you know a lot about SOC: is SOC2 Type I (point in time) enough to close enterprise sales? Is it worth getting for a new startup (seems super simple)?
It's complicated. In theory, SOC2 forces you to do some important stuff, like define your threat model and say "I can mitigate against the threats and prove that my mitigations are in place". The problem is always that the companies that care don't need it but are burdened with it while the companies that don't care will just checkbox their way through it. It sort of enforces a very baseline security posture, in theory, but the major win of "We've thought our security through" is more of a choice - SOC2 can't actually force you to care.
A ton of these SOC2 vendors take all of the potential good parts of SOC2 out of the equation, building the threat models for you and then you just hook up your gsuite/ github and they check boxes for you or tell you to flip a policy here or there. Delve took this to the extreme by not even asking you to flip the checkboxes.
That said, it doesn't matter if it's legit. Everyone is SOC2, and part of being SOC2 is that the vendors whose products you purchase are SOC2, so it's not a choice - you have to be SOC2 if you want to sell (industry/ product specific, but at some point it'll be clear if it applies). If your goal is security, well, SOC2 is irrelevant.
Ultimately, you'll end up having a separate compliance team to manage SOC2 and you'll actively try to keep "real security" from it because real security has to change over time. You'll encode the absolute minimum possible into your compliance for that reason so that you can easily pass every year and then, if you care about security, you'll invest in that separately.
You can get a long, long way without SOC2; virtually every prospective customer you run into that asks for a SOC2 will have an alternate on-ramp for vendors without it, and the ones that don't will sign a contingent PO on your Type I, which (again) you are guaranteed to get.
The idea that SOC2 forces you to do important stuff gets it backwards; SOC2 documents your existing practice, and demands only extremely high-level controls that you can deliver in any number of ways. Your security practice should (minimally) inform your SOC2, not the other way around.
Yes, that's true. I edited my post to be a bit clearer about this. When you need a SOC2 is going to depend a lot on your business. Lots of companies can make exceptions very easily. Type 1 is easy, I would highly recommend starting there pretty much no matter what since it'll be good practice before your SOC2.
> The idea that SOC2 forces you to do important stuff gets it backwards;
It's the goal behind SOC2. You're assuming a company has a security practice that informs the SOC2 but I think the idea is that companies have no security practice and the SOC2 is what forces them to sit down and build one. What you're describing is more like what happens when a company that actually cares about security goes through SOC2 - you take what you have, put it into a NIST format, and map minimal controls from your practices to the CCs. Most companies have nothing to start with.
Tangential to this but do ISO certifications make sense or are they security theater as well?
And another question but as a consumer, is there any certification which can meaningfully try to show if people/business take their security carefully or are all things security theater in that aspect and at some point, we just have to trust the enterprise and look for other signals of security (like for example blog posts which might show a deep-dive into security for example comes to my mind)
Not really. As long as current system where auditors are also clients of company being audited, the conflict of interest is too high.
Also, not to mention in many countries, the cost of getting breached is nothing so many companies are willing to just hope for the best and payout in case of the worst.
For enterprise sales you can get a SOC 2 Type I faster than any enterprise sale goes through. Typically, most enterprises are okay if you show them proof that you are "in the process" of getting the certification by showing them that you have signed up with one of those platforms (Delve, Vanta, etc.), so you would be okay to start only when you are about to close one of those enterprise deals.
Yeah, we got a signed letter of engagement from our auditor, which was enough to unlock a customer without having to go through any sidestepping process.
In my mind getting a clean report required three kinds of work:
1. Work that actively improved our security posture.
2. Work that didn't change much, but made our security posture easier to understand.
3. Busy work.
I think for most companies all three kinds of work will be required, but you can also make decisions that will push the percentages around. SOC 2 required us to start doing an annual security table top exercise. You could sit down, run a scenario, run it as fast as you can, and come up with a few pre-determined "improvements" that would help if you actually had that problem in the future. Or you could sit down and really put work into it, and see what works well and what doesn't.
As an example in our last tabletop I "exfiltrated" some data from one of our servers, and challenged the team to figure out what I'd done. The easy way out would have been for someone to say "We'll look at the logs and figure it out", but instead I asked them to actually try and find it. We discovered that the sheer volume of logs for that system made them hard to work with. So we made some changes to make them easier to work with and repeated the exercise later.
It could have been busy work, but instead we got real value from it.
It’s fine for what it is: some light guardrails that attempt to nudge you towards answering “is this all just a house of cards that will obviously collapse under a light breeze”.
Getting a SOC2 doesn’t mean you’re amazing or secure or stable. If a customer says they’ll write you a fat check but they need you to have a SOC2, tell them you’ll get it within a year if they start paying. Otherwise don’t bother.
It basically shows clients that you are not doing wildly incompetent things with their data, or if you are, they can more easily sue you, since you probably lied to your auditor about it.
But it’s ultimately not up to you if you do it or not. If all of your potential clients demand it, it’s generally easier to get it than it is to get on the phone with all of your potential clients’ IT departments and explain why you don’t have it.
SOC2 has been in trouble for a while now. Completely gamified. I was managing an acquisition of a healthtech company and asked if they did an internal risk assessment as part of their audit. Nope.
SOC2 certified, has never actually put to paper "here's what we know we're doing wrong, here is how we plan to remediate it."
Looking at our SOC 2 report (we don't use Delve, our auditor isn't on their list) I don't think this is quite the smoking gun it might look like if you're not reading SOC 2 reports for a living.
There's a fair amount of boiler plate language in these reports, and a bunch of re-stating the SOC 2 controls. I'd expect two reports (same auditors, same platforms) to be nearly identical. If they're both using AWS, Github, Stripe, Vetty, they're subbing a lot of the exact same thing out to the same companies, referencing the same set of internal controls.
Reading ours. There's a section titled $Company's Controls, followed by 20 pages listing the various SOC 2 controls. e.g.
---
CC9.0 Common Criteria Related to Risk Mitigation
CC9.1 The entity identifies, selects, and develops risk mitigation activities for risks arising from potential business disruptions.
IR-01 A Security Incident Response Plan that outlines the process of identifying, prioritizing, communicating, assigning, and tracking confirmed incidents through to resolution is accessible to all relevant employees and contractors and is reviewed annually.
---
Then there's another 20 pages of those same controls being listed, some language about how they tested the controls, and hopefully "No Exceptions Noted".
That's not going to change much between companies.
This mirrors my thoughts. A page of boiler play text with some check boxes, with some checked vs unchecked is going to be 99.8% similar between companies as well. A lot of audits are very much forms with boiler plate and fill in the blank. There is no point rewriting everything from scratch.
And those headlines read like Gemini's "punchy" writing.
boilerplate is one word. sorry for the nit, feel free to backpfeifengesicht
I don't think that is an important point.
it does highlight the efficiency value of boilerplate. you only have to proof it once, really well of course. all downstream instances get the benefit of that one very good review.
boilerplate overlap is expected, no one rewrites these from scratch
though when everything lines up the same way across hundreds of reports, it gets weird...
I mean look at those reports; same pagination, same auditor showing up almost everywhere, no exceptions across all clients.. not even efficient templates should be like that
you still expect variation in scope, findings, structure, even if the base language is reused
big signal
One word is two words.
The main issue isn't about the reports being copy-pasted - it's about they are created with no audit ever being done.
You literally paid to get your cert without ever getting audited.
But maybe you shouldn’t raise so much money and make a big fuss about it when all you’re selling is a template?
I mean it’s a template, but in theory someone went and checked stuff. Did you actually have a quarterly security team meeting? Was there minutes? Was there an invite?
Did someone actually go and confirm your role based access control matrix is up to date and user accounts have the right access? Were all of those screenshots watermarked with timestamps?
There is work to do, whether or not auditors are doing it is another question.
Why not?
yes. I think some overlap is normal, but this is not that, eg. seen:
• same pagination across hundreds of reports → 100% template output • same auditor license everywhere → either extreme concentration or just rubber stamping • zero exceptions across all clients → unrealistic, real audits always find something.. right? • system descriptions pulled from marketing sites → .. copy paste
at one point you’re really looking at reports that were never really produced per each company
and that’s the problem
> "We may receive compensation from vendors listed below. All recommendations are based on independent research."
this + new HN account? couldn't be more obviously a competitor. not to defend delve, but can’t be pushing this like some noble effort with the goal of transparency
also lol @ the fake realtime "just searched for" toasts on a setInterval in the bottom left.
> lol @ the fake realtime "just searched for" toasts on a setInterval in the bottom left.
There is intermittent XHR traffic, but it's most definitely not returning any such information and it's posting what looks (in FF's dev tools) like garbage binary data.
The CSS doesn't even load here:
https://trustcompliance.xyz/_next/static/chunks/17psh0.nytnh...: 404 Not Found
hello, this isn’t a competitor. I run a consulting company in the cybersecurity space and saw a chance to make this whole process more transparent.
I agree it came off a bit clickbaity, I'm sorry, Claude probably pushed it too far. but I don’t have an audience anywhere, no following on social, so I needed to ship something fast and make it engaging. the intent wasn’t just this Delve thing, the goal is to move away from it and turn it into a proper hub for compliance transparency over time. But i need a way to marketing this intially.
it’s been less than 24h, I built and pushed everything pretty quickly, so yeah there are rough edges. I’m already working through them and fixing things.
on the account being new, I get how that looks. I mostly use X and reddit, this is actually my first time posting on HN so I had to create an account.
> I agree it came off a bit clickbaity
Not clickbaity, but downright misleading and, for all we know about the XHR traffic, potentially malicious.
> But i need a way to marketing this intially.
Posting a site which blatantly fakes statistics, such as the ones mentioned two posts up from here, is not doing any favors.
Sending XHR requests with opaque binary data every few seconds is not doing you any favors. No information-only site has _any_ business XHR-posting opaque state every several seconds.
"Post it quickly at all costs, accuracy be damned," is not a viable strategy for a legitimate and believable site.
This site also has no identifiable contact information whatsoever, making it likely illegal in many places
" let r = ["Acme Corp", "CloudVault", "DataSync Pro", "NexGen AI", "SecureStack", "TrustLayer", "Vanta", "ComplianceIQ", "InfraSec", "ByteShield", "PipelineOps", "CyberNova", "TokenGuard", "ZeroTrust Labs", "Aether Security", "PrismData", "CloudArmor", "RiskLens", "AuditTrail", "ShieldIO"] , n = ["just checked", "searched for", "ran a scan on", "verified"] , a = ["San Francisco, CA", "New York, NY", "Austin, TX", "London, UK", "Berlin, DE", "Toronto, CA", "Seattle, WA", "Chicago, IL", "Denver, CO", "Boston, MA", "Singapore", "Sydney, AU"]; "
fake popups, xyz domain, recent zeitgeist, 100% straight vibecoded. good hustle I have to say. a domain that'll now get ranked on google for SOC 2 compliance which likely has a high CPC and good DR to piggyback off.
fair call on the popups, they’re not real-time. I added them quickly to make the page feel less empty while testing engagement, probably not the best call in hindsight and I’ll remove or replace them with something real.
on the "vibecoded" part, yeah I moved fast. this was built in under a day to get something out and see if people even care about this angle. that doesn’t mean the underlying data or direction is fake though.
the domain choice is just speed and availability, not some SEO master plan. if this turns into something real I’ll move it to a proper brand/domain.
and yeah I get why it looks like a growth/SEO play, but the actual goal is to push more transparency around these audits. if I just wanted traffic there are easier angles than going after something this niche and messy.
either way, appreciate you calling it out, some of it is fair and already being fixed.
Cybersecurity compliance, but .xyz site built under a day to get something out fast to drum up engagement and "test the vibes on the idea". Makes one wonder what became of this industry.
Just know that alot of startups with all star founders are closer to delve than not.
Its mostly marketing, "look at this MIT genius that noticed something about legacy xyz industry that no one else did"
Truth is venture funds are allocating a limited pie of what is really societies capital to people that dont deserve it
It's almost a cycle.
Something bad happens because of lack of regulation -> People strive for regulation -> Govt's actually regulates and sets some norms/procedures -> system works for a while -> Then someone takes the same idea and molds it into something else to bypass the regulation -> they get promoted because they are "clever" and get rewarded -> Then something bad happens as the tool is used by public.
From Prediction markets to Buy now, pay later to Delve to so many other things.
Is there a name to this particular phenomenon, because this just keeps on repeating in multiple industries.
> Truth is venture funds are allocating a limited pie
This pie does not seem that limited recently.
there are only a certain amount of series A rounds per year
The damage this will do to the reputation of the SOC2 Security Attestation is incalculable.
As someone unfamiliar with the topic, should I read this comment as very dry humour?
Based on my personal experience with security theater and its many talented actors, yes.
Incalculable due to division by zero error
Does SOC2 in general have a particularly high reputation?
The only security compliance frameworks that have any particular reputation with me are the ones associated with the department of defense where the consequences range between a slap on the wrist warning or a small 5 figure fine to execution for espionage (which only ever happened for Julius and Ethel Rosenberg, though one could imagine there may have been more, uh, unofficial consequences that nobody ever heard about). In other words, people actually care about the enforcement of security standards in meaningful ways and there are meaningful consequences.
Everything else... well they're all at least a little better than a participation trophy and the process proving you're trying isn't meaningless. It's just not been my experience with these things that they're particularly good guarantees that the spirit embodying the compliance program is actually being done particularly well.
It's the universal de facto standard at least in North America, and nobody takes it especially seriously. About the best thing you could say for it is that it verifies that you're an actual company and not 3 raccoons in a trench coat. But if you're savvy about how you manage your auditors, you can get an attestation for 3 raccoons as well.
Right, in general the compliance programs mean that you are at minimum put together enough to scam the auditors which is, in itself, not nothing.
i don't think you can really descend below zero?
We analyzed the leaked Delve audit reports and found some wild patterns:
- The same auditor license number (PAC-FIRM-LIC-47383) appears in 487 out of 494 reports
- Every Type II report has identical page numbers: Section 4 at page 30, tests at page 59, Section 5 at page 82
- 220+ "No exceptions noted" per report, across every single client
- The system descriptions were copy-pasted from each company's marketing website
We built tools to check this data:
- Search by company name to see if they're in the leaked database
- Paste any SOC 2 report text to scan for 10 template fingerprints
- A swipe game where you try to tell real audit excerpts from the fakes (harder than you'd think)
455 companies indexed, all free, no signup needed.
I'm also curious what the HN community thinks about the fingerprint detection approach, are there patterns we're missing?
Most of this isn't that damning. SOC IIs are already highly templatized, so pages matching up really isn't meaningful. In fact, an overly detailed or overly verbose template is more likely to have matching pages since you'd never have to add additional content to it.
System descriptions don't necessarily hold much weight. They're often more about giving a general shape of the system to help orient the reader, rather than providing a technically complete picture.
Most of the meat in these is about the controls being tested (which are semi-standardized within an auditor) and the results. Many of these controls are really basic and easy to get "no exceptions noted".
That being said, nearly everyone has at least one exception, even if it's minor. The fact that they didn't find any across all of their clients is a strong indicator they're not diving deeply enough.
Genuinely curious: if you just need an independent audit report to check a box, do you really care how good a job the auditor did?
Probably not, in fact your auditors not being terribly thorough might be a selling point. But your clients, who are the ones asking for the box to be checked, might.
In my experience, clients don't dig deeply into the report or the auditor, they just want to see that you 1) have the report 2) it doesn’t have any egregious exceptions. Perhaps if this makes big enough news, that’ll change.
"You" probably don't, but it's not just "you". There's also the counterparty who's asking to see that report. Maybe they're doing it for paper-pushing purposes of their own, but ultimately, somewhere up the chain, there's someone thinking "I can't personally audit all my suppliers, and I can't be sure they're doing the right thing, so I'm going to ask them to get an independent audit".
Of course, this shows that the entire system is a bit of a charade, but the point is that someone cares and they're gonna be annoyed when they find out that the audit appears to be a sham.
Whether they have a good alternative is a separate question. But here's another way to look at it: if we show blatant disregard for self-regulation, the government is eventually going to show up and come up with more onerous rules.
> but the point is that someone cares
Is it true, though? Or has everyone just been psyched into asking for that certification out of a vague fear of "consequences" or of being left behind?
It's not either-or. Companies care about security because of the consequences. If you're a big company contracting a small one, you don't want to get owned through that vendor because you know you'll be the one holding the bag (data loss, reputational damage, regulatory scrutiny, lawsuits).
Small vendors will tell you what you want to hear because they're desperate for your business. Independent auditing is, in theory, a way to get closer to the ground truth. Well, in theory.
As the company? No. In fact, it's likely better for you if they do a bad job. You potentially get shielded from blame, but don't actually have to put in the work.
As a user/customer/potential victim? Yeah, you do.
The swipe game idea is new to me - you have internal testers or some team use that to go through it?
The no exceptions noted piece is kinda funny. Most SOC2 auditors at least put in the minimal effort of finding one person who didn’t do their cybersecurity training, so the report’s not total boiler plate.
I'd find this more compelling if you looked at a few thousand Vanta or Drata reports grouped by auditor. You're going to find the same commonalities with only trivial language differences.
SOC2 reports are private between you and the auditor (that way if you "fail" you can just find another auditor or have a re-do, and no one is the wiser), and basically always gated behind a sales touchpoint (another hint about what utility they provide). I guess the Delve ones leaked which is why they can all be compared.
220 out of 494 "no exceptions" seems quite high to me. Nobody I've ever dealt with allows an exception to make its way into the report.
what do you expect? if you’re “automating” an audit, it already means you don’t care. the LLM is there to blur the calculus of responsibility, take the blame if someone cares enough to look. happy customers, until someone “delves” a little too deep (like you did) and ruins the slumber party.
This was my general reaction when it was determined that the "log review" portion of the PCI checklist (can't remember what level) could be satisfied by computer "review", and that newer PCI versions were moving towards preferring automated "review"
Thanks for compiling this. Will get used to every sufficiently-interesting data dump being beautifully analyzed shortly after release.
Delve's response blog post from two days ago: https://delve.co/blog/response-to-misleading-claims
I've worked with SOC2-certified companies where employees would email each other plaintext credentials, publish them in Notion pages, etc. You cannot cure stupidity by "complying".
There's no particular reason anyone's SOC2 DRL would cover "make sure people don't email credentials". It's not a technical certification.
Is SOC 2 legit? I have this on my roadmap but now I’m wondering if it’s just security theatre?
That's a difficult question to answer. It shouldn't be, but it is. The reality is, SOC2 is a sales-enablement tool. You should:
* Run a SOC2/compliance program that is entirely disjoint from your security practice.
* Defer SOC2 until the work required to sell into customers demanding it (phone calls, questionnaires) exceeds the cost of obtaining SOC2.
* Prepare for SOC2 by making simple best-practices engineering decisions, in particular single-signon for virtually everything and protected branches for all your repositories.
* Do not allow SOC2 to force any engineering decisions that you would not have intuitively made yourself (this is a big risk with the evidence-gathering platforms like Drata, Delve, and Vanta).
* Assume your SOC2 Type I report will suffice as a first attestation (ie: buy you 1 year of time) with all your customers, and understand that you cannot fail to obtain a Type I; your Type I is guaranteed.
Over 5-6 years of discussing SOC2 with other security practitioners pretty intensively, the overwhelming weight of the evidence is that ~practically nobody actually reads SOC2 reports; they just check the box for each vendor and move on. Plan accordingly.
Since you know a lot about SOC: is SOC2 Type I (point in time) enough to close enterprise sales? Is it worth getting for a new startup (seems super simple)?
Yes, it is, and no, you should not get it, not until you know you need it. If you have to ask, defer.
Thanks!
It's complicated. In theory, SOC2 forces you to do some important stuff, like define your threat model and say "I can mitigate against the threats and prove that my mitigations are in place". The problem is always that the companies that care don't need it but are burdened with it while the companies that don't care will just checkbox their way through it. It sort of enforces a very baseline security posture, in theory, but the major win of "We've thought our security through" is more of a choice - SOC2 can't actually force you to care.
A ton of these SOC2 vendors take all of the potential good parts of SOC2 out of the equation, building the threat models for you and then you just hook up your gsuite/ github and they check boxes for you or tell you to flip a policy here or there. Delve took this to the extreme by not even asking you to flip the checkboxes.
That said, it doesn't matter if it's legit. Everyone is SOC2, and part of being SOC2 is that the vendors whose products you purchase are SOC2, so it's not a choice - you have to be SOC2 if you want to sell (industry/ product specific, but at some point it'll be clear if it applies). If your goal is security, well, SOC2 is irrelevant.
Ultimately, you'll end up having a separate compliance team to manage SOC2 and you'll actively try to keep "real security" from it because real security has to change over time. You'll encode the absolute minimum possible into your compliance for that reason so that you can easily pass every year and then, if you care about security, you'll invest in that separately.
You can get a long, long way without SOC2; virtually every prospective customer you run into that asks for a SOC2 will have an alternate on-ramp for vendors without it, and the ones that don't will sign a contingent PO on your Type I, which (again) you are guaranteed to get.
The idea that SOC2 forces you to do important stuff gets it backwards; SOC2 documents your existing practice, and demands only extremely high-level controls that you can deliver in any number of ways. Your security practice should (minimally) inform your SOC2, not the other way around.
> You can get a long, long way without SOC2;
Yes, that's true. I edited my post to be a bit clearer about this. When you need a SOC2 is going to depend a lot on your business. Lots of companies can make exceptions very easily. Type 1 is easy, I would highly recommend starting there pretty much no matter what since it'll be good practice before your SOC2.
> The idea that SOC2 forces you to do important stuff gets it backwards;
It's the goal behind SOC2. You're assuming a company has a security practice that informs the SOC2 but I think the idea is that companies have no security practice and the SOC2 is what forces them to sit down and build one. What you're describing is more like what happens when a company that actually cares about security goes through SOC2 - you take what you have, put it into a NIST format, and map minimal controls from your practices to the CCs. Most companies have nothing to start with.
It's security theater. Friendly plug for Oneleet, who actually talked us out of getting it.
We were considering getting certified, but it only really makes sense if your customers require you to have it.
Tangential to this but do ISO certifications make sense or are they security theater as well?
And another question but as a consumer, is there any certification which can meaningfully try to show if people/business take their security carefully or are all things security theater in that aspect and at some point, we just have to trust the enterprise and look for other signals of security (like for example blog posts which might show a deep-dive into security for example comes to my mind)
Not really. As long as current system where auditors are also clients of company being audited, the conflict of interest is too high.
Also, not to mention in many countries, the cost of getting breached is nothing so many companies are willing to just hope for the best and payout in case of the worst.
What about enterprise customers / sales?
For enterprise sales you can get a SOC 2 Type I faster than any enterprise sale goes through. Typically, most enterprises are okay if you show them proof that you are "in the process" of getting the certification by showing them that you have signed up with one of those platforms (Delve, Vanta, etc.), so you would be okay to start only when you are about to close one of those enterprise deals.
Yeah, we got a signed letter of engagement from our auditor, which was enough to unlock a customer without having to go through any sidestepping process.
Thanks!
Great info, thanks!
We did SOC 2 a few years ago, I'm glad we did it.
In my mind getting a clean report required three kinds of work:
1. Work that actively improved our security posture. 2. Work that didn't change much, but made our security posture easier to understand. 3. Busy work.
I think for most companies all three kinds of work will be required, but you can also make decisions that will push the percentages around. SOC 2 required us to start doing an annual security table top exercise. You could sit down, run a scenario, run it as fast as you can, and come up with a few pre-determined "improvements" that would help if you actually had that problem in the future. Or you could sit down and really put work into it, and see what works well and what doesn't.
As an example in our last tabletop I "exfiltrated" some data from one of our servers, and challenged the team to figure out what I'd done. The easy way out would have been for someone to say "We'll look at the logs and figure it out", but instead I asked them to actually try and find it. We discovered that the sheer volume of logs for that system made them hard to work with. So we made some changes to make them easier to work with and repeated the exercise later.
It could have been busy work, but instead we got real value from it.
It’s fine for what it is: some light guardrails that attempt to nudge you towards answering “is this all just a house of cards that will obviously collapse under a light breeze”.
Getting a SOC2 doesn’t mean you’re amazing or secure or stable. If a customer says they’ll write you a fat check but they need you to have a SOC2, tell them you’ll get it within a year if they start paying. Otherwise don’t bother.
All of the audit / certifications are theatre. the only question is if your customer is required to participate in the show.
If you really care about security, you need to separate it from this stuff, it can only hurt you.
Do your own, real, security, and treat this compliance stuff as an opaque customer feature request.
It basically shows clients that you are not doing wildly incompetent things with their data, or if you are, they can more easily sue you, since you probably lied to your auditor about it.
But it’s ultimately not up to you if you do it or not. If all of your potential clients demand it, it’s generally easier to get it than it is to get on the phone with all of your potential clients’ IT departments and explain why you don’t have it.
SOC2 has been in trouble for a while now. Completely gamified. I was managing an acquisition of a healthtech company and asked if they did an internal risk assessment as part of their audit. Nope.
SOC2 certified, has never actually put to paper "here's what we know we're doing wrong, here is how we plan to remediate it."
I don’t mind AI. I mind slop. This website is slop. There is so much wrong
What is SOC2 ? I studied hardware electronics engineering
>The Biggest Compliance Fraud in SOC 2 History
How is it bigger than the auditors that Delve was using. Surely Delve wasn't there only client. Delve is just a drop in the bucket.
This has to result in jail time for multiple people… right?
YC company, on forbes, so I guess maybe a bonus, promotions and AI spinoff...