Our account was recently banned from OpenAI for a CSAM violation. We had an autonomous agent using an image classifier switch to OpenAI (instead of a local model it should have been using) which was classifying user-uploaded content. After classifying over 100,000 images, OpenAI banned our account.
We asked for further information so we could report the uploader to the NCMEC and remove the content from our systems. They refused to give us any information about this incident, denied our appeal, and have so far, illegally refused our privacy requests for data about our account.
OpenAI does not care about fighting child sexual abuse, this is completely performative.
EDIT: These image uploads are up to 20+ years old, and are being served through Cloudflare with their CSAM scanning product enabled. We've received no notification of any problems from Cloudflare. Something on OpenAI's system thinks that an image we sent to them for classification is CSAM, and they aren't willing to elaborate or communicate about it at all. For reference, our website is https://ytmnd.com
I've done some work in this space and it's mostly that Big Tech has regulators and parent advocacy groups already very angry with them and breathing down their necks, so they would rather lose customers than take a nuanced approach. The marginal value of having you as a customer is approximately zero, the regulatory risk of letting CSAM through is huge.
Another complicating factor is that it really blows to have humans look at stuff like that, so everyone would rather use automated classifiers; and if you assume good faith and share too much info with the affected parties, this will be exploited by the bad guys to gain insights into how the system works and how to circumvent it.
I don't think there's an elegant solution to this, other than "don't allow tech companies to get so big that they can afford not to care about individual customers".
I'm unsure if there's any particular size requirement for a company to not care about individual customers. Barring an exceedingly small amount of businesses who rely on a handful of customers for their entire bottom line, probably almost all B2C and most B2B companies have a certain point at which any particular customer is simply not worth the overhead to serve. For OAI I would assume anything that even smells like CSAM is probably more than enough for them to justify acting first and (maybe) asking questions later, especially with the regulators and IPO to consider.
> I don't think there's an elegant solution to this, other than "don't allow tech companies to get so big that they can afford not to care about individual customers".
On the contrary, that's an incredibly elegant solution.
I would posit almost no one really cares about fighting child sexual abuse; most of them just use this as pretence for more surveillance and data collection.
I would posit the exact opposite: it's really important to fight child sex abuse, which is why it's so frustrating that people are co-opting that cause to promote surveillance and data collection.
You know, when you stop to think about it, what does forcing the major file/image hosters to overtly delete CSAM (and ban the poster accounts) exactly do? That's right, it forces the people who create/share/exchange CSAM to go for the secured/encrypted messengers where that content is pretty much impossible to intercept and detect — which obviously makes the task of finding and arresting those people much harder. Like, just make those hosters to tacitly contact the relevant authorities and not tip off the criminals that they've been detected. Nah, let's push those guys to use Signal and TOR.
It also pushes big players (like FB or Apple) to add e2e encryption in their services so they can avoid being blamed for "knowing" that someone was planning an act of violence.
AI unfortunately currently can't be e2e encrypted at scale, so the the big players are forced to use surveillance to protect their legal butts from being sued.
Imagine if the government (USPS) decided they need to open and read every single letter or package you send or receive, for /safety/.
> Imagine if the government (USPS) decided they need to open and read every single letter or package you send or receive, for /safety/.
Of interest for those who might not know, the USPS does this for _metadata_ today.
The US Postal Service photographs every single article of mail it processes in the US through a program called Mail Isolation Control and Tracking[0]. It was created post-2001 anthrax scare, and is widely shared across LEOs.
But unlike most LEO intercept systems, you can actually /benefit/ from this one and get your own lens into their data feed.[1]
One key means of preventing the exploitation of children is placing children in situation where they aren't powerless, where their caregivers aren't in such a marginal position that they can't prevent predators from exploiting them.
Basically, keeping children from being miserably. Poverty isn't the only source of the exploitation of children but it's huge one and how little any anti-CSM crusaders care about this reveals a lot imo about their lack of sincerity.
But in our society, even in a much better society than this, bad things are going to happen at scale. At a certain point, you have abandon the "we'll do anything to stop this terrible thing, no matter how much what you do hurts people otherwise, no matter how ineffective what you do is, you must do your thing since it can be done".
It is complex and controversial. What is appropriate to what age? Who should do it? How much should it be driven by children's questions and how much regardless? Should all children by taught the same things at the same age? Should it be purely biological or do you teach them values at the same time? If so whose?
Reminds me of a newsroom (the AFP) who was constantly publishing on their social media account conspiracy theory debunking. I told them it was a problem, because they were actually exposing me to those theories: I would not have heard of them without reading the AFP feed, and now I would remember something about it months or years later, maybe not remembering clearly if it was true or not. I recommended than instead of posting their debunking in their main feed, they should post it in discussions on the debunked subject. They never answered and never stopped. And I understood why: they don't want to stop conspiracy theories. They want to be seen fighting them. It's all posturing as well.
I'd agree but I'd add a lot of things are done to get the child sex crusaders off a given corporation's back. Moreover, the crusaders will use every victory to widen their definition of bad things (which isn't me saying anything "good" is here) and number of areas they can trawl for this bad stuff. Enlarged enough, the agenda essentially would include criminalizing pornography and any "deviant" sexuality.
But this agenda is also ultimately about the power of those pushing it rather than just its goals, of course.
I agree. Because if they do they hold some sort of view on what is or isn’t in an image/video. I think it is simpler and easier to put your hands over your eyes and look the other way.
If OpenAI actually flagged something as CSAM, they were supposed to report it to NCMEC. I wonder whether they will confirm having done so, even if they won't say which image it was.
I've so far managed to resist using OpenAI and similar for anything, except having played around with duck.ai a little bit. I'd only want to use local models, preferably models that I trained myself. "Open weight" seems like perverse terminology, like "open binary" for closed source software.
I run a moodboard website (sort of like pinterest, but no social features) and I have been worrying about this as well. The site is picking up pretty decent traffic and has hit nearly 1TB in uploaded images, but I haven't yet started scanning incoming images for CSAM content.
Are there any resources or tools you'd recommend here for me to setup? It's already going through cloudflare, and I'm actually in the process of shifting my transform service from Supabase to a hosted container there, so I've got my images routing through there anyways (but storage on Supabase still instead of R2 for now).
I’m sorry that happened to you, but I would read this more as they are an organization not prepared for this. Why would they be? One minute you’re trying to make computers pass the turning test, the next minute child porn is getting involved.
The fact that they did not previously have proper procedures in place for something like this is not proof that they don’t care. They are one of the fastest growing companies of all time, it’s hard to get disparate parts of an organization aligned on something. Their organization has more than doubled in size in the last nine months.
Though of course fastest growing is not any excuse to not be prepared. Maybe the growing should happen in other directions too, than only investor money and evaluation. There are other axes of growth than the financial one.
I had the same experience with OAI. We were all-in with OpenAI as our primary provider. We used it exclusively for python and go programming requests. Received a ban for "inciting violence". Appeal denied, no refund, no human response or explanation, 3 years of data lost (privacy requests for data denied). We're on team Anthropic now. Still stings when I think about it.
Perhaps. But we aren't even riding the line of potentially ban-able usage. There is still that option if it happens again, but right now the benefits outweigh the risks. We were left with few options.
But part of that was trying to identify which content was flagged so that the uploader could be identified, but since it sounds like they didn't identify the content now the OP cannot report it to the authorities.
OpenAI was given a chance to help someone fight child sexual abuse. They declined to do so, potentially violating the law in the process. It's a pretty straight line, given the facts of the post.
If they were serious about stopping CSAM they'd spend 10 seconds looking at the prompt we used, the emails we've sent and the fact we're trying to get an NCMEC case number and report the uploader, but we can't because we literally don't know what image they're referencing. If a human was involved in any part of this process or decision, it isn't apparent because all we've received is automated responses.
Frankly I am not interested in retaining an OpenAI account anymore so the only reason we appealed was to try to get more information, not to restore the account.
Because CSAM is illegal to transmit in many jurisdictions, and sending a link to it is considered transmitting, it's quite possible that they don't think they can legally inform you. The presumption being, if NCMEC think it's CSAM, you will find out then, when they ask you to remove it.
They informed us our account was being deactivated for CSAM. We were seeking literally any metadata to help us track it down (a batch number, a content hash, a description, a date and time it happened so we could narrow down the search by looking in our logs, etc), and they have refused to provide any information to us, at all, despite our eagerness to quarantine the asset(s) and report the person who uploaded it to the proper authorities.
At the very least, they could provide us with an NCMEC case number so we can proactively work with the NCMEC to track the content down, but it appears their main motivation is not "strengthening the fight against child sexual abuse" but rather "reducing OpenAI's liability in the fight against child sexual abuse".
His point is that even if they do care, they are being performative in going completely silent, doing "nothing" as opposed to working together with others to remove content and report bad actors.
If you throw huge amount of work loads at an LLM and expect 100% correct work in order not to loose your account, you might want to reconsider your expectation, no?
Imagine you are OpenAI, with hundreds of thousands of business customers and trillions of actions taken through your system every day, in violation of your policies and often local laws.
How much patience would you have for some entity maybe "accidentally" processing something that (apparently) looks like child porn against your policies?
This is understandable. I think the core problem is that most big tech/social media have no working appeals system which would allow a false-positive situation like this to be resolved in a sensible fashion.
it's not performative if it's done to reduce or eliminate liability. Performative is about signally for the purpose of signalling. Businesses dont signal for it's own sake. They do it to either increase business or decrease liability.
So I've been working on having Claude using GPT images to illustrate various fairy tales. At some point there was a Russian one of a girl left in the forest, but in the first cut she looked peaceful and happy. I told Claude I didn't think that was realistic, and it said that it had already asked GPT images to show the girl "scared", and it refused 12/12 times. It also refused to have Frost (who in the fairy tale ends up giving her a coat and a chest of gold) standing over her; he had to be at her level, an arm's length away. (Showing the girl as "brave" and then "nervous" had better results.)
Gave me quite a funny feeling, tripping innocently over the empty shape that they've had to carve out. They wouldn't have spent the effort to do that if it wasn't pretty bad.
It is kind of wild reading these criminal affidavits around these National Center for Missing and Exploited Children integrations. The criminal uploads an image to their email, photo storage service, or gen AI service, it's detected as CSAM, automatically routes an alert to NCMEC, NCMEC generates a tip to the user's local law enforcement jurisdiction, and they're arrested within a short time period, like one or two weeks.
This is the problem. Unless an image is unambiguously a sexual image of a child it needs context, knowledge of intent, and common sense to classify correctly.
There will only be very few generations of people who can see photos of them running around at the beach as small kids. That window closes faster than it opened just a few decades earlier.
I can see how this the average, unthinking member of the general population thinks this level of tech surveillance is a good idea, but those people haven't paused to consider the implications in literally any other context.
Every mainstream vendor with a US presence does this on their ZDR/MAM AI endpoints. This also includes AWS and Azure. It's explained directly in the AWS ZDR docs, and it falls under the "synchronous" scanning pipeline in Azure with full MAM.
At least with AWS they're clear that this is the only type of proactive scanning they do on their ZDR AI endpoints. With Azure it's actually unclear exactly how much more falls into the synchronous scanning pipeline, though I may be missing some docs.
The way this was explained to me is although there is no legal requirement to do this, vendors are concerned that AI processing constitutes "actual knowledge" of the content unlike a lot of traditional cloud-based workloads, which would expose them to legal liability. Seems like a weird argument in a ZDR context, but anyway, there's also a public optics element too I'm sure.
I'm personally hopeful that as an industry we can move towards E2EE and attested secure enclave pipelines to remove the "actual knowledge" argument. NEAR AI and Tinfoil are the only publicly available providers in this area but I would be excited to hear about more advancements in this area because I personally think that these NCMEC integrated "ZDR" systems are major privacy violations and deceptive marketing.
Azure does offer nVidia's secure enclave architecture via "Azure Confidential GPU", but it's limited to a single GPU and a little less than 90GB after overhead, so basically maxes out at Mistral-Large-2 with 4 bit quantization.
This whole thing kind of puts the industry in a bind. On one hand, the big labs do have an incentive to get this kind of thing rolled out so as to avoid open weight on-prem AI gaining traction, and to remove remaining roadblocks to corporate adoption in regulated industries. But rolling it out risks attracting even stronger regulatory responses for them as well (that's even setting aside likely national security/law enforcement pushback), as well as likely being strongly opposed by the AI safety people. There's no one way to square the circle that will please everyone. My guess is the labs will take a third route: continue to lobby against open weight models, thus killing the possibility of on-prem, and then drag their heels on E2EE/secure enclave models except perhaps for military customers.
I'm hopeful that Tinfoil's enclave router, which does decrypt the payload but does so in a verifiably secure manner, can be used to scale to multi-gpu models (for example, Tinfoil does offer Kimi K3.1 Pro).
Because of economies of scale with batch inference as seen with DS4.1 Flash, this is the only avenue for true privacy from frontier models at scale/cost.
In terms of policy, I agree it's looking dire. There's unfortunately just more money/lobbying/mindshare on the 'safety at all costs' camp but I'm still advocating and educating where I can.
E2EE will never ever happen at this scale. If it does, it will necessarily have monitoring before the encryption, because CEOs don't like jail [1], with the alternative being terminating the service [2], which won't happen when billions are being made.
How are these companies able to legally train these models? Are the employees citizens of countries that this material is illegal to download or possess?
But they still have to obtain the images, look through the images and do tests with the images? Sounds like a totally fucked up thing to do. Reminds me of a fellow student that now works for a IT forensic company that does forensics for the police. He said that 98% of police requests involve checking storage of suspects for CP. He also has to look at the images/videos to write a report. He changed, no joke.
This is a positive development. People working for the NCMEC have had to deal with an insane increase in CSAM content (like upwards of 1,000% increase in 2025 alone) due to diffusion models dramatically multiplying the supply of CSAM.
When there's an active and urgent case to try and locate and rescue children in these situations, the volume of AI-generated content makes it that much harder to perform forensic investigations on background materials (often how they locate children being abused is by prosaic items like electrical outlets, household goods, etc. in the background of abuse photos and videos) because now they have to rule out whether an image is even real -- a task that often requires close attention and scrutiny to details of traumatizing and disturbing content (sometimes all the more traumatizing due to the 'creative freedom' of AI-generated material)
A model capable of shielding investigators from additional trauma is an excellent thing.
CSAM stands for Child Sexual Abuse Material. Using diffusion models to create naked pics is by any reasonable definition not CSAM since no child is being abused. It's baffling to me that I have to state this, since it's tautological. The reason why child pornography is treated like a horrific crime by all of society, is because it harms actual children - this used to be the actual legal criteria for it.. Extending the reach of the crime to something that has none of that impact seems like a transparent power grab.
Just to counter any obvious moralizing, I am not saying using models like that is okay, but it's an entirely different kind of crime.
Well, the argument has been that it's a kind of vaguely cosmological or conceptual abuse of children, through the normalisation of sexualised depictions of children and so forth. In the production of such material, children suffer as a class.
The problem is, this viewpoint is infinitely flexible, infinitely elastic and can do a lot of gymnastics. I'm not sure it's compatible with the very concrete and procedural orientation of Anglo-American jurisprudence, particularly.
Depends on the definition. It's become more frequent for predators to download images of real children from their parents' social media, then edit them with diffusion models to be nudes. This should still be illegal, and it is in the US.
Yup. This is such hysteria. Personally I’d argue that the existence of diffusion models “multiplying the supply of this material” means that the value of it goes way down, meaning it’s harder for the truly monstrous parties who have for decades harmed actual human beings to create that “content” now face collapsing value of what they make, which makes it far less lucrative to be in that business and should make it less attractive to get into it, and remove some incentive to make more and more of it.
This is no different than how something like basic poster design is not a real service you can sell anymore since ChatGPT can make a passable one for free.
If that kind of pervert can’t be simply fixed (or even exterminated) with a magic wand, it would be far better for our society if they’re looking at fake images than tempted to make their own by harming kids. But we as a society can’t accept that so we police this as aggressively as we did the kind that had an actual victim.
Not sure that multiplying the supply of content that pedophiles jerk it to is a great idea. Not only do you risk exposing people who (hopefully) never would've been exposed to it before, but research shows that increased access to CSAM can actually help normalize the abuse and support deviant behavior in online communities dedicated to it.
Some have argued the opposite, that "multiplying the supply of this material" further normalises sexual abuse of children, whereas before, it lurked in the shadows of the dark web and was treated like serious contraband, which imbued it with a certain gravity.
Not saying I agree with them, just that this is one of the main points trotted out in support of intrusively policing generation and editing models.
The reason why it's a specific problem is in my original comment: Signal to noise ratio.
The huge increase in volume of imagery is impeding operations to rescue children whom are actively being abused or held captive because every image, real or generated, takes up time and resources from the people trying to rescue the _actively abused children_
So your middlebrow refutal ("um ackshually no children are harmed when a diffusion model makes an image" ) of (one of) the many problems with additional CSAM volume is unconvincing and makes me think you've not actually considered the situation in any real detail.
> Isn't CSAM only material based on abuse of real humans?
If only.
The same "hard on crime" ratchet that happens to most every thing that's popular to punish and extremely unpopular to speak out against the tightening of said ratchet has happened here.
I don't think OP thinks that. I hear him as pointing out that this question can't be considered in acontextual, ahistorical vacuum. It's a set of simultaneous equations, where the other equations and terms are of a more general form:
"So you don't think it's morally wrong to generate _______ using AI?"
"So you don't think it's morally wrong to _______ using AI?"
The matter can't be considered in isolation, and although anyone voicing a healthy fear of where this is going can be portrayed, through demagoguery, as "pro-CSAM" or "anti-children" at this juncture, the logical endpoint of this kind of surveillance is in far more unpalatable places.
As the good folks fighting--a losing battle, at present--for digital freedom and privacy know, guarding dearly held liberties means securing those liberties for criminals and antisocial pursuits, too. That's just the cost of freedom.
To elaborate, I was reading it to mean that they hoped the generation would be allowed because it doesn't actually involve a human.
What do you think? Is it morally wrong to generate child porn using AI? My reading is that you think it should be allowed as a necessary cost for freedom. Is that accurate?
I guess calling it a cost recognizes there is something wrong with it at least.
> What do you think? Is it morally wrong to generate child porn using AI? My reading is that you think it should be allowed as a necessary cost for freedom. Is that accurate?
I don't have a great answer on whether it should be "allowed", but I see much wrong--as a matter of grave implications for freedom--with all the presently available and deployed techniques for "disallowing" it.
Offhand, I'm not sure what's special about it There are all kinds of horrors one can generate with AI: beheadings, murder fantasies, extreme sadism, material promotional of darkly extremist ideology, text and literature intended to mislead/manipulate/deceive/brainwash, depict or glorify self-harm and suicide, etc, for example, and one could argue that all of these further actual socially harmful impulses and behaviours, feed and nurture social clubs built around them, etc.
A number of European jurisdictions--and famously, the UK--do just that, and they aren't necessarily wrong. It is a tangible social good to keep the unthinkable cruelty of Nazi atrocities and the Holocaust alive in our collective and institutional memory. God forbid we ever forget these... actually, we seem to be well on our way, in that doom loop of history.
We haven't really had to confront, as a society, the ethics of photorealistic but utterly synthetic content, which can be generated quickly and at a very large scale, that contravenes our legal and social norms. There are clearly important nuances: purely synthetic content is not the same as deepfakes, which the courts have called "morphed" images.
I hear people who say it normalises sexual exploitation of children and makes us callous and indifferent to it. I hear people saying it's better than actual exploitation of real children, and that--reasoning from long-established free speech and privacy principles present elsewhere in our jurisprudence--the government shouldn't be telling people what they can enjoy in the privacy of their own home, provided that thing doesn't have a human victim, nor appears in any legally cognisable way to harm a class of victims (this is where much "hate speech" and "antisocial ideation" type laws dwell in the UK and EU).
I'm not entirely sure what to think. My first instincts are quite libertarian on this matter, and in a world that is substantially electronic, digital freedom may matter more than any other kind, and we clearly took it for granted in the free and open Internet of decades past. I can appreciate the arguments from the other side, but they all strike me as special cases of much larger principles at stake--it's just that nobody wants to impale themselves on the child porn question due to its incendiary character.
I'd prefer it be illegal to generate porn that looks like children. I also understand that threshold can be difficult to define. What I'd heard most recently was courts said CSAM must involve real people or it's just free speech.
> The most important point is also the easiest to lose in the headlines: the AI-CSAM ruling in the Anderegg case did not hold that AI-generated CSAM is generally protected by the First Amendment.
Are you not confusing your preferences with moral here? It is morally wrong because they depict children in sexual context. I mean these sickos come up with all kinds of things.
With your argument, why would it any child pornography be immoral? It’s just 0s and 1s in the computer, innit? They never took the photo. They just downloaded it from the dark web.
But wait, why is the bar different for this subject matter than for other depictions of immoral acts?
Creating a real video of a real person getting their head chopped off: someone got killed, so that's clearly a problem. Creating a fake video depicting the same decapitation in graphic realism: no one bats an eye - go eat popcorn while it happens on a big screen and laugh about it.
So, why is one violent, illegal depiction given a green light, but another is reasoned about completely differently? I think there may be an argument, there, which is why I'm asking, but it doesn't spring to mind. How would you explain the contradiction to a peaceful alien species who uses asexual reproduction (knows nothing about sex) and is shocked and horrified by the violence that passes as entertainment to us?
It is indeed a very dangerous ratchet. Who in their right mind would take an extremely unpopular position that is affront to the SafeTy of children?
And yet, we have no means of disputing the accuracy of these inscrutable classifiers, nor any reason to think that similar approaches won't be applied to other "objectionable" content in other contexts, as is currently the fashion in Europe in the UK with regard to anything that could be conceivably deemed to be inciting hate, animus or disorder.
Automated technological surveillance and censorship is really, really having a moment right now. I'm not holding my breath that the pendulum swings the other way, but I sure hope it does.
This is a real moral dilemma for people who think consent and harm are all that matters. Vitriolic disgust at the idea of generating child porn using AI, but no good way to explain why its wrong.
The motivation of people who make and consume this content is rarely monetary and almost always gratification. Do you really believe that a proliferation of child pornography will reduce the demand for child pornography, and reduce the abuse of children?
They must have amassed a large data set of things that are not pornography. Then trained the model on that. Anything that doesn’t match the model is pornography.
>A Bloom filter is a space-efficient, probabilistic data structure used to test whether an element is in a set. It can be “definitely not in the set” or “possibly in the set,” meaning false positives are possible but false negatives are not.
reminds me of the case where a dude made pics of his kid to show the doctor and his google account got permanently banned for CSAM (google photos involved)
Curious what people here think as digital technology makes it easier to spot indicators that someone may commit a crime.
I’m thinking about how it used to be extremely impractical to track most of the population, the the enforcement comes after the crime, and there is no pressure to have detected it before it happened and stop it.
How should society/law change in response? Is it purely a give and take between surveillance/right-to-not-be-watch and trying to prevent all bad things from happened before they happen?
>I’m thinking about how it used to be extremely impractical to track most of the population, the the enforcement comes after the crime, and there is no pressure to have detected it before it happened and stop it.
Privacy, unfortunately, is largely the result of it being impractical to violate it.
I see zero reason to believe that states wont read your thoughts if they can easily manage to.
In Minority Report (spoilers), the precrime department is shut down because of a single error proving the system is fallible, thus making the plot preposterous in the age of AI.
This kind of thing is always an excuse for creating surveillance tools that scan your files and report you to the government. It starts with CSAM because they know you don't want to argue against CSAM detection, but you absolutely should.
I deal with a lot of user-generated content, and I have been using OpenAI as a tool to help find offending content. Now I see that if CSAM ever makes it into one of my systems, OpenAI is going to send the authorities after me if their model sees it too, so that's a dealbreaker for that use case, and it was a big one!
It's also wild that they are trying to detect and report AI-generated CSAM. This does absolutely nothing to help protect actual children.
If you run a website with user-generated content, you absolutely will encounter CSAM. It may not be "normal" in that it shocks the conscience, but it is absolutely normal in that every single website that accepts user-submitted content will, at some point, encounter CSAM.
It's been like that for the entire 20 years I have run these kinds of websites. It sucks, but it's just a sad reality of the world we live in.
Frankly, and I don't mean this in a bad way about the problem, I do not think OpenAI actually gives a shit about the underlying problem. Especially as an OpenAI-hosted-service-only, this seems more like a data grabbing play. "Scan all your images through our CSAM detector" means "send all images to us, including the 99.99% that are not CSAM!"
Don't get me wrong: these are very necessary tools for the modern internet to function, and I'm sure _drastically_ reduce the review burden on humans, but the skeptic in me does not believe that OpenAI is being entirely altruistic here.
Remember when they wanted Apple to scan everybody’s devices locally?
I’m not sure what the solution is to deal with pedophiles but I’m sure it starts with parents monitoring their kids better and keeping young children from being isolated from real world social groups. I’m also sure it doesn’t involve using classifiers on a person’s own device to detect ‘possible’ child porn, which is where this is headed. Who can forget the father who had photos of his sick baby on Google Drive for his paediatrician to see and got reported to the police.
If it really is all about protecting kids then maybe the NCMEC should be lobbying to pass laws banning kids from the internet. Think about how many harms you could reduce instantly (bye bye Instagram, Snapchat, etc).
This is just public relations and no reasonable person would believe OpenAI will do anything to fight against online child sexual abuse. In fact, it's much more reasonable to assume they will continue to make the problem much worse.
Similar to Facebook and other platforms feeding self-harm content to vulnerable children and teenagers, and even connecting them to dealers. These companies know their platforms are not safe for kids. https://www.bloomberg.com/features/cant-look-away/
Finally Sam Altman found something good to do in terms of my security, I'm relieved. It fits perfectly well in a world where Saudi Arabia lead the UN Commission on the Status of Women (CSW).
If only there was a huge data mine of child abusers sitting on some sort of email service they could go after. Maybe some sort of jeevacation inbox.
Could be a great start for "rogue AI"
Jokes aside, I'm glad they are giving tools to help fight this as currently your options are limited because every provider will simply return a content warning if you try to use an LLM as a moderator.
Our account was recently banned from OpenAI for a CSAM violation. We had an autonomous agent using an image classifier switch to OpenAI (instead of a local model it should have been using) which was classifying user-uploaded content. After classifying over 100,000 images, OpenAI banned our account.
We asked for further information so we could report the uploader to the NCMEC and remove the content from our systems. They refused to give us any information about this incident, denied our appeal, and have so far, illegally refused our privacy requests for data about our account.
OpenAI does not care about fighting child sexual abuse, this is completely performative.
EDIT: These image uploads are up to 20+ years old, and are being served through Cloudflare with their CSAM scanning product enabled. We've received no notification of any problems from Cloudflare. Something on OpenAI's system thinks that an image we sent to them for classification is CSAM, and they aren't willing to elaborate or communicate about it at all. For reference, our website is https://ytmnd.com
I've done some work in this space and it's mostly that Big Tech has regulators and parent advocacy groups already very angry with them and breathing down their necks, so they would rather lose customers than take a nuanced approach. The marginal value of having you as a customer is approximately zero, the regulatory risk of letting CSAM through is huge.
Another complicating factor is that it really blows to have humans look at stuff like that, so everyone would rather use automated classifiers; and if you assume good faith and share too much info with the affected parties, this will be exploited by the bad guys to gain insights into how the system works and how to circumvent it.
I don't think there's an elegant solution to this, other than "don't allow tech companies to get so big that they can afford not to care about individual customers".
I'm unsure if there's any particular size requirement for a company to not care about individual customers. Barring an exceedingly small amount of businesses who rely on a handful of customers for their entire bottom line, probably almost all B2C and most B2B companies have a certain point at which any particular customer is simply not worth the overhead to serve. For OAI I would assume anything that even smells like CSAM is probably more than enough for them to justify acting first and (maybe) asking questions later, especially with the regulators and IPO to consider.
> I don't think there's an elegant solution to this, other than "don't allow tech companies to get so big that they can afford not to care about individual customers".
On the contrary, that's an incredibly elegant solution.
I would posit almost no one really cares about fighting child sexual abuse; most of them just use this as pretence for more surveillance and data collection.
I would posit the exact opposite: it's really important to fight child sex abuse, which is why it's so frustrating that people are co-opting that cause to promote surveillance and data collection.
Oh you misunderstood me; it’s important for sure.
I don’t think this is something that can be solved by more surveillance though.
Most abuse (i think something like 80%? Not sure but it was a big number) happens within the family and between minors (minors abusing minors)
Changing how easy or hard it is to distribute or generate CSAM will help exactly nothing here even if that’s the goal.
You know, when you stop to think about it, what does forcing the major file/image hosters to overtly delete CSAM (and ban the poster accounts) exactly do? That's right, it forces the people who create/share/exchange CSAM to go for the secured/encrypted messengers where that content is pretty much impossible to intercept and detect — which obviously makes the task of finding and arresting those people much harder. Like, just make those hosters to tacitly contact the relevant authorities and not tip off the criminals that they've been detected. Nah, let's push those guys to use Signal and TOR.
Obviously there's all flavors of coordination with law enforcement to do exactly what you're talking about when it's appropriate.
It also pushes big players (like FB or Apple) to add e2e encryption in their services so they can avoid being blamed for "knowing" that someone was planning an act of violence.
AI unfortunately currently can't be e2e encrypted at scale, so the the big players are forced to use surveillance to protect their legal butts from being sued.
Imagine if the government (USPS) decided they need to open and read every single letter or package you send or receive, for /safety/.
> Imagine if the government (USPS) decided they need to open and read every single letter or package you send or receive, for /safety/.
Of interest for those who might not know, the USPS does this for _metadata_ today.
The US Postal Service photographs every single article of mail it processes in the US through a program called Mail Isolation Control and Tracking[0]. It was created post-2001 anthrax scare, and is widely shared across LEOs.
But unlike most LEO intercept systems, you can actually /benefit/ from this one and get your own lens into their data feed.[1]
Modern problems, modern solutions, etc.
[0]: https://en.wikipedia.org/wiki/Mail_Isolation_Control_and_Tra...
[1]: https://www.usps.com/manage/informed-delivery.htm
Saying it's important is not the opposite of saying that it's rare to actually care.
How else do you prevent it tho?
One key means of preventing the exploitation of children is placing children in situation where they aren't powerless, where their caregivers aren't in such a marginal position that they can't prevent predators from exploiting them.
Basically, keeping children from being miserably. Poverty isn't the only source of the exploitation of children but it's huge one and how little any anti-CSM crusaders care about this reveals a lot imo about their lack of sincerity.
But in our society, even in a much better society than this, bad things are going to happen at scale. At a certain point, you have abandon the "we'll do anything to stop this terrible thing, no matter how much what you do hurts people otherwise, no matter how ineffective what you do is, you must do your thing since it can be done".
Another great way to prevent child exploitation is age-appropriate sex education for the children. But we don't see politicians promoting that.
It is complex and controversial. What is appropriate to what age? Who should do it? How much should it be driven by children's questions and how much regardless? Should all children by taught the same things at the same age? Should it be purely biological or do you teach them values at the same time? If so whose?
It may be important, but remains that nobody really cares.
I'm not sure your definitions of "posit" and "exact opposite" align with common usage.
Reminds me of a newsroom (the AFP) who was constantly publishing on their social media account conspiracy theory debunking. I told them it was a problem, because they were actually exposing me to those theories: I would not have heard of them without reading the AFP feed, and now I would remember something about it months or years later, maybe not remembering clearly if it was true or not. I recommended than instead of posting their debunking in their main feed, they should post it in discussions on the debunked subject. They never answered and never stopped. And I understood why: they don't want to stop conspiracy theories. They want to be seen fighting them. It's all posturing as well.
I'd agree but I'd add a lot of things are done to get the child sex crusaders off a given corporation's back. Moreover, the crusaders will use every victory to widen their definition of bad things (which isn't me saying anything "good" is here) and number of areas they can trawl for this bad stuff. Enlarged enough, the agenda essentially would include criminalizing pornography and any "deviant" sexuality.
But this agenda is also ultimately about the power of those pushing it rather than just its goals, of course.
I agree. Because if they do they hold some sort of view on what is or isn’t in an image/video. I think it is simpler and easier to put your hands over your eyes and look the other way.
To be fair, they're also trying to use it as PR to polish their rapidly tarnishing corporate image.
If OpenAI actually flagged something as CSAM, they were supposed to report it to NCMEC. I wonder whether they will confirm having done so, even if they won't say which image it was.
I've so far managed to resist using OpenAI and similar for anything, except having played around with duck.ai a little bit. I'd only want to use local models, preferably models that I trained myself. "Open weight" seems like perverse terminology, like "open binary" for closed source software.
I run a moodboard website (sort of like pinterest, but no social features) and I have been worrying about this as well. The site is picking up pretty decent traffic and has hit nearly 1TB in uploaded images, but I haven't yet started scanning incoming images for CSAM content.
Are there any resources or tools you'd recommend here for me to setup? It's already going through cloudflare, and I'm actually in the process of shifting my transform service from Supabase to a hosted container there, so I've got my images routing through there anyways (but storage on Supabase still instead of R2 for now).
I’m sorry that happened to you, but I would read this more as they are an organization not prepared for this. Why would they be? One minute you’re trying to make computers pass the turning test, the next minute child porn is getting involved.
The fact that they did not previously have proper procedures in place for something like this is not proof that they don’t care. They are one of the fastest growing companies of all time, it’s hard to get disparate parts of an organization aligned on something. Their organization has more than doubled in size in the last nine months.
Though of course fastest growing is not any excuse to not be prepared. Maybe the growing should happen in other directions too, than only investor money and evaluation. There are other axes of growth than the financial one.
uh actually it's good that they're bad and they're only bad because they're so good, cut them some slack will you it's not easy being rich
I had the same experience with OAI. We were all-in with OpenAI as our primary provider. We used it exclusively for python and go programming requests. Received a ban for "inciting violence". Appeal denied, no refund, no human response or explanation, 3 years of data lost (privacy requests for data denied). We're on team Anthropic now. Still stings when I think about it.
Anthropic acts the same way (from experience), so take the learning of backing up your data from this, if nothing else.
So just go from bad to worse? I thought the lesson should be trust neither and pick an open weight provider from the dozens?
Perhaps. But we aren't even riding the line of potentially ban-able usage. There is still that option if it happens again, but right now the benefits outweigh the risks. We were left with few options.
> OpenAI does not care about fighting child sexual abuse, this is completely performative.
That doesn’t really follow from your experience.
OpenAI could care about child sexual abuse and not respond to your appeal, even wrongfully.
But part of that was trying to identify which content was flagged so that the uploader could be identified, but since it sounds like they didn't identify the content now the OP cannot report it to the authorities.
OpenAI was given a chance to help someone fight child sexual abuse. They declined to do so, potentially violating the law in the process. It's a pretty straight line, given the facts of the post.
If they were serious about stopping CSAM they'd spend 10 seconds looking at the prompt we used, the emails we've sent and the fact we're trying to get an NCMEC case number and report the uploader, but we can't because we literally don't know what image they're referencing. If a human was involved in any part of this process or decision, it isn't apparent because all we've received is automated responses.
Frankly I am not interested in retaining an OpenAI account anymore so the only reason we appealed was to try to get more information, not to restore the account.
Because CSAM is illegal to transmit in many jurisdictions, and sending a link to it is considered transmitting, it's quite possible that they don't think they can legally inform you. The presumption being, if NCMEC think it's CSAM, you will find out then, when they ask you to remove it.
They informed us our account was being deactivated for CSAM. We were seeking literally any metadata to help us track it down (a batch number, a content hash, a description, a date and time it happened so we could narrow down the search by looking in our logs, etc), and they have refused to provide any information to us, at all, despite our eagerness to quarantine the asset(s) and report the person who uploaded it to the proper authorities.
At the very least, they could provide us with an NCMEC case number so we can proactively work with the NCMEC to track the content down, but it appears their main motivation is not "strengthening the fight against child sexual abuse" but rather "reducing OpenAI's liability in the fight against child sexual abuse".
His point is that even if they do care, they are being performative in going completely silent, doing "nothing" as opposed to working together with others to remove content and report bad actors.
Classifying unreviewed user-uploaded content with a service that does not accept CSAM is a footgun, even if they misclassified something this time.
If you throw huge amount of work loads at an LLM and expect 100% correct work in order not to loose your account, you might want to reconsider your expectation, no?
Imagine you are OpenAI, with hundreds of thousands of business customers and trillions of actions taken through your system every day, in violation of your policies and often local laws.
How much patience would you have for some entity maybe "accidentally" processing something that (apparently) looks like child porn against your policies?
I don't need to imagine. The postal service, telephone company, and internet service providers have all existed, and contnue to exist.
This is understandable. I think the core problem is that most big tech/social media have no working appeals system which would allow a false-positive situation like this to be resolved in a sensible fashion.
Performative and just try to shed liability.
it's not performative if it's done to reduce or eliminate liability. Performative is about signally for the purpose of signalling. Businesses dont signal for it's own sake. They do it to either increase business or decrease liability.
I mean, you're holding it wrong. You uploaded CSAM to OpenAI which is presumably against their TOS.
Allegedly. Or the discrepancy in Cloudflare's and OpenAI's detection is caused by a false positive on OpenAI's side. Who knows?
Oh my, ytmnd!
I used to view shit there a lot sometime in the mid-2000s.
Seems like they want to play in the big boys club (Google, Facebook, ... ) where this kind of treatment is standard.
So I've been working on having Claude using GPT images to illustrate various fairy tales. At some point there was a Russian one of a girl left in the forest, but in the first cut she looked peaceful and happy. I told Claude I didn't think that was realistic, and it said that it had already asked GPT images to show the girl "scared", and it refused 12/12 times. It also refused to have Frost (who in the fairy tale ends up giving her a coat and a chest of gold) standing over her; he had to be at her level, an arm's length away. (Showing the girl as "brave" and then "nervous" had better results.)
Gave me quite a funny feeling, tripping innocently over the empty shape that they've had to carve out. They wouldn't have spent the effort to do that if it wasn't pretty bad.
It is kind of wild reading these criminal affidavits around these National Center for Missing and Exploited Children integrations. The criminal uploads an image to their email, photo storage service, or gen AI service, it's detected as CSAM, automatically routes an alert to NCMEC, NCMEC generates a tip to the user's local law enforcement jurisdiction, and they're arrested within a short time period, like one or two weeks.
> A Dad Took Photos of His Naked Toddler for the Doctor. Google Flagged Him as a Criminal.
https://www.nytimes.com/2022/08/21/technology/google-surveil...
This is the problem. Unless an image is unambiguously a sexual image of a child it needs context, knowledge of intent, and common sense to classify correctly.
Companies declare that human moderators are augmented by AI, but humans make mistakes and even a photo lacks the context.
This is addressed in the model card: https://deploymentsafety.openai.com/csam-image-classifier/de...
> Family imagery: photos that are taken in private family settings where there may be instances of child nudity, such as a baby’s first bath.
I often take photos of my kids in the bath but they are neither babies nor is it their first so I guess that's CSAM then?!
There will only be very few generations of people who can see photos of them running around at the beach as small kids. That window closes faster than it opened just a few decades earlier.
I can see how this the average, unthinking member of the general population thinks this level of tech surveillance is a good idea, but those people haven't paused to consider the implications in literally any other context.
> reviewers confirmed 72% of the classifier’s high-confidence flags
They imply a 28% false-positive rate on their "high-confidence" flags. And that's an improvement? Yikes.
Relatedly, Openrouter now does CSAM scanning and retention on so called "Zero Data Retention" endpoints as well.
https://openrouter.ai/blog/announcements/csam-screening/
Every mainstream vendor with a US presence does this on their ZDR/MAM AI endpoints. This also includes AWS and Azure. It's explained directly in the AWS ZDR docs, and it falls under the "synchronous" scanning pipeline in Azure with full MAM.
At least with AWS they're clear that this is the only type of proactive scanning they do on their ZDR AI endpoints. With Azure it's actually unclear exactly how much more falls into the synchronous scanning pipeline, though I may be missing some docs.
The way this was explained to me is although there is no legal requirement to do this, vendors are concerned that AI processing constitutes "actual knowledge" of the content unlike a lot of traditional cloud-based workloads, which would expose them to legal liability. Seems like a weird argument in a ZDR context, but anyway, there's also a public optics element too I'm sure.
I'm personally hopeful that as an industry we can move towards E2EE and attested secure enclave pipelines to remove the "actual knowledge" argument. NEAR AI and Tinfoil are the only publicly available providers in this area but I would be excited to hear about more advancements in this area because I personally think that these NCMEC integrated "ZDR" systems are major privacy violations and deceptive marketing.
Azure does offer nVidia's secure enclave architecture via "Azure Confidential GPU", but it's limited to a single GPU and a little less than 90GB after overhead, so basically maxes out at Mistral-Large-2 with 4 bit quantization.
This whole thing kind of puts the industry in a bind. On one hand, the big labs do have an incentive to get this kind of thing rolled out so as to avoid open weight on-prem AI gaining traction, and to remove remaining roadblocks to corporate adoption in regulated industries. But rolling it out risks attracting even stronger regulatory responses for them as well (that's even setting aside likely national security/law enforcement pushback), as well as likely being strongly opposed by the AI safety people. There's no one way to square the circle that will please everyone. My guess is the labs will take a third route: continue to lobby against open weight models, thus killing the possibility of on-prem, and then drag their heels on E2EE/secure enclave models except perhaps for military customers.
I'm hopeful that Tinfoil's enclave router, which does decrypt the payload but does so in a verifiably secure manner, can be used to scale to multi-gpu models (for example, Tinfoil does offer Kimi K3.1 Pro).
Because of economies of scale with batch inference as seen with DS4.1 Flash, this is the only avenue for true privacy from frontier models at scale/cost.
In terms of policy, I agree it's looking dire. There's unfortunately just more money/lobbying/mindshare on the 'safety at all costs' camp but I'm still advocating and educating where I can.
E2EE will never ever happen at this scale. If it does, it will necessarily have monitoring before the encryption, because CEOs don't like jail [1], with the alternative being terminating the service [2], which won't happen when billions are being made.
[1] https://www.theguardian.com/technology/2016/oct/04/yahoo-sec...
[2] https://www.theguardian.com/technology/2013/aug/09/lavabit-e...
How are these companies able to legally train these models? Are the employees citizens of countries that this material is illegal to download or possess?
Most CSAM statutes have a carve-out exemption for things like this.
But they still have to obtain the images, look through the images and do tests with the images? Sounds like a totally fucked up thing to do. Reminds me of a fellow student that now works for a IT forensic company that does forensics for the police. He said that 98% of police requests involve checking storage of suspects for CP. He also has to look at the images/videos to write a report. He changed, no joke.
i'm pretty sure certain organizations are given authorization
This is a positive development. People working for the NCMEC have had to deal with an insane increase in CSAM content (like upwards of 1,000% increase in 2025 alone) due to diffusion models dramatically multiplying the supply of CSAM.
When there's an active and urgent case to try and locate and rescue children in these situations, the volume of AI-generated content makes it that much harder to perform forensic investigations on background materials (often how they locate children being abused is by prosaic items like electrical outlets, household goods, etc. in the background of abuse photos and videos) because now they have to rule out whether an image is even real -- a task that often requires close attention and scrutiny to details of traumatizing and disturbing content (sometimes all the more traumatizing due to the 'creative freedom' of AI-generated material)
A model capable of shielding investigators from additional trauma is an excellent thing.
CSAM stands for Child Sexual Abuse Material. Using diffusion models to create naked pics is by any reasonable definition not CSAM since no child is being abused. It's baffling to me that I have to state this, since it's tautological. The reason why child pornography is treated like a horrific crime by all of society, is because it harms actual children - this used to be the actual legal criteria for it.. Extending the reach of the crime to something that has none of that impact seems like a transparent power grab.
Just to counter any obvious moralizing, I am not saying using models like that is okay, but it's an entirely different kind of crime.
Well, the argument has been that it's a kind of vaguely cosmological or conceptual abuse of children, through the normalisation of sexualised depictions of children and so forth. In the production of such material, children suffer as a class.
The problem is, this viewpoint is infinitely flexible, infinitely elastic and can do a lot of gymnastics. I'm not sure it's compatible with the very concrete and procedural orientation of Anglo-American jurisprudence, particularly.
They probably use "uncensored" local models, though. I doubt commercial models could ever be used for that.
> diffusion models dramatically multiplying the supply of CSAM.
Isn't CSAM only material based on abuse of real humans? How could computer models multiply CSAM by that definition?
Some news links:
https://www.bloomberg.com/features/2026-grok-sexualized-chil...
https://www.ibtimes.co.uk/xai-lawsuit-grok-ai-generated-chil...
https://globalnews.ca/news/12077556/sherwood-park-artificial...
Depends on the definition. It's become more frequent for predators to download images of real children from their parents' social media, then edit them with diffusion models to be nudes. This should still be illegal, and it is in the US.
So deep fakes? That does sound horrific and challenging to reliably detect and stop.
Legally, if the computer generated content looks too real, its still criminal.
Yup. This is such hysteria. Personally I’d argue that the existence of diffusion models “multiplying the supply of this material” means that the value of it goes way down, meaning it’s harder for the truly monstrous parties who have for decades harmed actual human beings to create that “content” now face collapsing value of what they make, which makes it far less lucrative to be in that business and should make it less attractive to get into it, and remove some incentive to make more and more of it.
This is no different than how something like basic poster design is not a real service you can sell anymore since ChatGPT can make a passable one for free.
If that kind of pervert can’t be simply fixed (or even exterminated) with a magic wand, it would be far better for our society if they’re looking at fake images than tempted to make their own by harming kids. But we as a society can’t accept that so we police this as aggressively as we did the kind that had an actual victim.
Not sure that multiplying the supply of content that pedophiles jerk it to is a great idea. Not only do you risk exposing people who (hopefully) never would've been exposed to it before, but research shows that increased access to CSAM can actually help normalize the abuse and support deviant behavior in online communities dedicated to it.
https://link.springer.com/article/10.1007/s11896-023-09611-4
Some have argued the opposite, that "multiplying the supply of this material" further normalises sexual abuse of children, whereas before, it lurked in the shadows of the dark web and was treated like serious contraband, which imbued it with a certain gravity.
Not saying I agree with them, just that this is one of the main points trotted out in support of intrusively policing generation and editing models.
The reason why it's a specific problem is in my original comment: Signal to noise ratio.
The huge increase in volume of imagery is impeding operations to rescue children whom are actively being abused or held captive because every image, real or generated, takes up time and resources from the people trying to rescue the _actively abused children_
So your middlebrow refutal ("um ackshually no children are harmed when a diffusion model makes an image" ) of (one of) the many problems with additional CSAM volume is unconvincing and makes me think you've not actually considered the situation in any real detail.
> Isn't CSAM only material based on abuse of real humans?
If only.
The same "hard on crime" ratchet that happens to most every thing that's popular to punish and extremely unpopular to speak out against the tightening of said ratchet has happened here.
I cannot imagine being pro-child pornography.
Huh. So you don’t think it’s morally wrong to generate child porn using AI?
I don't think OP thinks that. I hear him as pointing out that this question can't be considered in acontextual, ahistorical vacuum. It's a set of simultaneous equations, where the other equations and terms are of a more general form:
"So you don't think it's morally wrong to generate _______ using AI?"
"So you don't think it's morally wrong to _______ using AI?"
The matter can't be considered in isolation, and although anyone voicing a healthy fear of where this is going can be portrayed, through demagoguery, as "pro-CSAM" or "anti-children" at this juncture, the logical endpoint of this kind of surveillance is in far more unpalatable places.
As the good folks fighting--a losing battle, at present--for digital freedom and privacy know, guarding dearly held liberties means securing those liberties for criminals and antisocial pursuits, too. That's just the cost of freedom.
I don't really understand what you are saying.
To elaborate, I was reading it to mean that they hoped the generation would be allowed because it doesn't actually involve a human.
What do you think? Is it morally wrong to generate child porn using AI? My reading is that you think it should be allowed as a necessary cost for freedom. Is that accurate?
I guess calling it a cost recognizes there is something wrong with it at least.
> What do you think? Is it morally wrong to generate child porn using AI? My reading is that you think it should be allowed as a necessary cost for freedom. Is that accurate?
I don't have a great answer on whether it should be "allowed", but I see much wrong--as a matter of grave implications for freedom--with all the presently available and deployed techniques for "disallowing" it.
>I don't have a great answer on whether it should be "allowed"
Im not asking about if it should be allowed, I'm asking if you find anything morally wrong about making child porn via AI, or consuming it.
Why can't you just say "yeah it's morally wrong?" Why is that not obvious?
Offhand, I'm not sure what's special about it There are all kinds of horrors one can generate with AI: beheadings, murder fantasies, extreme sadism, material promotional of darkly extremist ideology, text and literature intended to mislead/manipulate/deceive/brainwash, depict or glorify self-harm and suicide, etc, for example, and one could argue that all of these further actual socially harmful impulses and behaviours, feed and nurture social clubs built around them, etc.
A number of European jurisdictions--and famously, the UK--do just that, and they aren't necessarily wrong. It is a tangible social good to keep the unthinkable cruelty of Nazi atrocities and the Holocaust alive in our collective and institutional memory. God forbid we ever forget these... actually, we seem to be well on our way, in that doom loop of history.
We haven't really had to confront, as a society, the ethics of photorealistic but utterly synthetic content, which can be generated quickly and at a very large scale, that contravenes our legal and social norms. There are clearly important nuances: purely synthetic content is not the same as deepfakes, which the courts have called "morphed" images.
I hear people who say it normalises sexual exploitation of children and makes us callous and indifferent to it. I hear people saying it's better than actual exploitation of real children, and that--reasoning from long-established free speech and privacy principles present elsewhere in our jurisprudence--the government shouldn't be telling people what they can enjoy in the privacy of their own home, provided that thing doesn't have a human victim, nor appears in any legally cognisable way to harm a class of victims (this is where much "hate speech" and "antisocial ideation" type laws dwell in the UK and EU).
I'm not entirely sure what to think. My first instincts are quite libertarian on this matter, and in a world that is substantially electronic, digital freedom may matter more than any other kind, and we clearly took it for granted in the free and open Internet of decades past. I can appreciate the arguments from the other side, but they all strike me as special cases of much larger principles at stake--it's just that nobody wants to impale themselves on the child porn question due to its incendiary character.
I'd prefer it be illegal to generate porn that looks like children. I also understand that threshold can be difficult to define. What I'd heard most recently was courts said CSAM must involve real people or it's just free speech.
It's been reported and discussed that way, but the ruling was actually quite a bit more narrow and contingent.
This is a tendentious framing, but legally detailed:
https://enoughabuse.org/ai-csam-ruling-what-the-seventh-circ...
Specifically:
> The most important point is also the easiest to lose in the headlines: the AI-CSAM ruling in the Anderegg case did not hold that AI-generated CSAM is generally protected by the First Amendment.
I do not. I would much rather that those interested do exactly that rather than trade in the real thing.
Just don't put it in MY face.
Are you not confusing your preferences with moral here? It is morally wrong because they depict children in sexual context. I mean these sickos come up with all kinds of things.
With your argument, why would it any child pornography be immoral? It’s just 0s and 1s in the computer, innit? They never took the photo. They just downloaded it from the dark web.
But wait, why is the bar different for this subject matter than for other depictions of immoral acts?
Creating a real video of a real person getting their head chopped off: someone got killed, so that's clearly a problem. Creating a fake video depicting the same decapitation in graphic realism: no one bats an eye - go eat popcorn while it happens on a big screen and laugh about it.
So, why is one violent, illegal depiction given a green light, but another is reasoned about completely differently? I think there may be an argument, there, which is why I'm asking, but it doesn't spring to mind. How would you explain the contradiction to a peaceful alien species who uses asexual reproduction (knows nothing about sex) and is shocked and horrified by the violence that passes as entertainment to us?
What a bizarre strawman
It is indeed a very dangerous ratchet. Who in their right mind would take an extremely unpopular position that is affront to the SafeTy of children?
And yet, we have no means of disputing the accuracy of these inscrutable classifiers, nor any reason to think that similar approaches won't be applied to other "objectionable" content in other contexts, as is currently the fashion in Europe in the UK with regard to anything that could be conceivably deemed to be inciting hate, animus or disorder.
Automated technological surveillance and censorship is really, really having a moment right now. I'm not holding my breath that the pendulum swings the other way, but I sure hope it does.
Yes, who will cry for the people making virtualized child pornography which is, of course, sourced from actual child pornography? WHO???
This is a real moral dilemma for people who think consent and harm are all that matters. Vitriolic disgust at the idea of generating child porn using AI, but no good way to explain why its wrong.
If victimless AI child porn is flooding the market and making it harder/impossible to monetise actual CP then isn't that a massive positive?
The motivation of people who make and consume this content is rarely monetary and almost always gratification. Do you really believe that a proliferation of child pornography will reduce the demand for child pornography, and reduce the abuse of children?
My colleagues in Singapore trained pornography detection model.
Pornography is illegal to download or save in Singapore… I wonder how they were able to train a model without the data.
Maybe they trained with live camera feeds. Data doesn’t need to be saved or downloaded in order to exist. :)
Streaming is also banned.
Camera feeds aren’t automatically streaming.
Maybe they got it sent from outside the country, on fast disks, so that they neither need to download it, nor save it (again).
As with CSAM, most laws care about possession AND transmission.
They must have amassed a large data set of things that are not pornography. Then trained the model on that. Anything that doesn’t match the model is pornography.
/s
A reverse bloom filter
>A Bloom filter is a space-efficient, probabilistic data structure used to test whether an element is in a set. It can be “definitely not in the set” or “possibly in the set,” meaning false positives are possible but false negatives are not.
Funniest thing I read all week lol
Not hotdog
Interestingly enough, the "not hot dog" app could do this. At least for about 50.26% of the world population.
reminds me of the case where a dude made pics of his kid to show the doctor and his google account got permanently banned for CSAM (google photos involved)
Curious what people here think as digital technology makes it easier to spot indicators that someone may commit a crime.
I’m thinking about how it used to be extremely impractical to track most of the population, the the enforcement comes after the crime, and there is no pressure to have detected it before it happened and stop it.
How should society/law change in response? Is it purely a give and take between surveillance/right-to-not-be-watch and trying to prevent all bad things from happened before they happen?
>I’m thinking about how it used to be extremely impractical to track most of the population, the the enforcement comes after the crime, and there is no pressure to have detected it before it happened and stop it.
Privacy, unfortunately, is largely the result of it being impractical to violate it.
I see zero reason to believe that states wont read your thoughts if they can easily manage to.
There is a popular movie starring Tom Cruise that explores the consequences of this question.
In Minority Report (spoilers), the precrime department is shut down because of a single error proving the system is fallible, thus making the plot preposterous in the age of AI.
Psycho Pass is perhaps a better depiction of what could happen to a society where the system is broadly accepted.
If we're wading into this medium, then Shimoneta ought be mentioned too, albeit in an absurdist manner filtered through a comedic lens.
Haha. In the real world, it's closer to "one success justifies the program's existence" than "one failure closes it down."
Philip K. Dick: In my book I invented the Precrime Division as a cautionary tale.
Tech Company: At long last, we have created the Precrime Division from classic sci-fi novelette The Minority Report.
How is this different from omni-moderation, which OpenAI already has?
https://developers.openai.com/api/docs/models/omni-moderatio...
This kind of thing is always an excuse for creating surveillance tools that scan your files and report you to the government. It starts with CSAM because they know you don't want to argue against CSAM detection, but you absolutely should.
I deal with a lot of user-generated content, and I have been using OpenAI as a tool to help find offending content. Now I see that if CSAM ever makes it into one of my systems, OpenAI is going to send the authorities after me if their model sees it too, so that's a dealbreaker for that use case, and it was a big one!
It's also wild that they are trying to detect and report AI-generated CSAM. This does absolutely nothing to help protect actual children.
> if CSAN ever makes it into one of my systems
If that is a worry you carry around, I really hope you work harder to make it never be the case lol. FYI it’s not a normal thing to worry about.
If you run a website with user-generated content, you absolutely will encounter CSAM. It may not be "normal" in that it shocks the conscience, but it is absolutely normal in that every single website that accepts user-submitted content will, at some point, encounter CSAM.
It's been like that for the entire 20 years I have run these kinds of websites. It sucks, but it's just a sad reality of the world we live in.
How else do you check user generated content other than checking user generated content?
>It is well established that combatting online child predators requires coordination across many organizations.
It requires all devices integrated into one single consciousness.
Frankly, and I don't mean this in a bad way about the problem, I do not think OpenAI actually gives a shit about the underlying problem. Especially as an OpenAI-hosted-service-only, this seems more like a data grabbing play. "Scan all your images through our CSAM detector" means "send all images to us, including the 99.99% that are not CSAM!"
Don't get me wrong: these are very necessary tools for the modern internet to function, and I'm sure _drastically_ reduce the review burden on humans, but the skeptic in me does not believe that OpenAI is being entirely altruistic here.
NCMEC is cancer.
Remember when they wanted Apple to scan everybody’s devices locally?
I’m not sure what the solution is to deal with pedophiles but I’m sure it starts with parents monitoring their kids better and keeping young children from being isolated from real world social groups. I’m also sure it doesn’t involve using classifiers on a person’s own device to detect ‘possible’ child porn, which is where this is headed. Who can forget the father who had photos of his sick baby on Google Drive for his paediatrician to see and got reported to the police.
If it really is all about protecting kids then maybe the NCMEC should be lobbying to pass laws banning kids from the internet. Think about how many harms you could reduce instantly (bye bye Instagram, Snapchat, etc).
This is just public relations and no reasonable person would believe OpenAI will do anything to fight against online child sexual abuse. In fact, it's much more reasonable to assume they will continue to make the problem much worse.
I agree with you. To me, it’s all lip service and nothing gets done until people get hurt.
Similar to Roblox ignoring child grooming in their platform because they make money off of it. https://www.bloomberg.com/features/2024-roblox-pedophile-pro...
Similar to Facebook and other platforms feeding self-harm content to vulnerable children and teenagers, and even connecting them to dealers. These companies know their platforms are not safe for kids. https://www.bloomberg.com/features/cant-look-away/
Finally Sam Altman found something good to do in terms of my security, I'm relieved. It fits perfectly well in a world where Saudi Arabia lead the UN Commission on the Status of Women (CSW).
If only there was a huge data mine of child abusers sitting on some sort of email service they could go after. Maybe some sort of jeevacation inbox.
Could be a great start for "rogue AI"
Jokes aside, I'm glad they are giving tools to help fight this as currently your options are limited because every provider will simply return a content warning if you try to use an LLM as a moderator.