Yeah but does it actually work, though? There have been a lot of online tools claiming to be "AI detectors" and they all seem pretty unreliable. Can you talk us through what you look for, the most common failure modes and (at suitably high level) how you dealt with those?
We've actually deployed to several Tier 1 banks and large enterprises already for various use-cases (verification, fraud detection, threat intelligence, etc.). The feedback that we've gotten so far is that our technology is high accuracy and a useful signal.
In terms of how our technology works, our research team has trained multiple detection models to look for specific visual and audio artifacts that the major generative models leave behind. These artifacts aren't perceptible to the human eye / ear, but they are actually very detectable to computer vision and audio models.
Each of these expert models gets combined into an ensemble system that weighs all the individual model outputs to reach a final conclusion.
We've got a rigorous process of collecting data from new generators, benchmarking them, and retraining our models when necessary. Often retrains aren't needed though, since our accuracy seems to transfer well across a given deepfake technique. So even if new diffusion or autoregressive models come out, for example, the artifacts tend to be similar and are still caught by our models.
I will say that our models are most heavily benchmarked on convincing audio/video/image impersonations of humans. While we can return results for items outside that scope, we've tended to focus training and benchmarking on human impersonations since that's typically the most dangerous risk for businesses.
So that's a caveat to keep in mind if you decide to try out our Developer Free Plan.
What's the lead time between new generators and a new detection model? What about novel generators that are never made public?
I think the most likely outcome of a criminal organization doing this is that they train a public architecture model from scratch on the material that they want to reproduce, and then use without telling anyone. Would your detector prevent this attack?
There are three observations that are helpful to know about here:
A: High quality, battle tested architectures are sold via an API and samples are therefore easy to retrieve at scale.
B: lower quality, novel architectures are often published on GitHub and can be scaled on budget compute resources.
C: Often these models perform well at classifying content generated by architectures similar to those they were trained on, even if that architecture is not identical.
As for actual lead time associated with our actual strategy, that’s probably not something I can talk about publicly. I can say I’m working on making it happen faster.
I don't want to be rude is this not a question you get asked by potential customers? Is that your answer for them? It sounds a lot like 'I guess we will find out.'
We have been working on this problem since 2020 and have created an trained an ensemble of AI detection models working together to tell you what is real and what is fake!
First want to say that I sincerely appreciate you working on this problem. The proliferation of deepfakes is something that virtually every technology industry is dealing with right now.
Suppose that deepfake technology progressed to the point where it is still detectable by your technology, but is impossible for the naked eye. In that scenario (which many would call an eventuality), wouldn't you also be compelled to serve as an authoritative entity on the detection of deepfakes?
Imagine a future politician who is caught on video doing something scandalous, or a court case where someone is questioning the veracity of some video evidence. Are the creators of deepfake detection algorithms going to testify as expert witnesses, and how could they convince a human judge/jury that the output of their black box algorithm isn't a false positive?
Thank you. As an inference-based detection platform, our models go into every scan with the assumption that all files are both not the original/ground truth AND the files have been likely transcoded. We never say something is 0% or 100% fake because we don’t have that ground truth. That said, our award-winning models are able to say, with a confidence score of 1-99% — the higher being likely manipulated — which, in turn, is sent to the team using said detection to action as they will. Some use it as one of many signals to make an informed decision manually. Others have chosen to moderate or label accordingly. There are experts who’ve been called to testify on matters like this one, and some of them work on these very models.
As for synthetic content that is undetectable to the naked eye or ear, we are already there.
And we’ve published peer-reviewed research at top AI conferences E.g. CVPR, NeurIPS, ECCV, AAAI, Interspeech which are available at https://www.realitydefender.com/research
We see who signs up for Reality Defender and instantly notice traffic patterns and other abnormalities that allow us to see if an account is in violation of terms of service. Also, our free tier is capped at 50 free scans a month which will not allow for said attackers to discern any tangible learnings or tactics they can use to bypass our detection models.
How would you detect someone who tests a single image using a new free tier, and then (if successful) uses that image against a targeted customer account?
Working in a similar area (bot detection) I think it's very difficult to proactively stop such targeted attacks, but maybe in this space you can do something interesting like duplicate detection across a consortium.
We'd rather not tip our hand on any/all techniques used to discern actual users from bad actors and those seeking to reverse engineer, but suffice to say we do have are methods (and plenty of them).
Just based on the first post, where they talk about their API a bit, it sounds like a system hosted on their machines(?). So, I assume AI trainers won’t be able to run it locally, to train off it.
Although, I always get a bad smell from that sort of logic, because it feels vaguely similar to security through obscurity in the sense that it relies on the opposition now knowing what you are doing.
How easy is it to fool Reality Defender into making false positives?
Whenever I'm openly performing nefarious illegal acts in public, I always wear my Sixfinger, so if anyone takes a photo of me, I can plausibly deny it by pointing out (while not wearing it) that the photo shows six fingers, and obviously must have been AI generated.
In support of said nefarious illegal acts, the Sixfinger includes a cap-loaded grenade launcher, gun, fragmentation bomb, ballpoint pen, code signaler, and message missile launcher. It's like a Swiss Army Finger! You can 3d print a cool roach clip attachment too.
I understand this is in jest, but unfortunately AI generation tools more or less stopped the six-finger issue a couple of years ago. We are decidedly not a model used for the express detection of finger abnormalities, but a multi-model and multimodal detection platform — driven by our Public API (which you can try for free right now, btw) — which uses many different techniques to differentiate between content that is likely manipulated and likely not manipulated.
I feel like a much easier solution is enforcing data provinence. Ssl for media hash, attach to metadata. The problem with AI isnt the fact its ai, its that people can invest little effort to sway things with undue leverage. A single person can look like 100's with signficantly less effort than previously. The problem with ai content is it makes abuse of public spaces much easier. Forcing people to take credit for work produced makes things easier (not solved) kind of like email. Being able to block media by domain would be a dream, but spam remains an issue.
so, tie content to domains. A domain vouches for content works like that content having been a webpage or email from said domain. Signed hash in metadata is backwards compatible and its easy to make browsers etc display warnings on unsigned content, content from new domains, blacklisted domains, etc.
benefit here is while we'll have more false negatives, unlike something like this tool, it does not cause real harm on false positives, which will be numerous if it wants to be better tham simply making someome accountable for media.
AI detection cannot work, will not work, and will cause more harm than it prevents. stuff like this is irresponsible and dangerous.
I understand the appeal of hashing-based provenance techniques, though they’ve faced some significant challenges in practice that render them ineffective at best. While many model developers have explored these approaches with good intentions, we’ve seen that they can be easily circumvented or manipulated, particularly by sophisticated bad actors who may not follow voluntary standards.
We recognize that no detection solution is 100% accurate. There will be occasional false positives and negatives. That said, our independently verified an internal testing shows we’ve achieved the lowest error rates currently available for addressing deepfake detection.
I’d respectfully suggest that dismissing AI detection entirely might be premature, especially without hands-on evaluation. If you’re interested, I’d be happy to arrange a test environment where you could evaluate our solution’s performance firsthand and see how it might fit your specific use case.
> so, tie content to domains. A domain vouches for content works like that content having been a webpage or email from said domain. Signed hash in metadata is backwards compatible and its easy to make browsers etc display warnings on unsigned content, content from new domains, blacklisted domains, etc.
Okay, so I generate an image, open Instagram, take a picture of the generated image on a hi-res screen, and hit upload. Instagram dutifully signs it and shows it to the public with that signature. What does this buy us?
What problem are you pointing out? The only thing you’ve done is severed the audit trail, which removes any trust in the image that was imbued in it by the original poster. Now when people wonder if the image is authentic, they can only rely on how trustworthy you are, not how trustworthy the original source was. This is working as the GP intended as far as I can see. You can’t add unearned authenticity this way, only remove it.
I pointed out a way to silently sever the audit trail, at which point it sure seems like we've done a lot of work to roll out a whole new system that has such a gaping hole in it that there's no actual benefit.
Data provinence would be neat and a big benefit. But any solution that requires virtually all content publishers to change approach (here: add signing steps to their publishing workflow) is doomed to fail. There is no alternative way to do this than what OP is doing, which is to try to filter the fire hose of content into real vs not.
I feel like this will be the next big cat and mouse sega after ad-blockers;
1) Produce AI tool
2) Tool gets used for bad
3) Use anti-AI/AI detection to avoid/check for AI tool
4) AI tool introduces anti-anti-AI/detection tools
5) Repeat
This is definitely a concern, but this is more or less how the cybersecurity space already works. Having dedicated researchers and a good business model helps a lot for keeping detectors like RD on the forefront of capabilities.
Are you sure you guys want to tackle this problem?
This isn’t the type of thing where you build a product doing a lot of work up front, and then spend the rest of its life offering support and new features and relaxing a bit.
You are committing to a cat and mouse game. You will constantly have to stay on top of ever improving tech that gets harder to beat, you will never know peace. You will have to exert more and more effort each year.
This is something we're constantly updating, upgrading, iterating, and improving on. Every. Single. Day.
Whether it's introducing new models, deprecating old ones, or improving existing ones, there is an element of both staying current but also looking ahead at research. Many of the new models generating hyperreal content we catch on day one because they're based on existing technology and/or research.
It's sadly not often enough I see a young company doing work that I feel only benefits society, but this is one of those times, so thank you and congratulations.
Thank you! We’ve been working on this since 2021 (and some of us a bit before that), and we’re reminded every day that we are ultimately working something that helps people on the macro and micro level. We want a world free of the malevolent uses of deepfakes for ourselves, our loved ones, and everyone beyond, and feel all should be privy to such protection.
Nice. They’ve got a very good reason to keep the model as closed as possible. The second you make it open, it just becomes the fitness signal for the next batch of deepfakes
Make sure it works (most of the time) lock it down behind an well-guarded API and charge a lot of money :)
As noted elsewhere, we give confidence scores between 1-99%. We also use many different models for each modality for a more robust and complete answer with each scan, and each model has its own confidence score.
I'am also a bit shocked by this SDK approach, why not a simple API
where you upload a file, get an ID and wait till it's done?
Beside that, sometimes it works, sometimes not:
I find that most companies find that to be a feature, not a bug. People are more likely to hit accept if they can only see a small chunk of the content.
Yeah but does it actually work, though? There have been a lot of online tools claiming to be "AI detectors" and they all seem pretty unreliable. Can you talk us through what you look for, the most common failure modes and (at suitably high level) how you dealt with those?
We've actually deployed to several Tier 1 banks and large enterprises already for various use-cases (verification, fraud detection, threat intelligence, etc.). The feedback that we've gotten so far is that our technology is high accuracy and a useful signal.
In terms of how our technology works, our research team has trained multiple detection models to look for specific visual and audio artifacts that the major generative models leave behind. These artifacts aren't perceptible to the human eye / ear, but they are actually very detectable to computer vision and audio models.
Each of these expert models gets combined into an ensemble system that weighs all the individual model outputs to reach a final conclusion.
We've got a rigorous process of collecting data from new generators, benchmarking them, and retraining our models when necessary. Often retrains aren't needed though, since our accuracy seems to transfer well across a given deepfake technique. So even if new diffusion or autoregressive models come out, for example, the artifacts tend to be similar and are still caught by our models.
I will say that our models are most heavily benchmarked on convincing audio/video/image impersonations of humans. While we can return results for items outside that scope, we've tended to focus training and benchmarking on human impersonations since that's typically the most dangerous risk for businesses.
So that's a caveat to keep in mind if you decide to try out our Developer Free Plan.
What's the lead time between new generators and a new detection model? What about novel generators that are never made public?
I think the most likely outcome of a criminal organization doing this is that they train a public architecture model from scratch on the material that they want to reproduce, and then use without telling anyone. Would your detector prevent this attack?
There are three observations that are helpful to know about here: A: High quality, battle tested architectures are sold via an API and samples are therefore easy to retrieve at scale. B: lower quality, novel architectures are often published on GitHub and can be scaled on budget compute resources. C: Often these models perform well at classifying content generated by architectures similar to those they were trained on, even if that architecture is not identical.
As for actual lead time associated with our actual strategy, that’s probably not something I can talk about publicly. I can say I’m working on making it happen faster.
I don't want to be rude is this not a question you get asked by potential customers? Is that your answer for them? It sounds a lot like 'I guess we will find out.'
I think you misunderstood my answer. There is established research showing that deepfake detection models transfer learning within architectures.
Give it a try for yourself. It's free!
We have been working on this problem since 2020 and have created an trained an ensemble of AI detection models working together to tell you what is real and what is fake!
I tried, It required making an account to use.
In this day and age, everybody realises that forcing people to make an account does not count as free. It is paying with personal information.
I completely agree! Please see the bottom of our post where we offer free access in two ways:
1) Email up to 50 files to yc@realitydefender.com, we’ll scan them for you, no setup required
2) 1-click add to Zoom/Teams (via Appstore) to try detection live in your own calls immediately
https://marketplace.zoom.us/apps/OYu4CZuRSwy_ieJ-6xKcrA
First want to say that I sincerely appreciate you working on this problem. The proliferation of deepfakes is something that virtually every technology industry is dealing with right now.
Suppose that deepfake technology progressed to the point where it is still detectable by your technology, but is impossible for the naked eye. In that scenario (which many would call an eventuality), wouldn't you also be compelled to serve as an authoritative entity on the detection of deepfakes?
Imagine a future politician who is caught on video doing something scandalous, or a court case where someone is questioning the veracity of some video evidence. Are the creators of deepfake detection algorithms going to testify as expert witnesses, and how could they convince a human judge/jury that the output of their black box algorithm isn't a false positive?
Thank you. As an inference-based detection platform, our models go into every scan with the assumption that all files are both not the original/ground truth AND the files have been likely transcoded. We never say something is 0% or 100% fake because we don’t have that ground truth. That said, our award-winning models are able to say, with a confidence score of 1-99% — the higher being likely manipulated — which, in turn, is sent to the team using said detection to action as they will. Some use it as one of many signals to make an informed decision manually. Others have chosen to moderate or label accordingly. There are experts who’ve been called to testify on matters like this one, and some of them work on these very models.
As for synthetic content that is undetectable to the naked eye or ear, we are already there.
Have you checked the calibration of that confidence value? When it reports 99% confidence, are 99/100 of those manipulated?
I’m curious what awards the models have won?
We won multiple awards including RSA: https://www.rsaconference.com/library/press-release/reality-...
And we’ve published peer-reviewed research at top AI conferences E.g. CVPR, NeurIPS, ECCV, AAAI, Interspeech which are available at https://www.realitydefender.com/research
How do you prevent bad actors from using your tools as a feedback loop to tune models that can evade detection?
You would need thousands to tens of thousands of images, not just 50 to produce an adversarial network that could use the API as a check.
If someone wanted to buy it, I'm sure reality defender has protection especially because you can predict adversarial guesses.
It would be trivial for them to build "this user is sending progressively more realistic, rapid responses" if they haven't built that already.
We see who signs up for Reality Defender and instantly notice traffic patterns and other abnormalities that allow us to see if an account is in violation of terms of service. Also, our free tier is capped at 50 free scans a month which will not allow for said attackers to discern any tangible learnings or tactics they can use to bypass our detection models.
Yes, of course new accounts should be scrutinized.
But what I find more interesting is how you prevent someone from training a model adversarially via one of your legitimate customers.
Wouldn't any of your customers that use your service to make a decision about something uploaded by a user be an attack vector?
How would you detect someone who tests a single image using a new free tier, and then (if successful) uses that image against a targeted customer account?
Working in a similar area (bot detection) I think it's very difficult to proactively stop such targeted attacks, but maybe in this space you can do something interesting like duplicate detection across a consortium.
We'd rather not tip our hand on any/all techniques used to discern actual users from bad actors and those seeking to reverse engineer, but suffice to say we do have are methods (and plenty of them).
Won't this just become the fitness function for training future models?
Just based on the first post, where they talk about their API a bit, it sounds like a system hosted on their machines(?). So, I assume AI trainers won’t be able to run it locally, to train off it.
Although, I always get a bad smell from that sort of logic, because it feels vaguely similar to security through obscurity in the sense that it relies on the opposition now knowing what you are doing.
now -> not
True, haha. Although “the opposition now knowing what you are doing” is the big danger for this sort of scheme!
How easy is it to fool Reality Defender into making false positives?
Whenever I'm openly performing nefarious illegal acts in public, I always wear my Sixfinger, so if anyone takes a photo of me, I can plausibly deny it by pointing out (while not wearing it) that the photo shows six fingers, and obviously must have been AI generated.
In support of said nefarious illegal acts, the Sixfinger includes a cap-loaded grenade launcher, gun, fragmentation bomb, ballpoint pen, code signaler, and message missile launcher. It's like a Swiss Army Finger! You can 3d print a cool roach clip attachment too.
"How did I ever get along with five???"
https://www.youtube.com/watch?v=ElVzs0lEULs
https://www.museumofplay.org/blog/sixfinger-sixfinger-man-al...
https://www.museumofplay.org/app/uploads/2010/11/Sixfinger-p...
I understand this is in jest, but unfortunately AI generation tools more or less stopped the six-finger issue a couple of years ago. We are decidedly not a model used for the express detection of finger abnormalities, but a multi-model and multimodal detection platform — driven by our Public API (which you can try for free right now, btw) — which uses many different techniques to differentiate between content that is likely manipulated and likely not manipulated.
That said, neat gag.
I feel like a much easier solution is enforcing data provinence. Ssl for media hash, attach to metadata. The problem with AI isnt the fact its ai, its that people can invest little effort to sway things with undue leverage. A single person can look like 100's with signficantly less effort than previously. The problem with ai content is it makes abuse of public spaces much easier. Forcing people to take credit for work produced makes things easier (not solved) kind of like email. Being able to block media by domain would be a dream, but spam remains an issue.
so, tie content to domains. A domain vouches for content works like that content having been a webpage or email from said domain. Signed hash in metadata is backwards compatible and its easy to make browsers etc display warnings on unsigned content, content from new domains, blacklisted domains, etc.
benefit here is while we'll have more false negatives, unlike something like this tool, it does not cause real harm on false positives, which will be numerous if it wants to be better tham simply making someome accountable for media.
AI detection cannot work, will not work, and will cause more harm than it prevents. stuff like this is irresponsible and dangerous.
I understand the appeal of hashing-based provenance techniques, though they’ve faced some significant challenges in practice that render them ineffective at best. While many model developers have explored these approaches with good intentions, we’ve seen that they can be easily circumvented or manipulated, particularly by sophisticated bad actors who may not follow voluntary standards.
We recognize that no detection solution is 100% accurate. There will be occasional false positives and negatives. That said, our independently verified an internal testing shows we’ve achieved the lowest error rates currently available for addressing deepfake detection.
I’d respectfully suggest that dismissing AI detection entirely might be premature, especially without hands-on evaluation. If you’re interested, I’d be happy to arrange a test environment where you could evaluate our solution’s performance firsthand and see how it might fit your specific use case.
> so, tie content to domains. A domain vouches for content works like that content having been a webpage or email from said domain. Signed hash in metadata is backwards compatible and its easy to make browsers etc display warnings on unsigned content, content from new domains, blacklisted domains, etc.
Okay, so I generate an image, open Instagram, take a picture of the generated image on a hi-res screen, and hit upload. Instagram dutifully signs it and shows it to the public with that signature. What does this buy us?
What problem are you pointing out? The only thing you’ve done is severed the audit trail, which removes any trust in the image that was imbued in it by the original poster. Now when people wonder if the image is authentic, they can only rely on how trustworthy you are, not how trustworthy the original source was. This is working as the GP intended as far as I can see. You can’t add unearned authenticity this way, only remove it.
I pointed out a way to silently sever the audit trail, at which point it sure seems like we've done a lot of work to roll out a whole new system that has such a gaping hole in it that there's no actual benefit.
It’s not a hole though. By severing the audit trail, all you have done is remove trust from your copy.
Data provinence would be neat and a big benefit. But any solution that requires virtually all content publishers to change approach (here: add signing steps to their publishing workflow) is doomed to fail. There is no alternative way to do this than what OP is doing, which is to try to filter the fire hose of content into real vs not.
I worked in the fraud space and could see this being a useful tool for identifying AI generated IDs + liveness checks. Will give it a try.
I feel like this will be the next big cat and mouse sega after ad-blockers;
1) Produce AI tool 2) Tool gets used for bad 3) Use anti-AI/AI detection to avoid/check for AI tool 4) AI tool introduces anti-anti-AI/detection tools 5) Repeat
This is definitely a concern, but this is more or less how the cybersecurity space already works. Having dedicated researchers and a good business model helps a lot for keeping detectors like RD on the forefront of capabilities.
[flagged]
Are you sure you guys want to tackle this problem?
This isn’t the type of thing where you build a product doing a lot of work up front, and then spend the rest of its life offering support and new features and relaxing a bit.
You are committing to a cat and mouse game. You will constantly have to stay on top of ever improving tech that gets harder to beat, you will never know peace. You will have to exert more and more effort each year.
This is something we're constantly updating, upgrading, iterating, and improving on. Every. Single. Day.
Whether it's introducing new models, deprecating old ones, or improving existing ones, there is an element of both staying current but also looking ahead at research. Many of the new models generating hyperreal content we catch on day one because they're based on existing technology and/or research.
This sounds exhausting, and I don’t know what you will do when the day comes that you can’t keep up. Any day could be the end.
It's sadly not often enough I see a young company doing work that I feel only benefits society, but this is one of those times, so thank you and congratulations.
Thank you! We’ve been working on this since 2021 (and some of us a bit before that), and we’re reminded every day that we are ultimately working something that helps people on the macro and micro level. We want a world free of the malevolent uses of deepfakes for ourselves, our loved ones, and everyone beyond, and feel all should be privy to such protection.
Nice. They’ve got a very good reason to keep the model as closed as possible. The second you make it open, it just becomes the fitness signal for the next batch of deepfakes
Make sure it works (most of the time) lock it down behind an well-guarded API and charge a lot of money :)
This is the way. Don't become VirusTotal for deepfakers.
Is this gonna be Turnitin[1] all over again?
[1]: https://www.nytimes.com/2025/05/17/style/ai-chatgpt-turnitin...
As noted elsewhere, we give confidence scores between 1-99%. We also use many different models for each modality for a more robust and complete answer with each scan, and each model has its own confidence score.
That doesn't fix the fundamental potential for abuse, moral hazard, and accountability sink[1].
[1]: https://en.wikipedia.org/wiki/The_Unaccountability_Machine
Congrats on launching. Would be good to have a quick trial on the website for a sample image, rather than going through the SDK route.
I'am also a bit shocked by this SDK approach, why not a simple API where you upload a file, get an ID and wait till it's done? Beside that, sometimes it works, sometimes not:
I figure the VirusTotal approach would be bad in this case, as it'd be a vector for bad actors to test AI-generated images to see if they pass or not.
This is something for us to consider in the near future!
About time. Much needed. I just wish this was open source and built in public.
On my todo list to build a bot that finds sly AI responses for engagement farming
And fiction has become reality, this is a scenario explored in AI 2041 https://www.ai2041.com/
But can't the bad actors use the same APIs to ensure that it's passing this first?
See my other response on this, but this is something we are actively protecting against.
Interesting that fakes text is not an artifact they support. I can understand there isn’t enough entropy for the detection logic to work may be
It is in fact something we support with our enterprise clients and will roll out at a later date via Public API. Including the free tier.
On a 2k desktop using Chrome, your website font/layout is way too big, especially your consent banner--it takes up 1/3 of the screen.
I find that most companies find that to be a feature, not a bug. People are more likely to hit accept if they can only see a small chunk of the content.
Noted. Our marketing team only uses 640x480 CRTs and works exclusively in IE6, so will flag to them via Yahoo Messenger.
And the scrolling behavior is infuriating
please do not hijack the scroll wheel
It's like scrolling through molasses.