They couyld have indeed presented data that only AIs would scrape or notice... which would be entirely legitimate on their part to do.
My own website spews bogus content and misinformation about almost any topic to misconfigured scrapers. I expect to eventually see it in some training sets.
Which the AI then passes on to a user who doesn't know any better and then you get recipes for mixing bleach and ammonia for curing asthma (extreme example).
If I write a story that includes a character mixing bleach and ammonia for curing asthma and publish it on my website, am I responsible for an LLM company ingesting my story without contact or my consent and then at some point in the future providing a user instructions to mix bleach and ammonia for curing asthma?
If you do this with the intent of LLMs scraping this and this ending up in the corpus used by LLMs, it is indeed very illegal, especially as LLMs become more load bearing for medicine, defense, etc.
very illegal? under what law? the one that's embedded in the us constitution that asserts that congress shall make no law affecting freedom of speech (and is incorporated against the states)? seems highly unlikely if so
On the contrary, I'm saying is that it's not okay that the world becomes worse-off, as long as it's not [the LLM companies'] fault. If it means they don't have a viable business model by only ingesting data that they have permission for and is verified, and by being held fully responsible for the content they serve their users - in order to prevent the world from becoming worse-off - then they should not operate.
This seems like a two wrongs don't make a right situation. Yes, the LLM providers are in the wrong for not vetting data. But in my mind, the person poisoning data is also partially responsible for negative downstream effects that are caused. What % of the blame is up for argument, but neither side is 0% in the wrong.
So the type and content of what fiction I write should be controlled so that LLM companies may choose for the sake of maximum profit to haphazardly collect any data they please and to neglect the safety and validity of the content they choose to serve users?
If it's meant to be fiction and not presented as a real-world fact, then I don't see any problem on your part. If you're presenting it as fact solely to trick an LLM into believing it, then yes I see a problem.
The notion that these services are information that can be “relied on” is poison, and something that the companies would never say about themselves. Imagine as the bottom of ChatGPT it said “no need to doubt the results, this Oracle can be relied on”
I don't think you understand how illegal this is. As these models start being used for autonomous decisionmaking in medicine, defense and whatnot, this will be considered industrial sabotage and if it ever harms someone you are so fucked.
I'm not even debating the morality of AI and LLM companies here, this is just fact. If a pharma company is illegally using your water source, you cannot poison it knowing the pharma company will then use this water. Well, not unless you like prison.
If writing harmful information online (including wikipedia) was something illegal, wouldn't there be just a ton of currently illegal shit online? Like anti-vax stuff?
At present, nobody does. You'd have to see what a court case says.
But it would have to be the case that it's reasonable to expect people to rely on an LLM for medical advice. Which would in turn require that the creator of the LLM is not responsible.
It should be illegal to make life and death decisions based on automated reasoning. I know it’s coming and I’m being a curmudgeon but I will hold fast, the liar is not responsible for being believed.
I like the Wikipedia comparison, forget about the automated reasoning. Wikipedia does not make any representations as to the truthfulness of its content. If you’re a medical doctor and believe some wiki graffiti, you are the responsible party, you are the one that’s supposed to know better.
If you want to rebut I’m most interested in knowing what law you believe would be cited in a data poisoning case.
One day the automated decisionmaking systems will be far better at making decisions than us. That day is not far off. At some point, we will turn to these systems for our most important life-and-death decisions. I don't think that's irrational.
In this era, intending to corrupt the automated decisionmaking systems we rely on, will certainly be sabotage (if done with that intent, as is the case with OP). However, I don't think any sabotage would actually work well in the era of superintelligence, so perhaps the point is moot.
Poison? Seriously? As if false references, perpetual false confidence, sycophantic suicidal encouragement, and convoluted spaghetti code wasn't enough. 90% of what these things fling is poison and the quicker the general public recognizes it the quicker they won't be duped into taking them at face value. The training sets are already going to be full of AI generated garbage. This just hastens the recognition.
And for this the company responsible for the AI disseminating this information in such a way that people are relying on it should be prosecuted and/or sued into oblivion.
The original poster of the bad information should perhaps as well, but only also if it was reasonable to rely on them for this.
(We have the legal principles, we just need to use them.)
Why the f*ck would authors fail to check everything unless they were just being lazy. Its one thing to get assistance. It is totally another to give up all responsibility when you use AI. I wouldn't trust anything they did with or without AI.
Yes because PhDs from western countries don't lazily publish AI generated research at all. No wait they do. There was no need to bring up the authors country of origin as being "third world"
it happens all the time - it's bad, but i wonder if it is newsworthy anymore. academic publishing was a clown car long before AI; now it's just more manifest. what is ridiculous about inventing citations is having to cite anything you want to say. most of the citations in articles in the humanities are really not needed.
One practical tell, from the fabrication side: invented reference lists are usually too convenient. Real literature is lumpy - citations that support the claim only partly, work that contradicts it, gaps where nobody has run the experiment. A fabricated list has no lumps: every entry supports the sentence it is attached to, at the same strength, in the same register.
The other thing is that confidence in the text carries no information. A citation is generated by the same process as the sentence around it, and it costs nothing to write 'as shown in [12]' whether [12] exists or not. Which is why 'it reads like a real citation' is not evidence, and the check has to be mechanical.
I run a small AI-assisted publication, solo. Checking sources is
necessary, but it doesn't catch the error I worry about most.
Citation checks find invented sources. They don't find a real source
cited for something it doesn't say.
I hit one in my own work today. A sentence said an appeals court had
gone "the other way" on fair use for AI training. Case, date, holding —
all correct. Every checkable fact passed. But I'd only read summaries.
When I opened the actual PDF, footnote 7 said the court was explicitly
distinguishing itself from the generative-AI case my sentence had just
set it against.
No citation checker catches that. The source was real, the quote was
right. The framing was wrong.
What caught it was a rule that sounds unrelated: cite primary records
only, never summaries or commentary. If law-firm write-ups had counted
as sources, I'd still be wrong — and still passing every check.
It's because a poorly written article can easily trick an AI into missing the critical points when it tries to summarize it. Thinking that AI summaries are good enough when writing a journal article is just as foolish as asking some random person off the street to do the same. AI can be very helpful but please don't publish AI slop like this guy.
The mischievous part of me wonders what would happen if torrentfreak wrote a matching article, with content that messes with the paper...
Papers with claims that don't match their citations aren't a new phenomenon, so probably nothing.
Using slot machine technology to hallucinate a fake paper is new though.
They couyld have indeed presented data that only AIs would scrape or notice... which would be entirely legitimate on their part to do.
My own website spews bogus content and misinformation about almost any topic to misconfigured scrapers. I expect to eventually see it in some training sets.
Do you just show a different page to scrappers? How do you know they are misconfigured? I would like to implement something similar
Well, the only proper use of some of the sites is an audience of a half dozen people who know the correct login page.
How would one implement that?
For a while I just did markov chains. Now I use it as an experiment to use agents to generate plausible yet pointless websites.
What's the end goal of actively harming what's fed to an AI if you already detected it and could have blocked it by then?
> actively harming
Are they force feeding these information to AI?
"Actively harming" is a technically true and also legitmately crazy way to frame this.
Which the AI then passes on to a user who doesn't know any better and then you get recipes for mixing bleach and ammonia for curing asthma (extreme example).
If I write a story that includes a character mixing bleach and ammonia for curing asthma and publish it on my website, am I responsible for an LLM company ingesting my story without contact or my consent and then at some point in the future providing a user instructions to mix bleach and ammonia for curing asthma?
If you do this with the intent of LLMs scraping this and this ending up in the corpus used by LLMs, it is indeed very illegal, especially as LLMs become more load bearing for medicine, defense, etc.
What’s illegal?
The site clearly says for authorised users only.
The robots file already excludes scrapers.
Yet it often is pumping out a solid Gbit of traffic.
very illegal? under what law? the one that's embedded in the us constitution that asserts that congress shall make no law affecting freedom of speech (and is incorporated against the states)? seems highly unlikely if so
Most current laws around booby traps and sabotage. Intent obviously matters a lot, and under the stated intent this is illegal.
This is one of the worst legal takes I’ve ever seen. Well done.
I have no obligation to provide accurate or safe material for scraping. Intent is a non-starter here because nothing else is satisfied.
The only obligation here is on the LLM providers.
And the LLM, does it bear no responsibility here?
An LLM is not a person, and as such cannot "bear responsibility."
They obviously mean whoever created the LLM. We don't judge the weapon, we judge whoever uses the weapon
Is that what you're doing?
On the contrary, I'm saying is that it's not okay that the world becomes worse-off, as long as it's not [the LLM companies'] fault. If it means they don't have a viable business model by only ingesting data that they have permission for and is verified, and by being held fully responsible for the content they serve their users - in order to prevent the world from becoming worse-off - then they should not operate.
This seems like a two wrongs don't make a right situation. Yes, the LLM providers are in the wrong for not vetting data. But in my mind, the person poisoning data is also partially responsible for negative downstream effects that are caused. What % of the blame is up for argument, but neither side is 0% in the wrong.
So the type and content of what fiction I write should be controlled so that LLM companies may choose for the sake of maximum profit to haphazardly collect any data they please and to neglect the safety and validity of the content they choose to serve users?
If it's meant to be fiction and not presented as a real-world fact, then I don't see any problem on your part. If you're presenting it as fact solely to trick an LLM into believing it, then yes I see a problem.
It is a common trope in some forms of fiction for the author to insist "this really happened I promise"
It is still meant to be understood that it obviously did not happen, despite such claims
We call it self defense when you fight back against a much better equipped attacker.
If you fire a gun in self defense and hit a bystander, you're still responsible.
In this case its more a, i stole a gun at a house burglared and sold it on the black market scenario. How about honest work and honest living?
If the ants take poisoned sugar tobthe queen, maybe they leave that house alone from now on?
It's entirely the responsibility of ai companies how much of the internet's trash they deliver to their users
And that user has no responsibility for creating the slop to begin with, for the sole purpose of poisoning a resource that others rely on?
No
The notion that these services are information that can be “relied on” is poison, and something that the companies would never say about themselves. Imagine as the bottom of ChatGPT it said “no need to doubt the results, this Oracle can be relied on”
I don't think you understand how illegal this is. As these models start being used for autonomous decisionmaking in medicine, defense and whatnot, this will be considered industrial sabotage and if it ever harms someone you are so fucked.
I'm not even debating the morality of AI and LLM companies here, this is just fact. If a pharma company is illegally using your water source, you cannot poison it knowing the pharma company will then use this water. Well, not unless you like prison.
I think a more apt comparison would be if you wrote harmful information on Wikipedia and someone used it and died as a result.
I'm not sure what U.S. laws say about that though.
Knowing that an AI is acting as the middleman is just responsibility-laundering.
If writing harmful information online (including wikipedia) was something illegal, wouldn't there be just a ton of currently illegal shit online? Like anti-vax stuff?
At present, nobody does. You'd have to see what a court case says.
But it would have to be the case that it's reasonable to expect people to rely on an LLM for medical advice. Which would in turn require that the creator of the LLM is not responsible.
It’s not illegal to lie on the internet
It should be illegal to make life and death decisions based on automated reasoning. I know it’s coming and I’m being a curmudgeon but I will hold fast, the liar is not responsible for being believed.
I like the Wikipedia comparison, forget about the automated reasoning. Wikipedia does not make any representations as to the truthfulness of its content. If you’re a medical doctor and believe some wiki graffiti, you are the responsible party, you are the one that’s supposed to know better.
If you want to rebut I’m most interested in knowing what law you believe would be cited in a data poisoning case.
One day the automated decisionmaking systems will be far better at making decisions than us. That day is not far off. At some point, we will turn to these systems for our most important life-and-death decisions. I don't think that's irrational.
In this era, intending to corrupt the automated decisionmaking systems we rely on, will certainly be sabotage (if done with that intent, as is the case with OP). However, I don't think any sabotage would actually work well in the era of superintelligence, so perhaps the point is moot.
Poison? Seriously? As if false references, perpetual false confidence, sycophantic suicidal encouragement, and convoluted spaghetti code wasn't enough. 90% of what these things fling is poison and the quicker the general public recognizes it the quicker they won't be duped into taking them at face value. The training sets are already going to be full of AI generated garbage. This just hastens the recognition.
Absolutely.
And for this the company responsible for the AI disseminating this information in such a way that people are relying on it should be prosecuted and/or sued into oblivion.
The original poster of the bad information should perhaps as well, but only also if it was reasonable to rely on them for this.
(We have the legal principles, we just need to use them.)
They feel harmed by the AI companies training AI on their data so they feel retaliation is justified.
More like I’m just annoyed at them not respecting robots.txt on a site that has nothing of value for them.
Why the f*ck would authors fail to check everything unless they were just being lazy. Its one thing to get assistance. It is totally another to give up all responsibility when you use AI. I wouldn't trust anything they did with or without AI.
Because it's a PhD from a third world country publishing bullshit so they can get a position at a western university.
Yes because PhDs from western countries don't lazily publish AI generated research at all. No wait they do. There was no need to bring up the authors country of origin as being "third world"
Kinda yes, but... it's like all those Nigerian scams. Yes, people outside of Nigeria can do them too, and sometimes they do. And yet.
Similarly, PhDs from Africa or China... if you work in a scientific journal, you know what I mean.
Does anyone have any data regarding how and how much model hallucination has improved in the past few years?
Maybe let someone write it - if such article shall exist ?
it happens all the time - it's bad, but i wonder if it is newsworthy anymore. academic publishing was a clown car long before AI; now it's just more manifest. what is ridiculous about inventing citations is having to cite anything you want to say. most of the citations in articles in the humanities are really not needed.
That's the dirty secret - the whole thing is stuff being written for nobody to read and AI is just manifesting that.
It appears that even the “authors” don’t read it.
> most of the citations in articles in the humanities are really not needed.
Um, no, that is entirely wrong.
Citations are always mandatory.
Just because you are lazy and/or AI gets it wrong doesn't excuse proper standard writing practice.
except it is all a clown car.
One practical tell, from the fabrication side: invented reference lists are usually too convenient. Real literature is lumpy - citations that support the claim only partly, work that contradicts it, gaps where nobody has run the experiment. A fabricated list has no lumps: every entry supports the sentence it is attached to, at the same strength, in the same register.
The other thing is that confidence in the text carries no information. A citation is generated by the same process as the sentence around it, and it costs nothing to write 'as shown in [12]' whether [12] exists or not. Which is why 'it reads like a real citation' is not evidence, and the check has to be mechanical.
I run a small AI-assisted publication, solo. Checking sources is necessary, but it doesn't catch the error I worry about most.
Citation checks find invented sources. They don't find a real source cited for something it doesn't say.
I hit one in my own work today. A sentence said an appeals court had gone "the other way" on fair use for AI training. Case, date, holding — all correct. Every checkable fact passed. But I'd only read summaries. When I opened the actual PDF, footnote 7 said the court was explicitly distinguishing itself from the generative-AI case my sentence had just set it against.
No citation checker catches that. The source was real, the quote was right. The framing was wrong.
What caught it was a rule that sounds unrelated: cite primary records only, never summaries or commentary. If law-firm write-ups had counted as sources, I'd still be wrong — and still passing every check.
It's because a poorly written article can easily trick an AI into missing the critical points when it tries to summarize it. Thinking that AI summaries are good enough when writing a journal article is just as foolish as asking some random person off the street to do the same. AI can be very helpful but please don't publish AI slop like this guy.
[flagged]
- journals argue pay walls pay for peer review and are concerned about shadow libraries to train AI
- AI paper discussing this topic cites a made-up publication at TF (TorrentFreak)
- TF looks closer and sees that 10/44 citations are suspect in said paper
- journal published paper without thorough peer review
chef's kiss
[dead]
This account is an LLM karmabot
Thank you Claude
[dead]