> In cartography, a trap street is a fictitious entry in the form of a misrepresented street on a map, often outside the area the map nominally covers, for the purpose of "trapping" plagiarists of the map who, if caught, would be unable to explain the inclusion of the "trap street" on their map as innocent.
This is fascinating . I wonder what the mechanism was that punished someone who copied the map. Copyright laws? This appears to be an old strategy so im not sure thats the case.
Other than some sort of social punishment I don’t see the cost for the copier. For the original author there is a cost of making your map less accurate on purpose.
They're used by map makers to prevent other map makers from freeriding on the original work. They're less worried about personal copies and much more interested in deterring commercial reuse.
That doesn't answer my questions although I now realize I made a bad assumption. This is a modern technique. I misread the bit about Greece on the wikipedia article and somehow got it in my head that it was more ancient.
So I think to answer my question more directly, yes, it's just copyright laws.
My point, before understanding that, was that from the map consumer's perspective it doesnt make a difference. And from the original creator's perspective it has a little accuracy cost. So without an additional mechanism I didn't see the disincentive for copiers. So I wondered if it was some ancient form of copyright law or just strong sense of morals or what.
I'm reminded of a section of Cory Doctorow's latest non-fiction, The Reverse Centaur's Guide To Life After AI (it is of course an excellent read), where he discusses how the semantic web and having an internet set-up for agents and AI to use won't work:
>It also hasn't happened because there is a reward for cheating. Scammers will label their phone numbers as the customer service number for your cable operator or bank. Counterfeiters will label their batteries or phone cases as official accessories. It's hard enough to get people to correctly label their stuff—but fact-checking billions of labels created by millions of people, none of whom you know personally or whose identities you can verify? That's insurmountable.
There's more he says—the preceding paragraph is below—but that line "there is a reward for cheating" is something I keep thinking about. It's hard not to see in so many of the discussions of LLMs.
>For one thing, there are irreconcilable differences in the ways in which things should be described. A site run by transphobes isn't going to label every story about a trans woman as "a story about a woman," irrespective of what the standard says. Microsoft is never going to label Excel as equivalent to Apple's Numbers, Google Sheets, and LibreOffice's Calc. Hermès is never going to list a $40,000 handbag as being made of the same material and having the same capacity as a $30 leather handbag of the same size and shape.
it seems like end stage of LLMs is an MCP economy in which Google and Apple will have a massive advantage in verified locations, businesses, identities and representation of services accurately.
Of course. But when you see text in the descriptor of a satchel on a satchel seller's site, you contextualize it as marketing. If you saw the same text in a textbook on the history, design, and manufacturing of satchels, you would contextualize it very differently.
When an LLM is the lens through which you view that text, you often don't have the context of where it came from, and the LLM is often really bad at filtering trustworthiness based on context when retrieving information to answer a query with.
I wouldn't go to a random site with a sketchy URL to get the customer service number for my xyz. I'd go to their official domain. Thus successfully navigating the treacherous waters scammers attempt to muddy.
Yea if you ever want to test how gullible even latest SOTA models are ask it for financial advice on a specific IPO and watch it vaccume up sell side nonsense and spew it out back to you as absolute fact.
How do you get the poison out of the water in the well? I understand the way this is supposed to work on the web, there are support channels to deal with this sort of thing. But if the deceit is baked into the model, what is your recourse?
IMO, it's similar to what's happening with code. People have been shipping sloppy code and fake news forever. It's just that the cost of doing these things has fallen off a cliff.
Sure but the agents obfuscate the signals you might use to judge the authority/trustworthiness of search results. They get disinformation results, wrap it into a reasonably sounding report, present it to you as trustworthy. There are so many ways for bad actors to game the system
> Isn't all of this true for a human-used internet as well?
Yes, but if you read this "semantic" info directly from a site, you can judge the site and its potential biases, and decide for yourself "this sells Louis Vuitton, of course they will praise their handbags". Or to pick a more extreme example that nobody would ignore, if you somehow land on an extremist website spewing racist crap, your alarms will surely trigger by anything they say.
But through an LLM this gets whitewashed and presented as authoritative and "neutral", when it isn't. You might know not to trust the LLM either (if you're paying attention; we all have our unconscious biases) but I bet you most people will be caught off guard.
The primary purpose of Google is to stop that. You have 100% trust in Google results because of the effort they have put into cleaning the results you see.
Billions have been spent on it.
Try searching Bing sometime. First result for Firefox is a malware clone site (you’ll have to scroll past the malware ad to see it).
The top 10 results were getting pretty bad even pre-LLM. I remember tons of poorly written articles near the top, just gaming the system with SEO. Everyone was paying some guy on Fiverr $5 an article to churn out hundreds of articles with little regard for accuracy.
1. This is a 2022 era picture that LLMs just suck up what's on the Internet and regurgitate it. It hasn't been true for a long time.
2. There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text. Training classifiers is pretty manageable.
The other day I asked Gemini to evaluate if a Reddit post was true. It told me it was definitely true because it had been verified by legitimate news agencies that had reported on the incident. These "legitimate" news agencies were an AI aggregator that just had a direct summary of the post with a disclaimer that it wasn't independently verified and a yahoo news "article" that was just the same AI summary. The AI summary included hallucinations that weren't even part of the original reddit post (like giving the OP a gender when no genders were mentioned).
>1. This is a 2022 era picture that LLMs just suck up what's on the Internet and regurgitate it. It hasn't been true for a long time.
When I ask ChatGPT certain questions, it very clearly shows me the sources it's searching through in the form of favicons, and they're always the favicons of sites that are in the first 5-10 results for a Google search query that would fit what I've asked the LLM. It will then cite those websites as it's source, with a direct link to the page that also shows up in an equivalent Google search. It does that daily.
Why do you believe that? The favicons seem like they’re generated by the UI, not the LLM.
On the other hand, I recently asked ChatGPT for some information about a group of people, and the headshots it put with each summary were very clearly not the people it was describing. So hallucinations do still occur even on the latest models, it just seems unlikely that the favicons would be LLM-generated.
>1. This is a 2022 era picture that LLMs just suck up what's on the Internet and regurgitate it. It hasn't been true for a long time
if anything it’s more true now, AI chatbots rarely use raw LLM output, instead they have CoT and use search tool calls. and the speed at which webpages are crawled is in the matter of hours. I posted something here on hn, then searched for it on google 1 hr later and its AI answered with my comment
there’s several companies right now making a killing of organic placement on chatbots similar to the SEO rush of early google
> 2) There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text. Training classifiers is pretty manageable
i think it’s pretty naive to think that if LLM companies would develop LLM detection they would use it to skip training on that content.
instead, they use it to train their output to be less detectable.
LLM detection is not solvable because it is adversarial, an arms race
I'm not sure you get my point.
Just ask ChatGPT, "What are the top 10 xxx," and see which sources it cites.
Oftentimes, it's complete garbage with LLM tells.
> 2. There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text.
There's a tension there: they want you to use AI for everything, and if they get what they want then new information will be embedded in LLM text.
> Training classifiers is pretty manageable.
Hopefully such things can be opened up, so the slop can be iterated on until it passes. It doesn't need to be perfect, just enough so more garbage gets through.
> There's a tension there: they want you to use AI for everything, and if they get what they want then new information will be embedded in LLM text.
In the long term, this is probably a real concern that they have to tackle. They'll have to move away from just classifying things as LLM generated or not, to attempting to judge the underlying quality, and that's not an easy problem.
This problem is so bad that you can create a website just to review your own website so when LLMs want to pull out reviews for the website the fake reviews count
Pre Google internet problem was not about the quality but about spam in headers and keywords, which were used by the pre-google search engines.
Google came and worked wonderfully for a year or two, until people learned to game page rank algorithm. Soon first google pages were just SEO spam and now half of the google is just spam, big corp websites, etc.
All of that precious stuff from small hobby websites about cool things to blogs that used to be wonderful read like https://www.joelonsoftware.com.
AI were not hurt that much by spam, I guess there were at least some curation done, the question is if this curation will work and for how long. It seems we are throwing the towel here as in case of search.
How can this be fixed? Nothing prevents you from publishing straight BS, and LLMs will use any site that doesn't have a garbage reputation as a valid third-party source.
Of course, LLMs already use Reddit, Facebook and co. as a source, so it doesn't really matter...
> How can this be fixed? Nothing prevents you from publishing straight BS, and LLMs will use any site that doesn't have a garbage reputation as a valid third-party source.
Personally, I hope it can't, or the fix is onerous and kneecaps a lot of the utility of AIs.
I wonder if we can even have some adversarial competition, where competing businesses try to get AIs to ingest bad data to pollute their competitors' decision-making.
For starters: lets get some fake math papers out there. Given proof-checking systems, maybe they won't totally stop OpenAI, but hopefully the bad math could at least drive up the cost and slow AI down.
I don't care where the bad data comes from. It's actually kinda good that some people will have a financial incentive to push it out and keep pushing.
> Bad for humans in general, good for hucksters and con artists. Net negative for humanity.
It's a sad state of affairs, but it's what has to be done. If we've got to burn the web down to save humanity from obsolescence, I'm for it.
The "net negative for humanity" was the attempt to build machines to replace human intellect. I'm not going to let any distaste for marketers or hucksters or con artists distract from that worse thing.
Ah, yes, arrogance of humans is unbounded. That you, humble reader are happier to swim in a sea of misinformation on the premise that you are smarter and faster at disambiguating bullshit from reality.
Furthermore, you are content to pretend that the rest of humanity is capable of doing this on a regular basis. And not just the top 10% of those who can read, interpret and weigh biases, but a solid portion of it.
It is a completely reasonable interpretation if you are an accelerationist living in the elite classes. In fact, perhaps, they are even counting on a majority of people thinking this is a fine environment.
I don't have the answers, but a lot of this is just a continuation of a very bad trend.
> Ah, yes, arrogance of humans is unbounded. That you, humble reader are happier to swim in a sea of misinformation on the premise that you are smarter and faster at disambiguating bullshit from reality.
I think you misunderstand me: I want people to shit in the sea until it stinks so bad that everyone gets out of the water.
Ok but I don't have a lot of faith that it will ever be turned back without violence because there are vested interests in keeping everyone swimming in it.
> Ok but I don't have a lot of faith that it will ever be turned back without violence because there are vested interests in keeping everyone swimming in it.
I'm not sure if we're talking about the same thing.
I think an acceptable good-but-not-perfect outcome it one where the "sea" (AI) stinks so bad that enough people get out that there's a viable life outside of it. The thing is people, individually and in aggregate, are lazy and competitive, so self-discipline isn't sufficient to sustain that life. There need to be problems that push people in that direction.
The web is foundational to AI, so that's a place where it's vulnerable.
Just to be clear, you're proposing that we poison the internet with misinformation - which is just as misleading to humans using traditional information retrieval tools as to AIs - until it's completely unusable, and this will be good because it's bad for AI, and everyone who is misled in the meantime is acceptable collateral damage?
Here I'm quoting from the website:
> It should go without saying that you should never publish fake content. While my experiment relied on tens of thousands of unverified stats, I ran it in order to share the results with the public, and the website, while still live, now carries a disclaimer.
"now carries a disclaimer" sure suggests it didn't, so "you should never publish fake content" really doesn't seem like a standard the author actually held themself to.
I'm very well aware of that. Unfortunately, this technology was invented, and the logic of our competitive society leaves little chance to contain it except scorched earth tactics.
Dude. The internet has been poisoned for a very long time. Remember Viagra? It used to be easy to call out the BS. But if you are outsourcing your thinking to a LLM, it’s much harder.
I don't think it is up to us at this point. LLMs are now the lens ("filter for what people see"). I hope, they will fix this. I think that they will still rely on reasoning and better grounding, which should prevent straight lies. But I think two bigger problems here: 1. training data might have the BS which would steer reasoning; 2. grounding might still be limited by small number of sources.
I've been thinking for awhile it is very strange that claude/ openai etc google "What was the inflation rate last quater" instead of just having a tool to query BLS datasets and that sort of thing.
So hopefully most of these statistics move towards first party sources.
Defer to authority and trust your government to rank what is truth and what is not, so your LLM always gives you the right answer. Nothing can go wrong here ever.
It cannot be fixed. The most hilarious example of it is YouTube. They have a video summarisation feature, which automatically creates video summaries that then get fed into the search engine. The problem with those is that they are created using models that are good few months behind, and let's say someone published a video on Nikon ZR and the model used to create summaries does not know about Nikon ZR and inserts references to Nikon Z8 or Nikon Zf into the summaries. This decreases accuracy of YouTube search and negatively affect discoverability. It's the AI version of shitting into the well you drink from. Bon appetit!
People stop trusting what they read on the internet.
All alternates are just attempting to improve the baseline trustworthiness of random information from the internet (and/or LLM that's baby-birding it back to you). Any increase in how much people blindly trust random stuff from the internet only serves to increase the blast radius when it is wrong.
The fix that seems obvious to me is to only let the LLM see truthful (or at least non-malicious) data. The model trainers carefully curate the training data, and you don't give the LLM access to full-web search. Instead, you hook them up to an 'official' source like an encyclopedia, documentation for the software in question, etc.
It's a ton of effort and expense, but probably feasible from a technical point of view. So instead, the model providers will probably do something silly like wire up rules like "NYT=Reliable;my.blog.forbes.com=Unreliable" and they'll say the issue is solved.
I thought everybody knew this was a problem. Scrapers vacuum up data but don't have any way tell what's true or false.
It's interesting to have a specific case where obviously false data is being spread, but there's nearly an infinite number of more subtle variations around.
I wonder if Googles products will eventually be the least vulnerable to this after spending so many years optimizing how they filter quality content from bad content.
The page demonstrates that Google specifically delisted them.
That said, I expect that OpenAI will eventually have its own search infrastructure that rivals Google's, and the overall system will get smarter. Whatever Google did to recognize this site as trash, OpenAI will do as well.
Reminds me of the people whining about Wikipedia. AI can know or care if the data is fake, it just needs to be programmed to cross check or rate the source. You can even tell it to do that before accepting the data.
These are not parallel situations. Wikipedia is far more accurate and consistent, and a lot of the whiners whine because their personal politics make it hard to square the information on the page.
Often real people don't either, plenty of fraud or selective reporting out there... but with AI we can also launder it so that the people who do care about fake data don't know any better.
You can prompt it to fact check. Getting tired of these slippery slope arguments that never pan out in the hopeless way people think (printing press/cotton gin/steam engine/computers/calculators/etc. will take all the jobs!!)... but I don't think it will ever stop, it seems inherent to human behavior or something.
"History shows that when a slope appears, regulation steps in, technology evolves to solve the problem, or the culture shifts to reinterpret the tool. In almost every case, the feared bottom of the slope was never reached because humans constantly built ramps or bridges along the way."
The fact that AI can be duped more easily than a human while sounding more authoritative and being entrusted with more and more critical tasks is really not something you should feel compelled to defend or relativize. It's a problem and the industry doesn't seem to have a good fix.
It can certainly make judgements on the data and how it was obtained. You just need to specify the kind of data you will accept (or this can be hard coded)
falsifying statistics alone without the added stealing on top, was 3 years of labor camp in places. that was after they course corrected and stopped giving capital punishment for it.
good thing nobody falsified statistic just for the sake of it -- so being a sabotager and a foreign spy was always part of it too.
> In cartography, a trap street is a fictitious entry in the form of a misrepresented street on a map, often outside the area the map nominally covers, for the purpose of "trapping" plagiarists of the map who, if caught, would be unable to explain the inclusion of the "trap street" on their map as innocent.
https://en.wikipedia.org/wiki/Trap_street
This is fascinating . I wonder what the mechanism was that punished someone who copied the map. Copyright laws? This appears to be an old strategy so im not sure thats the case.
Other than some sort of social punishment I don’t see the cost for the copier. For the original author there is a cost of making your map less accurate on purpose.
They're used by map makers to prevent other map makers from freeriding on the original work. They're less worried about personal copies and much more interested in deterring commercial reuse.
That doesn't answer my questions although I now realize I made a bad assumption. This is a modern technique. I misread the bit about Greece on the wikipedia article and somehow got it in my head that it was more ancient.
So I think to answer my question more directly, yes, it's just copyright laws.
My point, before understanding that, was that from the map consumer's perspective it doesnt make a difference. And from the original creator's perspective it has a little accuracy cost. So without an additional mechanism I didn't see the disincentive for copiers. So I wondered if it was some ancient form of copyright law or just strong sense of morals or what.
when you're a big corporation, probably no consequences: https://www.youtube.com/watch?v=byfWscC87Vg
See Chavda’s paradox https://zencapital.substack.com/p/chavdas-paradox
I'm reminded of a section of Cory Doctorow's latest non-fiction, The Reverse Centaur's Guide To Life After AI (it is of course an excellent read), where he discusses how the semantic web and having an internet set-up for agents and AI to use won't work:
>It also hasn't happened because there is a reward for cheating. Scammers will label their phone numbers as the customer service number for your cable operator or bank. Counterfeiters will label their batteries or phone cases as official accessories. It's hard enough to get people to correctly label their stuff—but fact-checking billions of labels created by millions of people, none of whom you know personally or whose identities you can verify? That's insurmountable.
There's more he says—the preceding paragraph is below—but that line "there is a reward for cheating" is something I keep thinking about. It's hard not to see in so many of the discussions of LLMs.
>For one thing, there are irreconcilable differences in the ways in which things should be described. A site run by transphobes isn't going to label every story about a trans woman as "a story about a woman," irrespective of what the standard says. Microsoft is never going to label Excel as equivalent to Apple's Numbers, Google Sheets, and LibreOffice's Calc. Hermès is never going to list a $40,000 handbag as being made of the same material and having the same capacity as a $30 leather handbag of the same size and shape.
it seems like end stage of LLMs is an MCP economy in which Google and Apple will have a massive advantage in verified locations, businesses, identities and representation of services accurately.
Isn't all of this true for a human-used internet as well?
Of course. But when you see text in the descriptor of a satchel on a satchel seller's site, you contextualize it as marketing. If you saw the same text in a textbook on the history, design, and manufacturing of satchels, you would contextualize it very differently.
When an LLM is the lens through which you view that text, you often don't have the context of where it came from, and the LLM is often really bad at filtering trustworthiness based on context when retrieving information to answer a query with.
I wouldn't go to a random site with a sketchy URL to get the customer service number for my xyz. I'd go to their official domain. Thus successfully navigating the treacherous waters scammers attempt to muddy.
Yea if you ever want to test how gullible even latest SOTA models are ask it for financial advice on a specific IPO and watch it vaccume up sell side nonsense and spew it out back to you as absolute fact.
How do you get the poison out of the water in the well? I understand the way this is supposed to work on the web, there are support channels to deal with this sort of thing. But if the deceit is baked into the model, what is your recourse?
IMO, it's similar to what's happening with code. People have been shipping sloppy code and fake news forever. It's just that the cost of doing these things has fallen off a cliff.
Sure but the agents obfuscate the signals you might use to judge the authority/trustworthiness of search results. They get disinformation results, wrap it into a reasonably sounding report, present it to you as trustworthy. There are so many ways for bad actors to game the system
> Isn't all of this true for a human-used internet as well?
Yes, but if you read this "semantic" info directly from a site, you can judge the site and its potential biases, and decide for yourself "this sells Louis Vuitton, of course they will praise their handbags". Or to pick a more extreme example that nobody would ignore, if you somehow land on an extremist website spewing racist crap, your alarms will surely trigger by anything they say.
But through an LLM this gets whitewashed and presented as authoritative and "neutral", when it isn't. You might know not to trust the LLM either (if you're paying attention; we all have our unconscious biases) but I bet you most people will be caught off guard.
The primary purpose of Google is to stop that. You have 100% trust in Google results because of the effort they have put into cleaning the results you see.
Billions have been spent on it.
Try searching Bing sometime. First result for Firefox is a malware clone site (you’ll have to scroll past the malware ad to see it).
The frequently incorrect or misleading Gemini answer at the top of search results is busy eroding that trust.
People will have varying levels of trust in Google, but 100% is clearly too much, given the quality of results.
This is a known problem, when ChatGPT uses web search tool, top 10 results are often LLM generated sources as well.
They now train on their own content.
Eating your own d̶o̶g̶ f̶o̶o̶d̶ poo.
The top 10 results were getting pretty bad even pre-LLM. I remember tons of poorly written articles near the top, just gaming the system with SEO. Everyone was paying some guy on Fiverr $5 an article to churn out hundreds of articles with little regard for accuracy.
We built our own ways to filter these because *we* went to the website and gathered a lot of information on it.
Now, the model slips it in without any verification and casually builds on top of it.
Web search is pointless for certain topics now, if you don't want LLM filtered information. The first few pages of results are obvious slop.
Two things:
1. This is a 2022 era picture that LLMs just suck up what's on the Internet and regurgitate it. It hasn't been true for a long time.
2. There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text. Training classifiers is pretty manageable.
This is not true.
The other day for me Google search AI cited triggerbites which is a vibe data website.
The LLM unless explicitly prompted cannot distinguish the validity of a data source.
The other day I asked Gemini to evaluate if a Reddit post was true. It told me it was definitely true because it had been verified by legitimate news agencies that had reported on the incident. These "legitimate" news agencies were an AI aggregator that just had a direct summary of the post with a disclaimer that it wasn't independently verified and a yahoo news "article" that was just the same AI summary. The AI summary included hallucinations that weren't even part of the original reddit post (like giving the OP a gender when no genders were mentioned).
The article literally describes ChatGPT regurgitating something it sucked up on the Internet to this day.
>1. This is a 2022 era picture that LLMs just suck up what's on the Internet and regurgitate it. It hasn't been true for a long time.
When I ask ChatGPT certain questions, it very clearly shows me the sources it's searching through in the form of favicons, and they're always the favicons of sites that are in the first 5-10 results for a Google search query that would fit what I've asked the LLM. It will then cite those websites as it's source, with a direct link to the page that also shows up in an equivalent Google search. It does that daily.
It's possible that the favicons it shows and the web pages that are actually retrieved are not the same
Why do you believe that? The favicons seem like they’re generated by the UI, not the LLM.
On the other hand, I recently asked ChatGPT for some information about a group of people, and the headshots it put with each summary were very clearly not the people it was describing. So hallucinations do still occur even on the latest models, it just seems unlikely that the favicons would be LLM-generated.
I think they are talking about training not run time search.
>1. This is a 2022 era picture that LLMs just suck up what's on the Internet and regurgitate it. It hasn't been true for a long time
if anything it’s more true now, AI chatbots rarely use raw LLM output, instead they have CoT and use search tool calls. and the speed at which webpages are crawled is in the matter of hours. I posted something here on hn, then searched for it on google 1 hr later and its AI answered with my comment
https://share.google/aimode/PkQUVfA68AlHGb7Yr
there’s several companies right now making a killing of organic placement on chatbots similar to the SEO rush of early google
> 2) There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text. Training classifiers is pretty manageable
i think it’s pretty naive to think that if LLM companies would develop LLM detection they would use it to skip training on that content.
instead, they use it to train their output to be less detectable.
LLM detection is not solvable because it is adversarial, an arms race
I'm not sure you get my point. Just ask ChatGPT, "What are the top 10 xxx," and see which sources it cites. Oftentimes, it's complete garbage with LLM tells.
I'm responding to
> They now train on their own content.
They train on conversations which pull data from web.
> 2. There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text.
There's a tension there: they want you to use AI for everything, and if they get what they want then new information will be embedded in LLM text.
> Training classifiers is pretty manageable.
Hopefully such things can be opened up, so the slop can be iterated on until it passes. It doesn't need to be perfect, just enough so more garbage gets through.
> There's a tension there: they want you to use AI for everything, and if they get what they want then new information will be embedded in LLM text.
In the long term, this is probably a real concern that they have to tackle. They'll have to move away from just classifying things as LLM generated or not, to attempting to judge the underlying quality, and that's not an easy problem.
This has existed for a long time before LLM, although in a slower form. On Wikipedia it's called "citogenesis".
Too bad "citosis" is too close to "cytosis," I think it would have been cleverer.
Quote Wikipedia:
> "Citogenesis" redirects here; not to be confused with cytogenesis.
I had to read that three times. I thought it was a joke at first.
This problem is so bad that you can create a website just to review your own website so when LLMs want to pull out reviews for the website the fake reviews count
i’d say this is fraud and a crime too, it’s not a new problem
What? How would that be a crime? Most marketing websites already contain fake reviews of their products.
It’s hard to find and convict people who post actual illegal content online, much less fake reviews
LLMs are facing the same problem pre-google humans faced:
Is this information on this website true/high quality?
We need a google for LLMs!
Oh, but LLMs don't consume ads.
We need ads for LLMs!
Help.
Pre Google internet problem was not about the quality but about spam in headers and keywords, which were used by the pre-google search engines.
Google came and worked wonderfully for a year or two, until people learned to game page rank algorithm. Soon first google pages were just SEO spam and now half of the google is just spam, big corp websites, etc.
All of that precious stuff from small hobby websites about cool things to blogs that used to be wonderful read like https://www.joelonsoftware.com.
AI were not hurt that much by spam, I guess there were at least some curation done, the question is if this curation will work and for how long. It seems we are throwing the towel here as in case of search.
I suspect there are already many fake stats sites like this by the time I read this.
According to a highly reputable source, there are over 15 quintillion of them.
This is fine work, we need more communication to the general public about this
I wonder why did Google derank it. Did it correctly identify it as LLM written and sink it because of that?
This is insane.
How can this be fixed? Nothing prevents you from publishing straight BS, and LLMs will use any site that doesn't have a garbage reputation as a valid third-party source.
Of course, LLMs already use Reddit, Facebook and co. as a source, so it doesn't really matter...
> This is insane.
As a human, this is fantastic.
> How can this be fixed? Nothing prevents you from publishing straight BS, and LLMs will use any site that doesn't have a garbage reputation as a valid third-party source.
Personally, I hope it can't, or the fix is onerous and kneecaps a lot of the utility of AIs.
I wonder if we can even have some adversarial competition, where competing businesses try to get AIs to ingest bad data to pollute their competitors' decision-making.
For starters: lets get some fake math papers out there. Given proof-checking systems, maybe they won't totally stop OpenAI, but hopefully the bad math could at least drive up the cost and slow AI down.
> As a human, this is fantastic.
s/human/unscrupulous marketer/
Bad for humans in general, good for hucksters and con artists. Net negative for humanity.
>> As a human, this is fantastic.
> s/human/marketer/
I don't care where the bad data comes from. It's actually kinda good that some people will have a financial incentive to push it out and keep pushing.
> Bad for humans in general, good for hucksters and con artists. Net negative for humanity.
It's a sad state of affairs, but it's what has to be done. If we've got to burn the web down to save humanity from obsolescence, I'm for it.
The "net negative for humanity" was the attempt to build machines to replace human intellect. I'm not going to let any distaste for marketers or hucksters or con artists distract from that worse thing.
Ah, yes, arrogance of humans is unbounded. That you, humble reader are happier to swim in a sea of misinformation on the premise that you are smarter and faster at disambiguating bullshit from reality.
Furthermore, you are content to pretend that the rest of humanity is capable of doing this on a regular basis. And not just the top 10% of those who can read, interpret and weigh biases, but a solid portion of it.
It is a completely reasonable interpretation if you are an accelerationist living in the elite classes. In fact, perhaps, they are even counting on a majority of people thinking this is a fine environment.
I don't have the answers, but a lot of this is just a continuation of a very bad trend.
> Ah, yes, arrogance of humans is unbounded. That you, humble reader are happier to swim in a sea of misinformation on the premise that you are smarter and faster at disambiguating bullshit from reality.
I think you misunderstand me: I want people to shit in the sea until it stinks so bad that everyone gets out of the water.
Ok but I don't have a lot of faith that it will ever be turned back without violence because there are vested interests in keeping everyone swimming in it.
> Ok but I don't have a lot of faith that it will ever be turned back without violence because there are vested interests in keeping everyone swimming in it.
I'm not sure if we're talking about the same thing.
I think an acceptable good-but-not-perfect outcome it one where the "sea" (AI) stinks so bad that enough people get out that there's a viable life outside of it. The thing is people, individually and in aggregate, are lazy and competitive, so self-discipline isn't sufficient to sustain that life. There need to be problems that push people in that direction.
The web is foundational to AI, so that's a place where it's vulnerable.
Just to be clear, you're proposing that we poison the internet with misinformation - which is just as misleading to humans using traditional information retrieval tools as to AIs - until it's completely unusable, and this will be good because it's bad for AI, and everyone who is misled in the meantime is acceptable collateral damage?
Here I'm quoting from the website:
> It should go without saying that you should never publish fake content. While my experiment relied on tens of thousands of unverified stats, I ran it in order to share the results with the public, and the website, while still live, now carries a disclaimer.
"now carries a disclaimer" sure suggests it didn't, so "you should never publish fake content" really doesn't seem like a standard the author actually held themself to.
Dude.
Regardless of how the data is sourced, this is bad. It's poisoning the internet.
Besides, such issues come from a lack of due diligence, which have only been exacerbated by AI.
> It's poisoning the internet.
I'm very well aware of that. Unfortunately, this technology was invented, and the logic of our competitive society leaves little chance to contain it except scorched earth tactics.
Dude. The internet has been poisoned for a very long time. Remember Viagra? It used to be easy to call out the BS. But if you are outsourcing your thinking to a LLM, it’s much harder.
Bots and their slop (especially when it's hard to tell apart from human-made content) are the ones "poisoning the Internet."
I don't think it is up to us at this point. LLMs are now the lens ("filter for what people see"). I hope, they will fix this. I think that they will still rely on reasoning and better grounding, which should prevent straight lies. But I think two bigger problems here: 1. training data might have the BS which would steer reasoning; 2. grounding might still be limited by small number of sources.
I've been thinking for awhile it is very strange that claude/ openai etc google "What was the inflation rate last quater" instead of just having a tool to query BLS datasets and that sort of thing.
So hopefully most of these statistics move towards first party sources.
>How can this be fixed?
Defer to authority and trust your government to rank what is truth and what is not, so your LLM always gives you the right answer. Nothing can go wrong here ever.
It cannot be fixed. The most hilarious example of it is YouTube. They have a video summarisation feature, which automatically creates video summaries that then get fed into the search engine. The problem with those is that they are created using models that are good few months behind, and let's say someone published a video on Nikon ZR and the model used to create summaries does not know about Nikon ZR and inserts references to Nikon Z8 or Nikon Zf into the summaries. This decreases accuracy of YouTube search and negatively affect discoverability. It's the AI version of shitting into the well you drink from. Bon appetit!
>How can this be fixed?
People stop trusting what they read on the internet.
All alternates are just attempting to improve the baseline trustworthiness of random information from the internet (and/or LLM that's baby-birding it back to you). Any increase in how much people blindly trust random stuff from the internet only serves to increase the blast radius when it is wrong.
The fix that seems obvious to me is to only let the LLM see truthful (or at least non-malicious) data. The model trainers carefully curate the training data, and you don't give the LLM access to full-web search. Instead, you hook them up to an 'official' source like an encyclopedia, documentation for the software in question, etc.
It's a ton of effort and expense, but probably feasible from a technical point of view. So instead, the model providers will probably do something silly like wire up rules like "NYT=Reliable;my.blog.forbes.com=Unreliable" and they'll say the issue is solved.
this is actually insane. and people still use chatgpt to generate reports for their managers. snowball of fake data
What's the conclusion? Publish fake content (or not)?
Publish fake stats and scam investors, chatgpt will tell them your idea is huge, while in reality it's irrelevant
If I'm not wrong ChatGPT indexes on Bing, curious if the site and stats rank well on Bing
I thought everybody knew this was a problem. Scrapers vacuum up data but don't have any way tell what's true or false.
It's interesting to have a specific case where obviously false data is being spread, but there's nearly an infinite number of more subtle variations around.
AI doesn't know or care if the data is fake.
I wonder if Googles products will eventually be the least vulnerable to this after spending so many years optimizing how they filter quality content from bad content.
Or maybe they could have listed well on Google, too, if they had splurged on ads and SEO...
The page demonstrates that Google specifically delisted them.
That said, I expect that OpenAI will eventually have its own search infrastructure that rivals Google's, and the overall system will get smarter. Whatever Google did to recognize this site as trash, OpenAI will do as well.
Reminds me of the people whining about Wikipedia. AI can know or care if the data is fake, it just needs to be programmed to cross check or rate the source. You can even tell it to do that before accepting the data.
These are not parallel situations. Wikipedia is far more accurate and consistent, and a lot of the whiners whine because their personal politics make it hard to square the information on the page.
Often real people don't either, plenty of fraud or selective reporting out there... but with AI we can also launder it so that the people who do care about fake data don't know any better.
Yep, and the results are presented with the voice of a confident expert but it is actually missing the epistemological layer
You can prompt it to fact check. Getting tired of these slippery slope arguments that never pan out in the hopeless way people think (printing press/cotton gin/steam engine/computers/calculators/etc. will take all the jobs!!)... but I don't think it will ever stop, it seems inherent to human behavior or something.
"History shows that when a slope appears, regulation steps in, technology evolves to solve the problem, or the culture shifts to reinterpret the tool. In almost every case, the feared bottom of the slope was never reached because humans constantly built ramps or bridges along the way."
The fact that AI can be duped more easily than a human while sounding more authoritative and being entrusted with more and more critical tasks is really not something you should feel compelled to defend or relativize. It's a problem and the industry doesn't seem to have a good fix.
Fact check against what? Other facts it can find online that manipulate its behaviour into citing them?
How do humans fact check?
Mostly by way of confirmation bias.
if it works often enough to be a good heuristic, it's not really a bias until somebody uses it adversly
I've had AI lie about fact checking and can't really trust it anymore.
in which case it sees it's been cited, that it's the "source of truth" and approves the fake data.
It can certainly make judgements on the data and how it was obtained. You just need to specify the kind of data you will accept (or this can be hard coded)
You cannot reliably do it, no.
Nine out of ten stats are made up on the spot.
"You really think someone would do that? Just go on the internet and tell lies?"
don’t
if you just lie and an llm or a newspaper or a kid believe you, at that point it isn’t a criticism of their naivete, you are just lying
similarly hacking through systems or picking locks isn’t a showcase of weakness if systems, it’s just being a hacker
People have been astroturfing sites like Wikipedia, Reddit etc, well before LLMs were a thing.
Good job that only honest people post things on the internet otherwise we’d have a problem
falsifying statistics alone without the added stealing on top, was 3 years of labor camp in places. that was after they course corrected and stopped giving capital punishment for it.
good thing nobody falsified statistic just for the sake of it -- so being a sabotager and a foreign spy was always part of it too.
Still it needs to be done once publicly to prove and document this is possible. The experiment was not intended to spread this far.
Prove what? That it's possible to lie on the internet and then SEO these lies to the top of a search ranking? We kind of already knew that.
Yes “kind of, that’s the issue though. Clearly the extent of the problem isn’t being acknowledged.