To answer the first question, yes! Web operators and security professionals actively track and categorize AI agents separately from traditional search crawlers because they serve fundamentally different purposes and impact site resources in distinct ways.
I built a database website a few months ago and submitted it to Google, Bing, and Yandex. 2 months later, according to my Cloudflare dashboard, I have 1.5 million unique visitors monthly. I found that human visitors only accounted for about 10% of the total, followed by search engine crawl bots and then AI crawl bots. I also discovered that AI bots (like GPTBot, ClaudeBot, or PerplexityBot) scraped a lot rapidly without adhering to traditional crawl limits or deeply checking robots.txt files, resulting in high server loads.
That should also answer the second question, which is that AI retrieval systems index semantic structure faster than they index page content. You have to understand that AI doesn't just index your website like regular crawl bots, which index mostly your content, schema, and so on. AI bots go deeper by trying to understand your website structure, as this will also help in training other AI models.
I wonder if "traffic" is becoming the wrong metric. A human visit has always implied someone consuming the content. An AI crawler may never generate a pageview from a human, yet it can still become the mechanism by which someone discovers your work later through an assistant.
In that world, machine visits aren't necessarily noise, they're another distribution channel.
I'm curious if you're just tracking browser user agent, fingerprinting or some other method? For instance would someone using a tool to spider your site, would it be classed as an attack?
To answer the first question, yes! Web operators and security professionals actively track and categorize AI agents separately from traditional search crawlers because they serve fundamentally different purposes and impact site resources in distinct ways.
I built a database website a few months ago and submitted it to Google, Bing, and Yandex. 2 months later, according to my Cloudflare dashboard, I have 1.5 million unique visitors monthly. I found that human visitors only accounted for about 10% of the total, followed by search engine crawl bots and then AI crawl bots. I also discovered that AI bots (like GPTBot, ClaudeBot, or PerplexityBot) scraped a lot rapidly without adhering to traditional crawl limits or deeply checking robots.txt files, resulting in high server loads.
That should also answer the second question, which is that AI retrieval systems index semantic structure faster than they index page content. You have to understand that AI doesn't just index your website like regular crawl bots, which index mostly your content, schema, and so on. AI bots go deeper by trying to understand your website structure, as this will also help in training other AI models.
I wonder if "traffic" is becoming the wrong metric. A human visit has always implied someone consuming the content. An AI crawler may never generate a pageview from a human, yet it can still become the mechanism by which someone discovers your work later through an assistant.
In that world, machine visits aren't necessarily noise, they're another distribution channel.
I'm curious if you're just tracking browser user agent, fingerprinting or some other method? For instance would someone using a tool to spider your site, would it be classed as an attack?
I thought about tracking all this stuff for my personal server, then realized that I wouldn't bother doing anything with the knowledge anyway.
try using captcha,
and puzzle for human viewers