Over 99% of website traffic comes from bots; a battle against AI scrapers that load pages 214 times for every one human visit.



Nick Gray, who runs the donor information database 'PatronView,' has revealed his battle against AI crawlers and scrapers that have been swarming his website for about a year. Comparing the number of pages the server delivered with the access analytics figures, he found that for every one time it was recorded as a 'human page load,' there were approximately 214 page loads performed by AI.

99% of My Website Traffic Is Bots | PatronView

https://patronview.com/news/99-percent-of-my-website-traffic-is-bots/




PatronView features approximately 1.5 million personal profiles based on publicly available information. Because it handles a large amount of public data, it is easily accessed by search engines and various crawlers, and Mr. Gray has been continuously working to counter the massive influx of traffic from SEO crawlers and bots.

In one week, PatronView's servers delivered approximately 1.28 million full pages. On the other hand, the JavaScript-based web analytics software 'Plausible Community Edition' recorded only 5,977 page views. A simple calculation shows a ratio of approximately 214 to 1, and Gray explains that even the statement '99% of the access is from bots' is an understatement.

The following diagram visualizes the ratio of approximately 214 to 1, with each blue square representing a page load measured by a human. The remaining gray areas represent access by bots.



The most notable surge in access occurred on April 22, 2026. Approximately 3.62 million requests were made in a single day, distributed across approximately 361,000 IP addresses. As the majority originated from China, Gray blocked access from China the following day, April 23. Immediately afterward, the number of requests decreased significantly.



Among the AI crawlers, Anthropic's 'Claude-SearchBot' retrieved approximately 420,000 pages in one week, while 'Claude-User,' which accesses websites at the request of users, only made 12 visits. Comparing the two, Gray states that this works out to approximately 35,000 pages retrieved per visitor. Amazon's 'Amzn-SearchBot' also made a large number of visits, but no visitors to PatronView were observed. Meanwhile, Googlebot retrieved 46 pages per visitor, and Bingbot retrieved 406 pages.

When Gray compared the ratio of the number of pages retrieved by each crawler to the number of user-generated visits, he found a significant difference between AI search crawlers and traditional search engines.



Gray used Cloudflare's Web Application Firewall to block AI crawlers that he determined would not lead to user traffic, required authentication via Cloudflare's 'Managed Challenge' for suspicious access, and set limits on the number of accesses that retrieved a large number of pages in a short period of time. He allowed search crawlers that send users, such as Googlebot and Bingbot, and used 'how many pages were retrieved' and 'how many visitors were sent back' as criteria for evaluation.

However, when developer Kaka Ruto tried to access PatronView, he was denied access using his own web retrieval tool, but was able to retrieve the page when he switched to 'Chrome DevTools MCP,' which uses an AI agent to control the Chrome browser. This suggests that complete blocking is difficult.




Gray says that as the cost of scraping continues to fall, a system like 'Pay Per Crawl,' where crawlers pay for obtaining pages, may be necessary in the future. Until a market like Pay Per Crawl is established, he says they will continue to operate under the policy of 'blocking crawlers that don't send any visitors.'

Related Posts:

in AI, Posted by log1d_ts