Reddit blocks search engines other than Google from retrieving its content
According to a report from 404media, Reddit, a social message board site, has blocked search engines like Bing and DuckDuckGo from crawling its content, arguing that only Google can retrieve Reddit content and display it in search results.
Google's Exclusive Reddit Access
According to 404media, Reddit only allows Google to crawl its content, and uses robots.txt to block Bing, DuckDuckGo, Mojeek, Qwant, and others.
When I searched on Bing in my local environment, I did get no search results, but I did get some hits for content on Reddit.
However, this only shows information that was crawled in the past, so I'll go to 'Tools' and specify the time period as 'within one week.'
I got 'No results.'
I'll also check with Google.
Then, the most recent information was displayed as usual.
The reason behind the block is believed to be a contract between Google and Reddit to use its content to train AI. Google is believed to be paying Reddit $60 million a year in licensing fees.
However, in response to a request for comment from The Verge, Reddit spokesman Tim Rathschmidt said, 'This matter is unrelated to our partnership with Google. We are in negotiations with multiple search engines about the use of Reddit content, but we have not reached agreements with all of them because some are unable or unwilling to make enforceable commitments, including regarding the use of AI.'
Reddit is now blocking major search engines and AI bots — except the ones that pay - The Verge
https://www.theverge.com/2024/7/24/24205244/reddit-blocking-search-engine-crawlers-ai-bot-google
Related Posts:
in Web Service, Posted by logc_nt