TIME magazine delivers different pages to humans and AI crawlers, and AI-targeted advertisements have also been spotted.

Software developer Vincent Schmalbach has confirmed that TIME magazine is displaying its regular website to human readers while delivering a separate, ad-supported version to some AI crawlers. The AI-friendly pages are written in Markdown format, which is easier for the AI to process, and include ads that are not present on the human-facing pages.

TIME Is Serving AI Bots a Different Website, With Ads Built In - Vincent Schmalbach

https://www.vincentschmalbach.com/time-serves-ai-bots-a-different-website/

AI services such as ChatGPT and Claude may use AI crawlers to traverse websites and collect information necessary for search results and responses. However, typical web pages contain many images, menus, and display code, which is not always efficient for text-based AI.

TIME delivers a Markdown version of its articles and links to some AI crawlers. When Mr. Schmalbach investigated by changing the 'User-Agent,' which indicates the type of browser and crawler used to access the site, he found that regular browsers returned approximately 303KB of HTML, while ClaudeBot and PerplexityBot delivered approximately 13KB of Markdown.

The actual response header looks like this. The 'content-type,' which indicates the type of content, is set to 'text/markdown.'

The AI-friendly pages included descriptions indicating Mobian, an AI advertising technology company, as well as unique ad display IDs for each access. Feature pages and other sections included sponsor information in a question-and-answer format, seemingly intended to make it easier for the generating AI to read advertiser information.

The advertisements clearly indicate that they are sponsored content. However, the same ads are not displayed on the human-facing pages, making it difficult for human readers to see the content and display methods of the AI-facing ads.

According to TIME, on many days, the number of visits by AI bots exceeds the number of visits by humans. Schmalbach says this is an early example of what will happen when AI models become the primary readers of the web.