The rules of the web change today as Cloudflare cracks down on AI bots
Cloudflare's new rules distinguish search crawlers from AI training and agent bots
The rules governing how artificial intelligence companies crawl the web are changing from today, as Cloudflare begins enforcing new defaults designed to give website owners greater control over how their content is used.
From September 15, new domains joining Cloudflare will automatically block bots classified as being used for AI training or AI agents on pages displaying advertisements, while crawlers used for search will remain allowed by default.
Get updated faster and for FREE: Download the Gulf News app now - simply click here.
The changes will also apply to existing Cloudflare free-tier customers who have not changed their AI bot settings before the September 15 deadline.
The move could have significant implications for publishers, creators and AI companies as the battle intensifies over who can access online content — and what they can do with it.
Search, training or AI agent?
Rather than treating every AI crawler in the same way, Cloudflare now separates automated traffic into three main categories.
Search covers crawlers that index web content so users can later discover it through search.
Training includes crawlers that collect content to train or fine-tune AI models.
Agent covers automated systems acting on behalf of users in real time, including chat-fetching bots and browser-based AI agents.
Website owners can choose to allow each category, block it entirely or block it specifically on pages displaying advertisements.
Cloudflare argues the distinction matters because a publisher may want its articles to appear in search results while not wanting the same content collected for AI training without compensation.
Why September 15 matters
Cloudflare first announced the policy in July, but September 15 is the deadline for the new defaults to take effect.
For new customers and new sites added by existing customers, search crawling will be allowed by default, while training and agent activity will be blocked on pages carrying ads.
The same defaults will be applied to existing free customers who have not already selected their own preferences.
Users can still change the settings through their Cloudflare dashboard.
There is another potentially important consequence.
Cloudflare says crawlers performing more than one function will be subjected to the most restrictive applicable setting. That means a crawler used for both search and AI training could be blocked if a website has chosen to prohibit training.
Cloudflare specifically identifies Googlebot, Applebot and BingBot among crawlers that can have multiple purposes.
Why publishers care
The change comes as publishers increasingly question the trade-off between allowing AI companies to consume their content and the amount of traffic those systems send back.
Traditional search engines generally crawl websites, index their pages, and return links that can generate visits — and advertising or subscription revenue — for publishers.
Generative AI can work differently. An AI service may extract information from webpages and present an answer directly to users, potentially reducing the need to visit the source.
Cloudflare is consequently also giving some customers access to analytics showing how frequently individual AI operators crawl their websites compared with how much referral traffic they generate.
The company says its goal is not simply to block AI.
Instead, it wants website owners to distinguish between automated services that help people discover their content and those that consume it for other purposes.
For publishers, September 15 therefore represents another small but potentially important shift in the relationship between the open web and the rapidly expanding AI industry.
