Cloudflare announced a policy change that will block mixed-use web crawlers, combining search and AI training functions, from accessing ad-supported publisher websites starting September 15, 2026. This move is designed to give content owners more power to manage AI companies’ use of their content and encourages the division of crawler roles to protect intellectual property.
- Mixed-use crawlers blocked by default on ad-enabled sites starting Sept 15
- Publishers can charge AI companies via evolving Pay Per Use model
- Cloudflare partners with Ceramic.ai and You.com to pilot content monetization
What happened
Cloudflare has instituted a new policy requiring AI companies to separate their web crawlers used for traditional search from those deployed for AI training and agent purposes. Effective September 15, 2026, any crawler that mixes search and AI training uses will be blocked by default on websites that display advertisements unless the site owner opts otherwise.
This policy affects all new Cloudflare customers, new websites by existing customers, and all existing free-tier users. The change comes amid a significant shift in internet traffic, where automated bot access now surpasses human visitors, necessitating clearer control over how content is accessed and used.
Why it matters
Website owners have increasingly voiced concerns about their content being leveraged by AI companies without adequate compensation or control. Cloudflare’s move reflects a growing industry trend toward recognizing publishers’ intellectual property rights in the AI era and establishing sustainable practices for AI data consumption.
By blocking mixed-use crawlers by default, Cloudflare encourages AI firms to be transparent and deliberate about the purposes of their crawlers. The adjustment also addresses disparities in content access, noting that some search engines currently enjoy broader data reach, potentially impacting competition and fairness in AI development.
What to watch next
Publishers and AI companies will closely monitor the impact of Cloudflare’s new default crawler policies after the September deadline. Many will assess how separating crawler functions affects content discoverability, data quality, and operational costs for AI training.
Cloudflare’s pilot programs with platforms like Ceramic.ai and You.com—where publishers can monetize AI use of their premium content—may signal a new model for compensating content owners. These evolving Pay Per Use frameworks could set standards for future agreements between publishers and AI service providers.