Cloudflare will change its default bot-traffic rules on September 15, 2026, forcing AI companies to separate crawlers used for search from those used for AI training and autonomous agents. After that date, ad-supported pages will block "mixed-use" crawlers by default, while Search bots stay allowed — a policy applying to all new customers, new sites, and every existing free-tier user.
The move follows Cloudflare CEO Matthew Prince's warning that automated bot traffic has overtaken human traffic online, and that "AI was taking everything and sending back nothing, presenting an existential threat to website owners." Cloudflare says over half of AI crawler traffic is spent simply re-fetching pages that have not changed, adding load without giving publishers anything in return.
Alongside the deadline, Cloudflare expanded its monetization tools, launching "Pay Per Use" next to its existing "Pay Per Crawl" marketplace so publishers can charge AI firms for access. Ceramic.ai and You.com are the first launch partners, paying publishers when their content shows up in AI search results. Cloudflare sits in front of more than 20% of all web domains, giving the policy outsized reach across the internet.