Cloudflare's new default bot policy went into effect on September 15, 2026, blocking "mixed-use" crawlers — those that combine search indexing with AI training or AI-agent retrieval — by default on any page that carries ads. The change applies to new Cloudflare customers, newly created sites from existing customers, and every customer still on the free plan; paid customers who already configured their own bot rules keep those settings.
Under the new three-tier system, Cloudflare separates bot traffic into Search, Agent, and Training categories. Only Search stays allowed by default; Agent and Training are blocked unless a site owner opts back in. Because crawlers like Googlebot, Applebot, and Bingbot mix search indexing with model-training data collection, sites that keep Training blocked will also block those multi-purpose bots. Cloudflare framed the logic around a simple signal: an ad shows a site owner meant a human visitor to land there and see it.
The move comes alongside an expansion of Cloudflare's "Pay Per Crawl" marketplace into "Pay Per Use," aimed at compensating publishers when their content shows up in AI-generated answers rather than charging per individual fetch. Cloudflare has said bot traffic already makes up the majority of requests hitting the open web, a milestone the company hadn't expected to arrive until 2027.