Cloudflare’s New AI Policy Could Decide Whether AI Can See Your Website
Over the past year, businesses have been asking one important question:
“How do I get AI to cite my content?”
Cloudflare’s latest announcement raises an equally important one:
“Can AI even access my content in the first place?”
Cloudflare, the internet infrastructure company that powers roughly 20% of the web, has introduced a major change to how AI companies access publisher content. The new policy gives website owners greater control over whether AI crawlers can read their websites while also laying the groundwork for publishers to be compensated when their content is used by AI.
For businesses investing in SEO and AI visibility, this isn’t just another technical update. It could directly impact whether AI platforms can discover, understand, and recommend your content.
The relationship between publishers and AI is changing
For decades, the agreement between publishers and search engines was straightforward.
Search engines crawled websites, indexed their content, and sent visitors back to the original source.
AI has disrupted that model.
Today, AI assistants often read web content, generate complete answers, and satisfy the user’s question without sending nearly as much traffic back to publishers. As a result, many publishers have questioned whether AI companies are benefiting from their content without providing enough value in return.
Cloudflare’s new policy is an attempt to give publishers more control over that relationship.
Not all AI crawlers are the same anymore
One of the biggest changes is that Cloudflare now distinguishes AI crawlers based on what they’re designed to do.
Instead of treating every crawler the same, Cloudflare separates them into categories such as:
- Traditional search indexing
- AI assistants and autonomous agents
- AI model training
Beginning September 15, 2026, Cloudflare’s default settings for ad-supported websites will continue allowing traditional search engines to crawl content while blocking AI agent and AI training crawlers unless the website owner explicitly allows them. Crawlers that don’t clearly identify their purpose may also be blocked by default.
Website owners can review and change these settings at any time through their Cloudflare dashboard.
A new way for publishers to get paid
The most interesting part of Cloudflare’s announcement wasn’t the new blocking rules.
It was the introduction of a new approach to compensating publishers.
Cloudflare is expanding its earlier Pay Per Crawl initiative into a Pay Per Use model. Rather than charging AI companies every time they crawl a page, the new system is designed to compensate publishers when their content actually contributes to an AI-generated response.
Several companies, including Ceramic.ai and You.com, are already testing the model.
While it’s still early, this could mark the beginning of a new economic model for content creators in the AI era.
What businesses should do now
If your website runs on Cloudflare, now is a good time to review your AI crawler settings.
Here are a few practical steps to consider:
- Check whether AI crawlers are currently allowed or blocked.
- Decide which AI platforms you want accessing your content.
- Continue publishing original, authoritative content that AI systems can confidently cite.
- Monitor AI referral traffic alongside your traditional organic traffic.
- Stay informed as AI crawling standards and publisher compensation models continue to evolve.
For most businesses, blocking every AI crawler probably isn’t the best strategy. As AI becomes a larger part of how people discover businesses, being visible in AI-generated answers will become increasingly valuable. The goal is to strike the right balance between protecting your content and remaining discoverable.
This is about more than Cloudflare
Cloudflare’s announcement reflects a much bigger shift happening across the internet.
Publishers are no longer debating whether AI should use their content. Instead, they’re asking two new questions:
- Under what conditions should AI be allowed to access it?
- Should publishers be compensated when AI uses their work?
Those questions are shaping the future of online publishing, search, and AI.
As AI becomes one of the primary ways people discover information, publishers, technology companies, and AI providers are all working to redefine the rules that have governed the web for decades.
The bottom line
The conversation is no longer just about SEO or AI optimization.
It’s about content ownership, discoverability, and fair compensation.
Businesses that understand how AI crawlers access their websites, maintain strong technical SEO, and make informed decisions about crawler permissions will be in a much stronger position as AI continues to reshape search.
If your website uses Cloudflare, take a few minutes to review your AI crawler settings. You may have more control over your AI visibility than you realize, and those decisions could influence how easily customers discover your business through AI in the years ahead.
