The web has long operated on an unspoken agreement: publishers create content, and bots (from search engines to aggregators) index it for discovery. But with the rise of AI models scraping vast amounts of web data for commercial gain, that balance has started to look a little one-sided.
Now, Cloudflare is offering a bold new option for website owners: the ability to block, allow, or charge AI crawlers. And it’s doing so through a new initiative called Pay Per Crawl, now in private beta.
Let’s break down what it is, why it matters, and how it could reshape the way AI scrapes content from the web.
What is Pay Per Crawl?
Pay Per Crawl is a new tool from Cloudflare that allows publishers to set a price per crawl for AI bots and similar data scrapers. Rather than offering their content for free, or blocking AI bots entirely, websites can now charge for access.
This system is built around:
-
HTTP 402 “Payment Required” responses (a rarely used but officially recognised status code)
-
Verified crawler identity (via headers and keys)
-
Stripe-powered billing, with Cloudflare acting as the payment processor
Essentially, an AI crawler bot lands on your site → Cloudflare checks if it’s allowed → if set to charge, the bot sees a “payment required” message → the crawl proceeds once payment is confirmed.
Why do we want to charge AI crawlers now?
There’s growing frustration among publishers that AI models are training on their content without permission or compensation. Organisations like The Associated Press, TIME, The Atlantic, and Stack Overflow have all made headlines recently for striking paid licensing deals or for taking legal action.
Cloudflare’s Pay Per Crawl creates a middle ground. As Matthew Prince, CEO of Cloudflare, noted:
"This gives publishers fine-grained control over how and if AI companies access their content—and creates a direct path for monetisation."
Default blocking of AI crawlers
As part of this initiative, Cloudflare will now block AI crawlers by default, unless a site chooses to opt in or sets specific rules.
This is significant as Cloudflare sits in front of ~20% of the web, meaning a large portion of content will now become off-limits to AI bots unless they’re willing to pay or comply with publisher terms.
This new default stance flips the traditional expectation: AI bots must ask first, not assume access.
How Pay Per Crawl works at a glance
-
Crawler Registration: AI bots register with Cloudflare, using headers and cryptographic keys to prove their identity
-
Payment Rules: Publishers set a fee per crawl or grant free access to approved bots
-
HTTP 402 Trigger: If payment is required, bots receive a "Payment Required" response
-
Stripe Checkout: Payment is processed via Stripe and verified instantly
-
Access Granted: Once paid, the crawl proceeds as normal
Cloudflare has committed to releasing technical documentation and APIs for both publishers and bot developers.
Who is Pay Per Crawl actually for?
This initiative is ideal for:
-
Publishers who want to be compensated when AI uses their work
-
Developers and content creators who are tired of their content being scraped without consent
-
Enterprise teams who are concerned about data use and intellectual property
-
Anyone exploring new models for content licensing in an AI-driven world
Importantly Pay Per Crawl is open to all Cloudflare customers, though it’s in private beta for now.
A glimpse of the future?
Pay Per Crawl isn’t just a tool, it’s a signal. In an era where content is being consumed and reused by AI at unprecedented scale, platforms like Cloudflare are stepping up to offer new economic models and enforcement mechanisms.
Whether you believe AI should be able to train on public data or not, one thing is clear. The status quo is shifting. Tools like Pay Per Crawl give power back to the people who create the web's most valuable resource, its' content.
At Pixie, we've seen the negative impact that AI crawlers and bots can have on websites, and look forward to seeing how Pay Per Crawl develops as a tool for website owners.