Following our previous report on Cloudflare’s efforts to block AI bots from scraping publisher content and the click theft that results from the practice, the company has now moved to help publishers monetise AI bot scraping.

Cloudflare has introduced an option to force AI bots to pay for the content they scrape.

The functionality is in beta test mode and publishers can opt in through their Cloudflare account.

This is an important move for broadcasters, podcasters and publishers, who have, until now, been caught in the lose-lose situation of blocking content and leaving the field to the publishers of fake news and misinformation, or keeping their content searchable but leaving themselves open to click theft.*

Cloudflare is using the HTTP 402 protocol to implement the functionality. The company says:

“Many publishers, content creators and website owners currently feel like they have a binary choice — either leave the front door wide open for AI to consume everything they create, or create their own walled garden… 

“After hundreds of conversations with news organizations, publishers, and large-scale social media platforms, we heard a consistent desire for a third path: They’d like to allow AI crawlers to access their content, but they’d like to get compensated. Currently, that requires knowing the right individual and striking a one-off deal, which is an insurmountable challenge if you don’t have scale and leverage… Instead of a blanket block or uncompensated open access, we want to empower content owners to monetize their content at Internet scale.”

Pay per crawl integrates with existing web infrastructure, leveraging HTTP status codes and established authentication mechanisms to create a framework for paid content access.  Each time an AI crawler requests content, they either present payment intent via request headers for successful access (HTTP response code 200), or receive a 402 Payment Required response with pricing. Cloudflare acts as the Merchant of Record for pay per crawl and also provides the underlying technical infrastructure.

Pay per crawl grants domain owners full control over their monetization strategy. They can define a flat, per-request price across their entire site. Publishers will then have three distinct options for a crawler:

• Allow: Grant the crawler free access to content.
• Charge: Require payment at the configured, domain-wide price.
• Block: Deny access entirely, with no option to pay.

Which AI scraping bots are behaving?

This site currently tracks behavious in an attempt to answer that question.

radioinfo is testing this function and will report back on what we find.

In another related piece of research, Website Planet conducted a small research project to quantify click theft via Google’s AI Overviews. Out of 100 Google searches, they found that 39% returned with AI summaries. They also found thay 28% returned results with no AI-generated content, and the remaining 33 searches displayed multiple source links with thumbnails instead of summarizing a single page.

The Rise of “Zero-Click Searches”

According to Google’s Q1 2025 earnings report, AI Overviews are used by over 1.5 billion users every month. This, combined with featured snippets, knowledge panels, and other rich SERP elements provide users with instant answers, causing a significant decline in the share of searches that lead to website clicks.

Recent analyses from First Page Sage’s 2025 Google Click-Through Rate (CTR) report reveal that AI overviews now appear on roughly 31% of search result pages, seen most often in general information queries. 

This is consistent with Website Planet’s findings, which showed that 39 AI Overviews out of 100 Google searches and a near-even split between summary-based, multi-source, and traditional result formats.

The website Planet report also found that AI Overviews themselves have CTRs ranging from 29.5% to 38.9%, indicating that users often find their answers directly on the search results page without needing to click through.

Research conducted by Ahrefs suggests that AI Overviews significantly reduce click-through rates to traditional organic listings, especially for non-branded, informational queries. Their analysis of 300,000 keywords revealed a 34.5% drop in click-through rates for the top organic search result on informational queries when AI Overviews are present.

In all our publications, we employ real people with professional industry understanding, deep connections and long memories, who write real, researched stories. We don’t use AI or automated bots to generate stories. If we use AI for summaries or picture montages, we declare it. You can support our journalism by paying for deep access to our reporting via an annual subscription that supports our independent audio industry trade publication.

About the Author:

Steve Ahern is the founding editor and publisher of this trade journal.

*Click theft is a term coined by radioinfo’s Steve Ahern to describe the practice of AI content scraping which presents summaries of content that result in searchers no longer needing to click through the the publisher’s website to get the answer they are seeking.