radioinfo has previously covered the growing trend of click theft, where AI Bots scan websites and summarise the information in a way that searchers do not need to click through to the original website.
Click theft deprives websites of traffic and reduces the amount of eyeballs on advertising on its pages.
To help preserve the integrity of content creators’ work, Cloudflare has launched Bot Fight Mode to block AI bots from scraping publications.
When enabled, the bot fight mode:
- Identifies traffic matching patterns of known bots
- Issues computationally expensive challenges in response to these bots
- Notifies Bandwidth Alliance partners (if applicable) to disable bots
The popularity of generative AI has made the demand for content used to train models or run inference on skyrocket, and, although some AI companies clearly identify their web scraping bots, not all AI companies are being transparent. Google reportedly paid $60 million a year to license Reddit’s user generated content, and most recently, Perplexity has been accused of impersonating legitimate visitors in order to scrape content from websites. The value of original content has never been higher.
Last year, Cloudflare announced the ability for customers to easily block AI bots that behave well. These bots follow robots.txt, and don’t use unlicensed content to train their models, but other bots do not follow the rules. Cloudflare has added a one-click to block all AI bots. It’s available for all customers, including those on the free tier.
In June, AI bots accessed around 39% of the top one million Internet properties using Cloudflare, but only 2.98% of these properties took measures to block or challenge those requests. Dodgy bot operators attempt to appear as though they are a real browser by using a spoofed user agent. Cloudflare has monitored this activity over time, and its global machine learning model can accurately recognise this activity as a bot, even when operators lie about their user agent.
A full explanation of Cloudflare’s system is here. While there have been many individual tools to counter AI bot scraping, this is the first time that an internet wide enterprise level security platform has offered such a sophisticated feature.
An example of how the bots work is below. We asked AI for a summary of radioinfo reports on click theft. It scanned 4 of our articles and generated a summary that would be enough for a quick overview, withn no need to click through to our articles.
Sites that use Cloudflare can turn on the generative AI bot scraping protection within their account to activate the protection feature.
Journalists are also concerned about the changes AI is bringing to their work. The journalists union, MEAA, is campaigning for comprehensive policies to regulate AI through an Australian AI Act and for a levy on big tech developers who must be made to pay for the work they’ve stolen to train AI. A recent survey of MEAA members found that 71% are “extremely concerned about the loss of human-led creativity.”
At the same time, AI has turbocharged the proliferation of scams, deepfakes, disinformation and harmful content, whilst the big AI developers like Amazon and Meta are doing nothing to stop it. That’s why 69% of MEAA members strongly agree that greater government intervention is needed to regulate AI.
“Who will tell Australian stories if there are no artists, actors, journalists, or writers? Australians need to be able to trust that the news they read, the film and television they watch, and the music they listen to has been produced by artists and journalists whose work has not been compromised by AI,” says the union.
In all our publications, we employ real people with professional industry understanding, deep connections and long memories, who write real, researched stories. We don’t use AI or automated bots to generate stories. If we use AI for summaries or picture montages, we declare it. You can support our journalism by paying for deep access to our reporting via an annual subscription that supports our independent audio industry trade publication.
Related article: