Once Unimaginable, Publishers Are Preparing to Opt Out of Google Search
The nuclear option is gaining traction as web traffic collapses and Google refuses to negotiate with content creators
As publishers weigh blocking the Google crawler, the quality of Google Search results could decline.
This story was originally published in On Background with Mark Stenberg, a free, weekly newsletter that explores the key themes shaping the media industry. You can sign up for it here.
For decades, publishers have done everything in their power, from the legal to the not-explicitly illegal, to rank as highly in Google Search as possible. For many websites, traffic from the search engine was their single greatest source of audience and, as a result, revenue.
Now though, a handful of influential players in the digital media ecosystem have begun moving in the opposite direction, laying the groundwork for what was once unthinkable: removing themselves from Google Search.
Last week, the content delivery network Cloudflare, which hosts roughly one-fifth of the websites in the world, gave Google an ultimatum.
Beginning Sept. 15, all new websites signing up for Cloudflare, as well as all the customers on its free tier, will have the default settings in their bot management protocol set to block “multi-purpose crawlers” on any webpage that has ads. This means that any crawler that scrapes for both search indexing and AI training will be turned away at the door, unless the site owner decides otherwise.
“We’ve been clear about what we want,” said Cloudflare chief strategy officer Stephanie Cohen. “We want a technical solution that allows you to be discoverable without having to give your content away for free.”
While a handful of crawlers fit this description—Apple and Bing, among others—the primary, unnamed target of this action is Google, which infamously uses one crawler to both index sites and train its AI models.
In doing so, Google forces publishers to make an impossible choice: They either allow both functions, enabling Google to scrape their content to train the AI products that are regurgitating their data without compensation; or they shut off both functions and disappear from Google Search, presumably losing their largest source of traffic in the process.
To be fair, Google recently introduced an option called Google Extended, which nominally allows publishers to opt out of AI training without disappearing from Search. But publishers are wary that the program will penalize their search visibility, according to executives at two media companies.
Similarly, new controls introduced in the U.K. enable publishers greater agency over what Google can do with their scraped content, but publishers are similarly hesitant to trust a solution that relies on the discretion of Google rather than an outright block of its crawler.
“We provide web publishers with clear, granular controls to manage their content, including Google-Extended for AI training and a new Search Console control we are testing for generative AI Search features, neither of which impact traditional Search visibility,” said a Google spokesperson in a statement. “We are committed to designing AI experiences that highlight the web, drive valuable traffic to publishers, and provide the insights they need to succeed.”
Historically, abandoning Google Search would have been commercial suicide, according to SEO consultant Lily Ray. It is simply too valuable of a source of audience discovery and traffic.
“It’s a really hard tradeoff. Some publishers have already blocked OpenAI until it strikes partnerships, but with Google it’s hard,” Ray said. “Google is a different conversation because it has so many more users than other AI firms.”
But the gradual erosion of search traffic in recent years has, paradoxically, given publishers more agency to consider walking away from the platform.
USA Today Inc., which encompasses not just USA Today but a nationwide network of news sites, is weighing its options on the matter, according to CEO Mike Reed.
The company, like many others, has responded to declines in search traffic by bolstering audience from other sources, like newsletters, social media, and events. Its traffic has remained relatively stable in recent years, hitting its goal of 1 billion pageviews every month for the last three years, according to Reed.
Still, its monetization strategy going forward as it relates to AI will come from licensing agreements, which the company has struck with Meta, Microsoft, and Amazon, among others. Google, unlike its hyperscaler peers or pure-play AI firms like OpenAI and Anthropic, has not struck any licensing deals with any publishers.
As a result, USA Today Inc. is prepared to delist from Google in the next six to twelve months, according to Reed. Likewise, the creator network Beehiiv announced in a recent partnership with Cloudflare that its network of creators now has the ability to block the Google crawler.
“I wouldn’t call it a big decision because we’re blocking other crawlers,” Reed said. “For those with licensing agreements, they get our content. For those without, we block them.”
While USA Today Inc., Beehiiv, and Cloudflare are the first major players to take this step, executives at every major media company have a model for what it would look like if they blocked the Google Bot, according to one executive who wished to remain anonymous because of business engagements with Google.
The decision of when to make that call is mostly a matter of math: Once search traffic drops below a certain threshold, the value of appearing there becomes less than the value of withholding that content as a bargaining tactic. And to be clear, every publisher would prefer Google to come to the negotiating table.
But the alternative is bleak. If Google refuses to strike licensing agreements, more premium publishers could follow suit in denying its crawlers access. Doing so would degrade the quality of search results, making it harder for consumers to find accurate news. Just as social media has become a cesspool of misinformation, the open web could be overcome with untrustworthy content too.
“Our content is showing up and powering generative experiences on Google at least hundreds of millions of times a day,” said the media executive. “If we weren’t in it, every one of those experiences would be worse.”
Still, executives at other media companies are not so sure. The existing deal is a lose-lose situation for publishers: Allowing Google to scrape their content for free is self-defeating, but opting out has no clear material benefit either, according to one media operator who also wished to remain anonymous because of business dealings with the company.
Plus, if a company opts out now, there is no telling what negative externalities that could create in the future, the operator added. Google is not compensating publishers for their data now, but if it does in the future, could those payouts be harmed by restricting the Google crawler in the interim? The options are unappealing.
Nonetheless, that publishers are considering opting out of Google Search marks a symbolic milestone in the lifecycle of the open web. The posturing is all part of a broader negotiation, to be sure, but the threats are no longer empty.
No publisher wants to make do without Google, but many are increasingly prepared to do so. Google, on the other hand, has no such backup plan.
Editor’s Note: This story has been updated to include information on Google Extended and the new data controls available to websites in the U.K.