Google Search and AI Crawling: Why Publishers Need Separate Crawlers

Google’s search crawler gives the company access to far more web pages than competing AI crawlers, Cloudflare says, while publishers have strong reasons to keep that crawler enabled: it helps people find their sites through search. That combination sits at the heart of the UK Competition and Markets Authority’s proposed rules for Google. Cloudflare supports giving publishers more control over how their work appears in Google’s AI features, but argues the proposal won’t deliver that control unless Google separates search crawling from AI crawling.

The distinction has practical consequences. Publishers may want their pages indexed so they can receive search referrals, while declining to let the same material ground AI-generated answers. If one crawler serves both purposes, blocking it can put search visibility at risk. Cloudflare’s proposed remedy is to give each purpose a distinct crawler, so publishers can make separate choices about search and AI access.

Why search access and AI access are entangled

In January 2025, the Digital Markets, Competition and Consumers Act 2024 took effect, giving the CMA a way to designate businesses with substantial, entrenched market power as having Strategic Market Status, or SMS. The designation allows the CMA to impose conduct requirements aimed at improving competition in digital markets.

In October 2025, the CMA designated Google as having SMS in general search and search advertising, citing a 90 percent share of the UK search market. The designation covers AI Overviews and AI Mode, giving the CMA authority to set requirements for Google’s search services and related features. Requirements imposed at the end of the process would be enforceable.

The CMA’s consultation, published on January 28, 2026, proposes rules intended to address publishers’ limited choice, visibility and attribution when Google uses their material in AI-generated responses. The proposals apply broadly to publishers: parties that make content available on the web.

The CMA describes the underlying imbalance plainly: publishers have little practical room to block Google’s general search crawler, but Google can use material gathered through search for its generative AI features and wider AI services. Those uses include grounding answers in current web information, such as responses to live queries through AI Overviews and AI Mode.

That setup can put a publisher in a difficult position. Blocking Googlebot may reduce the chance that a site appears in search results and receives referral traffic. Leaving it open, however, can also let Google use the content in AI experiences that may send little traffic back to the source. Cloudflare argues that those experiences can compete directly with publishers by reproducing their material, often without attribution or payment.

For publishers whose business depends on advertising, the tension matters commercially. Search referrals can bring visitors who view ads on a publisher’s site. When an AI response answers a query without a visit, that route to an audience may weaken. The source article does not quantify the effect on publisher revenue, but it argues that the risk compounds when Google can draw on content that other AI developers must negotiate to access.

The crawler data behind Cloudflare’s argument

Cloudflare’s measurements suggest that Googlebot reaches more distinct pages than the other crawlers it compared. Across a two-month observation period, Googlebot crawled roughly 8% of the unique URLs sampled on Cloudflare’s network. The reported figures compare the number of unique URLs seen by Googlebot with the numbers seen by other named crawlers:

• About 1.70 times as many as ClaudeBot and 1.76 times as many as GPTBot.
• 2.99 times as many as Meta-ExternalAgent and 3.26 times as many as Bingbot.
• 5.09 times as many as Amazonbot and 14.87 times as many as Applebot.
• 23.73 times as many as Bytespider and 166.98 times as many as PerplexityBot.
• 714.48 times as many as CCBot and 1,801.97 times as many as archive.org_bot.

These are Cloudflare’s observations from its network over the period described, not a complete count of all web crawling. The figures compare distinct URLs reached, rather than crawler traffic volume or the importance of the pages each bot visited. They show a large access gap in this dataset; by themselves, they don’t establish why every crawler differs in reach.

Cloudflare points to another measure of publisher choice: site owners’ crawling rules. Its analysis says websites are much less likely to disallow Googlebot in robots.txt than other crawlers. That pattern fits the crawler’s dual role. Publishers may permit it because they want search referrals, even when they would prefer to restrict AI uses.

Robots.txt communicates a site’s crawling preferences, but it doesn’t technically prevent a crawler from accessing pages. For tighter control, site operators can use a web application firewall, or WAF, to block selected traffic. Cloudflare compared blocking behavior among customers using AI Crawl Control, its crawler-blocking tool integrated into its application security suite. Between July 2025 and January 2026, nearly seven times as many sites actively blocked other popular AI crawlers, including GPTBot and ClaudeBot, as blocked Googlebot and Bingbot.

Cloudflare notes that Bingbot, like Googlebot, combines search and AI crawling, although it has a much smaller share of search. The comparison helps explain why publishers might treat the two differently from crawlers associated mainly with AI: search access can bring visitors, while an AI crawler may offer no comparable referral benefit.

What the proposed rules would change

The CMA’s proposed publisher requirements would require Google to give publishers “meaningful and effective” control over whether their material is used in AI features such as AI Overviews. Google would also be barred from weakening those controls, for example by deliberately lowering the search ranking of publishers who use them.

The proposal also calls for clearer documentation describing how Google uses crawled content for generative AI, and what each publisher control covers. On attribution and measurement, Google would have to provide effective attribution and detailed, disaggregated engagement data. The proposed data includes impressions, clicks and “click quality,” which could help publishers assess what commercial value they receive when their content appears in AI-generated search summaries.

Cloudflare welcomes the focus on choice, transparency and attribution, but says controls designed and administered within Google’s own services leave publishers dependent on Google’s definitions and technical implementation. A publisher might signal a preference, yet still have limited ability to verify how it is applied or enforce it directly at the point of access.

The company also says it has heard from customers that existing Google controls, including Google-Extended and nosnippet, have not prevented content from being used in ways publishers can’t control. Those mechanisms, it argues, also don’t create a way for publishers to negotiate payment. These are customer reports presented by Cloudflare; the source article provides no independent assessment of how often the controls fail.

Why Cloudflare wants separate crawlers

Cloudflare’s broader principle is that AI crawlers should have a distinct purpose and identify that purpose, so website operators can decide who may access their content and why. It says Googlebot is used for search indexing as well as AI training and inference or grounding. By contrast, Cloudflare describes leading competitors such as OpenAI and Anthropic as using separate crawlers for their different purposes.

Under Cloudflare’s proposal, Google would split Googlebot into purpose-specific crawlers. A publisher could then allow the crawler used for conventional search indexing while blocking the crawler used to gather material for generative AI. The key change would happen before access: a site could set different conditions for different uses without having to sacrifice search visibility as the price of refusing AI access.

The CMA considered crawler separation an intervention that could be as effective as its proposed approach, but rejected a requirement after Google argued that the change would be too onerous. Cloudflare counters that Google already operates nearly 20 other crawlers, making separation technically feasible. It also says separate crawlers need not increase the total load on websites and could reduce it if publishers block AI crawling.

That claim addresses a practical concern: adding crawler identities could sound like adding more visits. Cloudflare’s argument is that separation changes how access is labeled and controlled, while publishers’ choices determine whether AI crawling continues. The source does not provide a technical implementation plan or independent cost estimate, so the difficulty and expense of the change remain matters for the CMA’s assessment.

Cloudflare also frames separation as a competition measure. If Google can use the crawler it relies on for search to collect content for AI services, while publishers can’t realistically block it, Google may have an access advantage over AI developers that need separate permission. A clearer distinction could give publishers a basis for setting access terms and put competing developers on more comparable footing.

The article says Daily Mail Group, the Guardian and the News Media Association have expressed public support for crawler separation. Cloudflare’s position is that such a requirement wouldn’t disadvantage Google or undermine AI investment; it would curb the advantage it says comes from combining dominant search access with AI use.

The CMA’s consultation leaves open what final requirements will look like. Its proposal could improve publisher controls, explain Google’s practices and give sites better engagement data. Cloudflare’s critique is that those measures leave the central access decision inside Google’s system. For publishers, the decisive test is whether they can keep search indexing while independently refusing AI use. Separate crawlers would make that choice legible at the point where content is collected—and would give the next round of AI content negotiations a clearer starting point.

Cloudflare says it plans to provide evidence during the consultation process. The eventual requirements will determine whether publishers receive controls they can apply and verify themselves, or tools whose practical reach depends on Google’s implementation. The distinction will shape how content owners weigh search visibility against AI access, and whether smaller AI developers can compete for material on comparable terms.

CMA consultation announcement · Google SMS decision · Cloudflare’s AI bot principles · Proposed publisher requirements · Cloudflare on crawler traffic · Consultation introduction · Robots.txt explainer · AI Crawl Control · AI Crawl Control with WAF · Publisher conduct requirements · Google crawler overview