Cloudflare Proposes Five Principles for Identifying and Managing AI Crawlers

AI search can give people answers without sending them to the websites that supplied the information. That creates a practical tension: faster access for users may mean fewer visits for publishers, weakening the revenue that supports original material. Cloudflare argues that clearer rules for AI crawlers could help address that tension while preserving useful access to web content.

The company’s proposal sets out five principles: crawler operators should disclose what their bots do; bots should identify themselves truthfully; each bot should have a declared, distinct purpose; bots should honor website preferences when technically feasible and proportionate; and operators should prevent excessive traffic and deceptive behavior. Cloudflare presents the principles as a starting point for discussion among publishers, creators, bot operators, and infrastructure companies.

The proposal focuses on AI crawlers because of generative AI’s effect on the web, though Cloudflare says the same practices can apply to other automated traffic. It is a framework for making crawler behavior easier to recognize and control, not a finished technical standard. Its practical value will depend on whether the groups involved agree on shared signals and put them into use.

Why crawler behavior needs clearer rules

AI chatbots and search summaries can help people find information more quickly. But when a summary answers a query directly, a user may have less reason to visit the source website. Publishers that depend on visits to sustain their work can then lose a route to supporting the creation of new material. Cloudflare points to a feedback loop: AI services need fresh, high-quality content, while publishers need an ongoing way to fund it.

Cloudflare says it has worked on bot management for years, helping customers handle unwanted and harmful automated traffic. More recently, it has introduced options for customers to charge AI crawlers for access or block them, published a verified bots policy, and proposed Web Bot Auth for verifying bot requests. The company says it has also acted against crawler behavior it considers deceptive.

Those measures give Cloudflare a direct role in how websites manage automated requests, but the proposed principles reach beyond its own customers. The company says it wants people to be able to see which AI bots follow the practices, whether fully, partly, or not at all. It also acknowledges that crawler operators are adapting to changing expectations and says many want to act responsibly while continuing to build their services.

Cloudflare frames the proposal as an invitation to discuss workable norms. Technical details and competing interests will complicate any agreement, and the company expects the principles to change as the technology and standards develop. The central aims are transparency, accountability, and respect for how content owners choose to allow access and use.

1. Disclose who operates each bot and what it does

Cloudflare’s first principle asks companies to make basic information about their AI crawlers easy to find. That information should cover the bot’s identity, the organization responsible for it, a contact for reporting problems, and the purpose for which it accesses content.

Identity details could include a user agent, the IP addresses used by the bot, or a cryptographic identifier. Purpose matters because a crawler might collect material for search, feed content into an AI service, or use it to train or fine-tune a model. A website operator can make a more informed access decision when these details are available together.

Cloudflare points to OpenAI’s published bot information as an example. Clear descriptions can help a site operator check whether a request appears to come from the named service and understand what the crawler is meant to do. That information can then inform choices expressed through mechanisms such as a robots.txt file.

Disclosure is useful only if it connects to what a bot actually sends. A company may publish a description, but site operators still need a practical way to match that description to incoming requests. That leads to Cloudflare’s second principle.

2. Make bots identify themselves truthfully

Cloudflare says a crawler should communicate its identity in its requests, for example through its user agent and traffic from IP addresses associated with that bot. The company regards these checks as imperfect, but as a useful interim measure while stronger verification becomes more widely available.

OpenAI’s GPTBot illustrates the approach described in the proposal. OpenAI publishes the expected user-agent details and describes the bot’s role in collecting material that may be used to improve and train its generative AI models. Cloudflare says it has observed GPTBot traffic from addresses OpenAI reports. Taken together, the published description and request signals give a site operator a basis for deciding whether to allow that traffic.

Cloudflare contrasts that with cases where identification is harder to verify. It says Anthropic publishes a user agent but, without other verifiable information, a site operator may have difficulty confirming which requests come from Anthropic. The company says xAI’s Grok bot does not identify itself, making it difficult for a website operator to recognize and block its requests. These examples reflect Cloudflare’s assessment of the services and practices described in the source article.

User agents and IP address lists have a weakness: another party can copy or spoof them. A deceptive crawler could present itself as a bot that a website permits, then avoid the site’s stated preferences. Cloudflare therefore argues that AI bots should ultimately use cryptographic verification based on an accepted standard.

Its proposed option is Web Bot Auth, which uses cryptographic signatures in HTTP messages to help verify that a request came from an identified automated bot. Cloudflare’s implementation draws on proposed IETF directory and protocol drafts. The proposal is still in development, so the details may change.

Cloudflare reports early support from Vercel, which announced support for Web Bot Auth in its bot verification, and from OpenAI, whose ChatGPT agent signs requests using Web Bot Auth alongside the HTTP Message Signatures standard. Wider adoption would give websites a stronger signal than a self-declared name or address list, though the source does not establish how quickly that adoption will happen.

3. Give each bot one declared purpose

Cloudflare’s third principle is about separating access by purpose. It describes three categories as a starting point: Search, for building an index and returning links or short excerpts; AI-input, for using content in real-time answers, retrieval-augmented generation, or other grounding; and Training, for training or fine-tuning AI models.

The distinction matters when one crawler’s activity serves several purposes. If search indexing and model training share a bot without a clear boundary, a website operator may have to choose between blocking useful search access and allowing a use of its material that it would otherwise decline. Cloudflare says crawlers should not combine purposes in a way that prevents site operators from making deliberate access decisions.

The company illustrates the issue through a hypothetical publisher, Vinny. OpenAI identifies OAI-SearchBot as a crawler for surfacing websites in ChatGPT search features, while a separate bot, GPTBot, handles crawling that may support model training. If those roles remain separate, Vinny can allow search crawling while restricting training access, assuming the crawler descriptions and preferences are honored.

Cloudflare’s example of Googlebot raises a different issue. The source says Googlebot gathers material for traditional search indexing and that the same content may also inform AI Overviews and AI Mode. A publisher may want search results to send people to its site while declining to provide material for an AI-generated answer that could satisfy the query without a visit. Cloudflare argues that this can leave site operators with an awkward choice between losing search visibility and accepting AI use tied to search.

The company notes that the Google-Extended control it describes applies to model training, not to AI Overviews attached to search. It also points to snippet controls as a possible measure, while saying that these can affect how content appears in search. The concern is that high-ranking, useful material may help produce answers that reduce the visits that support the publisher who created it.

Cloudflare also challenges the argument that combining purposes necessarily cuts crawler costs. It compares two requests for different tasks with making separate calls to order two pizzas: the extra connection may add some overhead, but the work needed to prepare both items remains. In the crawler case, the company argues, content still needs processing for each purpose, so separating requests does not eliminate that work; the cost of an additional connection is small by comparison.

As context for its view on request costs, Cloudflare says its network handles an average of 84 million requests per second. The company also recognizes that additional crawls impose costs on websites. Under its proposal, a site operator can weigh those costs against the value of a particular crawl when the bot’s purpose is clear.

4. Follow preferences expressed by websites

Website operators need ways to state how bots may access and use their material. Cloudflare points to the established robots.txt mechanism and to newer proposals such as HTTP headers that could carry machine-readable preferences for content.

The proposal asks bot operators to make a good-faith effort to retrieve robots.txt, follow the rules it contains, and comply with RFC 9309. It also asks them to track work on a draft vocabulary that aims to give site operators more detailed ways to express preferences in these files.

Other standards under discussion could let owners attach preferences to individual formats or assets. HTTP headers are one option for declaring preferences at the server level. Cloudflare says crawler operators should be prepared to support such approaches as they mature. How these signals will fit together—and how consistently different crawlers will interpret them—remains an open implementation question.

5. Keep traffic reasonable and avoid deception

The final principle asks companies to keep their crawlers from overwhelming websites and to avoid tactics that disguise what the bots are doing. Operators should monitor their systems for security problems and fix vulnerabilities that could put websites or users at risk. Excessive traffic that harms a site’s performance is outside the behavior Cloudflare considers acceptable.

The proposal also rules out stealth crawling: for example, changing a user agent to evade detection, shifting source ASNs to conceal activity, or ignoring robots.txt. Cloudflare argues that evasion would defeat the earlier principles by making disclosure, identification, and website preferences ineffective. The source describes an earlier case in which the company reported and acted on stealth crawling.

What comes next

Cloudflare says the proposal needs input from the people affected by crawler behavior, including content owners, AI companies, and infrastructure providers. It also describes work with rights holders, AI companies, policy-makers, and regulators, while arguing that neutral technical forums can help participants develop shared standards. The IETF is one example: its current work addresses some of the technical questions, though Cloudflare says those efforts do not cover every concern raised by the proposal.

The five principles point toward a practical test for AI crawlers: can a website operator tell who is making a request, what the crawler intends to do, and whether the operator’s preferences are being followed? Public disclosures and declared purposes provide the information; verifiable identity and preference signals make it actionable; reasonable traffic and honest behavior help keep the system usable.

That framework will need to evolve as crawler techniques and content uses change. Its success will hinge on whether operators adopt shared verification and purpose signals, and whether site owners can express distinct choices without losing access to services they want. Cloudflare’s proposal makes that coordination the next step in balancing AI access with the continuing production of web content.