Cloudflare Adds Search, AI Input and Training Signals to robots.txt

Cloudflare’s Content Signals Policy gives website operators a way to state what crawlers may do with content after they access it. The three signals cover search indexing, use as input to AI systems, and AI model training. The proposal fills a gap in robots.txt, which can guide access to pages but doesn’t describe how their contents may be used later.

That distinction matters as scraping grows more costly for publishers and can reduce the referrals and attribution they once received. The signals offer a shared, machine-readable vocabulary for expressing preferences while keeping pages accessible. They communicate those preferences; they do not technically prevent scraping.

What robots.txt controls

robots.txt is a plain text file hosted on a website. It implements the Robots Exclusion Protocol, allowing site operators to tell crawlers which parts of a site they may access. Many crawlers follow these instructions, but the file doesn’t compel every bot to comply.

For example, this directive invites all user agents to access every path:

User-agent: *
Allow: /

A user agent is the identifier a browser or bot supplies when requesting a resource. Here, * matches all user agents, and / refers to the whole site. Operators can narrow the instruction to a particular path or crawler. This example tells all user agents to skip the archives directory:

User-agent: *
Disallow: /archives/

And this one applies the same restriction specifically to Googlebot:

User-agent: Googlebot
Disallow: /archives/

Operators can also add comments after a # character. Crawlers ignore those comments, which makes them useful for human-readable explanations. Cloudflare’s own robots.txt file, for instance, includes a large block of ASCII art in comments.

The limitation is that access rules don’t express what a crawler may do with content once it has retrieved it. A site can permit crawling while still wanting to distinguish search indexing from AI-related uses. Cloudflare positions the Content Signals Policy as a way to state those separate preferences in the same file.

Three signals, with a neutral default

The policy defines three content signals: search, ai-input and ai-train. A site operator can mark each as yes or no. A crawler reading the signal is told that yes permits collection for that use and no says it isn’t permitted. If a signal is absent, the policy assigns it no meaning: the operator has neither granted nor restricted permission through that signal.

That neutral state lets publishers make a choice about one use without making assumptions about the others. In the policy’s definitions, search covers building an index and returning results such as links and short excerpts. It excludes AI-generated search summaries. ai-input covers feeding content into AI systems for uses such as retrieval-augmented generation, grounding, or real-time answers. ai-train covers training or fine-tuning models.

The policy’s explanatory comments also state that restrictions expressed through these signals are intended as express reservations of rights under Article 4 of the European Union’s 2019 copyright directive. The comments give automated systems context for interpreting the vocabulary; the actual preferences are set separately through machine-readable directives.

For example, a publisher could allow search, disallow model training, and leave AI input unspecified with this entry:

User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

Because the line omits ai-input, it makes no statement about that use. It doesn’t mean the operator has no preference; it means the preference hasn’t been expressed through this signal. The comma-delimited format lets a site state only the choices it has made.

Cloudflare’s rollout and setup options

Operators who already manage their own robots.txt can add the policy’s explanatory comments and then define their preferences. Cloudflare says it will also update the managed files already served for customers using its managed robots.txt feature. That feature is enabled for more than 3.8 million domains, whose operators have chosen to say they don’t want their content used for AI training. Cloudflare plans to add:

Content-Signal: search=yes, ai-train=no

Cloudflare won’t add an ai-input signal for these customers, because it says it doesn’t know their preference and won’t infer one. The distinction is consistent with the policy’s neutral default: an omitted signal carries no permission or restriction under the framework.

For free-plan zones without an existing robots.txt, Cloudflare will serve the human-readable policy comments. Those comments explain the signals but don’t include Allow or Disallow directives, or any actual content signals. Site operators must add their own preferences. Customers with an existing file won’t see it changed by this step. Free-plan users can disable the policy through the Security Settings or Overview sections of the Cloudflare dashboard.

Cloudflare also offers ContentSignals.org, where operators can generate text to copy into their file or deploy through a Cloudflare button. Another option is the managed robots.txt feature, which can express a preference against AI training.

Signals depend on crawler adoption

The policy’s practical reach depends on crawlers choosing to read and respect it. A content signal is a declared preference, not a technical barrier to a bot retrieving or reusing a page. Cloudflare recommends pairing the signals with WAF rules and Bot Management for publishers seeking stronger control over automated access. Its separate guidance on preventing web scraping describes the technical side of that problem.

The reason Cloudflare gives for introducing the signals is that data collection can impose costs on sites without returning the referral traffic or attribution that publishers historically gained from being indexed and linked. It points to the early blogging practice of linkbacks, as well as permissive licenses such as the MIT License and Creative Commons licenses, as examples of systems where attribution mattered. Cloudflare argues that scraped material can now be used in ways that compete economically with its original creators, leaving publishers to choose between restricting access and accepting less credit or traffic.

Cloudflare forecasts that bot traffic will exceed human traffic on the internet by the end of 2029, and that bot activity alone will surpass the sum of current internet traffic by 2031. Those are the company’s expectations, offered as context for why it sees the need for more ways to state reuse preferences. They aren’t measurements of the policy’s effectiveness.

To encourage broader adoption, Cloudflare is releasing the policy under a CC0 license, allowing anyone to implement it. The company says it will continue working on standards with the wider internet community. For publishers, the immediate decision is whether to state separate preferences for search, AI input and training—and whether to pair those signals with controls that can limit access. The next test is whether enough crawlers recognize the vocabulary for those preferences to shape how content is collected.