Cloudflare and OpenAI pilot helps ChatGPT discover hundreds of millions of new web pages

Cloudflare and OpenAI published early results from a joint research pilot launched in July, revealing that network signals from Cloudflare significantly improve how AI systems discover web content. The initiative aims to help AI find useful information more efficiently while giving website owners greater control over how their content is accessed and represented.

The pilot, which involved approximately one percent of Cloudflare’s total network, has allowed OpenAI to discover and index hundreds of millions of new web pages. According to Cloudflare, these previously unknown pages are now part of OpenAI’s search index, providing ChatGPT with more sources to draw upon. This expansion helps the AI system provide more relevant and timely answers to user queries by accessing a broader range of web content.

Improving Search Efficiency and Accuracy

Cloudflare’s network provides specific signals that assist OpenAI’s search crawler in identifying useful pages across the web. Once discovered, OpenAI independently evaluates and ranks these pages based on relevance and helpfulness. This process creates a feedback loop designed to improve both the discovery mechanism and the overall quality of information presented to users. By linking back to original sources, ChatGPT Search also allows users to explore the content directly, connecting them with the publishers behind the information.

The company noted that finding a URL is only the first step; systems must also determine if a page is accessible, distinct, and useful. The pilot helped researchers understand how automated traffic can distort page value. For example, bots repeatedly requesting nonexistent files or sites redirecting all addresses to a homepage can make low-value pages appear interesting. Recognizing these patterns allows the system to cut wasted work before fetching pages, improving efficiency for site owners.

Safety and Content Control

The joint effort also focuses on safety by testing signals that identify and flag potentially harmful pages. Cloudflare stated that a legitimate site might contain a single compromised page, such as a fake login form planted by an attacker. Adding these signals supports safer and more reliable discovery by highlighting risks before users or AI systems interact with the content. This work complements OpenAI’s broader efforts to make ChatGPT a better place for shopping, learning, and finding answers through more visual and interactive experiences.

As people increasingly rely on AI to find specific information, such as washing machine manuals or passport requirements, discoverability becomes critical. These pages may have few external links but hold high value for specific user needs. The pilot demonstrates that AI systems can better identify such niche content when supported by network-level signals, ensuring that accurate information reaches users who need it most.

What should you do?

Website owners and publishers should monitor their server logs for unusual automated traffic patterns that may indicate inefficient crawling or potential security issues. Reviewing how AI systems index and represent your content can help ensure accuracy and relevance. Pay attention to any new tools or settings provided by your hosting or CDN provider that allow you to control AI access to your site.

What to do and how to stay safe: Cloudflare

• Review server logs to identify and block malicious bots that request nonexistent files or exploit redirects.
• Ensure that your website’s content is accessible and distinct, avoiding duplicate pages that confuse crawlers.
• Monitor for compromised pages, such as fake login forms, that could harm your site’s reputation and safety.
• Check if your CDN or hosting provider offers new controls for managing how AI systems discover and index your content.

Step-by-step guide: Prompt Injection Warning Signs: How to Spot AI Manipulation

General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

How many new pages did the pilot help OpenAI discover?

The pilot enabled OpenAI to discover and index hundreds of millions of new web pages.

What percentage of Cloudflare’s network participated in the pilot?

Approximately one percent of Cloudflare’s total network participated in the joint research pilot.

Does this pilot affect how ChatGPT ranks search results?

OpenAI independently evaluates and ranks pages based on relevance and helpfulness after discovery.