Cloudflare Radar published crawl-to-referral ratios for the week ending June 1, 2026, and the numbers make the structural problem with AI crawlers concrete.
OpenAI registered 857 pages crawled for every single referral sent back, an improvement from a prior ratio of 1,252 to 1. Perplexity moved the wrong direction, worsening to 190 to 1 from 95 to 1. Anthropic posted the worst figure among major operators at roughly 11,122 to 1, though that is an improvement from approximately 13,528 to 1 measured in April. For context, traditional Googlebot runs near 5 to 1.
The asymmetry is not minor. Even Perplexity, a product built around cited answers, doubled its consumption-to-referral gap in a short window.
This data is the operational reason a growing number of publishers are adding AI crawlers to their robots.txt blocklists. Crawling without referring is indexing without compensating.
SEO and GEO teams should treat AI-crawler access as a deliberate policy choice, not a default. Weigh whether citation visibility in AI answers outweighs the near-zero referral return before opening or maintaining access for each operator separately.
Data sourced from Cloudflare Radar, covering crawl-to-referral ratios for the week ending June 1, 2026.
