Nick Gray, founder of the philanthropy-donor database PatronView, blocked Amazon’s Amzn-SearchBot at his firewall on August 5, after the crawler that feeds Rufus and Alexa reached roughly 117,000 requests a day against his 1.5 million page site. The bot feeds answers that surface inside Amazon properties, which means it never returns a visitor to whatever page it just read, a distinction Gray says settles the arithmetic before it starts: the visit is pure cost. That logic, drawn from a year of server logs Gray published on August 7, turns one operator’s firewall change into a working model for pricing a crawler by what it takes and what it gives back.
The number behind the block is what Gray calls pages crawled per human visitor referred, and it is the figure worth tracking across every crawler hitting a site, not just Amazon’s. Googlebot logged 46 crawls for every visitor PatronView’s Plausible analytics recorded, a rate Gray calls earned. Bingbot ran far heavier at 406 to 1, still tolerable in his framing because Bing referrals keep climbing. Gray is explicit that this is one operator’s log file, not an industry study, and his site, a specialist donor database, may not generalize to how AI crawlers treat a general news publisher.
Anthropic’s Claude-SearchBot supplied the benchmark Gray later applied to Amazon. In June, before he blocked it, the bot pulled 420,680 pages off PatronView in a single week, while Claude-User, the separate agent Anthropic runs when a person asks Claude to open one specific page, brought in 12 human visits over that same stretch, a 35,000-to-1 ratio. Bandwidth told the same story: 4.63 gigabytes went to the crawler against 175 kilobytes reaching the dozen humans it sent. Once Gray cut the bot off, its volume dropped to roughly two dozen attempts daily from a prior pace near 60,000, which Gray reads as a crawler honoring a 403 response rather than working around it.
Amzn-SearchBot has no comparable ratio to report, because it has no referral column to divide by. Its daily volume nearly doubled what Claude-SearchBot had reached in June, and Gray wrote that he doubted the bot would even credit PatronView as a source once it answered a shopper’s question. In the day after his newest rules went live, Cloudflare rejected 46,729 requests outright, and more than nine in ten, 43,150, traced to Amazon’s crawler still testing a block that had already been in place.
Gray’s single-site numbers land inside a range Cloudflare has already documented at scale. By the company’s count, Anthropic sat at 286,930 crawls per referral in January 2025 and had fallen to 38,000 by that July, a decline consistent with what Gray found independently a year later. He still leaves Googlebot and Bingbot untouched, a split that lines up with Cloudflare’s own policy taking effect September 15, which will block training and agent crawlers by default on new domains while leaving search crawlers on.
Gray says his preference was never a block at all. He would rather meter the 3.5 million page views his site serves each month and charge Amazon for them, and points to Cloudflare’s pay-per-crawl option as the market mechanism that does not yet exist for him at scale. Until it does, his standing rule is unilateral: no referral, no access. Search teams reviewing their own server logs should split that math by user agent rather than by company, since one AI operator can run a bot that earns its keep next to one that never will.
Per a PPC Land report by Luis Rijo, published August 8, 2026.
