Cloudflare introduced a Web Search API through AI Gateway on October 2, with Ceramic.ai, Exa, and Linkup as launch partners. The company says those partners have committed to its crawler standards. For site owners, those standards matter more than the developer plumbing.
The pitch to developers is simple. According to the Cloudflare blog, agents often guess a page’s URL and fetch it, which can end in a 404 Not Found. Web search lets an agent query for relevant data and ground its responses in live information, with fresh, structured snippets placed directly into the model’s context.
The crawler terms are the part that touches publishers. Cloudflare states that each provider’s crawler has to meet the “Verified bots” requirements it publishes in its developer documentation. Every web search response must also carry a link pointing to where the crawled content lives.
Cloudflare reduces the commitment to three behaviors: partners name their crawlers, honor robots.txt, and credit the page a result came from. The company frames this as giving site owners transparency and control over how their content is used.
Those are Cloudflare’s claims about its partners. The post does not describe how compliance is checked or what happens if a partner falls short. It also does not say whether agents must show the source link to end users, only that the search response carries it. That gap decides whether a cited page ever earns a visit.
The commercial mechanics are plainer. Requests appear in normal AI Gateway logs, and queries draw down AI Gateway credits. Cloudflare says it will identify partners that support Zero Data Retention. It also sells search straight through at each partner’s list API price, adding no markup, and it supports bring-your-own-key setups. The post gives no price figures.
Developers can call the service through a REST endpoint or a Workers binding, with a sample query in the post asking about things to do in Salt Lake City. Cloudflare also says native Server Tools are coming to AI Gateway, with web search among the first, so the search step may soon be a default part of agent builds on its platform.
That default matters because it standardizes who sits between your content and an agent’s answer. Instead of auditing dozens of unknown agent fetchers, a site owner can look at three named providers whose crawler behavior Cloudflare is publicly vouching for.
Teams should review robots.txt rules and server logs for crawlers from Ceramic.ai, Exa, and Linkup, and decide now whether they want to be reachable by agent search.
Based on Cloudflare’s blog post “Introducing Web Search API via AI Gateway,” published October 2, 2026.
