Industry4d ago
Cloudflare CEO Predicts Pay-To-Crawl Web Future
Bot traffic now exceeds human traffic on the internet, according to Matthew Prince. The Cloudflare CEO attributes this surge to AI agents, arriving years earlier than his 2027 forecast. This shift forces a transition toward a paid access model for web crawling. Site owners will likely charge bots to index their content.