Which AI Bots Should You Allow on Your Website? A Bot-by-Bot Guide

Three closed airport security gates at night, each lit with a red cross, representing the allow and block decision every website now has to make for AI search, agent and training bots

Key Takeaways

Block Training on Cloudflare and you also block Googlebot. Cloudflare names Googlebot, Applebot and BingBot as multi-purpose crawlers caught by that rule (Cloudflare).

OpenAI and Anthropic each run a separate search bot and training bot, so blocking by brand blocks both. Microsoft runs no training crawler at all.

Google-Extended and Applebot-Extended are control tokens, not crawlers. They never appear in your access logs, and that is expected rather than a sign your block failed.

Blocking Google-Extended costs nothing in Google Search. Google says it is not a ranking signal (Google).

Training crawlers rose from 22% of crawler requests in spring 2025 to 52% by June 2026 (Cloudflare).