Should You Block AI Crawlers? Search, Agent and Training Bots
Not every AI bot is the same problem. Some make your content discoverable in search and AI answers. Some act on a real person's behalf, fetching a page because somebody asked an AI assistant to do something with it. Others exist purely to train a model, with no visit at the end and no benefit back to you. Treating all three as one "AI bot" toggle, block everything or allow everything, throws away the information you need to make a sensible decision.
It also ignores a real cost. A crawler can consume the same server resources as a real visitor without necessarily delivering the same value: CPU, memory, database queries and bandwidth, whether the request came from a buyer or a bot. That cost isn't evenly spread either. A crawler hitting a cached static page is close to free; one working its way through WordPress search results, faceted WooCommerce filters, calendar archives or paginated category pages generates uncached, database-heavy requests that push PHP workers to their limit and slow the site for the people you actually want.
Deciding which bots you want is a policy question. Enforcing that decision is a separate problem. Choosing what to allow doesn't mean every bot will respect it: you still need a way to enforce that decision against traffic that ignores robots.txt, spoofs its identity, or behaves differently from the bots you meant to allow. That's what BotStopper is built to handle.
Search, agent and training is becoming the standard three-way split for AI traffic, and it's a more useful lens than a single block-or-allow toggle. Search crawls and indexes your content so it can be cited later. Agent traffic acts in real time on a specific person's behalf, fetching a page because somebody asked an AI assistant to. Training takes your content to improve a model, with no visit and no benefit back to you. Each one earns a different answer, and the rest of this piece treats them separately.
Does Blocking AI Crawlers in robots.txt Work?
Not on its own. BuzzStream analysed 4 million citations across 3,600 prompts spanning ChatGPT, Gemini, Google AI Overviews and Google AI Mode in March 2026. Among the top 50 news sites blocking specific crawlers in their robots.txt, 92.3% of sites blocking Google-Extended still appeared in AI citations, along with 82.4% of those blocking OAI-SearchBot and 70.6% of those blocking ChatGPT-User (PPC Land).
robots.txt was designed in the 1990s as a polite request. It has never been an access control. Blocking a crawler does not guarantee your site disappears from AI results. It can still surface through several routes:
- It was indexed before you added the rule.
- It reached the model through Common Crawl, an intermediary you never had a relationship with.
- The bot ignored the file, because nothing stops it.
- The AI found the URL through a third-party search provider or another indexed page rather than crawling your server directly.
If you are blocking AI crawlers in robots.txt to protect your content, you may be paying the visibility cost of blocking while receiving little of the protection you think you bought. And if you have sensitive material, robots.txt was never the right tool. That needs authentication, access controls or firewall rules; a file that politely asks bots not to look is not a security control, and treating it like one is how private URLs end up in search results.
None of this applies to controls that enforce the decision rather than asking politely: a WAF rule, a firewall deny, or a challenge like BotStopper's. Those can stop or challenge the request before it reaches your application. robots.txt only ever asks.
Bots that ignore robots.txt keep hitting your PHP workers, your database and your bandwidth anyway. BotStopper is that enforcement layer.
What Did Blocking Cost the Publishers Who Tried It?
Hangcheng Zhao of Rutgers and Ron Berman of Wharton studied how news publishers responded to generative AI, using a difference-in-differences design across 30 major newspaper sites including CNN, The New York Times, the BBC and The Guardian (arXiv 2512.24968, submitted December 2025, revised April 2026). Their finding, in the paper's own words: "large publishers who block GenAI bots experience reduced website traffic compared to not blocking."
Roughly a 7% decline in traffic within six weeks of blocking, consistent across SimilarWeb, Semrush and Comscore.
Comscore measures a human web-behaviour panel, tracking actual people browsing rather than bot requests, so that decline is not purely the mechanical removal of bot visits from the total. Some of the traffic publishers lost was real visitors.
The effect is concentrated in large publishers. When the researchers extended the analysis to a broader set of around 500 publishers, only the top 50 showed a statistically significant decline; sites ranked 51-100 showed no significant effect, and the smallest tier showed a small, non-significant increase (arXiv 2512.24968).
The cost of blocking scales with how much of your traffic depends on being discovered, and a national newspaper and a regional accountancy firm are not running the same experiment.
Should You Trust the Numbers Driving This Debate?
Not without checking who benefits from them. Many of the companies publishing the most alarming AI bot statistics also sell bot management, and the claim that bots now outnumber humans on the web is one of the most repeated, and most disputed, of those numbers.
Crawler volume is genuinely up, and the imbalance between crawling and referring is real: one widely cited dataset put a major AI crawler at over 38,000 pages crawled for every referral it sent back to the site, against roughly five for a standard search crawler (source). But "more than half the web is bots" is usually a statistic about observed HTTP requests, not about the internet's population, and it's worth treating any single vendor's framing of it with a healthy dose of scepticism.
So What Should You Do?
Match the decision to what your website is for. There is no universal configuration: a rule that fits a news publisher rarely fits an e-commerce catalogue or a SaaS marketing site.
If you run a professional services or B2B site, being found is usually worth more than restricting access to your public service pages. Allow Search and Agent. Training is a separate decision: crawler volume is unlikely to be a significant load problem for most sites at this scale, so the question is mainly whether you are comfortable with your public content being used for model training.
If you run e-commerce, allow Search and Agent: agent traffic can form part of a customer's purchase journey, and blocking it risks losing the sale, not just the crawl. Watch the crawl volume rather than the ethics. Your exposure is uncached faceted pages, not your product descriptions. Competitor price scraping is a different problem with a different answer.
If you publish original research, journalism or documentation, this is hard, and the Zhao and Berman study is your warning: the bigger and more discovery-dependent you are, the more blocking costs. Consider allowing Search and Agent and restricting dedicated Training crawlers. Check how your platform's Training control treats mixed-purpose crawlers first: Googlebot, Applebot and BingBot combine indexing with training, and a blanket Training block can take them down with it.
Running a SaaS product or customer portal? Your marketing pages want Search and Agent. Your authenticated areas should never have been relying on robots.txt in the first place.
Small brochure site? The load risk is minimal and the visibility upside is most of what you have. Blocking is close to all cost and no benefit.
If you run a high-traffic WordPress site, measure before you decide. This is the case where crawler volume does cost money, and where the answer is usually caching and rate limiting rather than an outright block.
Our view: we wouldn't block everything or allow everything by default. Agent traffic acts directly on a person's behalf; Search traffic feeds the systems people use to find you. Training is the one you need to make a conscious decision about.
Where Managed Hosting Fits
This becomes a hosting problem when the traffic starts affecting the site. You cannot decide what to allow until you can see what is arriving, what it costs you, and what it returns.
In practice, that's telling human traffic apart from useful automation and from traffic with no upside for you at all, protecting performance for real visitors when crawl volume spikes, applying bot controls without accidentally cutting your search visibility (exactly the mistake the Training and Googlebot overlap invites), and separating ordinary crawling from behaviour that's abusive.
That is the work, and it is ordinary infrastructure work. On a managed VPS, you are not left relying on analytics that only capture part of your automated traffic: if your site starts running unusually slowly, our support team can see what is actually hitting your server and tell you whether bots are behind it. A CDN with a WAF gives you somewhere to apply rate limits and rules that is not your origin.
If you want to go further, our bot-by-bot guide to which AI bots to allow covers what each named crawler does, how BotStopper stops the bots robots.txt can't covers the mechanics in more depth, and BotStopper vs Cloudflare Bot Management compared covers the different approaches to bot management.
Once you can see what's arriving, BotStopper is how you control it: it evaluates requests before they reach your application and can allow, challenge or deny them. £10 a month per serve. Our support team confirms bots are the cause before it's installed, then customises and tunes it for your server.
Frequently Asked Questions
Should I block AI crawlers from my website?
It depends which behaviour you mean. AI traffic splits into search crawlers, user-directed agents and training crawlers, and each trades something different. Blocking search crawlers can remove you from AI answers. Blocking training crawlers protects content that may never send you a visitor. Decide on the three separately, or simply contact our support team as a Layershift customer.
Does blocking AI crawlers in robots.txt actually work?
Often not. BuzzStream found 92.3% of sites blocking Google-Extended were still cited by AI systems. robots.txt is a request, not an enforcement mechanism. Blocking one crawler does not guarantee your site disappears from AI results: URLs or content can still surface through prior indexing, third-party sources, search providers, or bots that ignore the file.
Do AI crawlers slow down my website?
They can. Crawlers consume CPU, memory and bandwidth like any visitor, and the cost lands hardest on uncached dynamic pages such as WordPress search results, WooCommerce filters and large archives. The load is real regardless of your view on content ownership.
How do I stop bots that ignore robots.txt?
robots.txt only works on bots that choose to read and honour it. For traffic that spoofs a user agent or ignores the file entirely, you need a control that sits in front of your application: a WAF rule, a firewall deny, or a challenge like BotStopper, which evaluates each request and can allow, challenge or deny it before it reaches your site. ```
