How BotStopper Stops the Bots Your robots.txt Can't

Robots.txt tells cooperative crawlers what you want. It doesn't enforce anything: a robots.txt file is a request, not a lock, and it only works on crawlers that read it and identify themselves honestly. Our earlier post in this series covers whether to allow Search, Agent and Training crawlers; the bot-by-bot guide covers what each named crawler does.

Duane Forrester, writing for Search Engine Journal, checked self-declared bot identities in his own server logs against the IP ranges the real operators publish. Of 799 requests claiming to be Googlebot, only 107 checked out. Of 33 requests claiming to be an AI assistant fetching a page on a user's behalf, only 6 did. Some of the rest went straight for files like .env.production and secrets.yaml (Search Engine Journal, 25 June 2026).

A User-Agent header costs nothing to fake. That's what BotStopper is for: it stops trusting a string that anyone can type, and enforces access instead of asking for it.

How Does BotStopper Enforce Access?

BotStopper is Layershift's deployment of Techaro's open-source Anubis project. A self-reported identity isn't enough on its own: BotStopper challenges suspicious requests with proof of work, adding cost and friction to automated traffic at scale. For a normal visitor, the challenge should add very little delay.

Not every request gets the same check. BotStopper can also use a lighter meta-refresh challenge that works without JavaScript.

The User-Agent header isn't trusted either. BotStopper combines the challenge with other checks to verify known crawlers and flag suspicious behaviour: if something claims to be Googlebot, for example, it can check whether the request actually comes from Google's network rather than trusting the name in the header.

Why BotStopper Doesn't Rely on Proof of Work Alone

Proof of work can be beaten. In August 2025, the code-hosting platform Codeberg reported that AI crawlers had learned to solve the underlying challenge, describing the resulting traffic as functioning like a denial-of-service attack (The Register, 15 August 2025). That's why BotStopper combines the puzzle with the identity checks above rather than relying on it alone.

Does This Work?

The clearest public account of the underlying engine in production comes from Duke University Libraries, which piloted it in 2025 across three applications: the Duke Digital Repository, its Archives & Manuscripts platform, and its Books & Media Catalog. All three had been suffering outages from aggressive scraping before the pilot began.

With the challenge in place, Duke reported blocking roughly 90% of total traffic across the three applications: more than 4 million unwanted HTTP requests a day. Service was restored to real users on all three platforms (Duke University Libraries, Anubis pilot project report, 2025).

BotStopper is Layershift's deployment of that same engine on your server.

What Bot Traffic Costs WordPress

On a WordPress site, unwanted bot requests consume CPU, PHP workers, and bandwidth. BotStopper stops that traffic before it reaches WordPress, your database, or anything else running behind it, rather than making your application do the work of rejecting it. That leaves those resources available for the visitors and application traffic you actually want to serve.

Get BotStopper

robots.txt tells legitimate bots what you want. It cannot stop a bot that ignores the rules or lies about who it is. BotStopper closes that gap: it stops unwanted bot traffic before it gets the chance to consume your server resources.

For just £10 a month per server, along with a 30-day money-back guarantee. Talk to our support team: we'll confirm bots are the cause and, if BotStopper is the right fix, install, customise and tune it for your server.

Frequently Asked Questions

Does BotStopper block search engines like Google?

Not if configured correctly. Legitimate crawlers that identify themselves accurately and match expected network ranges pass without being challenged. The challenge is aimed at unverified or spoofed traffic, not verified Googlebot requests.

Is proof-of-work enough on its own to stop scrapers?

No. A determined operator can solve the underlying puzzle, which is why BotStopper combines it with other checks rather than relying on the puzzle alone.

Will real visitors notice anything?

Most should not. For a normal visitor, a challenge should add very little delay, and a valid pass can be remembered for later requests rather than repeated on every page.

How is this different from blocking bots by User-Agent?

A robots.txt rule is a request that an honest crawler chooses to honour. It performs no verification, so a scraper can claim to be anything it wants at zero cost. BotStopper can challenge unverified traffic and check whether a claimed crawler actually comes from the network it says it does.

Does this slow the server down?

BotStopper is designed to make the bot do the expensive work, not your server. Legitimate visitors normally pass with little or no noticeable delay.

Does robots.txt stop AI crawlers like GPTBot or ClaudeBot?

No better than it stops any other bot. OpenAI's GPTBot and Anthropic's ClaudeBot are both documented to respect robots.txt, but that is voluntary compliance from operators willing to publish and honour a user-agent string. A robots.txt rule has no way to verify that a request claiming to be GPTBot is actually GPTBot and not something borrowing the name.