The Layershift Logbook

Robots.txt tells cooperative crawlers what you want. It doesn't enforce anything: a robots.txt file is a request, not a lock, and it only works on crawlers that read it and identify themselves honestly. Our earlier post in this series covers whether to allow Search, Agent and Training crawlers; the bot-by-bot guide covers what each named crawler does.

BotStopper and Cloudflare Bot Management get compared because they're both trying to solve the same underlying problem: verifying which bots are worth letting through and stopping everything that robots.txt alone can't. But they're not built to solve it at the same scale.