The Layershift Logbook

If you're considering a move to a new hosting provider, there's usually a good reason behind it, and often a fair bit of hesitation alongside it. That's understandable. Your website or application matters to your business, and the idea of moving it somewhere new naturally raises questions. What if something breaks along the way? What if customers notice? What if it all takes longer than expected?

Not every AI bot is the same problem. Some make your content discoverable in search and AI answers. Some act on a real person's behalf, fetching a page because somebody asked an AI assistant to do something with it. Others exist purely to train a model, with no visit at the end and no benefit back to you. Treating all three as one "AI bot" toggle, block everything or allow everything, throws away the information you need to make a sensible decision.

Robots.txt tells cooperative crawlers what you want. It doesn't enforce anything: a robots.txt file is a request, not a lock, and it only works on crawlers that read it and identify themselves honestly. Our earlier post in this series covers whether to allow Search, Agent and Training crawlers; the bot-by-bot guide covers what each named crawler does.

BotStopper and Cloudflare Bot Management get compared because they're both trying to solve the same underlying problem: verifying which bots are worth letting through and stopping everything that robots.txt alone can't. But they're not built to solve it at the same scale.

Ask an LLM support agent why your server has slowed down and it will usually give you an answer straight away. Confidently, too.

That confidence is doing a lot of work. The agent hasn't seen your server. It doesn't know your history, and it has no way of checking whether what it's telling you is actually correct.

On a pricing page, most VPSs look remarkably similar. The bigger differences often only become obvious after you've bought one.

CPU, RAM, storage and bandwidth are straightforward to compare. What is much harder to judge is everything that happens around the server.