You won't realise it at the time, but the hundreds of decisions you silently make (or these days, maybe your AI agent makes on your behalf?) when starting a new project determine how painful you'll find the inevitable security patching it needs throughout the rest of its lifetime.
What do I mean, and how do you make things easier on yourself?
Bugs and patches are inevitable
Bugs and security issues exist in every piece of software. It's an inescapable fact, regardless of who (or what) writes the code, the design processes you follow, and however large or small your development organisation (or budget) is.
No software component exists in isolation. Even if you have a line of code written in 2005 that still does exactly what it should today, the rest of the system around it has to change and evolve over time (e.g. due to discovered bugs: security-impacting, reliability-impacting, or otherwise, and feature evolution of the rest of the system). That has the potential to cause your perfectly crafted line of code to malfunction.
If everything apparently works fine, why risk breaking it? You might seek to minimise your risk by simply not patching(!) - even if we assume that's a reasonable position, the insurance industry and regulations such as GDPR basically force what is acceptable or not here anyway; it's mostly not a subjective topic… in a business environment, you have to deploy security patches.
When a sufficiently critical bug is identified that makes patching non-negotiable, the level of business impact and risk associated with deploying a patch is all a result of prior decision-making. Mostly, it's driven by the decisions you made on day 1.
Do you get the foundations right?
When you first develop an application, you make choices about its foundations. Which OS will be used, which version of PHP and MariaDB etc. You'll make a lot of these decisions without thinking much about it, based on things like familiarity and convenience.
You're focussed on classes, libraries, lots of business domain problems and design decisions; you may have given some thought to PHP vs. Python, or MariaDB vs. PostgreSQL, but you never really thought should I use PHP 5.x or PHP 8.x: it was blindingly obvious. It was a subconscious choice that just happened.
Similarly, you probably didn't have a strong reason for why Debian vs. Alpine vs. Rocky. You have your preferred OS and that's what you develop/deploy all of your applications on. It's what you know.
One decision decides everything
Even though there are lots of different software components involved in your application, and everything that underpins it to run successfully in production, there is really only one decision that impacts on your future software maintenance journey:
- Use the "bleeding edge" version, or a version with long-term support (LTS)?
That one choice repeats again and again for every single component, and it single-handedly determines how often you might need to patch, and the risk levels (and associated business impact) you could face each time you do.
Risk management
Each time you change something in a system there is a risk that your change introduces bugs.
However, if the purpose of that change is to fix bugs (rather than to add features), the trade-off is that you're fixing known bugs at the same time as (only) the risk of adding more bugs; i.e. you're balancing a definitive (definitely fixing bugs) against a mere possibility (there might also be new bugs).
The possibility of a patch introducing new bugs at the same time as fixing them is not random. There are many factors to it, but an important one is the size of the change. Not just how many lines of code were changed, but how many parts of the code (different files, functions, methods, classes).
A small change is less likely to introduce bugs simply because fewer things are changed, but also because it's easier to test, and easier to reason about (smaller chance of mistakes: human error; or perhaps these days, also AI error).
The choice of "bleeding edge" or LTS determines the size of the change introduced in each patch.
Bleeding edge patches
Bleeding edge describes the latest version(s) of a particular piece of software. It's the one that receives new feature development, and contains all of the latest and greatest ideas.
There are basically two possibilities if you're using a "bleeding edge" version, and need to deploy a patch for a security issue or serious bug:
Fixed in the current branch
An update is available for your current version.
Updates of "bleeding edge" software versions typically combine a number of bug fixes with a number of feature changes or enhancements.
Therefore, the amount of code changed by any given release may range from moderate to fairly large.
Maintainers usually try to avoid making breaking changes within the same major version, but the underlying behaviour may shift in subtle ways that matter to your application, but were not foreseen to matter (so were not forewarned).
Fixed in a newer branch
Your branch is no longer maintained; a fix is only provided in a newer major (or minor) version.
This is an unusual scenario if there's a major issue, but it depends on the size of the gap vs. your current version, and of course the size of the maintainer's development team. Sometimes it simply isn't feasible to release a fix for unmaintained versions as well as the maintained ones.
Upgrading to a new version means potential breaking changes. The amount of code changed is likely substantial, and if you need to make changes to your own code too that compounds the risks; you might also introduce your own bugs.
LTS patches
Many software maintainers (or sometimes third parties, on a commercial basis) also offer long-term support (LTS) versions. These are versions where the maintainer has committed to provide security and bug fixes for an extended period (compared to their bleeding-edge version).
If you're running an LTS version and need to deploy a patch for a security issue or serious bug, the update is likely to have these characteristics:
- The code changes contained in the update are usually very small, isolated to those deemed necessary to fix the identified bug
- Trivial bugs are usually left untouched (unfixed) to avoid unnecessary code churn
- No new features are added
- No existing behaviours are changed: your own code doesn't require any modifications (so no added risk of new bugs entering your own code)
Consequently, I strongly advocate for using LTS versions whenever and wherever you can. The security patching process is simply far less risky to your business, and that in turn also allows security patches to be deployed more quickly (reducing the time your business is left exposed).
Security patching at Layershift
When you buy a Managed VPS from Layershift, part of what you're buying is our experience and expertise to set you up with a system that's actually maintainable.
We make a lot of these decisions about the underlying system for you, which:
- Gives you fewer problems to worry about (you only have to worry about these issues within your application code, rather than within the OS and its server applications too!)
- Allows us to maintain the server without breaking your application
It doesn't magically make security patching a risk-free process, but we do substantially reduce the risk by making a series of good choices: many small things that are only obvious through experience, and that ultimately lead to a more stable software environment for you to build and run your business upon.
Setting everything up the right way to minimise the risk and disruption of patching is only the beginning. Knowing what to patch and how to do it safely is underpinned by in-house tooling that helps our engineers manage risk down to the lowest possible level. Every patch deployment is fully tested. We validate your server's state immediately before and after, so if anything goes wrong we can identify it proactively and remediate discrepancies during the planned maintenance window before they impact your business.