Database Clustering and Failover in Enscale with Galera
Your architecture diagram has three application servers sitting behind a load balancer. It looks resilient. Everyone agrees it looks resilient.
Then one database node stops responding, and you find out what that diagram was actually worth.
The site still loads, which is almost worse, because it means the phone starts ringing before anyone realises what's happened. Customers can't log in. Checkouts fail at the last step. Account updates vanish. Your three healthy application servers are all sitting there with nowhere to read or write, and every one of them is returning the same connection error.
However good the rest of your architecture looks, that database is still a single point of failure.
Database clustering removes that dependence on one node. The hard part was never spinning up a second database server. It's replication, traffic routing, health checks, failover and getting a recovered node safely back into rotation, correctly, while production is running.
This guide covers what clustering actually does for database reliability, how failover differs between topologies, which one to pick, and what clustering won't fix. Enscale does most of the assembly for you, and where that matters is called out as we go.
Key Takeaways
Load balancing your application servers does nothing for you if all of them read and write to one database node.
Clustering is not one design. Primary-secondary, primary-primary and Galera solve different problems and fail in different ways.
Enscale deploys the whole topology: nodes, replication model, and a pair of ProxySQL nodes that route writes to the primary and spread reads across replicas.
Take a preconfigured topology and have it running in about 10 minutes, or build and tune your own. Starting preconfigured and customising later is usually the right order.
A cluster is not a backup, and it will not fix a slow query. It replicates your mistakes faithfully to every node.
Pick the topology from how your application actually reads and writes, not from how many nodes sound reassuring.
What does database clustering actually change?
A database cluster runs multiple nodes holding copies of the same data.
Depending on the topology, those nodes can:
- take over when another node becomes unavailable
- share read traffic
- absorb reporting or analytics work away from your live queries
- reduce the blast radius of maintenance
- protect committed data when a node fails
What it doesn't do is make the database unbreakable. What it does is cut down the number of situations where one server going dark takes the entire application with it. That's a smaller promise than "high availability" usually implies, and it's the honest one.
Enscale supports preconfigured clustering for MariaDB, MySQL and Percona, in primary-secondary, primary-primary and Galera topologies. On Percona, the Galera-based option is packaged as Percona XtraDB Cluster, so everything in the Galera section below applies to it as well. New nodes get discovered and joined to the cluster during horizontal scaling. ProxySQL distributes traffic and pulls unhealthy or high-latency nodes out of rotation, then adds them back once they recover.
This is built into how Enscale works, not a set of scripts bolted onto an ordinary VPS after deployment. Clustering, node discovery and automatic scaling are platform behaviour, which matters most when the cluster changes shape. Add a node and the platform already knows what to do with it, rather than waiting for someone to remember which config files need editing and in what order.
Enscale assembles the moving parts for you
Build a cluster manually and the shopping list gets long fast. Install several database servers. Configure replication between them. Add a proxy layer. Set up health checks. Decide, in advance, exactly what should happen when a node disappears at 3am.
Then handle the awkward one: making sure a replacement node has the correct data before it's allowed anywhere near application traffic.
With Enscale, the cluster deploys as a complete topology. The platform creates the nodes, configures the replication model, and puts a pair of ProxySQL nodes in front of them, so the entry point isn't quietly a single point of failure either. Your application connects to that proxy layer rather than to one database server's address.
That distinction is the whole point. Your application shouldn't need to know which node is healthy, which one is still synchronising, or where a given query ought to go. The proxy layer works it out. This is read/write splitting: writes go to the primary, SELECT traffic spreads across the available replicas, and your code talks to one stable connection point and nothing else.
Which is not the same as saying every request survives every failure. Use sensible connection timeouts, use retry logic, and check what your database driver offers, because most connectors have their own failover handling worth configuring. What you get from the proxy layer is that a failed node no longer means someone SSHing in to repoint the application before traffic can reach a healthy database.
Which database cluster topology should you choose?
"Clustered" is not a design. It's a category, and the options inside it behave very differently under load.
| Topology | Writes go to | Main trade-off | Start here if |
|---|---|---|---|
| Primary-secondary | One primary | Asynchronous replication, so secondaries can lag. Promoting a new primary is a separate problem from routing around a dead one | Your application reads far more than it writes, and you want reporting off the primary |
| Primary-primary | Either node | No automatic conflict resolution. Your application has to guarantee it won't write the same rows on both nodes | You have a specific reason for it, and you can hold that guarantee |
| Galera (three nodes) | Any node | Writes don't scale linearly, transactions can fail certification and need retrying, tables need InnoDB and primary keys | You want the database to survive losing a node without anyone making a failover decision |
The trade-offs are where the real decision lives, so they're worth understanding properly rather than picking from a table.
Primary-secondary replication
One primary takes the writes. Changes replicate out to one or more secondaries.
Those secondaries can serve read traffic, run reports, carry your analytical work, all without piling it onto the primary. One of them can also be promoted if the primary fails. If your application reads far more than it writes, and most do, this is usually the right starting point.
The trade-off is that standard replication is asynchronous. The primary can confirm a write before a secondary has applied it, which is where replication lag lives. And failover of the write role needs thinking about properly: pulling a dead primary out of the proxy is not the same thing as promoting a new one. Those are two separate problems, and only one of them is solved for you.
Primary-primary replication
Two nodes, both accepting writes, replicating changes to each other.
It can simplify recovery and spread some workloads around. It also asks more of your application. Replication here is still asynchronous, so your code has to avoid conflicting changes landing on the same records on different nodes. MariaDB's standard asynchronous multi-primary replication offers no automatic conflict resolution. If two nodes disagree, nothing steps in to referee.
Sometimes it's the right answer. It should never be chosen because two writable databases sound better than one.
MariaDB Galera Cluster
Galera works differently, and it's worth understanding why before you pick it.
It's a multi-primary, virtually synchronous cluster. Any active node accepts reads and writes. When a transaction reaches commit, its changes are bundled into a write set and put through certification: the cluster checks for conflicts and agrees a consistent transaction order before the commit is confirmed.
That sidesteps the replication-lag problem you get with asynchronous primary-secondary setups, and it protects you strongly against losing an acknowledged transaction when a node fails.
It does not make reads perfectly consistent everywhere, though, and this catches people out. A write set is certified across the cluster before the commit is acknowledged, but each node applies it from its own receive queue a moment later. A transaction starting on a different node can briefly fail to see a write that has just committed. MariaDB calls these stale reads, and where a read genuinely has to see the latest state, the wsrep_sync_wait session variable makes that node catch up before it answers. The window is far smaller than asynchronous replication lag, but it is not zero.
Galera also uses quorum, which stops two isolated halves of a cluster happily accepting contradictory writes. In a three-node cluster, two connected healthy nodes form a majority and the cluster carries on if one dies. A two-node cluster cannot survive losing either node without a separate voting member, so if you're planning a pair, plan three.
The trade-offs are real.
Every write has to be processed by every node, so adding nodes does not scale writes linearly. Concurrent transactions touching the same rows can fail certification, which means your application has to be able to retry them. Replicated tables need InnoDB and primary keys.
And one slow node is everyone's problem. Galera uses flow control to throttle replication when a node can't apply changes fast enough. That protects consistency, which is what you asked for, but it also means a single under-resourced node can drag the throughput of the entire cluster down with it.
Galera is a strong high-availability choice. It is not a way to skip testing your workload.
Should you use a preconfigured cluster or build your own?
There are two routes to a cluster on Enscale, and they suit different teams.
The preconfigured route is the quick one. Choose your database, choose a topology, and the platform assembles everything described above. A three-node Galera cluster that would cost you a careful day by hand takes about ten minutes. More to the point, it comes out configured the way it's supposed to be, rather than the way it ended up after a long Friday of following a tutorial and improvising the last two steps.
The custom route is there for when your workload doesn't fit a template. Build the environment yourself: set the node counts, tune the stack, shape the topology around your application, and scale each layer independently as it grows. You aren't locked into the packaged shape.
Most teams are best off starting preconfigured and customising later, and that's genuinely the right order rather than the cautious-sounding one. Get a known-good cluster running. Watch how your real workload behaves on it for a few weeks. Then change the things your traffic has shown you need changing, instead of the things you guessed at on day one.
This part is easier to judge by doing than by reading, and Enscale comes with a 14-day free trial if you want to put a cluster up and look at it.
Scaling the database on Enscale
Two directions to grow in.
Vertical scaling gives an existing node more CPU and RAM. Enscale nodes consume additional resources up to the limits you set, so they can ride out changing demand without migrating the database to a different server plan. Resources are measured and billed in cloudlets based on what you actually use, not on the ceiling you set. Worth pairing with a look at how the InnoDB buffer pool is sized, because extra RAM only helps if the database is configured to use it.
Horizontal scaling adds nodes. In primary-secondary, extra replicas buy you read capacity. In Galera, a new node joins and synchronises its copy of the data before it starts serving traffic. Enscale automates the node discovery and cluster configuration that step normally requires.
Worth saying plainly: more nodes need more resources, and every node needs enough storage for its own full copy of the database. Horizontal scaling should be solving a defined availability or capacity problem. Not making the topology diagram look more impressive in a board pack.
Is a database cluster a backup?
Replication copies changes between nodes. Including the changes you didn't mean to make.
Drop the wrong table and the cluster will replicate that deletion perfectly, to every node, in seconds. Application bugs do the same. So do compromised accounts and that one destructive query someone ran against production because they thought they were on staging.
A cluster keeps the database service available. Backups give you somewhere to go back to once the data itself is wrong. Different problems, and production systems generally need both.
Enscale takes a disk-level backup every six hours, so you always have a restore point from earlier the same day. Be clear-eyed about what that means, though: the worst case is losing up to six hours of writes, and a disk-level restore brings back the environment rather than one accidentally dropped table. Depending on your database and how tight your recovery requirements are, database-native logical backups are worth running on top of it.
A cluster inside one location isn't a disaster recovery plan either. Node placement, failure domains and how you'd come back from losing the whole cluster are still questions you have to answer.
Will database clustering fix slow queries?
A query that wastes CPU on one database server will waste CPU on three of them. You've simply bought the same inefficiency in bulk.
In Galera it can be worse than neutral. An expensive write creates work everywhere, because its changes have to be replicated and applied across every node. Slow queries build queues, queues trigger flow control, and flow control takes capacity away from the whole cluster.
Clustering buys resilience and, for the right workloads, capacity. It does not touch missing indexes, poor query patterns, or configuration that stopped matching your workload some time last year. If you don't already know which of your queries are the expensive ones, the slow query log is the place to start, and it's worth doing before you clone the problem across three nodes.
Which is an argument for fixing the query layer before you multiply it across three nodes, not an argument against clustering. Just don't expect the cluster to do that job for you.
That job is a different one, and it's what Releem does: watching the workload, finding the slow and expensive queries, and showing you where a configuration or indexing change would give back performance you're already paying for. Worth a look once the cluster is up and the question changes from "will this stay up" to "are these nodes being used well".
So is database clustering worth it for you?
Start with one question: if the database went down right now, what stops?
If the answer is sales, customer access, or a process the business genuinely can't pause, clustering is worth costing out. That covers most SaaS platforms, e-commerce stores, membership sites, high-traffic WordPress sites, and anything where maintenance can't just wait for Sunday morning.
If it's an internal tool with good backups, a single well-managed database node may still be exactly right. There's no prize for over-engineering it.
What's worth avoiding is the middle case: a business-critical platform running on one database node, not because anyone decided that was an acceptable risk, but because nobody ever sat down and decided anything. That risk gets accepted by default, and it usually gets reviewed for the first time during an incident.
Enscale makes the clustered option practical. Replication, the proxy layer, health checks and scaling controls arrive together in one environment, instead of as six things your team has to assemble and then keep alive.
The part that's still yours is choosing the topology that matches how your application actually uses its database. If you're not sure which one that is, that's a conversation worth having before the failover, not during it. We're happy to be on the other end of it. Talk to our cloud architects, describe how your application uses its database, and we'll tell you honestly which topology fits, including if the answer is that a single well-backed-up node is still right for you.
Frequently asked questions
What is database clustering?
Database clustering runs several database nodes that hold copies of the same data and work together as one logical database. If a node fails, the others keep serving the application, so the database stops being a single point of failure. Depending on the topology, the nodes can also share read traffic, absorb reporting work, and reduce the disruption caused by maintenance.
How does database failover work?
Failover is the process of moving work away from a failed database node to a healthy one. In practice it has two parts that are often confused. Routing is handled by a proxy layer such as ProxySQL, which health-checks the nodes and stops sending traffic to one that stops responding. Promotion is the separate question of which node takes over the write role. In a primary-secondary setup a secondary has to be promoted to primary; in a Galera cluster every node already accepts writes, so there is no promotion step and failover is faster to reason about.
Does database clustering improve database reliability?
It improves availability, which is one part of reliability. Clustering reduces the number of failures that take the whole application down, but it does not protect you from bad data. Replication copies every change faithfully, including an accidental DROP TABLE, so backups remain essential. Clustering also will not improve the reliability of slow or badly indexed queries; it runs them on more nodes.
How many nodes does a database cluster need?
For a Galera cluster, three. Galera uses quorum to prevent two isolated halves of a cluster accepting conflicting writes, so a majority has to remain connected. Three nodes tolerate the loss of one. A two-node cluster cannot lose either node without a separate voting member, which is why a pair is usually a false economy. Primary-secondary setups can run with two nodes, but you then need a plan for promoting a new primary.
What is the difference between database clustering and replication?
Replication is the mechanism; clustering is the system built around it. Replication copies changes from one database node to another. A cluster adds the parts that make those copies useful when something fails: a proxy layer that health-checks nodes and routes traffic, a rule for which node accepts writes, and a safe way to bring a recovered node back into rotation. You can run replication without a cluster, but you cannot build a useful cluster without replication.
Try it before you need it
None of this is really settled by reading about it. Build a cluster, point some real traffic at it, and see how it behaves.
Start a 14-day free trial of Enscale
Fourteen days is enough to deploy a three-node Galera cluster, restore a copy of your database into it, and then take a node offline on purpose during a quiet hour to watch what your application actually does about it. The cluster itself is about ten minutes of that.
That last part is the bit worth doing. It answers the question this post opened with, and it's a much better test than an architecture diagram that everyone agrees looks resilient.
