For the last stretch of work I've been running a 4-node Proxmox VE cluster in production, with 16 VMs and containers under HA. Here is what I learned. Most of it is not in the "click here to enable HA" tutorials.
1. Do the quorum maths first
Corosync needs a majority of votes to keep quorum. With 4 nodes that is 3 votes. If one node dies, the other three keep going. If the network splits 2–2, neither half has quorum and HA stops on both sides.
So with an even number of nodes you have two options:
- make a split very unlikely: I run Corosync over three separate links on separate networks;
- add a QDevice, a small external vote that breaks the tie.
Check your links with:
pvecm status
corosync-cfgtool -s2. Fencing is the whole point
HA is not "restart the VM somewhere else". It is "restart the VM somewhere else and be sure the old copy is dead". Proxmox does this with a watchdog. A node that loses quorum stops resetting its watchdog and reboots itself, and only then are its guests started elsewhere.
Without that, a node that is cut off from the cluster but still running could keep writing to a database while a second copy starts on another node. That is split-brain, and it is much worse than a few minutes of downtime.
Check that fencing is armed under Datacenter → HA (you want fencing: armed), or with ha-manager status.
3. With local storage, replication is your RPO
I use local ZFS on each node, not Ceph. HA can only start a guest on another node if its disk is there, so every HA guest has a ZFS replication job to its failover node.
The schedule is your recovery point. If it replicates every 15 minutes, you can lose up to 15 minutes of data on failover. I don't use one schedule for everything:
- critical guests every 15 minutes, one very busy one every 5;
- less important guests every 30 minutes or 2 hours.
Replication isn't backup, though. It happily copies a deleted table. Keep real backups as well.
4. Use affinity rules so failover is predictable
Proxmox VE 9 replaced HA groups with HA rules. I use strict node-affinity rules to pin each service to a pair of nodes, for example Redis on node 3 and node 4. The replica already lives on the partner node, so failover never lands somewhere without a disk.
The trade-off is that if both nodes of a pair are down, that service waits. For us, predictable beat flexible.
5. Upgrade one node at a time, and read the checker
For the 8 → 9 upgrade I first ran the checker on every node and fixed every warning:
pve8to9 --fullThen, one node at a time: migrate its guests to their partners, switch the repositories to Debian 13 (Trixie) and Proxmox VE 9, upgrade, reboot, and confirm the node is back with quorum, HA active and replication green before touching the next one. Mixed versions are fine for a short time. Rushing is what causes outages.
6. Test failover before you need it
An HA setup you've never tested is a hope, not a plan. Pick a quiet window, cut a node's power or network on purpose, and time how long its guests take to come back. Write down what you saw. That document is worth more than any screenshot.
Written by Ameer Hamza, Linux systems engineer in Lahore. Need help with something like this? Get in touch.