Proxmox VE HA cluster with automatic failover

Results at a glance

16critical VMs and containers restart automatically on a partner node
≤ 15 minreplication interval for critical guests (some every 5 min)
3independent Corosync links, so one failed switch or NIC can't split the cluster
8 → 9rolling, node-by-node Proxmox VE upgrade

The problem

Our telecom services (Redis, application servers and databases) run as VMs and LXC containers on Proxmox. Before this work, losing a host meant someone had to notice, log in and start its guests on another node by hand. At 3 a.m. that can take a long time.

I wanted a failed node to be a non-event: its services come back on a partner node by themselves, with at most a few minutes of data to catch up.

Constraints

Local storage, not a SAN. Each node has its own ZFS pool, so HA needs a recent copy of every guest on the node it fails over to.
Four nodes. With an even number of votes, a 2–2 split would leave neither half with quorum. The cluster network has to be very hard to split.
24/7 traffic. Maintenance and upgrades have to happen with the cluster running.

What I built

Proxmox HA status page: quorum OK, watchdog fencing armed, all four nodes active — HA status: quorum OK, fencing armed, a local resource manager active on every node. Names and IPs replaced with dummy values.

Quorum that is hard to lose

Corosync runs over three separate links on three networks. If one switch, cable or NIC fails, the nodes still see each other and the cluster keeps quorum.

HA with fencing

Watchdog fencing is armed on every node. A node that loses quorum resets itself before its guests start anywhere else. That stops two copies of the same VM writing to the same data (split-brain), which is the failure that really hurts.

Every service has a failover pair

I put 16 critical VMs and containers under HA and used Proxmox VE 9's node-affinity rules to pin each one to a pair of nodes, for example Redis on nodes 3 and 4. Failover is predictable, and the target node always holds a replica.

HA node affinity rules pinning each workload to a pair of nodes — Strict node-affinity rules keep each workload on its node pair.

Replication matched to how important the data is

ZFS storage replication copies each HA guest to its partner node every 15 minutes. One busy guest replicates every 5 minutes, and less critical ones every 30 minutes or 2 hours.

Replication jobs table with 15-minute schedules to DR nodes — Replication jobs: each critical guest has a copy on a second node.

Upgrading 8 → 9 with the cluster running

I ran the pve8to9 --full checklist on every node and fixed every warning first. Then I upgraded one node at a time:

Migrate its guests to their partner nodes.
Upgrade, reboot and check the kernel and ZFS.
Confirm quorum, HA status and replication before moving on.

Running mixed versions for a short time is supported, so guests kept running on their partner nodes while each node was upgraded.

Decisions and trade-offs

ZFS replication instead of Ceph. Ceph gives shared storage and zero data loss on failover, but it needs more nodes, a fast dedicated network and more care. With four nodes and local disks, replication is simpler to run. The price is a recovery point of up to one replication interval. For these services that was an acceptable trade.
Strict affinity. It makes failover predictable, but a guest can't move to a third node if both of its partners are down. I chose predictability.

What I'd improve next

Scheduled, documented failover drills (pull a node on purpose every quarter), and a QDevice as a tie-breaking fifth vote so even a 2–2 network split keeps quorum.