VCF High Availability, Explained as a Human Body

Arun Nukula
Arun Nukula
8 min read

I get asked about high availability for VCF components more than almost anything else, and the question is usually some version of the same thing: which bits do I need two of, and what happens when one dies?

The documentation answers it, but it answers it component by component, spread across a dozen design pages. So here is the whole thing as one picture, using a body, because a body solves exactly this problem and solves it in more than one way.

The sentence that decides everything

Almost all of this comes down to one line in the VCF Operations design page:

vCenter and SDDC Manager are always deployed as single node. You have the flexibility to choose whether to deploy NSX Manager, VCF Operations, VCF Automation, and VCF Management Services as single node deployments, or as three node deployments to provide high availability capabilities.

Two components never get the choice. Four of them do, and the choice is yours to make per component at deployment time.

An anatomical chart of VCF components. The brain is vCenter and the vertebral column is SDDC Manager, both always a single node. The branching nerves are NSX Manager, the eyes are VCF Operations, the hands are VCF Automation and the liver is VCF Management Services, all of which can be deployed as a three node cluster. The skull and ribcage drawn in green around them are vSphere HA, which restarts a failed node on another host

A body stays alive in three different ways

This is the part that makes the analogy worth using. A body does not have one availability strategy, it has three, and they map cleanly onto what VCF does.

The bodyVCFWhat it protects against
Organs you have two of, kidneys, lungs, eyesA three node clusterLosing one, with no interruption to service
Organs you have one of, protected by boneA single node plus vSphere HALosing the host underneath, at the cost of a restart
Two of you, in two placesDisaster recovery to another VCF InstanceLosing the whole site

The third row is where most confusion starts, and it is worth separating out early. A cluster is not a backup, and a second site is not high availability. I wrote about that distinction on its own in High Availability Is Not Disaster Recovery.

The organs

Here is the component map, with what each one actually does and what you get if you give it three nodes.

ComponentIn the bodyWhat three nodes buys you
vCenterThe brainNot available. Always a single node
SDDC ManagerThe spineNot available. Always a single node
NSX ManagerThe nerves, carrying signals and deciding where they goA three node cluster behind a VIP, with an anti-affinity rule keeping the nodes on different hosts
VCF OperationsThe eyes, noticing that something is wrongPrimary, replica and data node, with data redundancy and rapid recovery from a single node failure
VCF AutomationThe hands, the part that builds thingsApplication level availability, rather than none
VCF Management ServicesThe liver, the background work that keeps things currentHigh availability for the control plane and the worker nodes

A detail worth pulling out of that table: for VCF Automation the availability model is tied to the node size rather than being a separate switch. Small is a single node. Medium and Large are a three node cluster, and resizing a Small node upwards scales the deployment out to three automatically.

Why three, and not two

This is the question the body analogy answers best, by being wrong in an interesting way.

You have two kidneys, not three. Two lungs, not three. So why does a VCF cluster need an odd number?

Because organs never have to agree with each other. Your left kidney does not need your right kidney's permission to filter blood. If the right one stops, the left one carries on, and nothing has to decide which kidney is in charge, because neither of them is.

Cluster nodes do have to agree. They hold shared state, and exactly one view of that state has to be the true one. So the failure they are defending against is not only a node dying, it is a node becoming unreachable while still running.

Why a cluster is three nodes and not two. With two nodes, if the link between them breaks each node can still see itself and assume it should take charge, which is a split brain. With three nodes, a break leaves two on one side and one on the other, the side with two has the majority and keeps running, and the single node stands down

With two nodes, a broken link between them leaves each one able to see itself and nothing else. Neither can tell the difference between the other node having died and the other node being fine but out of reach. If both assume they should take over, you get two divergent copies of the truth.

With three, a break always leaves a majority on one side. That side keeps serving, the minority side stands down, and there is never an argument about which copy is real. Three is simply the smallest number that can settle a disagreement.

That is also why the pattern repeats everywhere once you start looking. NSX Manager is three. VCF Operations is three. VCF Automation is three. The number is not a sizing decision, it is arithmetic.

What protects the parts you only have one of

vCenter and SDDC Manager have no spare, which sounds alarming until you notice the body has the same problem and the same answer.

You have one brain and one spinal cord. Evolution's response was not to grow a second one, it was to wrap both in bone. The protection is structural, and recovery from a serious hit is slow rather than instant.

vSphere HA is that bone. If the ESX host running vCenter fails, vSphere HA restarts the vCenter VM on a surviving host in the cluster. The service is interrupted while that happens, which is precisely what the design pages mean when they describe the single node models as having slower recovery from failure.

The same protection sits under everything else too, including the clustered components. NSX Manager gets it with high restart priority, and the three node model adds an anti-affinity rule so the cluster nodes do not share a host in the first place. The cluster handles the node failing. vSphere HA handles the floor giving way.

VCF Operations gets a third option

One component goes further than the rest, and it is the one that watches everything else.

Alongside the Simple and High Availability models, VCF Operations has a Continuous Availability model: node pairs split across two availability zones, with a witness in a third. A primary and its replica sit in opposite zones, as do the data nodes, and the stated benefit is no service interruption if a single availability zone fails. The documented limit is a maximum of 10ms network latency between zones.

The witness is the part with no biological equivalent at all. It holds no data and serves no users. It exists purely to be a third opinion, so that when the two zones cannot reach each other, the one that can still reach the witness knows it is the side that should carry on. It is the same majority argument as before, stretched across sites.

Where the analogy stops working

Any analogy is a loan, and it is worth being honest about where this one has to be paid back.

A body has no witness. There is nothing in human anatomy whose only job is to break a tie. This is the one piece of the design that has no counterpart, and it is also the piece that makes stretched availability work.

A body cannot be restored from backup. VCF can, and the recovery options are a genuine part of the availability story rather than an afterthought: component level restore, full instance restore, fleet disaster recovery to another instance, and cyber recovery into an isolated clean room. A body has redundancy and healing, but no restore point. VCF is better off here than we are.

Organs do not get upgraded. One of the real benefits listed for the three node NSX Manager model is no impact to the control plane during upgrade operations. Planned maintenance is a large part of why clusters earn their keep, and it has no equivalent in the body at all.

You can choose. You do not get to decide how many kidneys to grow. With VCF the decision is yours, it is made per component, and for most of these components it is made at deployment time.

Sources

Never miss a post

New guides on VMware Cloud Foundation, Aria Suite, and infrastructure automation. Follow the blog in your feed reader and new posts show up as soon as they are published.

Using a different reader? Copy the feed URL and add it there.