High Availability Is Not Disaster Recovery

Somebody will tell you the environment is covered because it is highly available. Ask what happens if the datacentre loses power and you tend to get a pause, then a slightly less confident answer.

High availability and disaster recovery get used as though they are the same thing bought in different sizes. They are not. They protect against different failures, they cost different money, and having one of them says almost nothing about whether you have the other.

The difference in one line

High availability keeps you running when a component fails. Disaster recovery gets you running again when the whole place fails.

That is the entire distinction, and nearly everything else follows from it.

A server dies, a disk fails, a power supply gives up, a host stops responding. That is what HA is for, and it should happen so smoothly that nobody files a ticket. A region loses power, a fire suppression system goes off, a fibre cut takes out a site, a flood arrives. That is disaster recovery, and everybody is going to know about it.

The two numbers

Every conversation about either of these should end up at two figures. If it does not, nobody has actually designed anything.

RPO, the recovery point objective. How much data can you afford to lose, measured in time. An RPO of thirty seconds means that after a disaster, the last thirty seconds of work is gone.

RTO, the recovery time objective. How long can you be down before it really hurts.

They point in opposite directions from the same moment.

RPO looks backwards and is about data. RTO looks forwards and is about time. People routinely quote one and assume it covers the other, and then discover during an actual incident that they are back online quickly with four hours of orders missing.

Side by side

	High availability	Disaster recovery
Protects against	A component failing	A site or region being lost
Scope	A node, a disk, a host	The entire environment
Distance	Same site, or a nearby one	Deliberately far away
Replication	Usually synchronous	Almost always asynchronous
RPO	Zero	Seconds to minutes
RTO	Seconds	Minutes to hours
Who starts it	Nothing, it just happens	A person, after a decision
How often it runs	Constantly, invisibly	Rarely, ideally never
What it costs	Built into the design	Capacity you pay for and hope to waste
How it gets tested	By itself, all the time	By an exercise you keep moving

The row worth sitting with is the one about who starts it. HA is automatic and boring. DR involves somebody senior saying the words "we are invoking DR", and that is a decision people are reluctant to make, because invoking it has consequences and the alternative is hoping the site comes back in the next ten minutes. That hesitation is part of your real RTO, and it never appears in the design document.

Three failures, one shop

Take an online shop. Web servers, an application tier, a database holding orders. It runs in Region 1 with three nodes behind a load balancer, a database with a synchronous standby in a second availability zone, and an asynchronous replica sitting in Region 2, a few hundred kilometres away.

Tuesday, 2am. A node dies. The load balancer stops sending it traffic, the other two absorb the load, and the queue depth barely moves. Somebody picks it up in the morning. No data lost, no outage, no phone call. That is HA doing exactly what you paid for.

Thursday, 4pm. Region 1 goes dark. A power event takes out the whole facility. All three web nodes are gone. Both database copies are gone, because they were both in Region 1. HA bought you nothing here, because the failure took out the thing HA was protecting. You declare a disaster, promote the replica in Region 2, repoint DNS, and you are serving again forty five minutes later. The last thirty seconds of orders never made it across. That is DR, and note that it cost you both time and data.

Friday, 11am. A migration drops the orders table. HA replicated the DROP to the synchronous standby immediately. DR replicated it to Region 2 about thirty seconds later. You now have three perfectly healthy, perfectly synchronised copies of a database with no orders in it.

That third one is the whole point. Replication is obedient. It does not evaluate whether the change was a good idea, it just copies it. HA and DR both protect you from infrastructure failing. Neither protects you from the infrastructure working correctly on a bad instruction. Only a backup you can restore to a point in time before the mistake does that.

If you take one thing from this, take that. HA, DR and backup are three separate controls against three separate failures, and owning two of them is not the same as being covered.

Why you cannot simply have both

The obvious question is why the DR replica is asynchronous. Make it synchronous and the RPO goes to zero, problem solved.

Physics says no. Light in fibre covers roughly two hundred kilometres every millisecond, so a round trip adds about a millisecond for every hundred kilometres of separation. Synchronous replication means the write is not acknowledged until the far end confirms it. Put your second site three hundred kilometres away and you have added several milliseconds to every single write, forever, including the ones happening while nothing is wrong.

So you are squeezed from both sides. Far enough away to survive a regional event means far enough to hurt every write. Close enough for synchronous replication means close enough to share the same flood plain, the same grid, and the same regional network. Most people resolve it exactly the way that shop did, with synchronous replication over a short distance for HA and asynchronous replication over a long distance for DR, and they accept a small RPO as the price.

That is not a compromise anyone should apologise for. It is the design.

The parts people forget

Failback. Everybody rehearses the failover. Almost nobody rehearses the journey home. Region 1 comes back, and now you have to move a live, changed dataset back the other way without another outage. It is frequently harder than the failover was.

The dependencies. Your application replicates beautifully. Now check that the DR site has working DNS, an identity provider, certificates that have not expired, secrets, licence servers, and monitoring. DR plans fail on these far more often than on the application itself.

The test. An untested DR plan is a hypothesis. It is a document describing what somebody believed in the year it was written. If you have never failed over in anger, your real RTO is unknown, and it is not the number on the slide.

What to ask

• What is our RPO and our RTO, as numbers, for this specific system?
• Are the HA copies in the same failure domain? Same rack, same power, same region?
• When did we last actually fail over, not on paper?
• How do we get back afterwards?
• If somebody deletes the data rather than the datacentre, which of these helps?

Get the last one answered honestly and you will usually discover whether the environment is genuinely covered or just comfortably redundant.