Availability Sets

Backup Versus Resiliency Versus Disaster Recovery

Most of us are no strangers to the backup versus disaster recovery conversation. Each is a different problem, typically (but not always) with different business expectations. Lately, resiliency has crawled into the mix, and a lot of social media commentary isn’t helping. In this post, I’m going to explain how I define backup, resiliency, and disaster recovery, and discuss how they impact my Azure designs for service & data availability.

Essential Terminology

There are two essential terms that we have to understand to discuss these problems/solutions:

RPO: The recovery point objective is how much data, measured in time, is lost when our solution kicks in.
RTO: The recovery time objective is how long, measured in time, services are offline while the solution kicks in.

Backup/Restore

A backup is when we take a copy of our data and (ideally) store that copy elsewhere, and even in several places. The concept is that we can restore our data from a backup if the original data (files, database, VM files, etc) are deleted either accidentally or deliberately.

The base product for backup in Azure is Azure Backup, which supports:

Azure VMs
Managed disks
Azure Files
SQL Server in Azure VMs
SAP HANA databases in Azure VMs
Azure Database for PostgreSQL servers
Azure Blobs
Azure Database for PostgreSQL Flexible server
Azure Kubernetes service
Azure Database for MySQL – Flexible Server
SAP ASE (Sybase) database on Azure VMs
Azure Data Lake Storage
Azure Elastic SAN

Quite honestly, that list is much longer than the last time I searched for it! Azure Backup covers a lot, but it doesn’t cover everything. Some solutions, like Azure SQL, feature their own backup solution.

It’s not unusual for people to bring another backup tool to Azure. The one I hear most of is Veeam Backup for Microsoft Azure. While I’ve never used Veeam hands-on, its reputation is excellent, and it has the unique ability to be platform agnostic. Want to restore VMs from Azure to Hyper-V, Nutanix, or VMware if you’re that way inclined ;)? You can with Veeam.

RPO: Backup features the longest RPO here. The data loss is depdendent on how often your backup jobs run. Daily backups? You can lose up to 24 hours of data. Backups every 15 minutes? You might lose up to 15 minutes of data.
RTO: This is where the pain can be; the RTO is how long it takes to copy your data from the backup storage to the production storage? Restoring an Azure Backup snapshot recovery point is a disk-to-disk copy. Restoring a 10 TB VM from blob storage over the network is going to be a long wait.

Disaster Recovery (DR)

The purpose of DR is to recover from a disaster. Let’s define what a disaster could be using real examples:

Hurricane Katrina was a natural disaster that wiped out huge areas of the USA in 2005.
The “black summer” bushfires in Australia destroyed millions of hectares of land in 2019-2020.
The Indian Ocean Tsunami in 2004 caused devastation in the coastal areas of many countries.
Keeping it local for me: post-storm winter floods have caused widespread damage throughout Ireland in the last few years.
Three AWS data centres were hit by drone attacks in the UAE & Bahrain in March of this year.

Disasters can be natural or they can be man-made. Disasters rarely target 1 building; they wipe out an area. They are rare – but they happen. There is another kind of disaster, which few think about:

KNP Logistics Group, a 125-year-old UK transport firm with over 700 employees, was put out of business because of a ransomware attack in 2025.

Pending (and passed in some countries) EU regulations (NIS2) consider this a disaster that subject organisations must be prepared for.

For cloud planning, if we need to prepare for disaster recovery, then we must plan for the loss of the Azure region ny replicating services/data to another region – typically the paired region. There is no one solution, and there are plenty of complicating factors. Techs that will be in scope include:

Azure Site Recovery (ASR) for Azure VMs
Geo-redundant storage (GRS) and the various geo-variants
PaaS resources that include GRS
Database replication
DevOps pipelines/workflows to redeploy resources (but not data)

There is a fun grey area here. Veeam is not only a backup solution; it is also a DR solution! You will also find that some people use backup as a budget DR solution – they replicate data from the primary location to the secondary location (Azure Backup Geo-Redundant). The right solution for your organisation is often based on business requirements and budget, with budget being the big elephant in the room.

RPO: DR replication is typically based on asynchronous replication. The RPO is often measured in seconds/minutes.
RTO: The RTO really is dependent on the complexity of services, the quantity of services to restore, the interdependencies, and how automated the process is once it starts. The RTO should be measured in hours, but a backup solution might be measured in days/weeks.

Resilience

The purpose of resilience is to enable a service to survive a localised issue, such as:

A VM crashes.
Microsoft are patching an App Service compute instance.
An Azure host is getting a firmware update.
Microsoft had a networking issue in a single data centre building.

We use resilience to keep the service operational with no perceivable outage to the service consumer. There are many ways to tackle resilience, but they are all based on scaling out:

Availability Zones: Most Azure regions have multiple data centre buildings. The buildings are split into what we see as 3 Availability Zones. Each Availability Zone has independent external network connections, power, and cooling. The theory is that if I spread the tier of a service across 3 zones, then that tier can survive 2 zones going offline. Some PaaS services, like Bastion, default to using zones; some have to be opted in. Beware of some PaaS resources, like App Service Environment, that have minimum consumption requirements to be placed across Availability Zones.
Availability Sets: If we cannot use Availability Zones (more later on this), then we can place virtual machines in Availability Sets. We can think of Availability Sets as a form of anti-affinity; machines in the same set are placed into different update domains (Azure platform updates) and fault domains (racks) in the same room in the same data centre. Microsoft does this for multi-instance PaaS services that are not using Availability Zones.
Zone Redundant Storage (ZRS): Azure storage is based on the concept of storing each block 3 times. ZRS places the replica blocks across 3 different Availability Zones. Your data remains operational even if 2 of the data centres are lost.

There are many architectural considerations to handle when you start resiliency planning.

The old pain-in-the-a** is the legacy line-of-business app that supports just a single VM. There is no scaling out to gain resiliency. Traditionally, VMs used LRS (locally redundant storage) managed disks. LRS managed disks are stored in a single data centre with the VM. There have been issues in the past where storage in a single room has gone offline, taking all three LRS replicas of the disks’ blocks offline. You can choose to use ZRS managed disks. The VM will continue to primarily use the local replica, but two replicas are stored in other Availability Zones in the same region. If the primary storage cluster goes offline, you can perform a manual process to get the VM back online with another replica.

RPO: Depending on the architecture and technologies, there is either a zero-RPO (active/active services) or an RPO of a few seconds (replicated storage).
RTO: In most cases, the RTO is 0. The one exception that I can think of is the single VM with a ZRS where the RTO is how long it takes you to force-detach the disk and create a new VM with the existing disks in another Availability Zone.

By the way, there are whole areas on networking resiliency that I could type about for hours too!

Confusion

As I have alluded to, I’ve seen some discussions on LinkedIn recently stating that Availability Zones can be used for disaster recovery. They could. Can they? Should they?

What is the disaster that you are planning for? If it’s any of the above natural disasters then I would argue that spreading your services/data across data centres located beside each other is going to lead to a sudden career-ending meeting.

Don’t give me the “Availability Zones are spread apart from each other” line. Suuuuure they are – except any of the ones that I’ve located on Google Maps, such as North Europe, West Europe, or US East to begin with.

Now, let’s get on with the practical realities of following the concept of using Availability Zones for DR. When was the last time you tried to deploy Azure VMs across Availability Zones? What about a firewall? Or App Services? Did you get an “Internal Service Error”, a weird quota error, or at least some helpful message to inform you that there was no capacity in “zone 2”? That’s been my experience for the last 14+ months in any regions that I’ve worked in. So, don’t recommend me to use a technology for emergency DR if I cannot even use it for operational resiliency!

Yes, I know that capacity issues also impact inter-region DR designs. If West Europe were to be flooded, you can be all but sure that you are not getting into North Europe thanks to the instant massive demand from many customers. I know that’s an unlikely scenario – but it’s one that some organisations must plan for. For example, I had a central government customer ask me about Azure region choice. The country in question has an “aggressive” neighbour to the east that likes to wage war on its neighbours. The local Microsoft office asked them to move into the new local Azure region soon after Ukraine was invaded. I asked the customer: “Where would Ukraine be now if all of its IT services were based in a local Azure region under 300 KM from Russia?” I’d extend that with a follow-up question now: “What if you used Availability Zones in that single region for DR?” Yes, the scenario is real – see above. Or consider if a hurricane reached Boydton in Virginia, USA, or a bushfire ran rampant in New South Wales/Victoria, Australia.

Before you go planning, please:

Understand the risks you are planning for
Have a budget
Understand the technologies
Comprehend how or if the technologies counter the risks
If the technologies are available to you at all!

Azure Availability Zones in the Real World

I will discuss Azure’s availability zones feature in this post, sharing what they can offer for you and some of the things to be aware of.

Uptime Versus SLA

Noobs to hosting and cloud focus on three magic letters: S, L, A or service level agreement. This is a contractual promise that something will be running for a certain percentage of time in the billing period or the hosting/cloud vendor will credit or compensate the customer.

You’ll hear phrases like “three nines”, or “four nines” to express the measure of uptime. The first is a 99.9% measure, and the second is a 99.99% measure. Either is quite a high level of uptime. Azure does have SLAs for all sorts of things. For example, a service deployed in a valid virtual machine availability set has a connectivity (uptime) SLA of 99.9%.

Why did I talk about noobs? Promises are easy to make. I once worked for a hosting company that offers a ridiculous 100% SLA for everything, including cheap-ass generic Pentium “servers” from eBay with single IDE disks. 100% is an unachievable target because … let’s be real here … things break. Even systems with redundant components have downtime. I prefer to see realistic SLAs and honest statements on what you must do to get that guarantee.

Azure gives us those sorts of SLAs. For virtual machines we have:

5% for machines with just Premium SSD disks
9% for services running in a valid availability set
99% for services running in multiple availability zones

Ah… let’s talk about that last one!

First, we must discuss availability sets and what they are before we move one step higher. An availability set is anti-affinity, a feature of vSphere and in Hyper-V Failover Clustering (PowerShell or SCVMM); this is a label on a virtual machine that instructs the compute cluster to spread the virtual machines across different parts of the cluster. In Azure, virtual machines in the same availability set are placed into different:

Update domains: Avoiding downtime caused by (rare) host reboots for updates.
Fault domains: Enable services to remain operational despite hardware/software failure in a single rack.

The above solution spreads your machines around a single compute (Hyper-V) cluster, in a single room, in a single building. That’s amazing for on-premises, but there can still be an issue. Last summer, a faulty humidity sensor brought down one such room and affected a “small subset” of customers. “Small subset” is OK, unless you are included and some mission critical system was down for several hours. At that point, SLAs are meaningless – a refund for the lost runtime cost of a pair of Linux VMs running network appliance software won’t compensate for thousands or millions of Euros of lost business!

Availability Zones

We can go one step further by instructing Azure to deploy virtual machines into different availability zones. A single region can be made up of different physical locations with independent power and networking. These locations might be close together, as is typically the case in North Europe or West Europe. Or they might be on the other side of a city from each other, as is the case in some in North America. There is a low level of latency between the buildings, but this is still higher than that of a LAN connection.

A region that supports availability zones is split into 4 zones. You see three zones (round robin between customers), labeled as 1, 2, and 3. You can deploy many services across availability zones – this is improving:

VNet: Is software-defined so can cross all zones in a single region.
Virtual machines: Can connect to the same subnet/address space but be in different zones. They are not in availability sets but Azure still maintains service uptime during host patching/reboots.
Public IP Addresses: Standard IP supports anycast and can be used to NAT/load balance across zones in a single region.

Other network resources can work with availability zones in one of two ways:

Zonal: Instances are deployed to a specific zone, giving optimal latency performance within that zone, but can connect to all zones in the region.
Zone Redundant: Instances are spread across the zone for an active/active configuration.

Examples of the above are:

The zone-aware VNet gateways for VPN/ExpressRoute
Standard load balancer
WAGv2 / WAFv2

Considerations

There are some things to consider when looking at availability zones.

Regions: The list of regions that supports availability zones is increasing slowly but it is far from complete. Some regions will not offer this highest level of availability.
Catchup: Not every service in Azure is aware of availability zones, but this is changing.

Let me give you two examples. The first is VM Boot Diagnostics, a service that I consider critical for seeing the console of the VM and getting serial console access without a network connection to the virtual machine. Boot Diagnostics uses an agent in the VM to write to a storage account. That storage account can be:

LRS: 3 replicas reside in a single compute cluster, in a single room, in a single building (availability zone).
GRS: LRS plus 3 asynchronous replicas in the paired region, that are not available for write unless Microsoft declares a total disaster for the primary region.

So, if I have a VM in zone 1 and a VM in zone 2, and both write to a storage account that happens to be in zone 1 (I have no control over the storage account location), and zone 1 goes down, there will be issues with the VM in zone 2. The solution would be to use ZRS GPv2 storage for Boot Diagnostics, however, the agent will not support this type of storage configuration. Gotcha!

Azure Advisor will also be a pain in the ass. Noobs are told to rely on Advisor (it is several questions in the new Azure infrastructure exams) for configuration and deployment advice. Advisor will see the above two VMs as being not highly available because they are not (and cannot) be in a common availability set, so you are advised to degrade their SLA by migrating them to a single zone for an availability set configuration – ignore that advice and be prepared to defend the decision from Azure noobs, such as management, auditors, and ill-informed consultants.

Opinion

Availability zones are important – I use them in an architecture pattern that I am working on with several customers. But you need to be aware of what they offer and how certain things do not understand them yet or do not support them yet.

Azure Schedules Maintenance & Downtime For January 9th

Microsoft are currently distributing the following email template:

Performance, security, and quality are always top priorities for us. I am reaching out to give you an advanced notice about an upcoming planned maintenance of the Azure host OS. The vast majority of updates are performed without impacting VMs running on Azure, but for this specific update, a clean reboot of your VMs may be necessary. The VMs associated with your Azure subscription may be scheduled to be rebooted as part of the next Azure host maintenance event starting January 9th, 2018. The best way to receive notifications of the time your VM will undergo maintenance is to setup Scheduled Events <https://docs.microsoft.com/en-us/azure/virtual-machines/windows/scheduled-events> .

If your VMs are maintained, they will experience a clean reboot and will be unavailable while the updates are applied to the underlying host. This is usually completed within a few minutes. For any VM in an availability set or a VM scale set, Azure will update the VMs one update domain at a time to limit the impact to your environments. Additionally, operating system and data disks as well as the temporary disk on your VM will be retained (Aidan: the VM stays on the host) during this maintenance.

Between January 2nd and 9th 2018, you will be able to proactively initiate the maintenance to control the exact time of impact on some of your VMs. Choosing this option will result in the loss of your temporary disk (Aidan: The VM redeploys to another host and gets a new temporary disk). You may not be able to proactively initiate maintenance on some VMs, but they could still be subject to scheduled maintenance from January 9th 2018. The best way to receive notifications of the time your VM will undergo maintenance is to setup Scheduled Events <https://docs.microsoft.com/en-us/azure/virtual-machines/windows/scheduled-events> .

I have put together a list of resources that should be useful to you.

* Planned maintenance how-to guide and FAQs for Windows <https://docs.microsoft.com/en-us/azure/virtual-machines/windows/maintenance-notifications> or Linux <https://docs.microsoft.com/en-us/azure/virtual-machines/linux/maintenance-notifications> VMs.

* Information about types of maintenance <https://docs.microsoft.com/en-us/azure/virtual-machines/windows/maintenance-and-updates> performed on VMs.

* Discussion topics for maintenance on the Azure Virtual Machines forums.

I am committed to helping you through this process, please do reach out if I can be of any assistance.

Regards

<Insert signature>

In short, a deployment will start on Jan 9th that will introduce some downtime to services that are not in valid availability sets. If you are running VMs that might be affected, you can use the new Planned Maintenance feature between Jan 2-9 to move your VMs to previously updated hosts at a time of your choosing. There will be downtime for the Redploy action, but it happens at a time of your choosing, and not Microsoft’s.

For you cloud noobs that want to know “what time on Jan 9th the updates will happen?”, imagine this. You have a server farm that has north of 1,000,000 physical hosts. Do you think you’ll patch them all at 3am? Instead, Microsoft will be starting the deployment, one update domain (group of hosts in a compute cluster) at a time, from Jan 9th.

And what about the promise that In-Place Migration would keep downtime to approx 30 seconds. Back when the “warm reboot” feature was announced, Microsoft said that some updates would require more downtime. I guess the Jan 9th update is one of the exceptions.

My advice: follow the advice in the mail template, and do planned maintenance when you can.

Want to Learn About In-Place Migration, Availability Sets, Update & Fault Domains?

If you found this information useful, then imagine what 2 days of training might offer you. I’m delivering a 2-day course in Amsterdam on April 19-20, teaching newbies and experienced Azure admins about Azure Infrastructure. There’ll be lots of in-depth information, covering the foundations, best practices, troubleshooting, and advanced configurations. You can learn more here.

Restore An Azure VM to an Availability Set From Azure Backup in the Azure Portal

Microsoft has shared how to restore an Azure VM to an availability set using PowerShell from Azure Backup. It’s nasty-hard looking PowerShell, and my problem with examples of VM creation using PowerShell is that they’re never feature complete.

While writing some Azure VM training recently, I stumbled across a cool option in the Azure Portal that I tried out … and it worked … and it means that I never have to figure that nasty PowerShell out

The key to all this is to start using Managed Disks. Even if your existing VMs are using un-managed (storage account) disks, that’s not a problem because you can still use this restore method. The other thing you should remember is that the metadata of the VM is irrelevant – everything of value is in the disks.

Restore the Disks of the VM

Using these steps you can restore the disks of your VM, managed or un-managed, to a storage location, referred to as the staging account.. Each disk is restored as a blob VHD file, and a JSON file describes the disks so that you can identify which one is the “osDisk”.

Create Managed Disks from the Restored VHDs

In this process, you create a managed disk from each restored VHD or blob file in the staging location. You have the option to restore the disks as Standard (HDD) or Premium (SSD) disks, which offers you some flexibility in your restore (you can switch storage types!). Make sure you ID the osDisk from the JSON file and mark it as either a Windows or Linux OS disk, depending on the contents.

Create a VM From the OS Managed Disk

The third set of steps bring your VM back online. You use the previously restored/identified osDisk and create a new virtual machine using that managed disk. Make sure you select the availability set that you want to restore the VM to.

Clean Up

The last step is the clean up. If you had any data disks in the original machine then you need to re-attach them to the new virtual machine. You’ll also need to configure the network settings of the Azure NIC resource. For example, if the new VM is replacing the old one, you should enter the IP settings of the old VM into the new NIC Azure resource, change any NAT/load balancing rules, NSGs, PIPs, etc.

And that’s it! There’s no PowerShell, and it’s all pretty simple clicking in the Azure Portal that won’t take that long to do after the disks are restored from the recovery services vault.

Create a New VM From An Existing Managed Disk

In previous posts I have shown how to restore the disks of a VM to a storage account and how to create managed disks from those VHD blobs. In this post, I will show how to create a new VM from a managed disk. When these 3 steps are done together, this is an easy way to restore an Azure virtual machine from backup to an availability set.

I previously created a managed disk from a restored VHD blob, and stored it in a resource group called demorestore. I deliberately named the new managed disk after the VM that I am going to create.

You can only create a new VM from a managed disk that contains an operating system. In the below screenshot, you can see that this disk contains Windows. If this is an OS disk, then you can click the magic button called + Create VM.

What you are doing by clicking the button is shorting the usual Create Virtual Machine blade/wizard. A blade you probably know appears, but some of the features are greyed out because they’re already selected by choosing to create a VM from an existing managed disk.

Enter the name of the new VM, and select the resource group.

In the Size blade, choose the size of the new VM. In settings, choose the availability set (key to restoring a VM to an availability set) and then all the other stuff like network, subnet, extensions, etc.

When you complete the wizard, a VM (which is just metadata) is created using your pre-existing OS managed disk. If you have any data disks to re-use, open Disks in the settings of the VM and add those managed disks with the required host caching mode. And that’s all there is to it!