You might have heard that WS2012 R2 Hyper-V changed how it performs backups of virtual machines but not heard how. I’ll give you a glimpse of what that change was in this post.
First some terminology:
- Volume Shadow Copy Service (VSS) Snapshot: VSS performs a snapshot of a volume to get a consistent backup of a file or files/folders.
- Checkpoint: Formerly known as a (Hyper-V) snapshot, a checkpoint (matching the SCVMM term) creates a crash consistent copy of a VM at a point in time, using AVHD or AVHDX files that are linked to a parent VHD or VHDX file – or linked to a parent AVHD or AVHDX files in the case of nested checkpoints.
In the past, VSS is used to create a VSS snapshot of the volume(s) containing the files of a VM that is to be backed up. The snapshot was mounted and the required files (identified by the Hyper-V Writer) were backed up. The process was pretty complex and could lead to problems for some customers. The quality of storage hardware VSS writers had too much impact on the process. Once you have a pretty clean environment (network/storage design, patching, drivers and firmware), backup was the one problem that could hurt. CSV 2.0 sorted out most of that, but Microsoft wanted to simplify the backup process.
Thanks to live merging of checkpoints (snapshots) that was added in WS2012, and the experience Microsoft has gained by using checkpoints (snapshots) for other features (such as Hyper-V Replica), WS2012 R2 Hyper-V has switched to using checkpoints instead of VSS snapshots.
Now a checkpoint is created of every virtual machine that is to be backed up. Writes are temporarily redirected to the checkpoint’s AVHDX/AHVD file(s). This gives the backup requestor a clean & crash consistent copy of the virtual machine’s VHD or VHDX files that are safe to read. After the backup, the checkpoint is merged and the job is done.
Note: You might notice that a VSS snapshot is still taken of the relevant volume(s).
We’ve moved from application consistent backup to the next (small) step down with a crash consistent backup. This isn’t a big deal – not for backup experts anyway. For products like SQL Server or Exchange, restoring this VM is like someone reset the VM. The restored database starts up, does a quick cleanup, and carries on operating as it did before the backup operation. In return, we get a much simpler backup process that should prove to be more resilient and selective.
Thanks Aidan,
That’s interesting, I’m still having problems backing up on WS2012.
Should “WS2012 Hyper-V has switched to using checkpoints instead of VSS snapshots” read “WS2012R2 …”
Tim
Changed. THanks.
Is this a possible workaround to backup Lotus Domino servers?
The last I looked, Lotus were unaware that virtualization exists. They’re worse than Exchange, and that’s saying something. You’ll have to take it up with them. If crash consistent is good enough for Domino to recover, then it might be.
I agree that moving away from VSS hardware provider on HOST is good ( having support where a VSS hardware provider is involved ) is a nightmare i wonder why do not use VSS on guest OS so that Exchange and SQL and any other VSS aware app can do their work to have application consistency.
You say “YOU” … I do not work for Microsoft.
Thanks for the explanation! After having read many articles about Windows Server Backup and Hyper-V as far as I understand in order to be able to restore individual files from a backup (without having to restore complete VMs) I have to run WSB from inside my guest VMs. Additionally, Server 2012 R2 Essentials offers a wonderful capability to back up client computers. I suppose to use this feature I really need to perform those client backups from the Guest VM (running Server Essentials). So if I want to store those client backups locally on my server where is the recommended place to put those client backups? I suppose a pass-through disk attached to my Guest VM would be the best. Or do you still recommend to store backups inside a virtual hard disk (VHDX)? In the latter case I would have VHDX files (created by WBS) inside a VHDX. This is probably not recommended for performance reasons etc. Thanks for any recommendation.
What is the following for?
“Note: You might notice that a VSS snapshot is still taken of the relevant volume(s).”
Hi, appreciate that this post is a couple of years old now, but we’re running into a problem with Windows Server Backups of Hyper-V VMs. The partition that holds the .VHDXes for the VMs fills up every time a backup is run, and we can see that .AVHDX files are the culprit, the thing is we’ve turned off the Backup Hyper-V integration services and Windows Server Backup is showing that it’ll do offline backups, so surely this shouldn’t now happen? Also, we’ve change the checkpoint location to a drive with more space, but backup appears to ignore this and create the ‘checkpoint’ AVHDX files in the same location as the .VHDXes.
See above. Add more space and/or use Storage Live Migration to move VMs to other LUNs.
Very interesting! I’m running Hyper-V 2012 R2 Datacenter as a Hyper-V Host running several 2008 R2 SP1 Guest VM’s on iSCI volumes. None have checkpoints, yet many show a “Data-AutoRecovery.avhd” file next to their corresponding OS or Data VHD.
A shutdown does not merge the .avhd’s, so I’m assuming it’s to do with vssadmin. I look at configure shadow copies and it’s showing volume P: DISABLED next run time, used 193Gb on J:
We are using ARCSERVE r16.5 as our backup software and it has a Hyper-V agent on the HOST.
I cannot be sure but it smells like ArcServe screwed the pooch here.
Sirs,
We have two hyper-v clustered sharing a CSV on starwind. However, we managed to setup veeam for testing.
Tried to backup an online and offline virtual machine with no luck…
Error receiving:
unable to create snapshot (Microsoft Software Shadow copy provider 1.0; Microsoft CSV shadow copy provider) (mode: crash consistent) . details:
unknown status of async operation.
the shadow copy provider timed out while flushing data to the volume being shadow copied. this is probably due to excessive activity on the volume.
try again later when the volume is not being used so heavily. failed to create vss snapshot / failed to perform pre-backup tasks
Any hint or advice?
Open up a support call with Veeam.
Is there anyway to redirect the AVHDX/AVHD file to a separate volume?
The next thing I did was disable the Integration Service for backup and try another backup. It worked just the same as it did in the previous test. Of course, its contents would be backed up in crash-consistent rather than application-consistent fashion as it has no way of knowing that a backup is occurring.
Hi Aidan,
I have noticed that regardless what option I chose for optimize backup performance (normal, optimize or custom) my Hyper-V VM is always backed-up full, not incremental. Worth mentioning that its VHDX resides on a VSS-enabled volume.
Can you help me understand if W2K12R2 does only full backups to Hyper-v VMs or am I doing something wrong?
10x!
Sergiu.
Thanks Aiden really really useful read as I couldn’t for the life of me understand why differencing disks were being created and bloated – a combination of windows backup and overnight file migrations was the cause. Fantastic read thank you again!