Hyper-V Backup Strategies

Because you are dealing with virtual machines you have more options available to you than you did when backing up traditional tin servers.  What approach you take depends on whether you need to recover files, databases or just an entire server, what your budget is and how you configure the storage of your VM’s.  Oh yes, and your budget.

In-VM or On-Host Backup?

What does that mean?  There are two places you can do your backup from. 

On-Host Backup

This allows you to capture selected VM’s on the host as they are running.  There’s some catches to that which I’ll come back to later.  The benefit of this approach is that it’s a simple hammer that can hit everything.  If you need to recover all of your VM’s then you can do it.  But you have no knowledge of the VM’s contents nor the ability to recover single files from within the VM.  To do that with this approach you have to recover the entire VM to an isolated network, log into the VM, and then grab the files you need.

This on-host backup really needs to be Hyper-V aware, i.e. use the Hyper-V VSS (volume shadow copy service) writer.  When your backup software tries to backup the VM then the VM will be quickly brought to a “quiescent” state.  This is accomplished at two levels.  The parent partition uses VSS to access the VHD files.  The integration components feature a backup integration.  This allows the VSS writers in the VM to bring file services, Exchange, SQL (and any other VSS aware services) into a brief restful state too.  A snapshot of the VM can then be taken using VSS and the backup software get’s the VM’s running state.  Note that this quiescent state is not noticeable.  Odds are you are already using this VSS technology to backup Windows file servers, Exchange and SQL and haven’t noticed a thing.

You probably noticed a catch here.  The backup causes no noticeable downtime to the VM if (a) VSS is available in the VM operating system and (b) the backup integration component service is running in the VM.  That means you must be running Windows Server 2003 SP3 or later in the VM and you have installed the IC’s and left the backup integration service enabled.  All of the volumes in the VM must also have VSS enabled.

If you have VM’s that don’t meet both of those requirements then they must be stopped (saved state) before a backup can commence.  This will include, for example, VM’s that meet these conditions:

  • VM’s that do not have the VSS service, e.g. Linux or Windows 2000
  • VM’s that do not have the IC’s installed and the backup integration service enabled
  • VM’s that do not have VSS enabled on all of their volumes.

Some types of storage cannot be backed up in this way.  Passthrough storage is not a file like a VHD so that is excluded from this approach.  And you need to be aware of remote storage that is directly connected to the VM.  It is not connected to the parent partition so it cannot be backed up with this approach.

You should also be aware that virtual network configurations are reportedly not backed up with this approach.

However the two big benefits are:

  • You can do an “iron”-level backup of a VM.  If you lose the VM then you can instantly restore it to a known state with no need to build new VM’s, install software, patches, etc.
  • As I’ve mentioned you should not use snapshots in production.  Using a VSS backup on the host you effectively get snapshot functionality.

In-VM Backup

The second approach is to do an in-VM backup.  This is pretty much doing what you’ve always done with your physical servers.  You log into the VM and do the backup from there.  Here are the benefits:

  • You can use whatever backup tool you want that is installed in the VM.  It does not need to be Hyper-V VSS aware.  Although it doesn’t need to be Hyper-V VSS aware you should take steps to ensure you can still backup open files and backup databases (mail, Oracle, MySQL, etc) consistently.
  • You can backup remote storage that is not connected to the host, e.g. where a VM directly connects to iSCSI storage.
  • You can use this approach for Linux/Windows 2000/etc and where you do not (or cannot) install Integration Components, do/cannot not have VSS enabled on all volumes or do not/cannot enable the backup integration service.
  • Best of all, this approach allows you to selectively backup files and allows you to selectively recover files or databases.  This is because the backup is in the VM and thus is aware of the data in the VM.

Recovering a lost VM with just this approach will be time consuming.  You would have to:

  • Build a new VM and set up the operating system to be identical to the previous version including service pack.
  • Do a complete restoration of the backup data.
  • Test like crazy to ensure everything is OK.

Best of Both Worlds

The best solution is to do both types of backup.  You can do an on-host backup maybe once a day, once a week or once a month for all VM’s, depending on major changes on those VM’s.  Identify those VM’s that you need to backup/recover on a granular level, e.g. shared SQL servers, Exchange, file servers, etc.  For those machines you should configure in VM backup.  Of course, there are those VM’s that don’t meet the requirements for on-host backup.  Exclude them from the backup set and set up in-VM backups for them.  It might make sense to do an on-host backup once in a while for these VM’s.  This will require a scheduled maintenance window where you put the VM’s into a saved state to run the backup.  This will allow quicker recoveries in a major disaster for these VM’s.

Here’s how you can handle various recoveries now:

  • VM destroyed: Recover the last backup of the VM from the host level.  Restore data from in-VM backup that has changes since that on-host backup.  This will bring the VM back up to date, e.g. SQL databases.
  • Data lost from a VM, e.g. SQL database, files, etc: Recover the data from the in-VM backup.
  • Host destroyed/Office Destroyed: Recover the complete on-host backups to another host or another host in another office.  Remember to configure the virtual networks.

Backup Tools

If you are operating on a shoestring then the solution for you is Windows Server Backup.  You can use this to backup your host and VM’s.  It’s not the prettiest solution but it works.  VM’s that are backed up at the host level that are not compliant with all the requirements will need to be put into a saved state either manually or via a (PowerShell) script.  In VM backup is complicated because you need to provide storage for the backups.  That means using either iSCSI or VHD’s and that adds complexity to your storage solution.

The ideal solution in a Microsoft centric network is Data Protection Manager.  DPM 2007 SP1 can backup Windows Server 2008 hosts and clusters.  It can also backup Windows Server 2008 R2 hosts and clusters.  However the caveat for Windows Server 2008 R2 clusters is that it cannot backup VM’s that are stored on Cluster Shared Volumes (CSV) and it is not Live Migration aware.  DPM 2010 (expected to RTM in Q2 2010 and in beta now) will resolve that.

DPM installs agents on the host and in the VM’s.  Licensing costs are reduced with System Center Enterprise (host and 4 VM’s on the host) and Datacenter (host and all VM’s on the host) CAL’s/SAL’s.  You can configure protection sets with schedules of your choice and your hosts/VM’s/data will be backed up to the disk storage set(s) on the DPM server.  For those VM’s that are not compliant with the Hyper-V VSS/IC requirements, DPM will automatically put them into a saved state and do the backup.  A nice touch with DPM is that it will allow replication of the backed up data to another DPM server.  This could be in a remote location, e.g. a hosting company, and have a tape drive attached to stream data from disk to tape for archival purposes.  DPM is quite clever with backups.  It backs up at a block level.  It only backs up differences rather than entire files.  It can also compress data on the wire.

What if you’ve made an investment in other backup technologies and want to keep it simple or you have lots of non-Microsoft technology?  You have a few options:

  • If your backup vendor has Hyper-V VSS compliance then do what I’ve talked about above, picking and choosing between in-VM and on-host backups.  Windows Server 2008 R2 CSV is still pretty new so verify that the vendor also has compliance for that if you are deploying an R2 Hyper-V cluster.
  • If your backup vendor does not have Hyper-V VSS compliance then you can only do in-VM backups.  It’s not ideal but it’s what you’ve been doing up to now with your physical servers so nothing has changed.  You’re just not able to take advantage of snapshot style functionality at the host level for your VM’s.
  • Maybe add DPM into the mix for host-level backups only and do daily/weekly/monthly backups.  That way you get an “iron” level backup of the VM for those dreaded scenarios when you have to do a complete recovery.

Things To Watch Out For

  • Patches.  No matter what your backup solution is, get all of the latest patches.  DPM 2007 SP1 requires a hot fix for W2008 Hyper-V support.  Install the June 2009 rollup.  DPM 2010 requires a hotfix on W2008 R2 Hyper-V RTM clusters too.
  • DPM 2007 SP1 isn’t the completed solution for W2008 R2 clusters due to the lack of support for CSV and lack of Live Migration awareness.  If you are deploying DPM 2007 SP1 on W2008 R2 clusters then have your licensing set up to upgrade to DPM 2010 next year.
  • The Windows Server Backup approach requires a registry change on the host.  Complete instructions are on the MS site.
  • Even if you only do in-VM backups, ensure your vendor will support it.  Just because it’s in VM and should be pretty much identical to backing up a physical box, it doesn’t mean the vendor will actually support a VM backup.
  • Test the crap out of this stuff once you have a lab or a pilot set up.

Bare Metal Recovery of Windows Server 2008 with DPM SP1

Microsoft has released guidance on how to perform a bare metal or iron level recovery of W2008 using System Center Data Protection Manager Service Pack 1.

“This technical article outlines the steps of using DPM 2007 SP1 alongside the Windows Server Backup (WSB) utility to provide a supported bare metal recovery of Windows Server 2008.

System Center Data Protection Manager (DPM) 2007 is a key member of the Microsoft System Center family of management products designed to help IT professionals manage their Windows Server environments. DPM is the new standard for Windows Server backup and recovery – delivering continuous data protection for Microsoft applications, virtualization, file servers, and desktops using seamlessly integrated disk and tape media, as well as cloud repositories. DPM enables better backups with rapid and reliable recoveries for both the IT professional and the end-user. DPM helps significantly reduce the costs and complexities associated with data protection through advanced technology for enterprises of all sizes. Using complimentary technologies in addition to DPM’s actual software, DPM 2007 SP1 can perform a bare metal recovery (BMR) to restore an entire server without an operating system”.

Technorati Tags: ,,

Performance Tuning Guidelines for Windows Server 2008 R2

Microsoft has updated the Performance Tuning Guidelines document to include W2008 R2.  It covers all aspects of the server operating system but I’m going go focus on Hyper-V here.

The guidance for memory sizing for the host has not changed.  The first 1GB in a VM has a potential host overhead of 32MB.  Each additional 1GB has a potential host overhead of 8MB.  That means a 1GB VM potentially consumes 1056MB on the host, not 1024MB.  A 2GB VM potentially costs 2088MB on the host, not 2048MB.  And a 4GB VM potentially costs 4152MB, not 4096MB.

The memory savings for a Server Core installation are listed as 80MB.  That’s seriously not worth it in my opinion given the difficulty in managing it (3rd party software and hardware management) and troubleshooting it when things go wrong. “Using Server Core in the root partition leaves additional memory for the VMs to use (approximately 80 MB for commit charge on 64-bit Windows)”.

RAM is first allocated to VM’s.  “The physical server requires sufficient memory for the root and child partitions. Hyper-V first allocates the memory for child partitions, which should be sized based on the needs of the expected load for each VM. Having additional memory available allows the root to efficiently perform I/Os on behalf of the VMs and operations such as a VM snapshot”.

There is lots more on storage, I/O and network tuning in the virtualization section of the document.  Give it a read.

Can You Install Hyper-V in a VM?

The answer is sort of.  Strictly speaking it is possible.  You can indeed enable the Hyper-V role in a Server Core installation of Windows Server 2008 and Windows Server 2008 R2.  I’ve done it on both OS’s on both VMware Workstation 6.5 and on Hyper-V.  Logically this means you can deploy Hyper-V Server 2008 and Hyper-V Server 2008 R2 in a VM.

You can even create VM’s on the hosts.  However, the hardware requirements are not passed through to the VM’s and therefore the hypervisor never starts up.  That means you cannot start up those VM’s.

Why would you care?  You certainly cannot do it in a production scenario.  But you might find it handy when doing some demos, lab work or testing of clustering or VMM.


I have been told (but I have not tried this so I cannot say it will work) that you can get Hyper-V to install and run in an ESXi 3.X virtual machine.  The performance is said to be awful, but might be useful for a lab with limited hardware.

Cannot Delete Cluster Object From Operations Manager 2007

I recently decommissioned a Windows Server 2008 Hyper-V cluster.  It was monitored by OpsMgr 2007 R2.  When we shutdown the last cluster node I tried to remove both its agent object and the agentless managed cluster object from OpsMgr administration.  I couldn’t.  The cluster just refused to disappear.  The server agent would delete because there was a remaining dependency – the cluster object which relied on it as a proxy.

It had a red state (ruining my otherwise all green status view) and, more annoyingly, many of the migrated resources (VM’s) still seemed to be linked to the old cluster despite being moved to the new cluster.

I searched and found lots of similar queries.  The official line from MS is that there is no supported way to do this deletion.  There is a hack but the instructions didn’t work for me – I couldn’t find the key piece of info – plus it is unsupported.

So I uninstalled the agent manually.  No joy.  I waited.  No joy.  I rebuilt the server and added it to our Windows Server 2008 R2 Hyper-V cluster.  No joy.  I installed the OpsMgr agent and enabled the proxy setting.

That was yesterday.  This morning I logged in and the old cluster object is gone.  Vamoose!  I guess OpsMgr figured out that the server was now in a new cluster and everything was good.

How Hyper-V SCSI is Really IDE And It Doesn’t Matter

Ben Armstrong has done a good job at explaining how it doesn’t matter if you use IDE or SCSI disks in your VM.

It turns out that under the hood there’s no real difference between them.  And as you probably already know, the real decision is if you are using SAS or SATA disks underneath the virtualisation layer.

Finished Our W2008 R2 Hyper-V Cluster Migration

Last night we finished migrating the last of the virtual machines from our Windows Server 2008 Hyper-V cluster to the new Windows Server 2008 R2 Hyper-V cluster.  As before, all the work was done using System Center Virtual Machine Manager (VMM) 2008 R2.  The remaining host has been rebuilt and is half way to being a new member of the R2 Hyper-V cluster.

I also learned something new today.  There’s no supported way to remove a cluster from OpsMgr 2007.  Yuk!

Lots Of Operations Manager Updates

Microsoft released lots of updates for Operations Manager over the last couple of weeks.  There are lots of updates to management packs, too many for me to go posting them at this time of night.  Have a look on the catalogue and you’ll see them.  Or check your console if you’re using OpsMgr 2007 R2.

Most importantly is KB971541, Update Rollup for Operations Manager 2007 Service Pack 1.

“The Update Rollup for Operations Manager 2007 Service Pack 1 (SP1) combines previous hotfix releases for SP1 with additional fixes and support of SP1 roles on Windows 7 and Windows Server 2008 R2. This update also provides database role and SQL Server Reporting Services upgrade support from SQL Server 2005 to SQL Server 2008.

The Update Rollup includes updates for the following Operations Manager Roles:

  • Root Management Server, Management Server, Gateway Server
  • Operations Console
  • Operations Management Web Console Server
  • Agent
  • Audit Collection Server (ACS Server)
  • Reporting Server

The following tools and updates are provided within this update which may be specific to a scenario:

  • Support Tools folder – Contains SRSUpgradeTool.exe and SRSUpgradeHelper.msi (Enables upgrade of a SQL Server 2005 Reporting Server used by Operations Manager Reporting to SQL Server 2008 Reporting Server)
  • Gateway folder – Contains a MSI transform and script to update MOMGateway.MSI for successful installation on Windows Server 2008 R2
  • ManagementPacks folder – Contains an updated Microsoft.SystemCenter.DataWarehouse.mp which requires manual import

For a list of fixes and tools addressed by this update rollup, see KB971541.

This update is supported for application on System Center Operations Manager 2007 Service Pack 1 only.

Feature Summary

The System Center Operations Manager 2007 SP1 Rollup 1 contains:

  • All binary hotfixes released since Service Pack 1 release
  • Support for Windows 7 and Windows Server 2008 R2
  • Operational and DataWarehouse database support on Windows Server 2008 R2
  • Additional stability hotfixes”


  • Supported Operating Systems: Windows 7; Windows Server 2003; Windows Server 2008; Windows Server 2008 R2; Windows Vista; Windows XP
  • System Center Operations Manager 2007 Service Pack 1


This update must be applied to each computer that meets the following criteria:

  • Hosts a Microsoft Operations Manager Root Management Server
  • Hosts a Microsoft Operations Manager Management Server
  • Hosts a Microsoft Operations Manager Operations Console
  • Hosts a Microsoft Operations Manager Web Console Server
  • Hosts a Microsoft Operations Manager Reporting Server
  • Hosts a Microsoft Operations Manager Manually installed Agent
  • Hosts a Microsoft Operations Manager ACS Server

Before applying this update it is strongly recommended that Operations Manager databases, Management Server, Report Server and Web Console roles be backed up.

To extract the files contained in this update and installation of the update on the Operations Manager roles above:

  1. Copy the file – SystemCenterOperationsManager2007-SP1-KB971541-X86-X64-IA64-locale.MSI – To either a local folder or accessible network shared folder.
  2. Run the file – SystemCenterOperationsManager2007-SP1-KB971541-X86-X64-IA64-locale.MSI – locally on each applicable computer that meets the predefined criteria.
    You can run SystemCenterOperationsManager2007-SP1-KB971541-X86-X64-IA64-locale.MSI from either Windows Explorer or from a command prompt.
  3. Select the appropriate role to update from the Operations Manager 2007 Software Update dialog.

NOTE: To run this file on Windows Server 2008 you must run this file from a command prompt which was executed with the Run as Administrator option. Failure to execute this Windows installer file under an elevated command prompt will not allow display of the System Center Operations Manager 2007 Software Update dialog to allow installation of the hotfix”.

Upgrading Active Directory Domains to Windows Server 2008 and Windows Server 2008 R2 AD DS Domains

This guide explains the process for upgrading Active Directory domains to Windows Server 2008 and Windows Server 2008 R2, how to upgrade the operating system of domain controllers, and how to add domain controllers that run Windows Server 2008 or Windows Server 2008 R2 to an existing domain.

Upgrading your network operating system requires minimal network configuration and typically has a low impact on user operations. The upgrade process is straightforward, efficient, and allows your organization to take advantage of the improved security that is offered by the Windows Server 2008 and Windows Server 2008 R2 operating systems. This guide covers the process for upgrading domains and domain controllers, and how to add new domain controllers to existing Active Directory domains. It includes details about how to run Adprep.exe and resolve known issues and errors if they arise.