More on Private Cloud Academy

I presented session 2 in the Private Cloud Academy series last Friday in Microsoft Ireland.  That event focused on SCVMM 2008 R2 with SP1, Virtual Machine Servicing Tool 3.0, and Operations Manager 2007 R2 with PRO integration (with SCVMM).  It was a very demo driven session.  I had 25 slides but I probably only used half of them.  And as usual, there were lots of questions.

The next event was originally scheduled for March 18th but it has been rescheduled to March 25th.  Session 3 will focus on System Center Data Protection Manager 2010 and how you can use it in a virtualised environment.

I’ll start off with a high level view of backup and virtualisation.  For example, VM’s are usually “just” files making them easier to backup, restore, and replicate.  One of the biggest things people need to understand when backing up a Hyper-V cluster is how redirected I/O affects operations when using CSV.  And that means spending quite a bit of time on how a cluster should be designed.  That leads to backup strategy.

Once the theory is done we’ll get into the usual end-to-end demos.  I’ll be backing up VM’s on a CSV, backing up SQL workloads, and so on.  Then we move onto site-site replication of DPM, and maybe even automated restoration of VM’s in a secondary site.

If time permits, I’ll go on to talk about DR design possibilities, seeing as it is a related subject.

Sound interesting?  If so, go ahead and register if you can make it to Dublin (Ireland) on the day.

Backup VMM 2008 R2 Using PowerShell

I just found a handy page with a Microsoft-written PowerShell script for backing up a System Center Virtual Machine Manager 2008 R2 database.

You start off by writing a script with two lines:

$VMMServer = Get-VMMServer -ComputerName "<<insert the FQDN of your VMM server here>>"

Backup-VMMServer –Path "D:VMMBackups" -VMMServer $VMMServer

In the MS example, you save that as C:MyScriptsBackupVMMDatabase.ps1. I’ve tweaked it so it backs up the database to the D: drive which is probably the library drive and (hopefully) has plenty of free space.

You can create a scheduled task to run the following command (one line) for VMM 2008 R2:

PowerShell.exe -PSConsoleFile "C:Program FilesMicrosoft System Center Virtual Machine Manager 2008 R2bincli.psc1" -Command ".’C:MyScriptsBackupVMMDatabase.ps1’"

Otherwise, you can do this by hand in the Administration view –> General, and clicking the Back Up Virtual Machine Manager task in the Actions pane.  You can also use a traditional SQL backup of the database.  And don’t forget to backup the library file share!

EDIT #1:

You will need to change the PowerShell script execution policy.  You can sign the script and se the policy to AllSigned.  Or you can set it to Unrestricted.

Technorati Tags: ,,,

HP P4000 LeftHand SAN/iQ 9.0 Adds CSV Hardware VSS Provider Support

You may know that HP and Microsoft have formed a virtualisation alliance around Hyper-V.  One of HP’s key pieces in the puzzle the the iSCSI SAN formerly known as LeftHand, the P4000 series.

Cluster Shared Volume (CSV) can be backed up using a software VSS provider (i.e. Windows VSS) but this is slow.  When using DPM 2010, it’s recommended to use serialised backups.  If your hardware vendor support it, you can use their hardware VSS provider to take a snapshot in the SAN and then DPM (or whatever backup product) will use that feature for the backup.

Now back to the P4000.  Up until recently, the HP supplied DSM for MPIO was version 8.5.  The SAN/iQ software on the P4000 was also at 8.5.  Lots of people were using the 8.5 hardware VSS provider in the SAN to backup CSVs.  It seems that this was unsupported. by HP (nothing to do with MS).  In fact, it can even cause disk deadlock in a Hyper-V cluster, and lead to 0x0000009E blue screens of death (BSOD) on cluster hosts.  And that’s just the start of it!

HP did release DSM 9.0 and SAN/iQ 9.0 recently for the P4000.  These add support for using the hardware VSS provider for backing up a CSV.


So the SAN/iQ 9.0 release docs say that previous versions of SAN/iQ supported CSVs.  However, the Application Snashot Feature (hardware VSS provider/backup application) of the 8.5 release could not support quiecsed snapshots of CSVs.  In other words, it wasn’t supported to use DPM (or anything else) to perform a storage/host level backup of LeftHand with SAN/iQ 8.5 using the HP hardware VSS provider.  It is supported with v9.0.

Holistic Virtualisation Design

One of the biggest challenges I had when writing Mastering Hyper-V Deployment was choosing the ordering of the chapters. Some stuff needs to be understood before moving on. In the end I ordered it like a typical deployment. But I did make it clear that certain things needed to be considered.

One of the things I stressed was the storage. The choice of product, design, and implementation will affect what you can do, the performance, and stability. It must be considered as a central component of the entire implementation. Failure to do so will lead to project failure, maybe not today, but maybe 6-12 months down the road.

Bound to this, because of CSV and Redirected I/O, is backup. Host level backup will affect network performance. Huge CSVs being backed up will stress the CSV network and the CSV coordinator’s storage link. This means you need to consider sizing of CSVs and design backup protection groups accordingly. Hardware based VSS snapshots relieve this substantially.

Virtualisation is a foundation. We don’t build the roof of a house without doing the work on what is underneath. Identify your overall server objectives, such as DR or private cloud, and then do a holistic design.  Guess what – you’re going to have to talk to non-techies (the business) to figure out how to steer your design and to define those objects.  Have fun!

CIO’s Delaying Virtualisation Because They Don’t Trust Backup

With some incredulity, I just read a story on  where Veeam says that:

“44% of IT directors say they avoid using virtualisation for mission-critical workloads because of concerns about backup and recovery. At the same time, only 68% of virtual servers are, on average, backed up, according to the study of 500 IT directors across Europe and the US”.

That’s pretty damned amazing.  Why do I say that?  Because I know one MS partner here in Ireland sells Hyper-V because it makes backups easier and more reliable.

Hyper-V features a volume shadow snapshot service (VSS) provider.  This allows compatible backup solutions (there’s plenty out there) to safely backup VM’s at the host level.  This means that backing up a VM, its system state, its applications, and its data is a simple backup of a few files (it’s a bit more complicated than that under the hood).  From the admins perspective, it’s just like backing up a few Word documents on a file server. 

Here’s the cool bit.  When a Hyper-V VM is quiesced, the VSS providers within the VM also start up.  Any file services, Exchange services, SQL, and so on, are all put into a safe state to allow a backup to take place with no service interruption.  Everything is backed up in a safe, consistent, and reliable manner.  The result is that the organisation has a backup of the entire VM that can be restored very quickly.

Now compare being able to backup a VM by restoring a few files comapred to doing a complete restoration of a physical server when some 2-5 year old piece of tin dies.  You won’t get identical hardware and will have lots of fun restoring it.

BTW, if a physical piece of tin suddenly dies in a Hyper-V cluster then the VM just fails over to another host and starts working there.  There’s no comparison in the physical world.  Sure you can cluster there but it’ll cost you a whole lot more than a virtualisation cluster and be a lot more complicated.

Sounds good?  It gets better.  Backing up a Hyper-V cluster at the host level is actually not a good idea (sounds odd that something good starts with something bad, eh?).  This is because a CSV will go into redirected more during the backup to allow the CSV owner complete access to the file system.  You get a drop in performance as host I/O is redirected over the CSV network via the CSV owner to the SAN storage.  We can eliminate all of that and simplify backup by using VSS enabled storage.  That means choosing storage with VSS providers.  Now you backup LUNs on the SAN instead of disks on a host.  The result is quicker and more reliable backups, with less configuration.  Who wouldn’t like that?

Mastering Hyper-V Deployment Excerpts

Sybex, the publisher of Mastering Hyper-V Deployment, have posted some excerpts from the book.  One of them is from Chapter 1, written by the excellent Patrick Lownds (Virtual Machine MVP from the UK).  As you’ll see from the table of contents, this book is laid out kind of like a Hyper-V project plan, going from the proposal (Chapter 1), all the way through steps like assessment, Hyper-V deployment, System Center deployment, and so on:

Part I: Overview.

  • Chapter 1: Proposing Virtualization: How to propose Hyper-V and virtualisation to your boss or customer.
  • Chapter 2: The Architecture of Hyper-V: Understand how Hyper-V works, including Dynamic Memory (SP1 beta).

Part II: Planning.

  • Chapter 3: The Project Plan: This is a project with lots of change and it needs a plan.
  • Chapter 4: Assessing the Existing Infrastructure: You need to understand what you are converting into virtual machines.
  • Chapter 5: Planning the Hardware Deployment: Size the infrastructure, license it, and purchase it.

Part III: Deploying Core Virtualization Technologies.

  • Chapter 6: Deploying Hyper-V: Install Hyper-V.
  • Chapter 7: Virtual Machine Manager 2008 R2: Get VMM running, stock your library, enable self-service provisioning.  Manage VMware and Virtual Server 2005 R2 SP1.
  • Chapter 8: Virtualization Scenarios: How to design virtual machines for various roles and scales in a supported manner.

Part IV: Advanced Management.

  • Chapter 9: Operations Manager 2007 R2: Get PRO configured, make use of it, alerting and reporting.
  • Chapter 10: Data Protection Manager 2010: Back up your infrastrucuture in new exciting ways.
  • Chapter 11: System Center Essentials 2010: More than just SCE: Hyper-V, SBS 2008 and SCE 2010 for small and medium businesses.

Part V: Additional Operations.

  • Chapter 12: Security: Patching, antivirtus and where to put your Hyper-V hosts on the network.
  • Chapter 13: Business Continuity: A perk of virtualisation – replicate virtual machines instead of data for more reliable DR.

Hyper-V Backups Cause Large Registry Files

I saw this one in my tweet feed this morning.  It’s an important one for anyone doing host level backups of virtual machines on W2008 R2 Hyper-V host servers.  Ben Armstrong has explained how the registry file of the host will grow over time as you back up virtual machines.  This eventually leads to slow boot up times for the host.

Here’s how I understand the description:

  • A VSS backup at the host (W2008 R2) level backs up a VM
  • The backed up copy of each VHD is temporarily mounted by the parent partition for some cleanup work.
  • This mount causes a registry entry to be created.

Imagine doing this with a VSS enabled and optimized backup product like DPM 2010.   Maybe 15-20 VM’s, with an average of 2 VHD’s per VM, per host being backed up every 15 minutes.  That’s a lot of stuff going on.

Ben also warns that the fix, KB982210, is titled incorrectly.  It says Windows Server 2008 when it should say Windows Server 2008 R2.

By the way, this fix is thanks to the way that a number of MS people have embraced social media.  Ben asked people to contact him if they encountered issues with Hyper-V backups.  A number of people raised this issue, and we go this fix.

Data Protection Manager 2010 Webcast

I joined this late due to a phone conference.

This is a System Center Influencers briefing on Data Protection Manager (DPM) 2010.

The Aims

  • Single supported solution for Microsoft workloads
  • Single agent, no workload licensing
  • Enterprise scalability in the 2010 release

New Workload Additions

  • Cluster Shared Volume
  • Exchange 2010
  • SharePoint 2010

It supports OS’s going back to XP SP2.


  • Self-service end user restore from Explorer or Office
  • Self-service DBA restrore from within SQL
  • Auto protection of new databases
  • Protect 1000’s of databases per DPM server
  • Recover 2005 DB’s to SQL 2008
  • Auto protection of new content databases in SharePoint farms
  • Protect the farm, restore the document
  • Optimizations for the new and many Exchange architectures


  • CSV support
  • Item level recovery from within a VHD
  • Alternate host recovery

Client Protection

  • 1000 clients per DPM server
  • “User data only”.  Don’t protect the entire machine.
  • Uses VSS in Vista and Windows 7
  • Policy allows you to protect specific folders, so there’s no end user set up.
  • User can restore from local VSS while offline, or DPM while online.
  • While offline, the PC continues to make VSS copies and will sync them to DPM when it is online again.
Technorati Tags: ,,

HP Ireland Event at Redstone: Data Protector in The Virtualised Environment

Redstone is one of Ireland’s leading enterprise hardware providers in Ireland – I’ll be open and admit that I’m a (happy) blade and storage customer.  They are running this event today in cooperation with HP Ireland.  The goodie bag will in no way influence me 🙂

Today’s event will focus on Data Protector, HP’s backup solution, and how it can be used in a virtualised environment.  The majority of the attendees are using EVA/VMware.  About 1/4 are using Hyper-V.  A couple are using Xen and a couple are using XP SAN.  No one here is using Lefthand.  About 1/5 are using Data Protector for their backups.

  • Virtualisation solves some problems but complicates backups.
  • We need to reduce backup costs – storage amounts.
  • We need to be able to reliably restore business critical data and secure sensitive data.

A common problem is that people rush head first into virtualisation without considering the strategy for backup.


  • VM level backup: The argument by the HP speaker is that this is resource intensive.
  • Host level backup: This “doesn’t” impact the performance of the host. Hmm.  There is an issue with recovered data consistency, e.g. is there Volume Shadow Copy integration to Windows VM’s?  SQL and Exchange don’t support this.

The speakers says Data Protector allows you to take both approaches to meet suitable requirements for each VM.

Data Protector 6.11 has VMware VCB and Hyper-V support.  The core product has a license.  It has the traditional bolt-on license approach.  Virtualisation requires an “Online Backup” license.  The Zero Downtime Backup allows integration into the snapshot features of your HP storage array.

Note: that’s probably the approach you’d go with for backup of a Hyper-V CSV due to the CSV coordinator/redirected I/O issue with host level backups – assuming this is supported by Data Protector.

For storage I/O intensive applications, Data Protector can take advantage of the ability to snapshot the targeted LUN’s.  You identify a LUN to backup, the SAN creates a copy, Data Protector backups up the copy while the primary continues to be used by the application/users.  This can be a partial copy for normal backup/recovery to save storage space/costs on the SAN.  You can do a full copy of the LUN for “instant recovery”, i.e. Data Protector restores file(s) from the copy of the LUN.  This requires additional per TB licensing.  The partial copy cannot do “instant recovery” because it links back to the original storage and isn’t completely independent.  There’s a cost for these two solutions so you save it for the mission critical, storage performance sensitive data/applications.  You can do this on a replicated partner SAN to do backups in your DR site instead of in the production site.  These solutions require the VSS integrations for the storage arrays.  Note that this isn’t for VM snapshots.

Zero Time Backup and Instant Recovery can be done in VMware if the VM uses raw device mapping (pass through disks).

Hyper-V Backup Methods

  • In VM agent
  • VSS system  provider snapshots
  • VSS hardware provider snapshots
  • Full restore of VM
  • Partial restore of files
  • Offline backups for VM’s
  • Zero downtime backup
  • Instant recovery

I would guess the last two require passthrough disks.  Might be a solution for SQL/Exchange VM’s.

Really, you will end up with a combination of backup methods across the data centre, depending on VM’s, applications, and backup/recovery times/impacts.

After coffee, we had some demos of VMware backups that didn’t go so well for the HP speaker.

In summary, Data Protector gives you some HP storage integrated backup options.  Be careful and ensure that servers, OS’s, and applications support the backup type being used.

Although HP and Microsoft have announced their “Forefront” virtualisation alliance, there’s still a lot of catch up going on with regards to Hyper-V knowledge and sharing.  Thanks to Redstone for organising this up in their scenic office in the Wicklow mountains – not exactly a bad place to be just after sunrise.

Technorati Tags: ,,,