Clustered Hyper-V Host Virtual Machine Capacity Increases

Last week at TechEd, Microsoft announced an increase in the number of supported virtual machines in a Hyper-V Cluster.  You may know that a Hyper-V host supports up to 384 running VM’s.  But up to now, only 64 VM’s were supported on a clustered Hyper-V host.

That changes now.  Microsoft supports up to 1000 VM’s in a cluster, regardless of how many Hyper-V hosts are in the cluster.  With one exception,of course: You’ll see that a 2 node cluster is limited to 384 VM’s (N+1 hosts) because of the Hyper-V limit of 384 per host.

  • 16 nodes: ~62 VM’s/node
  • 8 nodes: 125 VM’s/node
  • 4 nodes: 250 VM’s/node
  • 3 nodes: 333 VM’s/node
  • 2 nodes: 384 VM’s/node

This is nicely timed with Dynamic Memory and new CPU’s allowing for greater numbers of VM’s per host.  Now we can host them in a supported manner.

The Sanbolic Melio FS (File System) With Hyper-V

I was contacted last month by Eva Helen in Sanbolic to see if I’d be interested in learning more about their Melio FS product.  I knew little about it and was keen to learn.  So Eva got David Dupuis to give me a demo.  Dave just ran an hour long demo on LiveMeeting with me and I learned a lot.  I gotta say, I’m impressed with this solution.

There were two Sanbolic products that Dave focused on:

  • La Scala = Cluster Volume Manager used instead of disk manager.  It’s a shared volume manager.  It is aware of what nodes are attached to it. 
  • Melio = Cluster File System. 

La Scala

  • La Scala can mirror volumes across 2 SAN’s, allowing for total SAN failure.  Each server has two controllers or a dual channel HBA, one path going to each SAN.  1 write is converted to two writes on two paths.  In theory, there’s no noticeable performance hit for amazing fault tolerance.
  • On the fly volume expansion
  • Can use any block based shared storage iSCSI or fibre channel system
  • You can set up a task, e.g. expand disk, and review it before committing the transaction.
  • Windows ACL’s are integrated in the interface to control volume access rights.

I’ve got to say, the SAN mirroring is pretty amazing technology.  Note the performance will equal the slowest SAN.  It can take cheap storage solutions that might not even have controller/path fault tolerance and give them really high fault tolerance via redundant arrays and mirrored storage with an unperceivable performance hit due to the mirroring being done by simultaneous writes by 2 independent controller paths.


  • This is 64-bit symmetrical cluster file system.
  • There is no coordinator node, management server, metadata controller, etc, that manages the overall system.  So there’s no redirected I/O mode *cheers from Hyper-V admins everywhere*
  • Metadata is stored on the file system and every node in the cluster has equal access to this.  This is contrary to the CSV coordinator in W2008 R2 failover clustering.
  • QoS (quality of service) allows per process or per file/folder file system bandwidth guarantees.  This allows granular management of SAN traffic for the controlled resources.  In the Hyper-V context, you can guarantee certain VHD’s a percentage of the file system bandwidth.  You can also use wildcards, e.g. *.VHD.  This is another very nice feature.
  • There is a VSS provider.  This is similar to how SAN VSS providers would work.  Unlike CSV, there is no need for redirected I/O mode when you snap/backup the LUN. 
  • There is a bundled product called SILM that allows you to copy (via VSS) new/modified files to a specified LUN on a scheduled basis.
  • Backups solutions like BackupExec that recognise their VSS provider can use it to directly backup VM’s on the Melio file system.
  • MS supports this system, i.e. Failover Clustering and VMM 2008 R2.  For example, Live Migration uses the file system.  You’ll see no CSV or storage in Failover Clustering.  The Melio file system appears as a normal lettered drive on each node in the cluster.
  • By using advance exclusive lock detection mechanisms that CSV doesn’t have, Melio can give near raw disk performance to VHD.  They say they have faster (57%) VHD performance than CSV!
  • You can provide iSCSI accessed Melio file systems to VM’s.  You can license the product by host –> gives you 4 free VM licenses.
  • Melio isn’t restricted to just Hyper-V: web servers, SQL, file servers, etc.
  • Issues seen with things like AV on CSV aren’t likely here because there is no coordinator node.  All metadata is available to all nodes through the file system.  You need to be aware of scheduled scans: don’t have all nodes in the cluster doing redundant tasks.  The tip here: put a high percentage guarantee for *.VHD and the AV has been controlled.

It’s got to be said that you cannot think of this as some messy bolt on.  Sanbolic has a tight relationship with Microsoft.  That’s why you see their Melio file system being listed as a supported feature in VMM 2008 R2.  And that can only happen if it’s supported by Failover Clustering – VMM is pretty intolerant of unsupported configurations.

Overall, I’ve got to say that this is a solution I find quite interesting.  I’d have to give it serious consideration if I was designing a cluster from scratch and the mirroring option raises some new design alternatives.

My $64,000,000 question has probably been heard by the guys a bunch of times but it got a laugh: “when will Microsoft buy Sanbolic and have you invested a lot in the company share scheme?”.  Seriously though, you’d think this would be a quick and superb solution to get a powerful cluster file system that is way ahead of VMFS and more than “just” a virtualisation file system.

Thanks to the kind folks at Sanbolic for the demo.  It’s much appreciated!

Live Migration Network Configuration Guide

Microsoft has released a configuration guide on how to set up networking cards in a Windows Server 2008 R2 Hyper-V cluster with Live Migration.  There’s been a bit of confusion on this topic for those who are new to Hyper-V and failover clustering.  It hasn’t been helped with various failover clustering posts making recommendations that don’t take account of CSV or Live Migration.  Once you get to know this stuff, it is actually not too bad at all.

Multi-Site Live Migration Clustering Whitepaper By HP

I just saw a tweet by one of the Microsoft virtualisation feeds, announcing that HP had released a white paper on how to do Hyper-V Live Migration/clustering across multiple sites using the HP StorageWorks Cluster Extension.

“This paper briefly touches upon the Hyper-V and Cluster Extension (CLX) key features, functionality, best practices, and various use cases. It also objectively describes the various scenarios in which Hyper-V and CLX complement each other to achieve the highest level of disaster tolerance.

This paper concludes with the step-by-step details for creating a CLX solution in Hyper-V environment from scratch. There are many references also provided for referring to relevant technical details”.

Can You Install Hyper-V in a VM?

The answer is sort of.  Strictly speaking it is possible.  You can indeed enable the Hyper-V role in a Server Core installation of Windows Server 2008 and Windows Server 2008 R2.  I’ve done it on both OS’s on both VMware Workstation 6.5 and on Hyper-V.  Logically this means you can deploy Hyper-V Server 2008 and Hyper-V Server 2008 R2 in a VM.

You can even create VM’s on the hosts.  However, the hardware requirements are not passed through to the VM’s and therefore the hypervisor never starts up.  That means you cannot start up those VM’s.

Why would you care?  You certainly cannot do it in a production scenario.  But you might find it handy when doing some demos, lab work or testing of clustering or VMM.


I have been told (but I have not tried this so I cannot say it will work) that you can get Hyper-V to install and run in an ESXi 3.X virtual machine.  The performance is said to be awful, but might be useful for a lab with limited hardware.

Cannot Delete Cluster Object From Operations Manager 2007

I recently decommissioned a Windows Server 2008 Hyper-V cluster.  It was monitored by OpsMgr 2007 R2.  When we shutdown the last cluster node I tried to remove both its agent object and the agentless managed cluster object from OpsMgr administration.  I couldn’t.  The cluster just refused to disappear.  The server agent would delete because there was a remaining dependency – the cluster object which relied on it as a proxy.

It had a red state (ruining my otherwise all green status view) and, more annoyingly, many of the migrated resources (VM’s) still seemed to be linked to the old cluster despite being moved to the new cluster.

I searched and found lots of similar queries.  The official line from MS is that there is no supported way to do this deletion.  There is a hack but the instructions didn’t work for me – I couldn’t find the key piece of info – plus it is unsupported.

So I uninstalled the agent manually.  No joy.  I waited.  No joy.  I rebuilt the server and added it to our Windows Server 2008 R2 Hyper-V cluster.  No joy.  I installed the OpsMgr agent and enabled the proxy setting.

That was yesterday.  This morning I logged in and the old cluster object is gone.  Vamoose!  I guess OpsMgr figured out that the server was now in a new cluster and everything was good.

How W2008 R2 Live Migration Works

Let’s recap the different types of migration that we can get with Windows Server Hyper-V and System Center Virtual Machine Manager:

  • Quick Migration: Leveraging Windows Failover Clustering, a VM is treated as a clustered resource.  To quick migrate, the running state is saved to disk (hibernating the VM), the disk failed over to another node in the cluster, and the saved state is loaded (waking up the VM).
  • Offline Migration: This is when we use VMM to move a powered down VM from one un-clustered Hyper-V server to another or from one cluster to another.
  • Quick Storage Migration: This is a replacement for Offline Migration for Windows Server 2008 R2 Hyper-V servers when using VMM 2008 R2. A running VM can be moved from one un-clustered host to another or from one cluster to another with only around 2 minutes.
  • Live Migration: This is the process of moving a virtual machine from one cluster node to another with no perceivable downtime to network applications or users.  VMware refer to this as VMotion.  It was added in Windows Server 2008 R2 Hyper-V and is supported by VMM 2008 R2.

Live Migration was the big stick that everyone beat up Windows Server 2008 Hyper-V.  A few seconds downtime for a quick migration was often good enough for 75%-90% of VM’s but not for 100%.  But you can relax now; we have Live Migration.  I’m using it in production and it is good!  I can do host maintenance and enable completely automated PRO tips in VMM without worrying of any downtime, no matter how brief, for VM’s.  How does Live Migration Work?  Let’s look at how it works.

imageAbove, we have a virtual machine running on host 1.  It has a configuration and a “state”.

imageWhen we initiate a live migration the configuration of the VM is copied from host 1 when the VM is running to host 2, the destination host.  This builds up a new VM.  The VM is still running on host 1.

imageWhile the VM remains running on host 1, the memory of the VM is broken down and tracked using a bitmap.  Each page is initially marked as clean.  The pages are copied from the running VM on host 1 to the new VM sitting paused on host 2.  Users and network applications continue to use the VM on host 1.  If a RAM page changes in the running VM on host 1 after it has been copied to host 2 then Windows changes the state from clean to dirty.  This means that Windows needs to copy that page again during another copy cycle.  After the first RAM page copy cycle, only dirty pages are copied. As memory is copied again it is marked as clean.  As it changes again, it is marked as dirty.  This continues …

So when does all this stop?

  1. The process will cease if all pages have been copied over from host 1 to host 2 and are clean.
  2. The process cease if there is only a tiny, tiny amount of memory left to copy, i.e. the state. This is tiny.
  3. The process will cease if it has done 10 iterations of the memory copy. In this scenario the VM is totally trashing it’s RAM and it might never have a clean bitmap or tiny state remaining.  It really is a worst case scenario.

Note: The memory is being copied over a GB network.  I talked about this recently when I discussed the network requirements for Live Migration and Windows Server 2008 R2 Hyper-V clusters.

Remember, the VM is still running on host 1 right now.  No users or network applications have seen any impact on uptime.

imageStart your stop watch.  This next piece is very, very quick.  The VM is paused on host 1.  The remaining state is copied over to the VM on host 2 and the files/disk are failed over from host 1 to host 2.

imageThat stop watch is still ticking.  Once the state is copied from the VM on host 1 to host 2 Windows will un-pause it on host 2.  Stop your stop watch.  The VM is removed from host 1 and it’s running away on host 2 as it had been on host 1.

Just how long was the VM offline between being paused on host 1 and un-paused on host 2?  Microsoft claims the time is around 2 milliseconds on a correctly configured cluster.  No network application will time out and no user will notice.  I’ve done quite a bit of testing on this.  I’ve pinged, I’ve done file copies, I’ve used RDP sessions, I’ve run web servers, I’ve got OpsMgr agents running on them and not one of those applications has missed a beat.  It’s really impressive.

Now you should understand why there’s this "long" running progress bar when you initiate a live migration.  There’s a lot of leg work going on while the VM is running on the original host and then suddenly it’s running on the destination host.

VMware cluster admins might recognise the above technique described above.  I think it’s pretty much how they accomplish VMotion.

Are there any support issues?  The two applications that come to mind for me are the two most memory intensive ones.  Microsoft has a support statement to say that SQL 2005 and SQL 2008 are supported on Live Migration clusters.  But what about Exchange?  I’ve asked and I’ve searched but I do not have a definitive answer on that one.  I’ll update this post if I find out anything either way.

Edit #1

Exchange MVP’s Nathan Winters and Jetze Mellema both came back to me with a definitive answer for Exchange.  Jetze had a link (check under hardware virtualization).  The basic rule is that a DAG (Data Availability Group) does not support hardware virtualisation if the hosts are clustered, i.e. migration of an Exchange 2010 DAG member is not supported.