Tag: Live Migration

Comparing TCPIP, Compressed, and SMB WS2012 R2 Hyper-V Live Migration Speeds

I’m building a demo for some upcoming events, blatantly ripping off what Ben Armstrong did at TechEd – copying is the best form of flattery, Ben Smile  In the demo, I have 2 Dell R420 hosts with a bunch of NICs:

  • 2 disabled 1 GbE NICs
  • 2 Enabled 1 GbE NICs teamed for Live Migration
  • 2 10 GbE iWARP (RDMA) NICs not teamed for cluster, SMB Live Migration, and SMB 3.0 storage
  • 2 10 GbE NICs teamed for VM networking and host management

It’s absolutely over the top for real world but it gives me demo flexibility, especially to do the following.  In the demo, I have a PowerShell script that will perform a measured Live Migration of a VM with 8 GB RAM (statically assigned).  The VM is a pretty real workload: it’s running WS2012 R2, SQL Server, and VMM 2012 R2.

The script then does:

  1. Configure the cluster to use the 1 GbE team for Live Migration with TCPIP Live Migration
  2. Live migrate the VM (measured)
  3. Configure the cluster to use the 1 GbE team for Live Migration with Compressed Live Migration
  4. Live migrate the VM (measured)
  5. Configure the cluster to use a single 10 GbE iWARP NIC Live Migration with SMB Live Migration (SMB Direct)
  6. Live migrate the VM (measured)
  7. Configure the cluster to use a both 10 GbE iWARP NIC Live Migration with SMB Live Migration (SMB Direct + Multichannel)
  8. Live migrate the VM (measured)

What I observed in my test runs:

  • TCP/IP: About 95% of a 1 GbE NIC is utilised consistently for the duration.
  • Compressed: The bandwidth utilisation has a saw tooth pattern up to around 98%, as one should expect with the dynamic nature of compression.  CPU utilisation is higher (as expected), but remember that Live Migration will switch to TCP/IP if compression is contending for resources with the host/VMs.
  • SMB Direct: Smile Nearly 10 Gbps over a single NIC.
  • SMB Direct + SMB Multichannel: Open-mouthed smile Nearly 20 Gbps over the two iWARP rNICs.

And the time taken for each Live Migration?

image

Over 78 seconds to move a running VM over a 1 GbE network without optimizations!  Imagine that scaled out to a host with 250 GB RAM of production VM memory, needing to be drained for preventative maintenance.  That’s over 40 minutes, but it could be longer.  That’s a long time to wait to get critical services off of a host before a hardware warning becomes a host failure.

As the Live Migrations get faster they get closer to the theoretical minimum time.  There are four operations:

  1. Build the VM on the destination host (that magic 3% point, where the VM’s dependencies are attempted to be prepared)
  2. Copy RAM
  3. Sync RAM if required
  4. Destroy the VM on the source host

The first and last operation cannot be accelerated, generally taking a couple of seconds each.  In fact, the first operation could take longer if you use Virtual Fiber Channel. 

This test with with a more common VM with 8 GB RAM.  Remember that I moved a VM with 56 GB RAM in 35 seconds using SMB Direct + Multichannel?  That test was 33 seconds earlier today on the same preview release.  Hmm, I think that hardware would take 2.5 minutes to drain 250 GM RAM of VMs, versus 42 minutes of un-optimised Live Migrations.  I hope the point of this post is clear; if you need dense hosts then:

  • Use 10 GbE Networking; If you can’t upgrade to WS2012 R2 Hyper-V and use compression
  • If you’re using rNICs for storage then leverage that bandwidth and offload for optimising Live Migration, subject to QoS and SMB Bandwidth Constraints

Passthrough Disks –ARE– Still Supported By WS2012 Hyper-V Live Migration, But With Limits

I recently reported on a new KB article that says:

Changes and improvements in Windows Hyper-V Server 2012 no longer support pass-through disks if Live Migration is to be used.

In other words, Live Migration was allegedly not supporting the use of passthrough disks.

That article was incorrect. 

The story is:

1) Hans Vredevoort told me that found a contradicting blog comment/response by Jeff Woolsey where he stated that Live Migration of VMs with passthrough disks would be supported in what I’ll call legacy scenarios:

Pass through disks are still supported with Windows Server 2012 Hyper-V Live Migration (just like they were with Windows Server 2008 R2 Hyper-V) as long as the migration of a clustered VM and the pass through disk is managed by the cluster. For migrations outside of a cluster, such as:

  • Shared Nothing Live Migration or
  • Using standalone hosts with file on a SMB share (without clustering enabled)

…pass through disks aren’t supported because the pass through disk doesn’t have a way to move between hosts.

That makes total sense.  The passthrough disk (which is not a CSV) has to be passed from one host to another, and only a cluster-managed disk can do this.

Therefore the new scenarios in WS2012 Hyper-V cannot support Live Migration.  Again, that makes total sense:

  • Non-clustered SMB-stored-VM Live Migration – you can’t store a LUN on a file share!
  • Shared-Nothing Live Migration – until you can transport a LUN from one server to another Star-Trek-style, it isn’t happening.

2) Who was correct, Jeff Woolsey or the KB article?  We needed clarity so I reached out to the Hyper-V group in Redmond.  They responded overnight and Jeff was right.  The KB article was … a … bit enthusiastic (I personally loved the message – I’m quite clear in my training that I’ll smack anyone who I find using passthrough disks).  In fact, that KB article has been deleted by Microsoft.

So those of you who are panicking about your passthrough disks, you can calm down now.

…

However …

The advice from everyone who knows anything about Hyper-V is that you should switch to using VHDX files.  This isn’t just me.  Check out any of the Virtual Machine MVPs on this topic.  Read what Didier Van Hoye or Hans Vredevoort (both being the top 2 storage guys in our group IMO) have to say on the topic.

  • VHDX scales out to 64 TB
  • It has new health features to limit corruption
  • It supports 4K sector matching for performance
  • It offers the legacy VHD types, where Fixed will be nearly the same speed as the underlying disk

I heard loads of complaints over the “death” of passthrough disks in the last 7 days.  To you I say, you need to put down your toys of the 2000s. and join us in the 2010s.  We virtualise now for flexibility.  The business demands it, and passthrough disks are not flexible.

The one argument I’ll give some credence to is “I can’t hot expand a VHDX”.  If you are hot expanding LUNs every couple of days then you’ve got other issues, but I too would like this feature.

Anywho, panic over.  You can Live Migrate a VM with passthrough disks as long as both the VM and the passthrough disks are managed by a Hyper-V cluster.  I’m going back to my lazy vacation day now.

Multi-Site Live Migration Clustering Whitepaper By HP

I just saw a tweet by one of the Microsoft virtualisation feeds, announcing that HP had released a white paper on how to do Hyper-V Live Migration/clustering across multiple sites using the HP StorageWorks Cluster Extension.

“This paper briefly touches upon the Hyper-V and Cluster Extension (CLX) key features, functionality, best practices, and various use cases. It also objectively describes the various scenarios in which Hyper-V and CLX complement each other to achieve the highest level of disaster tolerance.

This paper concludes with the step-by-step details for creating a CLX solution in Hyper-V environment from scratch. There are many references also provided for referring to relevant technical details”.

High Level Document On W2008 R2 Hyper-V and Live Migration

I just stumbled upon a very high level document on Windows Server 2008 R2 Hyper-V, featuring Live Migration.  It’ll be useful to newbies to Hyper-V and those wanting to learn a bit about Live Migration.  However, if you’re a tech and need to present to your boss, this might not be a bad start.

Technorati Tags: ,

Hyper-V Live Migration and Linux VM’s

Let’s get this out of the way quickly.  Yes, you can run Linux virtual machines on a Hyper-V cluster and you can Live Migrate them.  I have SUSE Enterprise Linux 10 SP1 VM’s running on our cluster.  I can live migrate them from one host to another and not lose a ping packet during the move.

There’s a configuration that you must to to ensure this stability.  I first read about it online and it is in the Microsoft documentation for the Linux integration components.

You need to set the MAC (Ethernet) address of the virtual machine to be static.  VMM makes that quite easy. 

image

Above you can see the properties of a SLES VM on our Hyper-V cluster.  You can see that I’ve put the VM into a VLAN so I can firewall it.  I’ve also set the NIC to have a static MAC address.  Unlike most controls for networking, this must be set while the VM is powered off.  There’s a button on the right which allows you to generate a MAC address.  This is created from a pool of MAC addresses.

image

VMM allows you to specify what that pool of MAC addresses is.  It must be a range that does not exist on any hardware – there’s always the chance that you could otherwise accidentally set a MAC address for a VM that clashes with that of an actual Ethernet network card and cause all sorts of ARP issues.

Once you have that setting configured, boot up your VM, install the OS, install the IC’s and test.  Here I have run a ping from within the VM to the default gateway while running a live migration from VMM:

64 bytes from 192.168.100.31: icmp_seq=15 ttl=64 time=0.469 ms
64 bytes from 192.168.100.31: icmp_seq=16 ttl=64 time=0.052 ms

— 192.168.100.31 ping statistics —
107 packets transmitted, 107 received, 0% packet loss, time 106053ms
rtt min/avg/max/mdev = 0.030/0.090/0.469/0.083 ms

Zero packets lost and no massive spikes in latency.  Failing to set the MAC to be static can cause issues where the VM appears to go offline.  There is an example of this on the MS support site (KB976724).  In this scenario, SLES 10 SP2 live migrates, changes MAC address on the new host and then loses it’s IP configuration.  This is because the Linux distro binds the IP configuration to the MAC address.

By the way, there’s usually no reason to configure this setting for Windows guests.

Technorati Tags: ,,,

Live Migrations Are Serial, Not Concurrent

Normally when you move 2 VM’s from one host to another using Live Migration they move one at a time.  Yes, the VMM job pauses at 50% for the second machine for a while – that’s because it hasn’t started to replicate memory yet.  The live migrations are serial, not concurrent.  The memory of a running VM is being copied across a network so the network becomes a bottleneck.

I ran a little test across 3 Windows Server 2008 R2 Hyper-V cluster nodes to see what would happen.  I started moving a VM from Host A to Host C.  I also started moving a VM from Host B to host C.  The first one ran straight through.  The second one paused at 50% until the first one was moved – just like moving 2 VM’s from one host to another.

How Well Does Live Migration Perform?

Excellently.  But why believe me?  I’ve just added a node to our cluster and moved a VM throughout the infrastructure in every combination I could think of, from host A to B, from B to A, from A to C, from C to A, from B to C … you get the idea.

While I was going this I was RDP’d into that VM that was being moved using Live Migration.  I ran a continuous ping from that session to the physical default gateway, a Cisco firewall.  This is the result of the ping:

    Packets: Sent = 1174, Received = 1174, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 4ms, Average = 0ms

What was lost?  Zip!  Nada!  How many interruptions did I experience during my RDP session?  Zip!  Nada!

‘Nuff said.

Adding a Node To A VMM Managed Hyper-V Cluster

I’ve just gone through this process so I thought I’d document what I did:

  • Have a test VM ready and running on the cluster.  You’ll be moving it around to/from the new node. Don’t use a production machine in case something doesn’t work.
  • Built the new node.  Set up hardware, drivers and patches, making sure the machine was identical to the other nodes in the cluster.  I mean identical.
  • Enable Hyper-V role and Failover Clustering feature.
  • Configure the virtual networks to be identical as the other nodes – VMM won’t do this in the “add” step and we know it messes up the configuration of External networks.
  • Used the SAN manager to present all cluster disks to the new node.
  • Put the cluster, Hyper-V cluster nodes and VMM server into maintenance mode in OpsMgr.
  • Add the new node to the cluster in Failover Clustering.  Modified the cluster quorum settings to be recommended.
  • Refreshed the cluster in VMM 2008 R2.  Waited for the new node to appear under the cluster in a pending state.
  • Right-clicked on the new pending node and selected Add Node To Cluster.  Entered administrator credentials (good for all nodes in the cluster).  VMM ran a job to deploy the VMM agent.
  • If everything is good and matches up (watch out for virtual networks) then you won’t see the dreaded “Unsupported Cluster Configuration” error.
  • Move that test VM around from the new node to all the other nodes and back again using Live Migration.
  • Re-run the validation tests against your cluster ASAP.

All should be well at this point.  If so, deploy your OpsMgr agent and take the OpsMgr agents out of maintenance mode.

How W2008 R2 Live Migration Works

Let’s recap the different types of migration that we can get with Windows Server Hyper-V and System Center Virtual Machine Manager:

  • Quick Migration: Leveraging Windows Failover Clustering, a VM is treated as a clustered resource.  To quick migrate, the running state is saved to disk (hibernating the VM), the disk failed over to another node in the cluster, and the saved state is loaded (waking up the VM).
  • Offline Migration: This is when we use VMM to move a powered down VM from one un-clustered Hyper-V server to another or from one cluster to another.
  • Quick Storage Migration: This is a replacement for Offline Migration for Windows Server 2008 R2 Hyper-V servers when using VMM 2008 R2. A running VM can be moved from one un-clustered host to another or from one cluster to another with only around 2 minutes.
  • Live Migration: This is the process of moving a virtual machine from one cluster node to another with no perceivable downtime to network applications or users.  VMware refer to this as VMotion.  It was added in Windows Server 2008 R2 Hyper-V and is supported by VMM 2008 R2.

Live Migration was the big stick that everyone beat up Windows Server 2008 Hyper-V.  A few seconds downtime for a quick migration was often good enough for 75%-90% of VM’s but not for 100%.  But you can relax now; we have Live Migration.  I’m using it in production and it is good!  I can do host maintenance and enable completely automated PRO tips in VMM without worrying of any downtime, no matter how brief, for VM’s.  How does Live Migration Work?  Let’s look at how it works.

imageAbove, we have a virtual machine running on host 1.  It has a configuration and a “state”.

imageWhen we initiate a live migration the configuration of the VM is copied from host 1 when the VM is running to host 2, the destination host.  This builds up a new VM.  The VM is still running on host 1.

imageWhile the VM remains running on host 1, the memory of the VM is broken down and tracked using a bitmap.  Each page is initially marked as clean.  The pages are copied from the running VM on host 1 to the new VM sitting paused on host 2.  Users and network applications continue to use the VM on host 1.  If a RAM page changes in the running VM on host 1 after it has been copied to host 2 then Windows changes the state from clean to dirty.  This means that Windows needs to copy that page again during another copy cycle.  After the first RAM page copy cycle, only dirty pages are copied. As memory is copied again it is marked as clean.  As it changes again, it is marked as dirty.  This continues …

So when does all this stop?

  1. The process will cease if all pages have been copied over from host 1 to host 2 and are clean.
  2. The process cease if there is only a tiny, tiny amount of memory left to copy, i.e. the state. This is tiny.
  3. The process will cease if it has done 10 iterations of the memory copy. In this scenario the VM is totally trashing it’s RAM and it might never have a clean bitmap or tiny state remaining.  It really is a worst case scenario.

Note: The memory is being copied over a GB network.  I talked about this recently when I discussed the network requirements for Live Migration and Windows Server 2008 R2 Hyper-V clusters.

Remember, the VM is still running on host 1 right now.  No users or network applications have seen any impact on uptime.

imageStart your stop watch.  This next piece is very, very quick.  The VM is paused on host 1.  The remaining state is copied over to the VM on host 2 and the files/disk are failed over from host 1 to host 2.

imageThat stop watch is still ticking.  Once the state is copied from the VM on host 1 to host 2 Windows will un-pause it on host 2.  Stop your stop watch.  The VM is removed from host 1 and it’s running away on host 2 as it had been on host 1.

Just how long was the VM offline between being paused on host 1 and un-paused on host 2?  Microsoft claims the time is around 2 milliseconds on a correctly configured cluster.  No network application will time out and no user will notice.  I’ve done quite a bit of testing on this.  I’ve pinged, I’ve done file copies, I’ve used RDP sessions, I’ve run web servers, I’ve got OpsMgr agents running on them and not one of those applications has missed a beat.  It’s really impressive.

Now you should understand why there’s this "long" running progress bar when you initiate a live migration.  There’s a lot of leg work going on while the VM is running on the original host and then suddenly it’s running on the destination host.

VMware cluster admins might recognise the above technique described above.  I think it’s pretty much how they accomplish VMotion.

Are there any support issues?  The two applications that come to mind for me are the two most memory intensive ones.  Microsoft has a support statement to say that SQL 2005 and SQL 2008 are supported on Live Migration clusters.  But what about Exchange?  I’ve asked and I’ve searched but I do not have a definitive answer on that one.  I’ll update this post if I find out anything either way.

Edit #1

Exchange MVP’s Nathan Winters and Jetze Mellema both came back to me with a definitive answer for Exchange.  Jetze had a link (check under hardware virtualization).  The basic rule is that a DAG (Data Availability Group) does not support hardware virtualisation if the hosts are clustered, i.e. migration of an Exchange 2010 DAG member is not supported.