KB2894032 – Clustered VM Cannot Access Fiber Channel LUN After Performing Live Migration on WS2012 Hyper-V

FYI, Windows Server 2012 R2 allows VMs to have virtual fiber channel adapters that use the bandwidth of the hosts’ physical HBAs.  This means that VMs can have their own WWN (actually 2 WWNs per virtual HBA) and connect to zoned LUNs on an FC SAN.  This supports both Live Migration of those VMs, and the ability to use the FC LUNs as the shared storage of a guest cluster.

The first phase of Live Migration (that first 3% of the progress bar) is when Hyper-V attempts to build up a VM’s spec & dependencies on a destination host.  This includes connecting to any FC LUNs using the alternative WWN (hence 2 WWNs per virtual HBA).

Microsoft released a hotfix to deal with an issue on WS2012 Hyper-V where one of those FC enabled VMs loses connectivity to an FC LUN.

Symptoms

Consider the following scenarios:

  • You have two Windows Server 2012-based computers that have the Hyper-V role installed.
  • You install a virtual machine on one of the Windows Server 2012 Hyper-V hosts.
  • You set up a guest failover cluster, and then you make the virtual machine a cluster node.
  • The virtual machine is configured to access logical unit numbers (LUNs) over a Synthetic Fibre Channel.
  • You try to perform live migration to move the virtual machine to another Hyper-V host.

In this situation, the virtual machine on the target Hyper-V host cannot access the LUNs over the Synthetic Fibre Channel. 

Cause

This issue occurs because the target Hyper-V host cannot restore the Synthetic Fibre Channel LUN on behalf of the virtual machine during live migration.

More Information

You might receive the following error event and warning event when this issue occurs:

  • On the target Hyper-V host:

    Error event:
    Hyper-V SynthFC-Admin ID 32214 with description like
    Failed to reserve LUN with Instance Path ‘\?SCSI#VMLUN&Ven_HP&Prod_HSV360#5&17efa605&0&070002#{6f416619-9f29-42a5-b20b-37e219ca02b0}’ to virtual machine ‘WS2012-1’ with error: The data is invalid. (0x8007000D). (Virtual machine ID C799C113-B153-4E49-B0C5-F9E24774EB9A)
    Hyper-V SynthFC-Admin ID 32216 with description like
    Failed to register LUN with Instance Path ‘\?SCSI#VMLUN&Ven_DGC&Prod_RAID_5#5&378d83c&0&080200#{6f416619-9f29-42a5-b20b-37e219ca02b0}’ to virtual machine ‘SERVER2012R2-STD-64-1’ with error: The data is invalid. (0x8007000D). (Virtual machine ID 86FA60B1-8B40-45C5-A88F-1F024BECA8F0)

  • On the virtual machine:

    Warning Event:
    Microsoft-Windows-Ntfs ID 140
    The system failed to flush data to the transaction log. Corruption may occur in VolumeId: F:, DeviceName: DeviceHarddiskVolume82.
    (A device which does not exist was specified.)
    Event ID:50
    {Delaled Write Failed} Windows was unable to save all the data for the file. The data has been lost. This erorr may be caused by a failure of your computer hardware or network connection. Please try to save this file elsewhere.

A hotfix has been released to fix this problem.

KB2893986 – Vulnerability in WS2012 & Windows 8 Hyper-V Could Allow Elevation of Privilege

In November, Microsoft released the MS13-092 (aka CVE-2013-3898) security fix for Windows Server 2012 Hyper-V and Windows 8 Client Hyper-V. 

An attacker who successfully exploited this vulnerability could execute arbitrary code as System in another virtual machine (VM) on the shared Hyper-V host. An attacker would not be able to execute code on the Hyper-V host, only on guest VMs on the same host. The vulnerability could also allow denial of service in Hyper-V on the same platforms, allowing an attacker to cause the Hyper-V host to stop responding or restart.

We’ve only had a handful of security fixes for Hyper-V since it was released on Windows Server 2008.  This looks like the most important one of the lot.

Please test/deploy as quickly as you can.

Please Welcome 5nine!

I’m pleased to announce that 5nine have become a sponsor of my blog.  5nine is a key player in the Hyper-V partner ecosystem, making of security, networking, and management tools & extensions.

Manager Banner 235x235 (Free)

You’ll find an alternative to SCVMM for small/medium businesses, extensibility to SCVMM and the Hyper-V virtual switch for the medium to large enterprise, and a number of very useful free tools.

Please take the time to evaluate 5nine’s tools and see what they could add to your network.

WS2012 R2 Hyper-V Manager Can Be Used With WS2012 Hyper-V

Most everything we get told at MVP Summit is under NDA, but this is an exception.  In fact, Ben Armstrong asked a few of us to blog about this last night at the Summit party.

The new generation of Hyper-V Manager (Windows Server 2012 R2 and Windows 8.1) can be used to manage Hyper-V of the most recent generation (Windows Server 2012 and Windows 8). 

This is a first for Hyper-V.  We’ve always needed to use a matching version of Hyper-V Manager on our PC.  For example, you needed RSAT for Windows 7 to manage W2008 R2 Hyper-V from your PC.  Windows Server 2012 R2 and Windows 8.1 introduced an anomaly; Windows 8.1 is a free upgrade from Windows 8 so it was expected that many administrators would upgrade their PCs.  However, Windows server is not a free upgrade and businesses are often slower to upgrade servers OSs, even on hosts.  This could lead to a situation where an IT department upgrades their PCs but could no longer use their shiny new 8.1 Hyper-V Manager to manage their recently new WS2012 hosts.

The new Hyper-V Manager won’t do anything noticeably different (with one tiny exception for the eagle-eyed) when you use it normally.  The GUI calls either the 2012 or the 2012 R2 binaries depending on the generation of the host you are configuring.  That teeny exception?  Hyper-V snapshots will appear as “snapshots” in the UI for 2012 hosts, and they’ll appear as “checkpoints” (the new term to cause less confusion) for 2012 R2 hosts.

There is one thing to be careful of and this will affect very few people.  A few people launch the connect tool by running the executable directly.  I’ve only ever done this by accident when searching for “Hyper-V” on Windows 8/8.1.  I normally launch connect from Hyper-V Manager or Failover Cluster Manager.  If you do run this tool directly, then you need to run VMCONNECT.EXE for the older host versions and run VMCONNECT.6.2.EXE for VMs running on Windows 8.1 or Windows Server 2012 R2 hosts.

Note that everything I have said here for Hyper-V Manager also applies to Failover Cluster Manager.  The clustering team used the same approach as the Hyper-V team.

And no, you cannot manage legacy versions such as W2008 R2.  This is because of significant changes to the underlying WMI; WS2012 introduced WMIv2.

And before you ask: you need VMM 2012 R2 to manage WS2012 R2 Hyper-V from System Center.

Double-Take Move Partners With Microsoft For Easy Migration From VMware To Hyper-V Via System Center

Makers of Double-Take Move, Vision Solutions Inc, recently released a press release to announce that:

Double‐Take Move®, will help enterprise customers and service providers move from VMware to Windows Server 2012 Hyper-V and/or Windows Azure … With the capabilities of Windows Server 2012, Hyper‐V is becoming the virtualization platform of choice for customers running mission critical applications. Customers and partners are benefiting from significant cost savings over VMware when migrating their datacenters to Hyper‐V, and with Double‐Take Move, the migration is easier and more productive than ever.

The concept of the partnership is that Double-Take Move will integrate into System Center and make it easy for administrators to V2V migrations:

  • Near‐zero downtime migrations (downtime is when the VM comes online after the migration)
  • Automated cutover
  • Test cutover
  • WAN optimization
  • Milestone notifications
  • Automatic VM provisioning

image

According to the data sheet, this is a service-centric approach, leveraging Orchestrator to automate the process and Service Manager to reveal the process as a request to cloud tenants.  There is also a video on the solution.

This will be good news to large organizations that have made big investments in vSphere in the past and want to have an easy switch over to a more efficient platform.

Configuring The Paging File On Hyper-V Hosts

A paging file creates virtual RAM, allowing the OS to “page out” inactive memory and “page in” memory from the file when it is required.  This allows the OS to make better use of the available RAM in a computer.

Hyper-V has little use for a paging file.  Unlike certain hypervisors, Hyper-V does not do blind second level paging.  That’s because Hyper-V’s Dynamic Memory works differently to other memory techniques.  That’s a whole other blog post.  But long story short: the paging file in a Hyper-V management OS (the host OS) is there only for the management OS itself … and that management OS doesn’t need much RAM or performance.  You save capacity and performance for your VMs.

There was a time when we used to advise configuring the paging file on Hyper-V hosts.  But that all changed with the release of WS2012.  Lots of little things that we advised tweaking (like memory reserve) should never be tweaked now.  That’s because Microsoft built in automated management for Hyper-V management OS installations.

Now the official line on the paging file is:

For Hyper-V servers that run Windows Server 2012 or Windows Server 2012 R2, the page file should be left at the default of setting of System Managed. This is per the Hyper-V Product Group.

That comes from a KB article called How to determine the appropriate page file size for 64-bit versions of Windows that gives you guidance on how to setup or configure the paging file on x64 Windows installations. 

So that makes it official & simple: leave the paging file on Hyper-V hosts with the default configuration.

For those of you who mistakenly call their VMs “Hyper-V servers” this guidance has nothing to do with the guest OS of your VMs.  Configure the paging file of your VMs according to best practice for the OS and the services that it is running.

5nine Software Releases V2.0 Of VMware-To-Hyper-V V2V Conversion Tool (FREE!)

5nine Software announced the release of the free 5nine V2V Easy Converter v2.0 yesterday.  This will allow you to convert VMware VMs into Hyper-V VMs with support for VHD and VHDX and the following guest OSs:

  • Windows Server 2008
  • Windows 7
  • Windows Server 2003 (x86 and x64)
  • Most of Advanced Server 2000
  • Ubuntu Linux
  • CentOS Linux

Yes: V2V conversions to Hyper-V for Linux guests!

The last three versions of Hyper-V are supported:

  • Windows Server 2008 R2 SP1 Hyper-V
  • Windows Server 2012 Hyper-V
  • Windows Server 2012 R2 Hyper-V

If you have any questions, please contact 5nine Software Information at info(at)5nine(dot)com or via phone +44 (20) 7048-2021 from 7am to 5pm GMT

Windows Azure Backup Is Generally Available & Other Azure News

The following message came in an email overnight:

Windows Azure Backup is now generally available, Windows Azure AD directory is created automatically for every subscription, and Hyper-V Recovery Manager is in preview.

What does that mean?  Some backup plans charge you based on the amount of data that you are protecting.  Personally, I prefer that approach because it is easy to predict – I have 5 TB of data and it’s going to cost me 5 * Y to protect it.  Azure Online Backup has gone with the more commonly used approach of charging you based on how many GB/month of storage that you consume on Microsoft’s cloud.  This is easy for a service provider to create bills, but it’s hard for the consumer to estimate their cost … because you have elements like deduplication and compression to account for.

The pricing of Azure Online Backup looks very competitive to me. 

Windows Azure Backup is billed in units based on your average daily amount of compressed data stored during a monthly billing period.

Some plans get the first 5GB free and then it’s €00.3724 per GB per month.  In the USA, it will be $00.50 per GB per month.  Back when I worked in backup, €1/GB per month was considered economic.

In other Azure news:

A Windows Azure AD directory is created automatically for every subscription:

Starting today, every Windows Azure subscription is associated with an autocreated directory in Windows Azure Active Directory (AD). By using this enterprise-level identity management service, you can control access to Windows Azure resources.

To accommodate this advancement, every Windows Azure subscription can now host multiple directories. Additionally, Windows Azure SDK will no longer rely on static management certificates but rather on user accounts in Active Directory. Existing Active Directory tenants related to the same user account will be automatically mapped to a single Windows Azure subscription. You can alter these mappings from the Windows Azure Management Portal.

Take advantage of the new Windows Azure Hyper-V Recovery Manager preview.

Windows Azure Hyper-V Recovery Manager helps protect important applications by coordinating the replication of Microsoft System Center clouds to a secondary location, monitoring availability, and orchestrating recovery as needed.

The service helps automate the orderly recovery of applications and workloads in the event of a site outage at the primary data center. Virtual machines are started in an orchestrated fashion to help restore service quickly.

The Euro GA pricing for Hyper-V Recovery Manager was included in the email.  It will cost 11,9152€ per virtual machine per month to use this service.  The website is not updated with GA pricing.

How To Avoid Common Networking Issues In Hyper-V

This subject seems to rear it’s ugly head with every new version of Hyper-V.  We get new people trying the product for the first time who don’t know the best practices.  And we get people who forget the best practices from last time around.

Here is my advice:

Use Logo Tested NICs

Only NICs that are on the HCL for your version of Windows Server (or Windows client OS if using Client Hyper-V) should be used.  Make sure that your hardware manufacturer supports your version of Hyper-V (i.e. Windows Server) for all components that you are using from them.  This is a starting point because …

If At All Possible, Do Not Use Broadcom NICs

Sure, that might preclude you from using certain stuck-in-2009 server manufacturers.  And maybe that’s not a bad thing 😀

Broadcom seems to be a name that comes up over and over, version after version, when there are problems.  My lab is all Chelsio, but that moves you into the territory of 10 GbE iWARP, which is more expensive than what comes onboard a host.  But it allows me to get great SMB 3.0 performance for storage and Live Migration over converged networks.  Intel seems to produce decent NICs, firmwares, and drivers, which leads me to …

Update Your Firmware

Assumption … ARGH!  Do not assume that your firmwares are up to date.  I don’t care how new your server is.  This goes for every part of the server.  Update the firmware.  Faulty older firmware is a common cause of issues.

Update Your Drivers

Use the latest drivers from the manufacturer.  The drivers you got with the server might be old.  The drivers in Windows Server might be old too.  For example, the automatically detected Chelsio drivers in WS2012 R2 give poor performance.  The fix is to download the drivers from Chelsio.  If you bought a HP NIC then get the latest drivers from HP.  If you bought a NIC from Dell then get the drivers from Dell.

Install The Recommended Updates For Windows Server

Windows Update is not enough. Let me repeat that: WINDOWS UPDATE IS NOT ENOUGH. If you think it is, then please stop reading, print off a resignation letter, and go home now. Download and install the recommended updates for Windows Server. This link will lead you to 3 sets of updates for WS2012 R2.

Understand And Use Microsoft NIC Teaming

Third-party NIC teaming has never been supported by Microsoft for anything.  Why?  Because third-party NIC teaming pulls out the guts of Windows NIC teaming, pushes itself in, squeezes the networking stack back in, finds a few of those “spare screws” and throws them aside.

Take the time to learn Microsoft NIC teaming (there’s more to any teaming than the default options) and use it instead of the third-party unsupported crapware.

EDIT (28/07/2014):

One more to add ..

Don’t Use Blade Servers

You have no choice but to use what the manufacturer gives you. For many of you, that will be Emulex NICs … and using those are like going fire-walking after taking a bath in petrol (benzine for Europeans, and gas for Americans).

Disable VMQ on Virtual Switch NICs

This is a workaround, and not a solution. We want and may even need VMQ for networking performance and scalability. However, we have seen NICs (Emulex and Broadcom) where having VMQ enabled has caused issues. This should be fixed by the manufacturer using firmware/driver updates … but some manufacturers don’t give a damn about their customers. Emulex – care to prove me wrong?

Microsoft Releases Remote Desktop For Apple iOS

You don’t need pricey third party RDP apps anymore.  Microsoft has finally released a Remote Desktop app for iPhone and iPad.  The features are:

  • Access to remote resources through the Remote Desktop Gateway
  • Rich multi-touch experience with remote desktop protocol (RDP) and RemoteFX supporting Windows gestures
  • Secure connection to your data and applications with breakthrough Network Layer Authentication (NLA) technology
  • Simple management of all remote connections from the connection center
  • High quality video and sound streaming with improved compression and bandwidth usage
  • Easy connection to external monitors or projectors for presentations

The price is good: free.  And all the gesture stuff works – now I don’t feel stupid for swiping from the right on an iPad 🙂

IMAG0097 An abomination: Windows 8.1 “running” on an Apple iPad

Source: Wes Miller (@getwired).

EDIT:

Microsoft also released the Microsoft Remote Desktop app for Android and updated it for Mac OS X.