Don’t Deploy KB3161606 To Hyper-V Hosts, VMs, or SOFS

Numerous sources have reported that KB3161606, an update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (WS2012 R2), are breaking the upgrade of Hyper-V VM integration components. This has been confirmed & Microsoft is aware of the situation.

As noted below by the many comments, Microsoft eventually released a superseding update to resolve these issues.

The scenario is:

  1. You deploy the update to your hosts – which upgrades the ISO for the Hyper-V ICs
  2. You deploy the update to your VMs because it contains many Windows updates, not just the ICs.
  3. You attempt to upgrade the ICs in your VMs to stay current. The upgrade will fail.

Note that if you upgrade the ICs before deploying the update rollup inside of the VM, then the upgrade works.

My advice is the same as it has been for a while now. If you have the means to manage updates, then do not approve them for 2 months (I used to say 1 month, but System Center Service Manager decided to cause havoc a little while ago). Let someone else be the tester that gets burned and fired.

Here’s hoping that Microsoft re-releases the update in a way that doesn’t require uninstalls. Those who have done the deployment already in their VMs won’t want another painful maintenance window that requires uninstall-reboot-install-reboot across all of their VMs.

EDIT (6/7/2016)

Microsoft is working on a fix for the Hyper-V IC issue. After multiple reports of issues on scale-out file servers (SOFS), it’s become clear that you should not install KB3161606 on SOFS clusters either.

42 thoughts on “Don’t Deploy KB3161606 To Hyper-V Hosts, VMs, or SOFS”

  1. Great info; thanks! Do you know if the WS2012 version, KB3161609, which also includes the same IC update, has the same issue?

  2. This update also broke my SOFS cluster which is serving backing virtual disk files to a Hyper-V cluster. Caused access to the fileshares hosted on the SOFS node where this was applied to freeze and reboot those VMs.

    1. Yours was the first of 2 reports I’ve heard via this blog. I’ve contacted PMs in Microsoft to let them know.

  3. Any news on Microsoft side? They have yet to acknowledge this issue (it happened to my systems as well). Should we wait on them or just remove the whole KB3161606 ?

  4. This update broke two of our Storage Spaces Hyper-Converged Cluster Systems. We uninstalled KB3161606. After that everything went fine again.

    1. That’s the second report I’ve heard via the blog. I’ve contacted PMs in Microsoft to let them know.

  5. We’ve had issues on our SOFS, specifically with HV and SQL Clusters losing access to the Quorum Shares hosted on the SOFS. Subsequently, the clusters lost their config information and services crashed (VMs and SQL instances).

  6. We made the mistake of installing this onto our Hyper-V cluster today. Massive issues with live migration, VMs getting locked, vmwp.exe getting locked, vmms.exe getting locked, VMs disconnecting from the network, the RHS deadlocking and crashing, and cluster nodes refusing to shutdown.

  7. Has MS fixed this issue? What is the current best workaround, so we can the benefit of the other hot-fixes in this rollup?

    1. I haven’t heard anything about a fix 🙁

      The workaround is messy. Take ownership of winsxs\Manifests. Then rename the manifest to *.old and install the updated integration components. I’d wait for an update from Microsoft.

  8. Thanks, Aidan. I’ve found that there are no constraints on live migration after installing the rollup on Hyper-V hosts and guests, as long as I don’t try to install the updated IC’s. Hyper-V Mangler whines mightily, but that seems to be the extent. What are other folks seeing? (We don’t have any production SOFS.)

  9. Following up on my prior question, the June rollup for WS2012, KB3161609 does, in fact, have the same issue. KB31361609 was re-released 12 Jul, and the KB article reflects that the revised update fixes problems installing the new IS on guests.

  10. I have installed this update at about half the schools I look after before trying to run the integration services update, at which point I stopped.

    At the schools I did install the update should I un-install or wait for MS to release some info.

    At the schools I have not installed the update should I install or wait, as it has other fixes in the updated.

    Thanks for your help

  11. The July 2016 rollup has been released, KB3172614:

    MS do not explicitly state that there is a fix included for the IC’s issue in the June rollup, but the notes state that this rollup replaces both May and June rollups. Have you heard anything from the MS folks about this issue as this morning?

  12. This update rollup is superseded by July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (KB3172614) that was released on July 21, 2016.

  13. Microsoft just released an update to KB3161606 on 22-Jul-2016 that corrects the issue. I applied the update to the host, rebooted, then I was able to successfully upgrade the IC on the hosts by inserting the virtual update DVD from the console without first uninstalling the prior update or the IC services. I then applied the update to the VM guests for good measure. Unfortunately this required a reboot of the host and 2 reboots of each VM. Very messy.

  14. AH HA:
    Just updated the July KB3161606:
    Known issues in this update

    •Issue 1

    Running data deduplication on a Cluster Shared Volume (CSV) on Windows Server 2012 R2 based-Hyper V cluster may impact live migration where resource hosting subsystem (RHS) may deadlock and crash with 0x9E Stop error.

    This issue occurs because of an incorrect status returned by data deduplication driver to the CSV file system causing a 0x9E Stop error.

    To fix this issue, download and install July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (KB3172614).
    •Issue 2

    After this update is installed, you may receive an error message in certain configurations on a Hyper-V guest when you install Integration Services. The error message resembles the following message:

    An error has occurred: One of the updated processes returned error code 14101.
    To fix this issue, download and install July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (KB3172614) on the Hyper-V host.
    •Issue 3

    After you install this update on a Hyper-V host, you may be unable to install this update in certain configurations on a Hyper-V guest if Integration Services have been updated on the guest.

    To fix this issue, download and install July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (KB3172614).

  15. Looks like the kb3172614
    appears to be the updated version that incorporates KB3161606. I did not see the issue in a test environment I saw with KB3161606. I will delay deployment to a production environment for a month or so if no issues crop up.
    “The July 2016 update rollup includes some new improvements and fixes including the improvements from June 2016 update rollup KB3161606 and May 2016 update rollup KB3156418 for the Windows RT 8.1, Windows 8.1, and 2012 R2 platform. We recommend that you apply this update rollup as part of your regular maintenance routines. Before you install this update, check out the Prerequisites and restart requirement section”
    Update replacement information
    This update replaces the previously released updates 3161606 and 3156418.”

  16. should also add to my comment that the integration service is a newer version than that of KB3161606

    6.3.9600.18339 = KB3161606
    6.3.9600.18398 = KB3172614

    Is there any publication that outlines what is new with the newer integration service updates

  17. Looks like Microsoft superseded this update with a new one (KB3172614). It would be great if you could do a followup blog post on this issue.

  18. This latest update rollup seems to fix this problems. I have two 2012 r2 Hyper-V hosts and both of them had huge problems after KB3161606. Lots of network issues with quest VM’s and not being able to update hyper-v integration components. After installing 3172614 to host I was able to update the VM’s integration components and for now I have not seeing any network issues.

  19. The KB article is updated and the July 2016 update rollup KB3172614 is supposed to fix the issues. Did any of you already verify if this update indeed fixes the issues and is safe to install?

  20. It looks like KB3161606 caused the NFS to crash also. Whenever we copy a large file from Linux system to our server 2012 r2 it crash the server. I updated the new 3172614, no luck. It still crash. 🙁

Leave a Reply to AFinn Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.