Don’t Deploy KB3161606 To Hyper-V Hosts, VMs, or SOFS

Image Credit: Max Wolfe, https://www.flickr.com/photos/leeadlaf/

Numerous sources have reported that KB3161606, an update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (WS2012 R2), are breaking the upgrade of Hyper-V VM integration components. This has been confirmed & Microsoft is aware of the situation.

As noted below by the many comments, Microsoft eventually released a superseding update to resolve these issues.

The scenario is:

  1. You deploy the update to your hosts – which upgrades the ISO for the Hyper-V ICs
  2. You deploy the update to your VMs because it contains many Windows updates, not just the ICs.
  3. You attempt to upgrade the ICs in your VMs to stay current. The upgrade will fail.

Note that if you upgrade the ICs before deploying the update rollup inside of the VM, then the upgrade works.

My advice is the same as it has been for a while now. If you have the means to manage updates, then do not approve them for 2 months (I used to say 1 month, but System Center Service Manager decided to cause havoc a little while ago). Let someone else be the tester that gets burned and fired.

Here’s hoping that Microsoft re-releases the update in a way that doesn’t require uninstalls. Those who have done the deployment already in their VMs won’t want another painful maintenance window that requires uninstall-reboot-install-reboot across all of their VMs.

EDIT (6/7/2016)

Microsoft is working on a fix for the Hyper-V IC issue. After multiple reports of issues on scale-out file servers (SOFS), it’s become clear that you should not install KB3161606 on SOFS clusters either.

Please follow and like us:

38 Comments on Don’t Deploy KB3161606 To Hyper-V Hosts, VMs, or SOFS

  1. Why can’t Microsoft revoke updates with issues, or at least post this on the KB?

  2. Great info; thanks! Do you know if the WS2012 version, KB3161609, which also includes the same IC update, has the same issue?

  3. This update also broke my SOFS cluster which is serving backing virtual disk files to a Hyper-V cluster. Caused access to the fileshares hosted on the SOFS node where this was applied to freeze and reboot those VMs.

    • Yours was the first of 2 reports I’ve heard via this blog. I’ve contacted PMs in Microsoft to let them know.

  4. Alexander Kanakaris // July 4, 2016 at 11:35 AM // Reply

    Any news on Microsoft side? They have yet to acknowledge this issue (it happened to my systems as well). Should we wait on them or just remove the whole KB3161606 ?

  5. This update broke two of our Storage Spaces Hyper-Converged Cluster Systems. We uninstalled KB3161606. After that everything went fine again.

    • That’s the second report I’ve heard via the blog. I’ve contacted PMs in Microsoft to let them know.

  6. Jan Olbrecht // July 5, 2016 at 12:21 PM // Reply

    We’ve had issues on our SOFS, specifically with HV and SQL Clusters losing access to the Quorum Shares hosted on the SOFS. Subsequently, the clusters lost their config information and services crashed (VMs and SQL instances).

  7. We made the mistake of installing this onto our Hyper-V cluster today. Massive issues with live migration, VMs getting locked, vmwp.exe getting locked, vmms.exe getting locked, VMs disconnecting from the network, the RHS deadlocking and crashing, and cluster nodes refusing to shutdown.

  8. Loel Wooldridge // July 12, 2016 at 7:12 PM // Reply

    Has MS fixed this issue? What is the current best workaround, so we can the benefit of the other hot-fixes in this rollup?

    • I haven’t heard anything about a fix 🙁

      The workaround is messy. Take ownership of winsxs\Manifests. Then rename the manifest to *.old and install the updated integration components. I’d wait for an update from Microsoft.

  9. And why isn’t there known issues documented anywhere in the KB?

  10. Loel Wooldridge // July 13, 2016 at 6:25 PM // Reply

    Thanks, Aidan. I’ve found that there are no constraints on live migration after installing the rollup on Hyper-V hosts and guests, as long as I don’t try to install the updated IC’s. Hyper-V Mangler whines mightily, but that seems to be the extent. What are other folks seeing? (We don’t have any production SOFS.)

  11. Jeff Vandervoort // July 14, 2016 at 2:20 PM // Reply

    Following up on my prior question, the June rollup for WS2012, KB3161609 does, in fact, have the same issue. KB31361609 was re-released 12 Jul, and the KB article reflects that the revised update fixes problems installing the new IS on guests.

  12. I have installed this update at about half the schools I look after before trying to run the integration services update, at which point I stopped.

    At the schools I did install the update should I un-install or wait for MS to release some info.

    At the schools I have not installed the update should I install or wait, as it has other fixes in the updated.

    Thanks for your help

  13. Loel Wooldridge // July 21, 2016 at 7:40 PM // Reply

    The July 2016 rollup has been released, KB3172614:

    https://support.microsoft.com/en-us/kb/3172614

    MS do not explicitly state that there is a fix included for the IC’s issue in the June rollup, but the notes state that this rollup replaces both May and June rollups. Have you heard anything from the MS folks about this issue as this morning?
    TIA

  14. Please be aware that the June update has now been rolled up into https://support.microsoft.com/en-us/kb/3172614 The July update rollup. I can’t tell if the issue has been fixed in the July release. Can you confirm with your sources?

  15. Peter Camps // July 22, 2016 at 11:32 AM // Reply

    This update rollup is superseded by July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (KB3172614) that was released on July 21, 2016.

  16. Microsoft just released an update to KB3161606 on 22-Jul-2016 that corrects the issue. I applied the update to the host, rebooted, then I was able to successfully upgrade the IC on the hosts by inserting the virtual update DVD from the console without first uninstalling the prior update or the IC services. I then applied the update to the VM guests for good measure. Unfortunately this required a reboot of the host and 2 reboots of each VM. Very messy.

  17. Microsoft has released KB3172614 for replace KB3161606.
    https://support.microsoft.com/en-us/kb/3161606

  18. Jeff Vandervoort // July 22, 2016 at 3:53 PM // Reply

    https://support.microsoft.com/en-us/kb/3161606 says the June WS2012R2 Rollup has been replaced by the July WS2012R2 Rollup, released 21 Jul 2016, which includes fixes for the issues discussed in this blog and its comments. The July WS2012R2 Rollup is https://support.microsoft.com/en-us/kb/3172614.

  19. AH HA:
    Just updated the July KB3161606:
    Known issues in this update

    •Issue 1

    Symptoms
    Running data deduplication on a Cluster Shared Volume (CSV) on Windows Server 2012 R2 based-Hyper V cluster may impact live migration where resource hosting subsystem (RHS) may deadlock and crash with 0x9E Stop error.

    Cause
    This issue occurs because of an incorrect status returned by data deduplication driver to the CSV file system causing a 0x9E Stop error.

    Resolution
    To fix this issue, download and install July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (KB3172614).
    •Issue 2

    Symptoms
    After this update is installed, you may receive an error message in certain configurations on a Hyper-V guest when you install Integration Services. The error message resembles the following message:

    An error has occurred: One of the updated processes returned error code 14101.
    Resolution
    To fix this issue, download and install July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (KB3172614) on the Hyper-V host.
    •Issue 3

    Symptoms
    After you install this update on a Hyper-V host, you may be unable to install this update in certain configurations on a Hyper-V guest if Integration Services have been updated on the guest.

    Resolution
    To fix this issue, download and install July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 (KB3172614).

  20. Lester Clayton // July 22, 2016 at 9:20 PM // Reply

    It looks like KB3161606 has now been superseded by KB3172614. I just wonder how safe it is 🙂 https://support.microsoft.com/en-us/kb/3172614

  21. Looks like the kb3172614
    (https://support.microsoft.com/en-us/kb/3172614)
    appears to be the updated version that incorporates KB3161606. I did not see the issue in a test environment I saw with KB3161606. I will delay deployment to a production environment for a month or so if no issues crop up.
    “The July 2016 update rollup includes some new improvements and fixes including the improvements from June 2016 update rollup KB3161606 and May 2016 update rollup KB3156418 for the Windows RT 8.1, Windows 8.1, and 2012 R2 platform. We recommend that you apply this update rollup as part of your regular maintenance routines. Before you install this update, check out the Prerequisites and restart requirement section”
    Update replacement information
    This update replaces the previously released updates 3161606 and 3156418.”

  22. should also add to my comment that the integration service is a newer version than that of KB3161606

    6.3.9600.18339 = KB3161606
    6.3.9600.18398 = KB3172614

    Is there any publication that outlines what is new with the newer integration service updates

  23. Chris Jones // July 24, 2016 at 4:38 PM // Reply

    Looks like Microsoft superseded this update with a new one (KB3172614). It would be great if you could do a followup blog post on this issue.

  24. Seppo Anttola // July 25, 2016 at 10:10 AM // Reply

    This latest update rollup https://support.microsoft.com/en-us/kb/3172614 seems to fix this problems. I have two 2012 r2 Hyper-V hosts and both of them had huge problems after KB3161606. Lots of network issues with quest VM’s and not being able to update hyper-v integration components. After installing 3172614 to host I was able to update the VM’s integration components and for now I have not seeing any network issues.

  25. Chris Walker // July 25, 2016 at 2:49 PM // Reply

    It looks like Microsoft has released an update. Has anybody tried this?

    https://support.microsoft.com/en-us/kb/3172614

  26. This update has now disappeared from my WSUS

  27. Stefan Ebenhoch // July 26, 2016 at 7:05 AM // Reply

    Hi, looks like Microsoft fixed it.

    https://support.microsoft.com/en-us/kb/3172614

  28. Roland Noordermeer // July 26, 2016 at 7:05 AM // Reply

    The KB article is updated and the July 2016 update rollup KB3172614 is supposed to fix the issues. Did any of you already verify if this update indeed fixes the issues and is safe to install?

  29. Jacob M Jensen // July 26, 2016 at 7:33 AM // Reply

    Any news on this?

  30. It looks like KB3161606 caused the NFS to crash also. Whenever we copy a large file from Linux system to our server 2012 r2 it crash the server. I updated the new 3172614, no luck. It still crash. 🙁

4 Trackbacks & Pingbacks

  1. Issue with KB3161606
  2. Updating of Hyper-v Integration tools
  3. July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 - Working Hard In IT
  4. July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2 – Michael Stoica

Leave a comment

Your email address will not be published.


*


This site uses Akismet to reduce spam. Learn how your comment data is processed.