{"id":15193,"date":"2013-07-12T09:37:51","date_gmt":"2013-07-12T08:37:51","guid":{"rendered":"https:\/\/aidanfinn.com\/?p=15193"},"modified":"2013-07-12T09:37:51","modified_gmt":"2013-07-12T08:37:51","slug":"something-has-gone-very-wrong-with-microsoft-patch-testing","status":"publish","type":"post","link":"https:\/\/aidanfinn.com\/?p=15193","title":{"rendered":"Something Has Gone Very Wrong With Microsoft Patch Testing"},"content":{"rendered":"<p>I suspect I\u2019m going to get some \u201cunwanted attention\u201d for this post but what I\u2019m going to say has to be said publicly \u2026<\/p>\n<p>Something has gone wrong with the testing process for Microsoft hotfixes since the release of WS2012.\u00a0 There has been a number of really bad releases in those 10 or months.\u00a0 The latest is KB2855336, aka the July 2013 update rollup, which causes hosts to bug check as <a href=\"http:\/\/www.hyper-v.nu\/archives\/hvredevoort\/2013\/07\/red-alertavoid-kb2855336-rollup-with-nic-teaming-and-vlans\/\" target=\"_blank\">Hans Vredevoort<\/a> and <a href=\"https:\/\/aidanfinn.com\/?p=15179\" target=\"_blank\">some of you<\/a> reported.\u00a0 There is also a <a href=\"http:\/\/social.technet.microsoft.com\/Forums\/lync\/en-US\/772562bc-4913-4d62-b304-3364706a0c5f\/hyperv-host-node-bughceck-on-mslbfoprovidersys\" target=\"_blank\">thread<\/a> on the Hyper-V TechNet forum.<\/p>\n<p>People like me and Microsoft are trying to encourage people to:<\/p>\n<ul>\n<li>Install security hotfixes with <em>minimal<\/em> delay<\/li>\n<li>Embrace a process of updating their Hyper-V hosts with <a href=\"https:\/\/aidanfinn.com\/?p=14227\" target=\"_blank\">fixes<\/a> for Hyper-V and Failover Clustering to prevent issues<\/li>\n<\/ul>\n<p>This string of updates that break hosts (and this is exponentially worse than breaking an occasional physical server here or there) is embarrassing and dangerous.\u00a0 The latest failure is in an update rollup that is issued via Windows Update.\u00a0 This just feeds the argument that patching is ba-ad and shouldn\u2019t be done \u2026 and creates a security mess for, not just for those companies but, everyone in the community.<\/p>\n<p>I\u2019d love to say I have a fix.\u00a0 I\u2019d love to say, hey use the automatic approval process in System Center Configuration Manager where we can:<\/p>\n<ul>\n<li>Delay approval of updates for X days \u2013 letting others find the bugs and Microsoft issue a superseding update<\/li>\n<li>Force the deployment<\/li>\n<\/ul>\n<p>That will work for non-clustered hosts but:<\/p>\n<ul>\n<li>Folks with clusters will want to use Cluster Aware Updating.\u00a0 ConfigMgr does not have a plug-in for CAU integration.\u00a0 Someone in MSFT will respond with VMM baselines.\u00a0 Tell \u2018em to go take a long walk off of a short pier; no one should have to do that amount of clicking every month.<\/li>\n<li>Most businesses are SMEs and SMEs cannot afford System Center anymore.<\/li>\n<\/ul>\n<p>So what\u2019s left?\u00a0 Manual approval and patching.\u00a0 And as I\u2019ve said before: that means patching just does not happen \u2026 at all.\u00a0 I\u2019m not being cynical; I\u2019m being pragmatic and basing this on experience in the real world.<\/p>\n<p>Let me tell you a story \u2026<\/p>\n<p>I used to work for a consulting company that specialised in Computer Associates software.\u00a0 I was certified and consulted in CA Unicenter, their huge enterprise monitoring system.\u00a0 I also dabbled in a few x-IT management products.\u00a0 CA were shite when it came to product quality and patch management.\u00a0 The process for installing a new product version was:<\/p>\n<ul>\n<li>Install from the media<\/li>\n<li>Test<\/li>\n<li>Find the broken basic functionality<\/li>\n<li>Log into Support and download <em>lots<\/em> of patches and install them for this <em>new<\/em> release<\/li>\n<li>Find the broken functionality that had been patched\/fixed 2 months before in the previous release<\/li>\n<li>Open up support calls to get them to update the previous release\u2019s fixes for the new release<\/li>\n<li>Try cover your ass with the angry customer<\/li>\n<\/ul>\n<p>I once had a CA tested over in the office to introduce us to a new beta version of Unicenter.\u00a0 I asked about the huge number of patches that would appear within a week of release because basic features didn\u2019t work.\u00a0 He explained that CA couldn\u2019t possibly test more than 75% of features before release.\u00a0 That\u2019s why I\u2019ve flat-out refused to work with CA software since 2001.<\/p>\n<p>Let\u2019s get back on track here.\u00a0 The problem with KB2855336 is that it breaks hosts that:<\/p>\n<ul>\n<li>Connect a virtual switch to a NIC team<\/li>\n<li>VMs are on different VLANs<\/li>\n<\/ul>\n<p>Hmm, seems like one of the most basic configurations for Hyper-V if you ask me.\u00a0 How the hell was this not tested?<\/p>\n<p>This litany of mistakes cannot continue.\u00a0 We (the community) cannot continue to recommend fixes if they break stuff in basic or default configurations.\u00a0 Microsoft, you want to be a cloud company; learn from how hosting companies have been very public with explanations and apologies.\u00a0 This actually <em>reassures<\/em> the customers of those hosting companies \u2013 I once worked for a company that blacked out 1\/3 of the hosted Irish internet for over a day, and that openness saved the day.\u00a0 Needless to say, I was amazed.\u00a0 Something must change, Microsoft, and you must be very public with the apology and the explanation of the process changes \u2013 and don\u2019t just hide this in a forum response.<\/p>\n<p>In the meantime, Microsoft should:<\/p>\n<ul>\n<li><span style=\"text-decoration: line-through;\">Remove this update rollup from the catalog to prevent further failures<\/span> Hans <a href=\"http:\/\/www.hyper-v.nu\/archives\/hvredevoort\/2013\/07\/red-alertavoid-kb2855336-rollup-with-nic-teaming-and-vlans\/\" target=\"_blank\">reported<\/a> just now (09:43 GMT) that this was done.<\/li>\n<li>Instruct employees to modify blog posts and retract recommendations to deploy this update rollup<\/li>\n<\/ul>\n<p>I hope any now-angry persons in Microsoft understand that I am writing this <em>in support<\/em> of Microsoft.\u00a0 A friend is honest with criticism and wants change for improvement.\u00a0 I\u2019m not writing this to score points.\u00a0 I\u2019m writing this because I care.<\/p>\n<p>EDIT:<\/p>\n<p>A fix was <a href=\"https:\/\/aidanfinn.com\/?p=15198\" target=\"_blank\">released <\/a>(allegedly) in an updated version of the July 2013 update rollup.<\/p>\n<p>EDIT (27\/7\/2013):<\/p>\n<p>It looks like UR3 for DPM 2012 SP1 joins the ranks of bad updates. \u00a0Almost immediately people reported that they could not upgrade their agents after upgrading DPM servers. \u00a0The update was withdrawn several days later, as noted in <a href=\"http:\/\/blogs.technet.com\/b\/dpm\/archive\/2013\/07\/23\/update-rollup-3-for-dpm-2012-service-pack-1-is-now-available.aspx?PageIndex=2#comments\" target=\"_blank\">these comments<\/a>.<\/p>\n<div id=\"scid:0767317B-992E-4b12-91E0-4F059A8CECA8:5ec09022-1f3a-4d67-9358-9a87deeee6a3\" class=\"wlWriterEditableSmartContent\" style=\"float: none; margin: 0px; display: inline; padding: 0px;\">Technorati Tags: <a rel=\"tag\" href=\"http:\/\/technorati.com\/tags\/Microsoft\">Microsoft<\/a>,<a rel=\"tag\" href=\"http:\/\/technorati.com\/tags\/Hyper-V\">Hyper-V<\/a>,<a rel=\"tag\" href=\"http:\/\/technorati.com\/tags\/Windows+Server+2012\">Windows Server 2012<\/a>,<a rel=\"tag\" href=\"http:\/\/technorati.com\/tags\/Virtualisation\">Virtualisation<\/a><\/div>\n","protected":false},"excerpt":{"rendered":"<p>I suspect I\u2019m going to get some \u201cunwanted attention\u201d for this post but what I\u2019m going to say has to be said publicly \u2026 Something has gone wrong with the testing process for Microsoft hotfixes since the release of WS2012.\u00a0 There has been a number of really bad releases in those 10 or months.\u00a0 The &hellip; <a href=\"https:\/\/aidanfinn.com\/?p=15193\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Something Has Gone Very Wrong With Microsoft Patch Testing&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"footnotes":""},"categories":[20],"tags":[181,185,195,118],"class_list":["post-15193","post","type-post","status-publish","format-standard","hentry","category-hyper-v","tag-hyper-v","tag-microsoft","tag-virtualisation","tag-windows-server-2012"],"aioseo_notices":[],"jetpack_featured_media_url":"","amp_enabled":true,"_links":{"self":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/15193","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=15193"}],"version-history":[{"count":0,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=\/wp\/v2\/posts\/15193\/revisions"}],"wp:attachment":[{"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=15193"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=15193"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aidanfinn.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=15193"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}