My Patching Recommendation Is Discussed on TWiT Windows Weekly

In case you don’t know, Windows Weekly on the TWiT online channel is probably the biggest Windows “podcast” (it’s also a live show) on the net. It is hosted by Leo Laporte with top tech journalists Paul Thurrott and Mary Jo Foley. Last night, they discussed the recent patching issues and Mary Jo brought up my advice to delay deploying updates for 1 month – I normally try to watch live but I listened in the car this morning.

Go to around the 34 minute mark to hear for yourself

Leo didn’t like my advice – Leo also hosts Security Now and hears on a weekly basis about the various ways that computers can be attacked from Steve Gibson. Leo was worried about “zero day” attacks. Paul appeared to have a very pragmatic view on things, wishing that we didn’t have this problem in the first place.

So here’s my views on the discussion. I understand why Leo doesn’t like my recommendation. I don’t like my recommendation to delay release of updates for 1 month. But I’ve been seeing for the last 2 years how bad updates for Windows Server (and thus Windows client) and System Center have been. We’re seeing failures and release withdrawals almost on a quarterly basis. And these aren’t just niche scenarios like a shortcut to a font file in the wrong place on Windows 7 Home Premium. This is widely used designs, basic installs, and so on. To be honest, I see the approval of new updates from Microsoft as a bigger risk than malware at this point; releasing an untested update (if I was still an admin) to 100 VMs and 1000 desktops is sure to get me fired within 3-6 months when the business goes in the dark a couple of times because of bad updates. On the other hand, I’ve never had a malware breakout on a network I owned in my career – I’ve only seen malware get trapped by well-managed AV.

I wish I could recommend approving MSFT updates for near-instant deployment, as Leo has suggested. But I cannot – I’ve heard of and reported on too many failures. And any business that needs to rely on their IT cannot take risks.

Paul has it right; Microsoft management is pushing releases (patches, rollups, full product milestones) faster than they should be – and testing is taking second place. I know that technical people that I have great respect for in Redmond are embarrassed by what is going on. Unfortunately, it’s going to take something really bad for Satya Nadella to undo the damage that is happening under his watch, that I guess is probably his doing.

Leo (not that you’ll ever read this), I completely understand your point of view. I used to be a person who said “get the updates out within a week”. But because of the events of the last 2 years, I respectfully have to disagree with you.

BTW, you can take the approach I recommended using SCCM ADRs and tweak it so you create ADRs to approve “critical” updates more rapidly. That will give you a middle ground for security updates, but the risk is yours to measure and take. This is a management decision!

Microsoft News Summary – 20 August 2014

The headline news from yesterday is that Steve Ballmer has resigned his new position from the Microsoft board to focus on “teaching” and his duties as the new owner of the Los Angeles Clippers NBA basketball franchise. He’s still the largest independent owner of MSFT stock.

Microsoft

Virtual Machine Manager

  • VMM 2012 Self-Service users cannot open a console session to a virtual machine: When you try to connect to the console session of a virtual machine (VM) that is running in Windows Server 2012 by using Microsoft System Center 2012 R2 Virtual Machine Manager or Microsoft System Center 2012 Virtual Machine Manager Service Pack 1 (SP1), the connection fails, and you receive the following error message – Virtual Machine Manager lost the connection to the virtual machine for one of the following reasons.

Azure

Office 365

See You At TechEd Europe 2014

Speaking at TechEd has been one of my career ambitions for years – it is the pinnacle of speaking in the Microsoft world. I started of presenting at MSFT community events and had no such goal. But eventually I reached the point with my knowledge of Hyper-V that I felt like I could contribute and that I wanted to speak on the bigger stage; certainly presenting one of the sessions at the WS2012 launch in London (1000 attendees in the room) fired me up even more. I submitted sessions to TechEd, but never got anywhere. I gave up on my goal last year.

Then things fell into place at TechEd North America. I wasn’t going to do Speaker Idol. But when I was asked, I had an idea and I said to myself “frak it, do it! It’ll be fun to do”. And I ended up winning a slot in “TechEd” int he USA next year. I also talked to some folks and they gave me some advice about submitting sessions for TEE14. I submitted one session and …

Getting good news is always a nice way to finish the day. Early yesterday evening I received an email informing me that Microsoft had picked their sessions/speakers for TEE14. I followed the link to check the status of my submission and there it said:

Approval Status: Approved

Yes; I did my happy dance 😀 My guess is that we cannot talk about our sessions yet, but you can safely guess that I’ll be talking about Hyper-V.

Hopefully I’ll see some of you there when I present … at TechEd!

Avoiding Microsoft “Fast Fail” Updates Using SCCM 2012/R2 Automatic Deployment Rules

I know there’s a risk in telling you to delay deploying updates for 1 month. Some think that means switching to manual approval – and that is an oxymoron because manual approval rarely happens. No; I would rather see large enterprises use a model that automatically deploys updates after delaying them for 1 month, just as you can do with System Center 2012 (R2) Configuration Manager (SCCM).

I’m going to refer you to the excellent guides by SCCM MVP, Niall C. Brady. SCCM uses WSUS to download the Windows Catalog. When I configure SCCM I configure WSUS to automatically sync and to automatically supersede updates. That means if Microsoft releases a replacement update, the old version is automatically replaced. That’s important so keep that in mind when reading the rest of the solution.

I will configure automatic deployment rules (ADRs) for each product. The ADR will be set up as follows:

  • Software Available Time: Set this to something like 21 days. That means that SCCM will hold back any applicable update for 3 weeks. That gives Microsoft lots of time to fix an update and the replacement will supersede the dodgy update.
  • Installation Deadline: With this set to 7 days, we have 4 weeks before updates are pushed out … and that assuming that we haven’t applied maintenance windows to any collections (servers, VMs, call centre PCs, etc) that might further delay the deployment.

image

With the above configuration, the dodgy August updates would not have been deployed to PCs or servers on your network. Instead, a tested and fixed update will be released, SCCM will sit on it and automatically approve it at a later date.

BTW, I do a similar thing with Endpoint Protection updates by delaying approval for 4 hours with immediate deployment.

I don’t know of a method for accomplishing this in Windows Intune – I’d like to see it. The same goes for WSUS, but a commenter suggested using cmdlets from this site for WSUS to write a script; I’d rather see a clean solution from Microsoft similar to what we have in ConfigMgr but less granular.

Microsoft News Summary – 19 August 2014

Does “fail fast” = “fail predictably often”? Automated testing of software for cloud services needs to be investigated and questioned. First we had the clusterfrak August updates for Windows. Then a significant chunk of Azure went offline.

image

Told Ya So Munich – Linux Sucks

How I laughed back in 2003 when I read that Munich was “dumping” Windows to migrate all servers, desktops and productivity software to Linux and open source. At the time I was deploying an XP and Windows Server 2003 network in a German group, headquartered in Munich. I saw up close, how dumb some local IT people could be (hello Marco of HVB and Hypo Real Estate IT! – another case of “I told you so” muppetry).

You see, the Munich city government decided to dump all Microsoft software. Everyone, other than penguin huggers, told them that they were nuts. If you value productivity and collaboration, you go with Microsoft. Even a college student, educated with an open mind instead of brainwashed by a “son of Linus”, can tell you that off-the-shelf software that you pay for is cheaper to buy and own than free software that you have to customise and maintain.

And that’s the lesson that Munich has learned in the last 10 years.

Firstly it took from 2003 until 2013 for Munich to complete the migration. Sounds mad, right? The whole story is mired in secrecy, political rhetoric, and bullshi1t marketing. What we do know is that employees are complaining that they cannot get work done. They can’t figure out Linux workstations. Their productivity software is inferior to Office. And what they produce is incompatible with their customers/suppliers/partners.

Oh well! I guess Munich can find some open source scheiße to use over the next 10 years to migrate back to Microsoft. Or maybe they can hire a giant consulting firm that will cost too much.

Microsoft News Summary – 18 August 2014

The big news this morning is that Microsoft has had to withdraw 4 of last weeks automatic updates. But in other news:

Microsoft Fraks Up Patches AGAIN

I’m sick of this BS.

Microsoft is investigating behavior in which systems may crash with a 0x50 Stop error message (bugcheck) after any of the following updates are installed:

2982791 MS14-045: Description of the security update for kernel-mode drivers: August 12, 2014
2970228 Update to support the new currency symbol for the Russian ruble in Windows
2975719 August 2014 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2
2975331 August 2014 update rollup for Windows RT, Windows 8, and Windows Server 2012

This condition may be persistent and may prevent the system from starting correctly.

If you are affected by any of the above then the repair process (see Known Issue 3) is an ungodly nightmare.

This is exactly why I tell people to delay deploying updates for 1 month. That’s easy using SCCM (an approval rule will do the delaying and supersede for you). WSUS – not so easy and that requires manual approval, which sadly we know almost never works.

Feedback, private and public from MVPs hasn’t worked. Negative press from the tech media hasn’t worked. What will, Microsoft? Nadella oversaw this clusterfrak of un-testing before he was promoted. Is sh1te quality the rule from now on across all of Microsoft? Should we tell our customers to remain un-patched, because catching malware is cheaper than being secure and up-to-date? Really? Does Microsoft need to be the defendant of a class action suit to wake up and smell the coffee? Microsoft has already lost the consumer war to Android. They’re doing their damndest to lose the cloud and enterprise market to their competition with this bolloxology.

Microsoft News Summary – 15 August 2014

Here’s the latest from the last 24 hours: