MS10-015 / KB977165 Causing BSOD For Some – How To Deal With The Issue

I came home tonight to see reports of blue screens of death and failed boot up/reboots for XP machines that had installed the MS10-015 security patch or hotfix.  There is a long thread on Microsoft Answers.

After installing the patch you have to reboot.  You are then greeted with:


Technical Information:
STOP: 0x00000050 (0x80097004, 0x00000001, 0x80515103, 0x00000000).

The posted fix by Microsoft in the thread is:

  1. Boot from your Windows XP CD or DVD and start the recovery console (see this Microsoft article for help with this step).  Once you are in the Repair Screen..
  2. Type this command: CHDIR $NtUninstallKB977165$spuninst       
  3. Type this command: BATCH spuninst.txt
  4. When complete, type this command: exit

There is what appears to be some misinformation or hysteria about this.  For example:

  • Some news articles are claiming that Windows 2003 and Vista are reported in this thread as being affected.  I saw no mention of those operating systems.
  • I saw one article (a random find) that tried to make it look like that this affected Hyper-V.  Pah!  It does not from everything I have read.  There are no reports of issues with Windows Server 2008 or Windows Server 2008 R2.  Put the Kool-Aid down and step away from the cup.

I cannot claim there are not problems there but they are not in that thread.

EDIT: Overnight after this blog post was originally written, some people did post about Vista and W2003 suffering issues with blue screens caused by the update.

It is bad that a patch has affected many.  I’m sure MS will be making someone feel very uncomfortable overnight about this.  It’s bad that it happened at all.  But let’s face it.  Not everyone is affected.  There is some combination in factors that is contributing to the blue screen.  There is some scenario that MS didn’t test or couldn’t predict.  These things happen.  It could be some niche piece of software or driver that reacts badly to the patch.

EDIT: I’ve read on one site that some people are finding an issue with the ATAPI.SYS file not looking like the genuine file supplied by MS.  They suspect an old malware issue causes an incompatibility with the fix!!!

This situation (whatever the actual cause of the blue screens) is why I think people like Steve Reilly who preach that we should all push out security updates immediately and without question are wrong for me (maybe not wrong for you).  How many zero day exploits have there ever been?  Not many.  Think of the big bad attacks … Nimda, SQL Slammer, MS Blaster, Conficker.  They all attacked vulnerabilities that were fixed with patches long before hand.  What’s a couple of weeks?  It’s because of the rare occasion when a patch goes wrong that I run a 3 phase process for patches.

I have three groups in WSUS.  I configure my Windows Update agents either via group policy (AD members) or registry edits (.REG files for workgroup members) to be members of 1 of 3 groups:

  1. Testing – contains VM’s with various blends of OS and application
  2. Management – Our production AD, management systems, and online applications
  3. Hosting– Hosted customer servers

We’re a hosting company.  WSUS has an automatic approval policy for the Testing group.  The machines in that group are VM’s on my Hyper-V lab server.  They patch in the late morning/early afternoon (around lunch) so we can see how they reacted.

Ideally that group would contain samples of the various bits of hardware you have on the network to include drivers in the mix.  I was lucky enough to be able to do that with one employed in the past – but we did push out updates in less than a week from release.  However, I need to be cost conscious and that is not an option now.

When we’re happy we sit and watch the news.  If all is well, change control happens, and then we approve the updates for the management network.  Stealing a line from Microsoft, we eat our own dog food.  Over the 3 nights of the following weekend (Friday, Saturday, Sunday), machines are patched and reboot automatically.  Some services are clustered/replicated and we do them on different nights or time slots.  We have scheduled scripts on the OpsMgr RMS to put machines into maintenance mode.

Now we watch how that went and continue to watch the news wires.  If there’s no more problems then we approve the updates for the hosting customers after another change control process.  Patches then deploy according to their pre-agreed time windows.

The end result is that within 2-3 weeks all security updates are deployed.  You could compress this down to a week.  We are totally minimizing the risk of being stung by a “bad” update.  Like I said earlier, MS probably did test the update as far as is realistically possible.  There is always the chance that something bad happens.

Steve Reilly’s argument was that if you get a bad update then you call easily rollback your server farm because it’s probably 90% virtual.  In my opinion you shouldn’t really use snapshots in production on Hyper-V.  They’re supported but they suck the life from your VM’s.  DPM or 3rd party solutions that are using the Hyper-V VSS writer are cool for this.  But really, do you want to risk your production network going down for hours while you recover (starting at 3am when your patch failed) because of the rush to deploy an update that will likely not have an attack vector for quite some time?

Weigh the various risks and make an informed decision for yourself.  Maybe Steve Reilly’s approach to push out updates without testing is right for you.  Maybe my phased and cautious approach is.   Maybe there is a middle ground that you prefer.  Do the research and be sure you know why you make your decision and that it is based on fact.


There is strong suspicion that the BSOD’s are actually happening on machines that were already infected by a rootkit called TDSS.  It attacks ATAPI.SYS and replacing that file appears to fix the BSOD issue as well.  Microsoft Security Essentials appears to be able to detect it.

Technorati Tags: ,,,

That Was The First Security Fix For Hyper-V

By the way, when I posted about the security fix for Hyper-V last night, I should have mentioned that it was the first such on for the hypervisor itself in the 18 or so months since it RTM’d in Windows Server 2008.  Not bad!

There’s some debate about how important it is.  Basically, if someone can log into a VM running on a host and has admin rights in that VM, then they can run a DOS attack on the hypervisor on that host.  Most scenarios will probably be safe enough.

I would guess that most companies that deployed virtualisation are running it for internal server virtualisation purposes.  The people who log into those machines are trusted administrators and extremely unlikely to go postal.

Funny phrase that.  I once worked with a guy who was the son of post employees and he didn’t know what it meant.  He got highly offended!

Virtualised “terminal services”, or to put it correctly using the current phrase, Remote Desktop Services Session Hosts *gasps for air*,  will likely only have users logging in with limited rights so they will be safe.

Some VDI implementations will have users logging in with administrative rights.  That means that they are vulnerable.  And those operating cloud services (server hosting) based on Hyper-V are vulnerable.  Those operating private clouds with large numbers of unknown administrators also face a risk.  It’s inevitable that someone will write an attack script/program for this.

I fall into one of those vulnerable scenarios so our normal patching process was put to one side today.  The update was approved in WSUS for all groups, not just our testing group.  Using Operations Manager and VMM we put clustered hosts into maintenance mode.  This allows VMM to use Live Migration to move VM’s from the host that will be worked on to another host.  If you don’t have VMM then you need to Live Migrate each of the VM’s, one by one.  OpsMgr maintenance mode prevents false alarms.  This is done in turn with all hosts in the cluster.  No customers have down time and the security fix gets deployed.  Nice and tidy.

Technorati Tags: ,

KB977894: VERY Important Hyper-V Security Update

One of the patches released by Microsoft is a critical security fix for Hyper-V.  It affects all installation types on both Windows Server 2008 and Windows Server 2008 R2.

“This security update resolves a privately reported vulnerability in Windows Server 2008 Hyper-V and Windows Server 2008 R2 Hyper-V. The vulnerability could allow denial of service if a malformed sequence of machine instructions is run by an authenticated user in one of the guest virtual machines hosted by the Hyper-V server. An attacker must have valid logon credentials and be able to log on locally into a guest virtual machine to exploit this vulnerability. The vulnerability could not be exploited remotely or by anonymous users”.

Basically, if a person has rights to log into a VM hosted on a vulnerable server, then they could cause a Denial-of-Service (DOS) attack. 

The update is supplied via Windows Update.  Check your updates either on the host, Configuration Manager, WSUS or whatever your update service is.

I’ll be pushing it out first thing tomorrow morning.  Live Migration with VMM 2008 R2 maintenance mode makes it really easy to update clustered hosts.  Standalone hosts will have some downtime for their VM’s.  Most VM’s will be set up to go into a saved state when the host shuts down.  That limits interruption to them in a way.

Worried About Hyper-V Security or Reliability? Don’t!

I’ve previously mentioned that Windows Server 2008 Hyper-V is secure.  It was designed from the ground up to be secure.  The German Federal government certainly thought so.  I’ve had excellent experiences.  But why take my word?

There’s one country out there that takes security more seriously than any other.  I’m not talking about the USA, the UK, China or France, and certainly not Ireland.  It’s Israel.  And what part of the nation do you think would value reliability more than anything else?  Yeap, the military.  The Israeli Defence Forces (IDF) just spent a bunch of money to deploy a Hyper-V cluster.  If it’s secure and reliable enough for them then it should be for you too.

Microsoft – Cloud Computing Security Considerations

One or two people out there are talking about something called Cloud Computing.  I doubt it’s important 😉

Microsoft published a document on cloud computing security:

“A high-level discussion of the fundamental challenges and benefits of cloud computing security, plus some of the questions that cloud service providers and organisations using cloud services need to consider when evaluating a new move, or expansion of existing services, to the cloud. This document presumes that the reader is familiar with the core concepts of cloud computing and basic principles of cloud security. It is not the goal of this paper to provide all the answers to the questions of security in the cloud or to provide an exhaustive framework for cloud security“.

Technorati Tags:

Setting Up Public Key Based SSH Access To SLES

I needed to set up key based, rather than password based, access to SUSE Linux Enterprise Server.  It’s more secure because it uses a public/private key pair rather than a password.  The user’s private key is stored on the client.  The private key for the user is stored on the Linux machines.  When they connect using an SSH client there is no need to enter a password.  You can optionally (and it’s recommended) store a passphrase with the private key so that it cannot be used without knowing the private key.

The solutions starts at the client.  I normally used Putty but I couldn’t get it to work properly with this type of solution.  Instead I turned to Poderosa.  Using it I create a public and private key pair.  From there I saved the public key in OpenSSH format and the private key.

Save the private key somewhere safe, e.g. a backed up location on your PC or on your home drive on a file server.  Make sure the location is secure.

Now you need to copy the text of the public key.  Note that it is a single line.  Log into the SLES machine and browse to your home directory.  For example:

  • For root browse to ~/.ssh
  • For any other user browse to /home/<username>/.ssh

Use a text editor (like vi) to create a file called authorized_keys in that home directory.  Copy the text from your private key and paste it into the file.  Save it.

You now need to enable SSH to allow logons using keys.  The configuration for SSH is stored in a text file: /etc/ssh/sshd_config.  Edit that and you’ll have a few entries to modify.  We’ll start by allowing public keys to be used for authentication.  This is done by setting PubkeyAuthentication to “yes”.  I had to remove the # (comment/remark) symbol from the start of the line.

PubkeyAuthentication yes

I restarted the SSH daemon or service by running rcsshd restart.  That’s required to load the new settings for authentication. 

I configured the SSH client to log in as my user to this server with my private copy of the key.  I started the connection and I was logged in without using a password.  It authenticated me using the private key (and the passphrase for the key if you set it).

Now it is possible to disable log via SSH on using passwords.  You’ll do this to force people to us their private key instead of a weaker password that could be subject to brute force attacks.

The first is to change PasswordAuthentication to have a value of “no”.  You may need to remove the comment/remark symbol of # from the start of the line.  I also found that I had to set UsePam to a value of “no”.  That meant these two lines were in the file in different locations:

PasswordAuthentication no

UsePam no

Again I restarted SSH using rcsshd restart.  Now I tested two things:

  1. I tried to login using Putty and my username and password.  The initial connection failed.
  2. I logged in using my private key.  That worked.

Perfect.  Now I can use SSH to log into the Linux box without the worry of weak passwords being used by users on the machine.  They are forced into using stronger public/private key pairs.  And I can sleep safe knowing that the machine is not vulnerable to brute force password attacks.

Technorati Tags: ,

The BitLocker “Crack”

Microsoft releases a new operating system and everyone wants to make a headline.  It happened 2 years ago and it’s happening again.

This time some people are claiming they’ve broken BitLocker.  Their attack vectors work two ways:

  1. Attack the machine while it’s running and a user is logged in.  That way they can scan the RAM for cached BitLocker keys.  If you have the machine while it’s logged in then you have access to the data.  Pointless.
  2. Gain access to the machine to attack the hardware.  Install something to capture the PIN as the machine boots up.  Then steal the machine or gain access to it again and use the captured data to access the hard disk data.

That last one would be a threat, admittedly.  It’s a far fetched one for laptops but it feasible.  I’m guessing that BitLocker with a Smart Card would beat that one assuming the smart card is not kept with the laptop.  We know how lazy people can be so – eek.  And potentially the latter approach is one to attack on-premises physical servers. 

I guess we’ll see.

Technorati Tags: ,

WSUS: The Update Could Not Be Found

I was re-installing the WSUS role on our security server (W2008 R2) today and hit this error as soon as the installation started:

“The update could not be found”.

It’s a bit of a weird one for a role installation.  I hadn’t the foggiest so I did a quick search and found the solution:

  • Delete the “WindowsUpdate” key from the registry at HKLMSoftwarePoliciesMicrosoftWindows.  I’d recommend you export this to a .reg file to be safe.
  • Restart the Windows Update service.

Now you can go ahead and install WSUS.

The problem and fix applies to previous versions of Windows.  The issue is that the installer is checking Windows Updates but it has found a circular reference.  You’ve uninstalled WSUS from the server and it is configured to update from itself.  How can it?  Make sure you do the install before GPO applies those settings again during an automatic refresh.

Technorati Tags: ,

Microsoft Responds to Black Screen of Death Claims

It was widely reported that a UK company was claiming that one of last weeks security updates by Microsoft was causing a “black screen of death” where Explorer would show nothing when you logged in.  Microsoft responded overnight:

“While these reports weren’t brought to us directly, from our research into them, it appears they’re saying that our security updates are making permission changes in the registry to the value for the HKEY_LOCAL_MACHINESoftwareMicrosoftWindows NTCurrentVersionWinlogonShell  key.

We’ve conducted a comprehensive review of the November Security Updates, the Windows Malicious Software Removal Tool, and the non-security updates we released through Windows Update in November. That investigation has shown that none of these updates make any changes to the permissions in the registry. Thus, we don’t believe the updates are related to the “black screen” behavior described in these reports.

We’ve also checked with our worldwide Customer Service and Support organization, and they’ve told us they’re not seeing “black screen” behavior as a broad customer issue. Because these reports were not brought to us directly, it’s impossible to know conclusively what might be causing a “black screen” in those limited instances where customers have seen it”.

There you have it.  Prevx didn’t do the responsible thing, i.e. contact Microsoft directly, and instead decided to generate some publicity for themselves.  Their claims have been refuted so this leads me to wonder: are these developers more of the same who don’t comply with documented standards and just write rubbish code and to hell with their customers?  I don’t know them, never dealt with them and certainly never heard of them before yesterday.  You decide 🙂

Technorati Tags: ,

Black Screen of Death

I’ve read a few reports on this today.  It’s being claimed by Prevx that one of Microsoft’s recent security updates causes issues on machines.  The machine boots up OK, but you get nothing but a black screen when you log in.  The have posted a “fix” (it’s not something MS has authorised).  Note that this is an issue that can be caused by a lot of things.  I’ve personally seen it a few times over the years.

Microsoft are reported to be investigating.

Technorati Tags: ,