Tag: Security

KB976323 Wipes the SMTP Configuration

The Windows update MS10-24 for SMTP will wipe the SMTP configuration on Windows Server 2008.  I discovered this today when we found SMTP was no longer relaying email (or accepting local connections) on a couple of servers.  One server and I was scratching my head.  The second one and I knew there was only one common denominator.

It took me a couple of different search attempts to find the culprit.  Even then, I went to the official page for this update and I had to click through 3 pages to find a warning that there might be an issue (I linked the eventual page above).

The developer of this automatic update expects you to magically script a solution to run before the update and after it.  This will backup your SMTP configuration and restore it.  That’s even assuming that your crystal ball has warned you of a problem.  The next time I hear a MS security evangelist talk about instant approval and deployment of updates … …

I know the issue with this update is an exception.  But I am not impressed.  Believe me – I am holding back on how unimpressed I am.

*counting down from 10, 9, 8 …*

Microsoft Guidance: Anti-Virus Exclusion Lists

This one comes on foot of a thread on the Minasi forum related to how AV screwed up a VM on a Hyper-V host. 

My recommendation is to not put antivirus on a Hyper-V host.  Unfortunately there are times when the techies get overruled on that one.  If you have to install AV on a Hyper-V host then you must read every word of this page.  It deals with how to avoid the dreaded 0x800704C8, 0x80070037 or 0x800703E3 errors when you start up virtual machines.  Do not skim over the page, do not make any assumptions about knowing where VM files are, do not undervalue the “shortcuts” you might see, and be aware the hidden folder that is referred to.  Long-time readers might remember my first W2008 RC Hyper-V lab host and how a bunch of VM’s disappeared on me.  This was caused by AV scanning the VM files (including those “shortcuts”) after a host reboot.  The VM’s disappeared from the Hyper-V console, even though all the files were still there.

In a more general AV & Windows conversion, you should also pay attention to Microsoft’s doctrine on AV exclusions for enterprise deployments of Windows.  A long time ago, I had a Sysvol issue on a few DC’s in branch offices.  We ended up believing that our AV caused the issue.  Check out that site when you are putting in AV or configuring scan exclusions.

MuckAfee Update Breaks PC’s

Thanks to Tim Bolton for making me aware of this.  It appears that MuckAfee (aka McAfee) distributed a bad AV update that breaks PC’s.  The malware definition file quarantined a critical XP system file.  They admit that “The problem occurs with the 5958 virus definition file (DAT) that was released on April 21 at 2:00 P.M. GMT+1 (6:00 A.M. Pacific).”

You know what?  I really don’t get why people still use MuckAfee or Sin-Mantec software.  Both have had these issues in the past.  This sort of failure makes me wonder about a complete lack of quality control in the release process.  People have been burned and they continue to hand over money for this trash.  Would you really go to a doctor who amputates your leg instead of extracting your appendix and then return to him again with an ingrown toe nail?  Seriously?  Get real and buy some decent software. 

You do have options!  Trend Micro’s up front cost may seem expensive but it is per user based.  AVG’s business product is fine – and very cheap.  Microsoft’s corporate and home solutions are easy to manage and lightweight on the machine. 

But I guess I’m wasting wear and tear on my keyboard.  Those same people who’ve just seen their XP PC’s die will renew their support contracts because they just don’t want to know better.  So be it.

Technorati Tags:
EDIT:
Microsoft blogged a method to make a published fix that can run as a ConfigMgr task sequence.  Now didn’t I say ConfigMgr was powerful?!?!?

MS10-015 Blue Screen: Microsoft Confirms It Was Malware

As was first reported by people on SANS and the Microsoft Answers forum, the cause of the MS10-015 blue screens of death was actually malware called Alureon or TDSS, i.e. a root kit that was already on the machine and had damaged it.  The update legitimately updated the system and the root kit failed, causing a blue screen.

This highlights a few things:

  • Get your anti-malware installed and keep it up to date.  AVG has a free product.  Avast has a free product.  Best of all, Microsoft Security Essentials is also free.  Microsoft’s free anti virus would have protected those people who were impacted.  Their business product would have protected those business users.
  • 64-bit computers have a built in process that defends system files.  They were not affected.  Walk into a PC store and have a look at the Windows 7 PC’s.  I bet you all (not the netbooks maybe) are running a 64-bit edition of Windows 7.
  • This situation was a great story for Microsoft Answers, the forum aimed at supporting the consumer.  This was where consumers raised awareness of the problem, Microsoft responded with a workaround, and they could gather information to identify the cause.
  • Make sure your patches are kept up to date.  Configure Windows Updates on home/small office PC’s and use WSUS/ConfigMgr to manage updates in the business/enterprise.  Quite often, these malware attacks leverage vulnerabilities that MS would have released patches for quite some time before. 

By the way, Microsoft Windows lets you know of issues in the System Notification area (bottom right) with a red shield.  That allows you to quickly get at a tool to view your security setup where any update or configuration problems will be highlighted.

Microsoft have explained the issue in depth.  The long and short of this one was that people did not defend their PC’s adequately.  Free tools that were marketed and advertised would have protected those people.  Take the time now to check your PC and your security settings.

Microsoft’s Initial Response To MS10-015 / KB977165 Blue Screen of Death

Microsoft’s security operation have issued an initial response to the issue with machines blue screening and failing to reboot correctly after installing MS10-015.

While we work to address this issue, customers who choose not to install the update can implement the workaround outlined in the bulletin. CVE-2010-0232 was publicly disclosed and we previously issued Security Advisory 979682 in response. Customers can disable the NTVDM subsystem as a workaround and we have provided an automated method of doing that with a Microsoft Fix It that you can find here.

Customers who are experiencing issues after installing any of our security updates can get help resolving the issues by either going to this site or by calling 1-866-PCSafety (1-866-727-2338). International customers can find local support contact numbers here.

Technorati Tags: ,

Burn The Witch! Hyper-V Security Fix And Hyperventilating

Ah, it takes a patch to find out who’s really thinking what :)  As you are now aware, Hyper-V had it’s very first (ever) security patch this week.  Not bad (typical Irish understatement) after a year and a half of being the most accessible hypervisor ever.  Just think of how many volume license, OEM, TechNet, MSDN, evaluation and pirated copies of Windows Server 2008 and Windows Server 2008 R2 must be out in the world, not to mention the free to download Hyper-V server, and that it can run on most hardware around in the last few years.  I’m betting people in parent’s basements were attempting to find vulnerabilities since the emergence of the first beta for Hyper-V, around 2 years ago.

And after all that time and opportunity, 1 security hole was found.  It isn’t even the dreaded “break out” where a VM is capable of reaching out and accessing the host and other VM’s.  No, it was a DOS attack where the hypervisor would shut down.  And you had to be logged into a VM on the host with admin rights!

I’ve noticed a lot of tweets in the last 48 hours of people writing with glee about a dreaded problem, implying that Hyper-V is inferior.  Oh, get over it!  I can think of another hypervisor from a certain company that has suffered from a break out attack.  Its patches are a complete OS upgrade and they break the host on a way too frequent basis.  So much so, in fact, that experts in that technology run 1 “service pack” behind the latest release to stay safe.

It’s a secure platform.  Think of all those attackers who hate Microsoft and have the chance to attack the most available hypervisor around and we get 1 patch in 2 years (since beta).  That’s unbelievable.  The basic architecture requirements (DEP) prevent buffer overrun attacks on the host from a VM.  The German government has certified it as being secure … trust me if you are unfamiliar with working in Germany … that doesn’t happen by accident.  Every piece of complex software has vulnerabilities and bugs.  If you didn’t learn that in programming classes in college then you need to ask for a refund.  The fact is that Hyper-V is so well designed and implemented that it’s taken quite some time for one to be found.  And Microsoft reacted perfectly about it.

So before you go running to the woods to get some kindling for the witch burning, sit back, breath into a brown paper bag and realise that this is not the end of the world for Microsoft virtualisation.  It’s actually not bad at all.  It was one small patch that was quick and easy to download and installed reliably. 

MS10-015 / KB977165 Causing BSOD For Some – How To Deal With The Issue

I came home tonight to see reports of blue screens of death and failed boot up/reboots for XP machines that had installed the MS10-015 security patch or hotfix.  There is a long thread on Microsoft Answers.

After installing the patch you have to reboot.  You are then greeted with:

PAGE_FAULT_IN_NONPAGED_AREA

Technical Information:
STOP: 0x00000050 (0x80097004, 0x00000001, 0x80515103, 0x00000000).

The posted fix by Microsoft in the thread is:

  1. Boot from your Windows XP CD or DVD and start the recovery console (see this Microsoft article for help with this step).  Once you are in the Repair Screen..
  2. Type this command: CHDIR $NtUninstallKB977165$spuninst       
  3. Type this command: BATCH spuninst.txt
  4. When complete, type this command: exit

There is what appears to be some misinformation or hysteria about this.  For example:

  • Some news articles are claiming that Windows 2003 and Vista are reported in this thread as being affected.  I saw no mention of those operating systems.
  • I saw one article (a random find) that tried to make it look like that this affected Hyper-V.  Pah!  It does not from everything I have read.  There are no reports of issues with Windows Server 2008 or Windows Server 2008 R2.  Put the Kool-Aid down and step away from the cup.

I cannot claim there are not problems there but they are not in that thread.

EDIT: Overnight after this blog post was originally written, some people did post about Vista and W2003 suffering issues with blue screens caused by the update.

It is bad that a patch has affected many.  I’m sure MS will be making someone feel very uncomfortable overnight about this.  It’s bad that it happened at all.  But let’s face it.  Not everyone is affected.  There is some combination in factors that is contributing to the blue screen.  There is some scenario that MS didn’t test or couldn’t predict.  These things happen.  It could be some niche piece of software or driver that reacts badly to the patch.

EDIT: I’ve read on one site that some people are finding an issue with the ATAPI.SYS file not looking like the genuine file supplied by MS.  They suspect an old malware issue causes an incompatibility with the fix!!!

This situation (whatever the actual cause of the blue screens) is why I think people like Steve Reilly who preach that we should all push out security updates immediately and without question are wrong for me (maybe not wrong for you).  How many zero day exploits have there ever been?  Not many.  Think of the big bad attacks … Nimda, SQL Slammer, MS Blaster, Conficker.  They all attacked vulnerabilities that were fixed with patches long before hand.  What’s a couple of weeks?  It’s because of the rare occasion when a patch goes wrong that I run a 3 phase process for patches.

I have three groups in WSUS.  I configure my Windows Update agents either via group policy (AD members) or registry edits (.REG files for workgroup members) to be members of 1 of 3 groups:

  1. Testing – contains VM’s with various blends of OS and application
  2. Management – Our production AD, management systems, and online applications
  3. Hosting– Hosted customer servers

We’re a hosting company.  WSUS has an automatic approval policy for the Testing group.  The machines in that group are VM’s on my Hyper-V lab server.  They patch in the late morning/early afternoon (around lunch) so we can see how they reacted.

Ideally that group would contain samples of the various bits of hardware you have on the network to include drivers in the mix.  I was lucky enough to be able to do that with one employed in the past – but we did push out updates in less than a week from release.  However, I need to be cost conscious and that is not an option now.

When we’re happy we sit and watch the news.  If all is well, change control happens, and then we approve the updates for the management network.  Stealing a line from Microsoft, we eat our own dog food.  Over the 3 nights of the following weekend (Friday, Saturday, Sunday), machines are patched and reboot automatically.  Some services are clustered/replicated and we do them on different nights or time slots.  We have scheduled scripts on the OpsMgr RMS to put machines into maintenance mode.

Now we watch how that went and continue to watch the news wires.  If there’s no more problems then we approve the updates for the hosting customers after another change control process.  Patches then deploy according to their pre-agreed time windows.

The end result is that within 2-3 weeks all security updates are deployed.  You could compress this down to a week.  We are totally minimizing the risk of being stung by a “bad” update.  Like I said earlier, MS probably did test the update as far as is realistically possible.  There is always the chance that something bad happens.

Steve Reilly’s argument was that if you get a bad update then you call easily rollback your server farm because it’s probably 90% virtual.  In my opinion you shouldn’t really use snapshots in production on Hyper-V.  They’re supported but they suck the life from your VM’s.  DPM or 3rd party solutions that are using the Hyper-V VSS writer are cool for this.  But really, do you want to risk your production network going down for hours while you recover (starting at 3am when your patch failed) because of the rush to deploy an update that will likely not have an attack vector for quite some time?

Weigh the various risks and make an informed decision for yourself.  Maybe Steve Reilly’s approach to push out updates without testing is right for you.  Maybe my phased and cautious approach is.   Maybe there is a middle ground that you prefer.  Do the research and be sure you know why you make your decision and that it is based on fact.

EDIT:

There is strong suspicion that the BSOD’s are actually happening on machines that were already infected by a rootkit called TDSS.  It attacks ATAPI.SYS and replacing that file appears to fix the BSOD issue as well.  Microsoft Security Essentials appears to be able to detect it.

Technorati Tags: ,,,

That Was The First Security Fix For Hyper-V

By the way, when I posted about the security fix for Hyper-V last night, I should have mentioned that it was the first such on for the hypervisor itself in the 18 or so months since it RTM’d in Windows Server 2008.  Not bad!

There’s some debate about how important it is.  Basically, if someone can log into a VM running on a host and has admin rights in that VM, then they can run a DOS attack on the hypervisor on that host.  Most scenarios will probably be safe enough.

I would guess that most companies that deployed virtualisation are running it for internal server virtualisation purposes.  The people who log into those machines are trusted administrators and extremely unlikely to go postal.

Funny phrase that.  I once worked with a guy who was the son of post employees and he didn’t know what it meant.  He got highly offended!

Virtualised “terminal services”, or to put it correctly using the current phrase, Remote Desktop Services Session Hosts *gasps for air*,  will likely only have users logging in with limited rights so they will be safe.

Some VDI implementations will have users logging in with administrative rights.  That means that they are vulnerable.  And those operating cloud services (server hosting) based on Hyper-V are vulnerable.  Those operating private clouds with large numbers of unknown administrators also face a risk.  It’s inevitable that someone will write an attack script/program for this.

I fall into one of those vulnerable scenarios so our normal patching process was put to one side today.  The update was approved in WSUS for all groups, not just our testing group.  Using Operations Manager and VMM we put clustered hosts into maintenance mode.  This allows VMM to use Live Migration to move VM’s from the host that will be worked on to another host.  If you don’t have VMM then you need to Live Migrate each of the VM’s, one by one.  OpsMgr maintenance mode prevents false alarms.  This is done in turn with all hosts in the cluster.  No customers have down time and the security fix gets deployed.  Nice and tidy.

Technorati Tags: ,

KB977894: VERY Important Hyper-V Security Update

One of the patches released by Microsoft is a critical security fix for Hyper-V.  It affects all installation types on both Windows Server 2008 and Windows Server 2008 R2.

“This security update resolves a privately reported vulnerability in Windows Server 2008 Hyper-V and Windows Server 2008 R2 Hyper-V. The vulnerability could allow denial of service if a malformed sequence of machine instructions is run by an authenticated user in one of the guest virtual machines hosted by the Hyper-V server. An attacker must have valid logon credentials and be able to log on locally into a guest virtual machine to exploit this vulnerability. The vulnerability could not be exploited remotely or by anonymous users”.

Basically, if a person has rights to log into a VM hosted on a vulnerable server, then they could cause a Denial-of-Service (DOS) attack. 

The update is supplied via Windows Update.  Check your updates either on the host, Configuration Manager, WSUS or whatever your update service is.

I’ll be pushing it out first thing tomorrow morning.  Live Migration with VMM 2008 R2 maintenance mode makes it really easy to update clustered hosts.  Standalone hosts will have some downtime for their VM’s.  Most VM’s will be set up to go into a saved state when the host shuts down.  That limits interruption to them in a way.

Worried About Hyper-V Security or Reliability? Don’t!

I’ve previously mentioned that Windows Server 2008 Hyper-V is secure.  It was designed from the ground up to be secure.  The German Federal government certainly thought so.  I’ve had excellent experiences.  But why take my word?

There’s one country out there that takes security more seriously than any other.  I’m not talking about the USA, the UK, China or France, and certainly not Ireland.  It’s Israel.  And what part of the nation do you think would value reliability more than anything else?  Yeap, the military.  The Israeli Defence Forces (IDF) just spent a bunch of money to deploy a Hyper-V cluster.  If it’s secure and reliable enough for them then it should be for you too.