VMM 2012 Distributed Key Management (DKM)

Virtual Machine Manager 2012 (VMM/SCVMM) 2012 adds something that was lacking in VMM 2007/2008/20008 R2: clustered VMM servers.  VMM 2012 is the gateway to the private cloud and you want that gateway to be fault tolerant at the hardware, OS, and service level.  If you want to have a clustered VMM server then you will need to get to grips with some new concepts.

The VMM database contains a lot of information.  Some of that information can be sensitive, such as product keys or administrator passwords.  You don’t want just anyone getting a copy of that database (from offsite stored backup tapes, for example [which should be encrypted anyway]) and figuring out a way into gaining administrative rights to your network.  For this reason, VMM uses encryption to protect the contents of this database. 

By default the decryption keys for accessing the encrypted data are stored on the VMM server.  Now imagine you have set up a clustered VMM server and those keys are stored locally, as seen below.

image

The first node with the local keys would encrypt the SQL data and access it with no issue at all.  But what would happen after a failover of the VMM service from Node 1 to Node 2?  The decryption keys are unavailable, on Node 1, and Node 2 has no way to read the encrypted data in clear text.  There goes the uptime of your cloud!

image

That’s why we have a new concept called Distributed Key Management (DKM) in VMM 2012.  Instead of storing the decryption keys on the server, they’re stored in a specially created container in Active Directory.  This means that the decryption keys can be accessed by both of the VMM cluster nodes, and either node can read the encrypted data in clear text.

You can configure the option to enable DKM when you install the first member of the VMM cluster.  You can optionally do this even if you’re setting up a non-clustered VMM server.  It’ll mean the keys are safe in AD, and it gives you the flexibility to easily set up a cluster without too much mucking around.

When you enable the option to use DKM, you have two choices:

  • Installing as a Domain Administrator: You can enter the LDAP path (e.g. CN = VMMDKM, CN = System, DN = demo, DN = local) and the installer will use your rights to create the VMM container inside of the default System container.
  • Not Installing as a Domain Administrator: You can get a domain admin to create the container for you, ensuring that your new user account will have Read, Write, and Create all child objects permissions.  You can enter the LDAP path (as above) that is provided by the domain administrator.

I like SystemVMMDKM for two reasons:

  1. ConfigMgr uses SystemSystemsManagement for its advanced client objects
  2. VMMDKM is quite descriptive. 

Now Node 1 of the VMM server cluster will use the DKM/AD-stored decryption keys and access the secured data in the SQL Server instead of storing them locally.

image

After a failover, Node 2 can also read those DKM/AD-stored decryption keys to access the encrypted data successfully:

image

Decryption keys; I bet your security officer is concerned about that!  I haven’t mentioned the protection of these keys yet.  Note how we didn’t do anything to lock down that container?  Normally, Authenticated Users will have read permissions.  We sure don’t want them to read those decryption keys!  Don’t worry, the VMM group has you covered.

In the new container, you will find an object called DC Manager <unique GUID>.  This is a container that DKM has created and contains the protected keys for the VMM server/cluster you just set up.

clip_image002

It is protected using traditional AD permissions.  VMM is granted rights based on what account is running VMM.  I prefer to install VMM using a domain user account, e.g. demoVMMSvc.  That account was granted full control over the container object and all descendent (contained) objects:

clip_image001

Note that Authenticated Users is not present.  In fact what you will find is:

  • Self: Inherited with apparently no rights
  • System: Full Control on the container object only
  • Enterprise Domain Controllers: Read tokenGroups (Descendent User Objects), Read tokenGroups (Descendent Group Objects), Read tokenGroups (Descendent Computer Objects)
  • Enterprise Admins: Full Control on this and descendent objects
  • Domain Admins: Full Control on this and descendent objects
  • Administrators: It’s long but basically it’s not Full Control and no delete rights on this and descendent objects
  • Administrator: Full Control on this and descendent objects

In other words, VMM 2012 DKM is a pretty sure way to:

  • Enable a SQL database to securely store sensitive data for a highly available VMM cluster running across multiple servers
  • Allow those nodes of a highly available VMM cluster to share a single set of decryption keys to access the encrypted data in the SQL database

Now you have some very special data in your AD – like you didn’t already!  But if you’re “just” a virtualisation administrator/engineer or a consultant, you better make sure that someone is backing up AD.  Lose your AD (those DKM keys), and you lose that sensitive data in the SQL database.  While you’re verifying the existence of a working AD backup (System State Backup of a few DCs, maybe), make sure that the backup is secure in terms of access rights to data and encryption.  You’ve got sensitive encryption keys in there after all.

So … I Prefer No Antivirus on Hyper-V Hosts?

Waiver: What you do following reading this post is up to you. 

After my earlier post on “Top Hyper-V Implementation Issues” I had some feedback on my preference to keep antivirus (AV) off of the Hyper-V hosts.

The configuration that you should have is in KB961804.  That article also says what can happen if you do install AV on your hosts, not follow that guidance, and scan everything.  One day you’ll end up with nasty errors such as 0x800704C8, 0x80070037 or 0x800703E3 and find lots of VMs (with their business apps and data) have:

  • Disappeared from your Hyper-V console
  • Disappeared from your VMM console
  • Are not running

The files are still there but, damn, the VMs will not start up or appear in a management tool.  That’s because AV has gotten in the way and screwed up with things.  I blogged about this back during the W2008 Hyper-V beta (can’t find the post now) in early 2008.  It happened to me.  I was unlucky; I set the required exclusions and restarted the host in question (a lab machine).  My VM configuration files were corrupted.  The solution was the recreate the VM’s and point them at the existing VHD’s containing the safe OS, programs, and data.  Time consuming – and how many people document/remember their VM configurations?  And come to think of it, how many businesses would be OK with their LOB applications being offline for half a day or more while admins do this?

I learned something in 2004.  There is a balancing act between security and business.  Sometimes it has to swing one way, sometimes another.  This is one of those cases.

I do not trust any antivirus product completely.  They are stupid assassins.  They are given rules of engagement, get a target list, and they attack.  But all too often, program updates, definition file updates, or dumb human operator error make mistakes.  It is not unknown for one of these to reset the exception list.  Yes; it has happened – and even happened recently.  Do you really want one of these things to undo the necessary configurations of your Hyper-V cluster – a thing that is effectively a mainframe running many/most/all of your LOB applications, and putting them at risk?

So I say: do not install AV on the parent partition or host OS.  Sure, go ahead and install it in the VMs.  If you can, choose an AV product that is aware of things like virtualisation and minimises redundant scanning.  On the host, make sure you apply security fixes.  Keep the service pack up to date.  And keep the Windows Firewall running.  Finally, restrict who has logon rights to the hosts.  If you can, prevent the hosts from having proxy/web access.  People should never browse from a server but I just don’t trust human nature.  All that should secure the parent pretty well.

Now let’s get back to why you’re installing AV on the parent partition.  Odds are there is a security officer who has a list of things that [booming voice] “must be done to all Windows computers” [/booming voice].  And if you do not do these things you will be fired!   One of them is: “you must install anti virus and scan everything because Windows is a threat to life itself”.  Hmm, someone’s been reading the SANS website again!  I hate checklist security experts.

Here’s my response to that person:

  • I’d point them to KB961804.  In fact, you might even want to show them the Microsoft required exceptions list.  It says “recommended” in the title but try having that argument with a MSFT support engineer when your SYSVOL is corrupted!
  • If they insist, then say you’ll comply but you have one requirement.  Never say “no” because that’s career suicide.  Give them a waiver form.  This form will clearly state that you the operator/administrator/engineer/consultant will not be held responsible for any corruption or loss of virtual machines because of the mandate to scan all things on the Hyper-V hosts.  All responsibility will lie with the undersigned security officer – and demand their signature.  Then keep a copy for yourself, give one to your boss, and one to the CIO.  At least then you know who will get fired when incorrectly configured AV causes your VMs to disappear.

It’s funny; security officers are usually career politicians.  And politicians do not like being nailed down to a something like that.  Taking responsibility is not in a politician’s nature.  I bet you get your way after that.

Maybe as a compromise, you might offer to take a host offline once in a while to perform a complete system scan of the C: drive.

Anyway, that’s my opinion on the matter.

Recommended Updates or Hotfixes for W2008 R2 SP1 Hyper-V

It used to be that we had an official page on TechNet for updates for Windows Server 2008 R2 Hyper-V.  It has since been decided to move the Windows Server 2008 R2 Service Pack 1 Hyper-V recommended updates list over to the TechNet wiki where it is community driven.

Book Review – Daemon by Daniel Suarez

The story of Daemon  is that a games development genius dies, but that doesn’t stop him from wreaking havoc on the world.  Before he dies, he uses the AI from his games to create a distributed network to enact his will.

This book has what Zero Day didn’t: a hook, something to keep you turning the pages.  In fact, I found it quite addictive.  I was reading it before work, at lunch, and going to bed early to read more.  I finished it this morning and immediately ordered/downloaded the sequel, Freedom.

Whereas Zero Day featured an extremely believable scenario, Daemon goes a little bit more into the sci-fi end of things to add an element of danger.  However, it is still rooted in the believable.  I can’t watch a movie or read a book that features “go hack now” scenarios.  But this book was based on things like trojans, in-game AI, RSS feeds, GPS, and so on.  It just stretched what we know about a little to enable the plot, but kept this acceptable an acceptable limit for me.

Over and over, in this book, you’ll see how hacks take advantage of poor patch control.  Spotting a trend?

I reckon that if you work in IT, or find computers interesting, then there’s a really good chance that you’ll like Daemon.  This book can be ordered on Amazon.com.

Technorati Tags: ,

Finished Reading Zero Day by Mark Russinovich

One of the nice things about not having constant deadlines is that I can “chillax”.  I’ve been getting a lot of reading done on my Kindle/iPad combination.  And the latest book I’ve read is Zero Day, the debut novel by famous Windows insider guru Mark Russinovich.

The book centres around an independent IT security consultant who stumbles on a worldwide IT security threat, and then goes on from there.  I normally cannot stand any form of entertainment that features IT.  There are usually so many holes in the technology that is the centre of the plot that I focus on those rather than on the story.  Not so here, as you would expect.  The IT stuff appears accurate to me, and technical terms like a rootkit are dealt with at a high enough level that your granny will know all about them when she finishes the book.

The story is OK.  I think it was missing a little something, a hook, … I dunno, I’m no novelist!  It’s just that I finished it and was left wanting something more from it.  But that’s just my opinion; lots of others have loved it and Mark Russinovich broke the news yesterday that a publisher has agreed to publish a follow up.

Where the book scores points is that it gets across that businesses are failing to get the most basic IT security practices right.  Things like patching and antivirus still are not being done.  And that probably goes back to an old soapbox rant of mine: many decision makers don’t value IT, and therefore don’t understand how it can benefit a business if dealt with strategically or put it at the risk of complete destruction if the right staff aren’t hired and best practices aren’t implemented. So if you are in IT and want a Secret Santa gift for the CIO/CEO, give them a copy of Zero Day Smile

I’m now reading Daemon by Daniel Suarez.  I’m just a short way into it but it’s started out well.  Leo Laporte and Steve Gibson both recommended it on the TWiT security podcast a few weeks ago.  I’ll blame them if it sucks Winking smile

Technorati Tags: ,

ConfigMgr 2007 Management Point Won’t Install – Failed to Create the CCM_Incoming Virtual Directory

I’ve been working on a customer site for the last few days in my old stomping ground: System Center Configuration Manager (SCCM) 2007.  It’s a new deployment in a mature Windows XP network.  Today started out as a nightmare.  I had all the prereqs done but the install of the primary site server was not going well.  The management point just would not install.  The SMS_MP_CONTROL_MANAGER was reporting that:

“MP Control Manager detected MPsetup has failed to create the CCM_Incoming Virtual Directory.

Possible cause: The IIS IWAM account has expired, been disabled, or has invalid or too restrictive logon hours. You may verify this information by running the net user command line for the IWAM account. (i.e.: "net user IWAMMachineName)

Solution: Use the output to verify that the account is enabled, and logon is possible during the time of installation. Note: You can use "net user" to modify the account properties.
Possible cause: The IIS IUSR account has expired, been disabled, or has invalid or too restrictive logon hours. You may verify this information by running the net user command line for the IUSR account. (i.e.: "net user IWAMMachineName)

Solution: Use the output to verify that the account is enabled, and logon is possible during the time of installation. Note: You can use "net user" to modify the account properties.
Possible cause: The designated Web Site is disabled in IIS.

Solution: Verify that the designated Web Site is enabled, and functioning properly”.

I knew that all IIS components were installed and configured correctly: I use my Zero Touch chapter from Mastering Windows 7 Deployment as my ConfigMgr prereqs check list!  Using that, I can normally get an all green install.  But something here was wrong.  I suspected a security issue … who knows what’ll impact you in a mature network.  I googled and a number of people reported corrupt IIS metabases caused issues.  A reinstall of IIS and ConfigMgr ensued but no result.

Now I was sure an external factor was at fault.  I’d heard that some security feature had screwed up the XP machines in the past.  Something to do with Conficker.  I had GPO, antivirus, and a 3rd party management product in my sights.  We started deploying a new VM that would be dropped into an OU with blocked inheritance to prevent anything from screwing with the clean OS.  Meanwhile, I returned to the already deployed (and new) VM and Google. 

Then I found this thread on MS TechNet Forums.  The user, tymque, had found that a hack to prevent Conficker had changed some permissions to the SVCHOST registry key and the WindowsTasks folder and this broke the management point installation.  I found the default permissions on MS Support (on a Conficker subject page).  I compared the default permissions with what was in place.  They were different!  I made the required changes manually and then the management point installation (manually running mp.msi) worked.  To be safe, I ended up doing a clean reinstall of the entire site server … and got an all green as expected.

I never did find out what hacked those permissions: a bit of time pressure on this project.

Using RSA Security Tokens for VPN, etc?

Then get them replaced now.  RSA were hacked and lost control over their master keys.  This has led to hacks against RSA customers – confirmed by RSA in an open letter to their customers.

I’ve never been keen on the concept of RSA tokens.  Now we learn that they stored the master keys live on the network with a route to the net by the looks of it!!!! Even the most basic certification training course on CA admin will teach you to use an offline root CA.

Technorati Tags:

How HM Treasury Was Allegedly Attacked & How to Defend Against It

I was listening to The Guardian’s Tech Weekly podcast on the way into work this morning and they were discussing some of the recent announcements from the British government about the cyberwar research that the MoD/GCHQ is doing.  In the discussion they mentioned that there was a recent attempted attack on HM Treasury (department of finance), and that the attacks allegedly came in two forms:

  • Drive-by browsing: this is where a user innocently goes onto a legitimate website, but an outsourced advert uses a browser vulnerability to inject some software onto the user’s computer.
  • Malware attachments: Some piece of dodgy software is sent as a normal looking attachment in an email.  This file has some sort of built in attack, like a trojan downloader, and the PC becomes a bot (something the attacker can remotely control by commands that the downloader will pull down from a service or website running on the Internet).

I am not a security expert.  In fact, most of the self-proclaimed security experts that you meet are not security experts.  I have met real security experts.  They speak a whole other language that we IT Pros don’t understand.  I’ve also met “security experts” with their recently downloaded checklists who do more damage than good.  The good news is that there is lots that you can do to protect yourself from attacks such as the above.  The bad news is that there is no 100% perfect defence.  For example, antivirus scanners detect already known threats.  Someone has to get hit somewhere before a threat becomes known.  Let’s stay positive and see what could be done to protect against the above two attacks.

Defending Against Drive-By Browsing

Drive-by browsing has been around for some time.  I’ve attended presentations by Microsoft’s Roger Grimes (serious security dude), where he talked about the website of a certain conservative news broadcaster.  They sold advertising space on their website.  Other than the space, they had no control over content.  That was done by the online advertiser.  And they probably did more outsourcing or bidding.  Allegedly, browsing this website could cause you to become a victim of an attack that was built into one of these outsourced adverts.  You’d just browse the site and *BANG* your PC downloaded a trojan downloader.  In other words, it was 0wned.

The most basic defence against drive-by attacks is to keep your browser up to date with security fixes.  Don’t be a fanboy sheep: all browsers are vulnerable.  I remember listening to another podcast (TWiT Windows Weekly) a few months ago where they discussed how Safari took seconds to smash, and Chrome/IE lasted a bit longer but eventually gave in at some hack-athon.  Google and Microsoft are constantly releasing updates.  Google do it via new versions of Chrome.  Microsoft do it through security hotfixes.

If you run anything but the smallest business then you need to manage these updates.  This is one of IE’s strengths because it can be updated immediately (or after internal testing) via Windows Updates, WSUS, and System Center (Configuration Manager 2007 or System Center Essentials 2010).  There really is no excuse for a business not to be doing this, monitoring patch update levels, and remediating any deployment issues.

This adverts are effectively downloading a trojan installer.  A proxy malware scanner can help defend against this.  Forefront Threat Management Gateway (TMG) includes a Malware Inspection Filter, as do many other firewall and proxy products.  I’ve always like the ISA (now TMG) family because they are AD integrated, and I can reuse security groups and user accounts for rules and exceptions.

Malware Attachment

The problem with email is that is pretty open, and trusting.  If I know the name or IP address of your SMTP gateway then there’s nothing to stop me from creating a malformed email that appears to come from someone you know and trust, and attaching a piece of malware to do bad things to your PC (and then your network).

Last night I read about some executive of a large corporation who sent out a memo to all employees to instruct that they should confirm the source of all emails before opening them. That certainly is one way to prevent the opening of an attachment. I just wonder if this executive answered the 20,000+ phone calls from his employees when they called to confirm that he really sent that email. Let’s get real – people have jobs to do and they cannot spend 3/4 of the day calling people to see if so’n’so really sent an email. Why would we have email at all in that case?

Sure we can do a bit of user education.  I don’t need to open an attachment with a .EXE file extension.  I don’t need to read an email from the wife of some deposed king.  And I really don’t need pills for you-know-what Smile  Common sense education helps.  But as Steve Riley has said in presentations in the past: the vulnerability lies in the meat that sits between the chair and the keyboard.  If we cannot fix that. then maybe we need to wrap our email system in defences to counter those weaknesses.

Lets start with the mail server.  Stick some malware scanning on there, like Forefront for Exchange (or another solution).  That will protect the server against external threats.  I know you’ll interject here with another suggestion (and I’ll get there).  Think about how IT is changing.  Consumerisation of IT has caused users to bring all sorts of devices onto our networks.  Lord knows what they connect to when they are not on our network.  And those same devices will be used to connect to the company’s mail services.  You need to protect the company’s email (and reputation) against those internal threats.

Next up is the online malware scanning service, such as Forefront Online Protection for Exchange (FOPE) or others.  The company’s MX record points to this, all incoming email is scanned for spam and malware, and then sent on to the company’s SMTP gateway.  You’re in complete control – you can even integrate the management of Forefront for Exchange with FOPE via a free (I believe) management console (it also can manage Forefront for SharePoint).  Now you can filter out the incoming rubbish before it gets to the company’s expensive Internet connection, and you have a layered defence.

Third Party Update Catalog

We aren’t finished yet.  Antivirus scanners are not perfect, especially when it comes to custom written or brand new threats.  The more serious attacks out there are not done by script kiddies in a basement; they’re done by organised crime, your competitors, and state agencies.  They have the time and money to create new programs to leverage discovered vulnerabilities.  For example, it’s one thing to scan for Conficker, it’s another thing to fix the vulnerability that it attacks so you can prevent anyone else from attacking it.

So you can use Windows Update, WSUS, ConfigMgr, or SCE to patch Windows.  Great!  The attachment that was used in the allegedly attack on HM Treasury was allegedly based on an Adobe product.  How often do you see Adobe products looking to update themselves to fix some security issue?  It feels to me like it happens a few times a week.  I bet most of you, and your users, disable these annoying updates – and that’s what the attacker is betting on!  They can write a custom attack, build it into a PDF (or whatever), send it to a user in your organisation using a crafted email that appears innocent enough, it’ll sail through the scanners (because it is an unknown attack), the attachment is opened in a vulnerable reader, and *badda bing* the attacker now has control of a PC on your network.

*PANIC* This is where you uninstall Adobe Reader, Flash, etc, and use third party alternatives – not so fast, my friend! (Why do I keep quoting Lee Corso?).  Adobe products, like every other, has vulnerabilities.  If you think those other readers don’t then you’re fooling yourself.  If you’re a big enough target, then an attacker will figure out what third party reader you use via social engineering, and craft an attack for that.  With Adobe, you at least have a way to force updates for your users.

No, we cannot trust users to run Adobe updates by themselves, just like we cannot trust them to run Microsoft updates for themselves.  Adobe has created software update catalogues that we can use in System Center Configuration Manager (MSFT’s main way to adopt/control consumerisation of IT) and System Center Essentials.  This will allow you to centrally download, test, approve, and deploy updates to relevant machines in an automated, and scheduled manner, with deployment deadlines.  Now you can force those vulnerable PCs to update, and secure your network against those vulnerabilities.

Summary

With all this you get layered defences.  Is it 100% secure?  No.  Like I said, I’m honest enough to say that I’m not a security expert but I know that with the above systems, you could protect yourself against the same attack that allegedly targeted HM Treasury (based on the information that I heard this morning).  Combine this with protection for PCs, servers, SharePoint, Lync, and so on, and you’ll have a nice fortress.  You can’t rely on people to protect the castle, and that’s why you need an automated portcullis approach like this.  The responsibility then falls on you as the gatekeeper to ensure that the gate is built correctly.

Note: I don’t know why some people always assume that virtual machines (on any hypervisor) assume that security should be any different for them.  The virtualised workloads still need the same levels of protection that they physical alternative would.

Microsoft Exchange Online Services Unbundled

This post is as much for me as anyone else – the array of products from Microsoft is mind boggling.

Once upon a time, Microsoft launched Exchange Hosted Services.  Based on the name alone, it sounded like Microsoft were now hosting Exchange mailboxes, like some of their hosting customers had already been doing.  But no, it was in fact a service that provided:

  • Online filtering of spam and malware
  • Archiving
  • Limited mailbox DR
  • Web portal
  • Mail encryption

Then along came BPOS (the product naming team strikes again) and that included mailbox hosting.  At least that’s been renamed to Office 365 –> but I’m encountering a lot of people who thing that’s just an alternative to the Office you install on your PC!

You can subscribe to a bunch of online services for Exchange

Exchange Hosted Services is still sold as a bundle including those 3 products.

Technorati Tags: ,,

System Center Update Catalogs for Third Party Products

Ever notice how many problems are caused by drivers or firmware?  Ever notice how often Adobe releases a new version of Reader or Flash to solve a security issue, and how many legacy versions are running on your network – thus making your Windows Updates process pretty irrelevant?  Ever wish you had a way to centrally deploy fixes for those problems?

One of the nice things about System Center Configuration Manager and System Center Essentials is that up can potentially distribute updates for just about anything.  For example, SCE 2010 has a wizard for adding catalogs for Dell, HP and Adobe products.  That means their system updates become something that can be distributed via Windows Updates!

Note: You would not want to do this for Hyper-V hosts – remember to treat them like change controlled mainframes.  Use your ability to filter update approvals using groups to control which machines will receive these updates automatically via Windows Update.

You are not limited to catalogs from the above companies.  You can even create your own catalog using the System Center Updates Publisher.  And some companies like IBM provide catalogs that you can add using their provided URLs.