Month: February 2010

System Center Capacity Planner End of Life

I just read a blog post by Microsoft staff that said System Center Capacity Planner is end of life and will have no new developments.  That’s a pity.  It’s a concept that was a really good idea, and could have extended to all products in the planning stages.

The idea was that Microsoft would model how their different products would work.  You would then describe your network and it would help you design a solution.  A simulation could be run to see how your hardware would handle the work loads.  Unfortunately it suffered from a few problems.  The product set never really grew beyond Exchange, SharePoint and Operations Manager.  The newer versions weren’t added.  And the available hardware models weren’t kept up to date.  Maybe if the model was open source it could have helped but it really needed Microsoft’s assistance.

On the Hyper-V front, I would have loved to see it integrate with data generated by MAP and OpsMgr/VMM to help you design a Hyper-V farm managed by VMM.  But that never came to be.

In the end, the results were just a recommendation and not to be 100% relied upon.

The post describes alternative sources for planning OpsMgr, SharePoint and Exchange.

Technorati Tags:

Unicast Mode Network Load Balancing on Hyper-V

This Microsoft blog post discusses a problem you might encounter when you enable NLB in unicast mode on Hyper-V virtual machines.  Clients may not be able to access the virtual IP address of the NLB cluster or the VM’s themselves.  This is because MAC spoofing is disabled by default on W2008 R2.  The blog post shows you what change to make to each VM in the NLB cluster to resolve the issue.

I Hate Flying … To The USA, Irrational Databases, and Air Conditioning

Before I get started, let me just say I like visting the USA, wandering about the countryside, the national/state parks, and enjoy the company of the people there.

I am sat in Dublin airport right now.  I am on my way to Seattle via Chicago.  My Dublin – Chicago flight will take around 7 hours.  Correction … let’s count the amount of time that the airline wants me to spend in the airport.  My Dublin-Chicago flight will take 10.5 hours.  One third of the 1600 mile flight will be spent in an over heated excuse for an over priced mini shopping centre where I cannot go outside for some fresh air.

But all airports are like that.  Stuffy, dry, cramped, oh … there’s a vending machine that sells cool drinks for twice their normal price, and would you like an hour of wifi for $7?  Security queues are too long because only a small fraction of the scanning machines are staffed.  Restaurants are full of dried, heated food that tastes worse than their airplane food and seems like it’s less of a bargain.  And there’s the people … *£$^£&£”% … I cannot stand the idiots who seem to pack these places.  I long for the days when Aer Fungus had a reasonable card scheme and I could hide away in a lounge.

But the experience to travelling to the USA usually starts long before you get into your car, taxi, bus or train.  It starts once you book your plane ticket.

Let me first say I have no problem filling in forms for a visa waiver or customs declarations.  I’m totally OK with that.  I cannot stand wasted effort.

The first thing you need to do when travelling to the States is ESTA.  Basically, that’s an application to fill in an application.  You know, it’s like a meeting about a meeting.  ESTA is an online form where you answer the questions that are on the paper visa waiver form.  It must be completed in order to submit a paper visa waiver form.  Both have identical questions.  And of course, the answers must be identical.  I thought that was pretty mad.  How about giving people the chance to do one or the other.

I once had an argument with someone about ESTA.  The other person said it was necessary to gather information about the passenger.  Fine … but there is nothing different between this and the green visa waiver card.  The next response was that it’s the 21st century and we should go digital.  Fine, but why do we also need to do the green paper form that is identical?  It was an unwinnable argument for the opposing view.  It is there just because it is there.  You know, there must be a massive amount of data gathered for ESTA every year.  I bet that requires a lot of SAN storage, replication and backup.  Someone like HP or EMC is making a stack of cash on that.

But it gets worse.

When I booked my flight I soon found out (I forget how) that I needed to complete another online form.  I think it was called the advanced passenger information form.  I logged in and was greeted by the same questions I had just answered for ESTA.  The USA Homeland Security department needed a third copy of this data.  This seems like a crude form of replication.  Can’t they order SAN controller replication instead, or maybe do it at the database level?  I’m told SQL is quite good at this.

Time goes by and 3 days before my flight, Aer Fungus sends me a reminder email of my flight.  They also remind me about the need for ESTA, the advanced passenger information form, and the passenger address information form.  The what?  Here we go again.  No wait; no one is that stupid to ask for the same set of information a fourth time.  I log in and there are the same questions again: name, passport number, issue date, expiration date, address in the USA, etc.  There is also a warning that the answers I give on this form must match those on the other forms or I will be declined access to the USA.  It’s incredible.

I am no DBA but I did do database classes in college back in the early 90’s.  Most of the lectures we got were cogged from books written in the 1970’s and 1980’s – they were in the library, fading and falling apart for any one to use.  We learned about data storage optimization for relational databases called normalisation.  1st normalised form, 2nd, 3rd, all the way to Boyce-Codd, and even on beyond that where it stopped making sense to me.  It seems that who ever is consulting for the USA department of Homeland Security uses irrational databases instead where data needs to be inputted 4 times: green waiver form, ESTA, advanced passenger information and passenger address information.  It’s pure stupid and the only people gaining anything are the storage companies who must be laughing all the way to the bailed out banks. 

I just got it!  This must be how the Fed props up the hardware companies during the recession?  Wasted  and duplicated petabytes of data that will never be used.  The data isn’t being used for security reasons.  We know that.  A known terrorist suspect managed to get on a plane to the USA and tried to set fire to his explosive crotch … damn that sounds like the plot to a bad Jean-Claude Van Dam movie.

I’ve got some advice for them.  In the northwest USA, there will be 1,300 of some of the world’s best IT people around next week.  Conveniently, they’ll be in Microsoft’s Redmond campus.  That’ll include software developers, security experts and DBA’s.  Send someone on up and ask a few questions.  You might learn how to gather the information you need with one form and store it just once.  You’ll accomplish a few things with that.  You’ll save millions every year and have a thankful American citizen/tax payer.  You’ll also stop alienating the legal visitors to your country who are bringing money to spend on your economy.  The experience of flying is bad enough without making it feel like a tax audit from hell that you might find in a book like 1985.

Microsoft’s Initial Response To MS10-015 / KB977165 Blue Screen of Death

Microsoft’s security operation have issued an initial response to the issue with machines blue screening and failing to reboot correctly after installing MS10-015.

While we work to address this issue, customers who choose not to install the update can implement the workaround outlined in the bulletin. CVE-2010-0232 was publicly disclosed and we previously issued Security Advisory 979682 in response. Customers can disable the NTVDM subsystem as a workaround and we have provided an automated method of doing that with a Microsoft Fix It that you can find here.

Customers who are experiencing issues after installing any of our security updates can get help resolving the issues by either going to this site or by calling 1-866-PCSafety (1-866-727-2338). International customers can find local support contact numbers here.

Technorati Tags: ,

Download OpsMgr 2007 R2 Documentation

You can now download Microsoft’s TechNet documentation for System Center Operations Manager 2007 R2.

The Operations Manager 2007 R2 technical documentation helps you plan, deploy, operate, and maintain Operations Manager 2007 R2. For information about the specific guides available in the library.

Hardware Monitoring Using System Center Operations Manager

Hardware management is the one thing I am most worried about.  Sure, I could deploy the manufacturers management solution.  But do I want consoles to manage lots of different systems?  Really, you don’t.  You want one central point and that can be the Operations Manager console.

I’m most familiar with what HP does so I’ll explain it.  They provide and Insight Manager agent that detects health and performance issues of the hardware.  This includes all of the components, e.g. CPU, fans, disks, network cards, etc.  You can deploy and OpsMgr agent to this server.  If you install the HP Insight Manager management pack then, after discovery, OpsMgr will be aware of the Insight Manager agent.  All data collected by that agent will be detected by OpsMgr.  So now, if a disk fails you learn about it in OpsMgr.  If memory degrades, you learn about it in OpsMgr.  This is so handy – because this is where you also get performance and health alerts for Windows, SQL, Exchange, Red Hat Enterprise Linux, SUSE Linux Enterprise Server, etc.  You can extend with 3rd party solutions to include your Cisco network, etc.  Heck, there’s even a coffee pot management pack!!!

Back in the day, there appeared to be only support from HP and Dell.  But that has changed.

  • HP: Hewlett Packard has management packs for ProLiant servers, BladeSystem, and Integrity.  There is also a management pack for StorageWorks systems (e.g. EVA SAN).
  • Dell: I’ve never managed Dell machines with OpsMgr.  But I am told that Dell did a very nice job.  They are significant Microsoft partners.
  • IBM: I’m not the biggest fan of IBM – we have some X series stuff which I detest.  We had to get a IBM employee to download the management pack because all external links failed.  At the time, it appeared their “shared” download was only available from the IBM corporate network. A Dutch friend had the same issue and I ended up sending him what I was given by IBM.  I’ll be honest, the IBM Director management pack is poor compared to the HP one.  IBM wants you to spend lots of money on consultancy led Tivoli.  IBM Director is pretty poor too.  IBM Ireland employees have been unable to figure out how to monitor IBM DAS nor give me the documentation to do it.
  • Fujitsu: I have not seen a Fujitsu server since 2005.  Back then there was no MOM management pack for the Fujitsu Siemens servers; they wanted you to use a native solution only.  That has changed.  They have ServerView Integration for Microsoft System Center Operations Manager 2007 and System Center Essentials 2007 and ServerView Integration Pack for Microsoft System Center Operations Manager 2007.

That should get you started.  Each of the manufacturers seems to do things differently.  HP, for example, uses the above system for ProLaints.  But blade enclosures require a piece of middleware.  Make sure you read the accompanying documentation from the OEM before you do anything.

Thanks to fellow MVP Mark Wilson for finding the links for Fujitsu. 

Technorati Tags: ,

Burn The Witch! Hyper-V Security Fix And Hyperventilating

Ah, it takes a patch to find out who’s really thinking what :)  As you are now aware, Hyper-V had it’s very first (ever) security patch this week.  Not bad (typical Irish understatement) after a year and a half of being the most accessible hypervisor ever.  Just think of how many volume license, OEM, TechNet, MSDN, evaluation and pirated copies of Windows Server 2008 and Windows Server 2008 R2 must be out in the world, not to mention the free to download Hyper-V server, and that it can run on most hardware around in the last few years.  I’m betting people in parent’s basements were attempting to find vulnerabilities since the emergence of the first beta for Hyper-V, around 2 years ago.

And after all that time and opportunity, 1 security hole was found.  It isn’t even the dreaded “break out” where a VM is capable of reaching out and accessing the host and other VM’s.  No, it was a DOS attack where the hypervisor would shut down.  And you had to be logged into a VM on the host with admin rights!

I’ve noticed a lot of tweets in the last 48 hours of people writing with glee about a dreaded problem, implying that Hyper-V is inferior.  Oh, get over it!  I can think of another hypervisor from a certain company that has suffered from a break out attack.  Its patches are a complete OS upgrade and they break the host on a way too frequent basis.  So much so, in fact, that experts in that technology run 1 “service pack” behind the latest release to stay safe.

It’s a secure platform.  Think of all those attackers who hate Microsoft and have the chance to attack the most available hypervisor around and we get 1 patch in 2 years (since beta).  That’s unbelievable.  The basic architecture requirements (DEP) prevent buffer overrun attacks on the host from a VM.  The German government has certified it as being secure … trust me if you are unfamiliar with working in Germany … that doesn’t happen by accident.  Every piece of complex software has vulnerabilities and bugs.  If you didn’t learn that in programming classes in college then you need to ask for a refund.  The fact is that Hyper-V is so well designed and implemented that it’s taken quite some time for one to be found.  And Microsoft reacted perfectly about it.

So before you go running to the woods to get some kindling for the witch burning, sit back, breath into a brown paper bag and realise that this is not the end of the world for Microsoft virtualisation.  It’s actually not bad at all.  It was one small patch that was quick and easy to download and installed reliably. 

Give Me Your Microsoft Virtualisation Feedback For Microsoft

Tomorrow (Feb 13th) I make my way to Bellevue in Washington State for the annual Microsoft Valuable Professional (MVP) Summit.  The event is four days where MVP’s get to interact with and give feedback to the product groups in Redmond.  Microsoft goes to great expense for this event, both in terms of money and time.  It’s their chance to get feedback on new stuff from the MVP’s and to get feedback from the communities we work in.

As I’m a “Virtual Machine” MVP, about 50% of my time will be with the folks behind Microsoft’s machine virtualisation technologies, e.g. Hyper-V, VMM, and Virtual PC.  I believe they have a good idea of what people are looking for in the future.  Folks like Mike Briggs, Ben Armstrong, Mike Sterling, Edwin Yuen, etc are all quite visible and are great netizens.  Microsoft Connect is also a good tool for gathering suggestions from the public.  But it won’t do any harm to hear from anyone out there who has additional feedback.  So fire ahead, post any comments you want to make and I’ll do my best to relay.

As the 4 day event is 100% under NDA, I will not be tweeting, talking, blogging or anything about the content of those 4 days.  Everything will stay under wraps if or until MS decides to make things public.

Technorati Tags: ,,

MS10-015 / KB977165 Causing BSOD For Some – How To Deal With The Issue

I came home tonight to see reports of blue screens of death and failed boot up/reboots for XP machines that had installed the MS10-015 security patch or hotfix.  There is a long thread on Microsoft Answers.

After installing the patch you have to reboot.  You are then greeted with:

PAGE_FAULT_IN_NONPAGED_AREA

Technical Information:
STOP: 0x00000050 (0x80097004, 0x00000001, 0x80515103, 0x00000000).

The posted fix by Microsoft in the thread is:

  1. Boot from your Windows XP CD or DVD and start the recovery console (see this Microsoft article for help with this step).  Once you are in the Repair Screen..
  2. Type this command: CHDIR $NtUninstallKB977165$spuninst       
  3. Type this command: BATCH spuninst.txt
  4. When complete, type this command: exit

There is what appears to be some misinformation or hysteria about this.  For example:

  • Some news articles are claiming that Windows 2003 and Vista are reported in this thread as being affected.  I saw no mention of those operating systems.
  • I saw one article (a random find) that tried to make it look like that this affected Hyper-V.  Pah!  It does not from everything I have read.  There are no reports of issues with Windows Server 2008 or Windows Server 2008 R2.  Put the Kool-Aid down and step away from the cup.

I cannot claim there are not problems there but they are not in that thread.

EDIT: Overnight after this blog post was originally written, some people did post about Vista and W2003 suffering issues with blue screens caused by the update.

It is bad that a patch has affected many.  I’m sure MS will be making someone feel very uncomfortable overnight about this.  It’s bad that it happened at all.  But let’s face it.  Not everyone is affected.  There is some combination in factors that is contributing to the blue screen.  There is some scenario that MS didn’t test or couldn’t predict.  These things happen.  It could be some niche piece of software or driver that reacts badly to the patch.

EDIT: I’ve read on one site that some people are finding an issue with the ATAPI.SYS file not looking like the genuine file supplied by MS.  They suspect an old malware issue causes an incompatibility with the fix!!!

This situation (whatever the actual cause of the blue screens) is why I think people like Steve Reilly who preach that we should all push out security updates immediately and without question are wrong for me (maybe not wrong for you).  How many zero day exploits have there ever been?  Not many.  Think of the big bad attacks … Nimda, SQL Slammer, MS Blaster, Conficker.  They all attacked vulnerabilities that were fixed with patches long before hand.  What’s a couple of weeks?  It’s because of the rare occasion when a patch goes wrong that I run a 3 phase process for patches.

I have three groups in WSUS.  I configure my Windows Update agents either via group policy (AD members) or registry edits (.REG files for workgroup members) to be members of 1 of 3 groups:

  1. Testing – contains VM’s with various blends of OS and application
  2. Management – Our production AD, management systems, and online applications
  3. Hosting– Hosted customer servers

We’re a hosting company.  WSUS has an automatic approval policy for the Testing group.  The machines in that group are VM’s on my Hyper-V lab server.  They patch in the late morning/early afternoon (around lunch) so we can see how they reacted.

Ideally that group would contain samples of the various bits of hardware you have on the network to include drivers in the mix.  I was lucky enough to be able to do that with one employed in the past – but we did push out updates in less than a week from release.  However, I need to be cost conscious and that is not an option now.

When we’re happy we sit and watch the news.  If all is well, change control happens, and then we approve the updates for the management network.  Stealing a line from Microsoft, we eat our own dog food.  Over the 3 nights of the following weekend (Friday, Saturday, Sunday), machines are patched and reboot automatically.  Some services are clustered/replicated and we do them on different nights or time slots.  We have scheduled scripts on the OpsMgr RMS to put machines into maintenance mode.

Now we watch how that went and continue to watch the news wires.  If there’s no more problems then we approve the updates for the hosting customers after another change control process.  Patches then deploy according to their pre-agreed time windows.

The end result is that within 2-3 weeks all security updates are deployed.  You could compress this down to a week.  We are totally minimizing the risk of being stung by a “bad” update.  Like I said earlier, MS probably did test the update as far as is realistically possible.  There is always the chance that something bad happens.

Steve Reilly’s argument was that if you get a bad update then you call easily rollback your server farm because it’s probably 90% virtual.  In my opinion you shouldn’t really use snapshots in production on Hyper-V.  They’re supported but they suck the life from your VM’s.  DPM or 3rd party solutions that are using the Hyper-V VSS writer are cool for this.  But really, do you want to risk your production network going down for hours while you recover (starting at 3am when your patch failed) because of the rush to deploy an update that will likely not have an attack vector for quite some time?

Weigh the various risks and make an informed decision for yourself.  Maybe Steve Reilly’s approach to push out updates without testing is right for you.  Maybe my phased and cautious approach is.   Maybe there is a middle ground that you prefer.  Do the research and be sure you know why you make your decision and that it is based on fact.

EDIT:

There is strong suspicion that the BSOD’s are actually happening on machines that were already infected by a rootkit called TDSS.  It attacks ATAPI.SYS and replacing that file appears to fix the BSOD issue as well.  Microsoft Security Essentials appears to be able to detect it.

Technorati Tags: ,,,

The Exchange Server 2010 Setup On Hyper-V Fails With 2147504141 Error

Thanks to Dutch Exchange MVP, Jetze Mellema, for raising this one.  Jetze is doing a lot of Exchange 2010 work over in the Netherlands.  He tweeted today about something he’d seen.

When you try to set up Microsoft Exchange Server 2010 on a Hyper-V Virtual Machine, the setup process may fail. Additionally, you receive the following error message:

"An error occurred with error code ‘2147504141’ and message ‘The property cannot be found in the cache.’"

KB980050 discusses the solution:

To resolve this problem, disable time synchronization in the Hyper-V Manager console. To do this, follow these steps:

  1. Open the Hyper-V Manager console.
  2. Locate and Right-click the virtual machine on which you want to install Exchange Server 2010, and then click Settings.
  3. Click the Management section in the Settings tab, and then click Integration Services.
  4. Click to clear the Time synchronization check box, and then click OK.
  5. Install Exchange Server 2010 on the virtual machine.
Technorati Tags: ,