Avoiding Microsoft “Fast Fail” Updates Using SCCM 2012/R2 Automatic Deployment Rules

I know there’s a risk in telling you to delay deploying updates for 1 month. Some think that means switching to manual approval – and that is an oxymoron because manual approval rarely happens. No; I would rather see large enterprises use a model that automatically deploys updates after delaying them for 1 month, just as you can do with System Center 2012 (R2) Configuration Manager (SCCM).

I’m going to refer you to the excellent guides by SCCM MVP, Niall C. Brady. SCCM uses WSUS to download the Windows Catalog. When I configure SCCM I configure WSUS to automatically sync and to automatically supersede updates. That means if Microsoft releases a replacement update, the old version is automatically replaced. That’s important so keep that in mind when reading the rest of the solution.

I will configure automatic deployment rules (ADRs) for each product. The ADR will be set up as follows:

  • Software Available Time: Set this to something like 21 days. That means that SCCM will hold back any applicable update for 3 weeks. That gives Microsoft lots of time to fix an update and the replacement will supersede the dodgy update.
  • Installation Deadline: With this set to 7 days, we have 4 weeks before updates are pushed out … and that assuming that we haven’t applied maintenance windows to any collections (servers, VMs, call centre PCs, etc) that might further delay the deployment.

image

With the above configuration, the dodgy August updates would not have been deployed to PCs or servers on your network. Instead, a tested and fixed update will be released, SCCM will sit on it and automatically approve it at a later date.

BTW, I do a similar thing with Endpoint Protection updates by delaying approval for 4 hours with immediate deployment.

I don’t know of a method for accomplishing this in Windows Intune – I’d like to see it. The same goes for WSUS, but a commenter suggested using cmdlets from this site for WSUS to write a script; I’d rather see a clean solution from Microsoft similar to what we have in ConfigMgr but less granular.

Microsoft Withdraws Update Rollup 2 for System Center 2012 R2 DPM

Why do I tell people not to deploy update rollups until they are one month old? This is why. I have been told that the fix for DPM 2012 R2 UR2 is not a permanent fix, but more like a hack for a short term solution that you will have to repeat.

Microsoft has been forced to pull UR2 for DPM 2012 R2.

image

This is a complete mess. It appears that the problems we have been seeing with quality since System Center 2012 SP1 are continuing to cause trouble for customers.

Technorati Tags: ,

Reminder: Re-Download Configuration Manager & Endpoint Protection 2012 SP1

You may have noticed some issues being reported with System Center 2012 SP1 Configuration Manager deployments.  It turns out that there were also some issues with the Linux/iOS Endpoint Protection agents.  As a result, you might need to re-download your media.  Check out this blog post to get more information.

Technorati Tags: ,

Remember To Set Your Network Speed in ConfigMgr 2012 Multicast

I’m currently on a 2 week project on a customer site to install System Center 2012 Configuration Manager with a focus on OS deployment to bare metal and application installation.  It’s been fun dong my first production install of ConfigMgr 2012.  You can really only push it so far in a virtualised lab and quite a bit has changed since 2007 R3 – it’s kind of like moving from XP to Windows 8 Smile  The biggest challenge is finding where things have moved to.

Today we moved to physical machine testing, verifying the drivers were installed, and IDing/importing those that got missed out.  Interestingly, the recently released HP all-inclusive driver pack for PCs/laptops is missing quite a few drivers.  We’re finding them in the per-model archives with no issues, as we are for the Dell machines.

One of the nice finds today was that I’d forgot to turn on Multicast on the distribution point and set the network speed in Multicast.  By default it is 100 Mbps.  I switched that sucker up to 1 Gbps.  Two things happened:

  1. All deployments that were on-going broke as the DP was updated.  This wasn’t instant either, taking a couple of minutes.
  2. Damn, OS deployment became so much quicker afterwards, as one would expect.

One of the nasties was a 3G modem “driver” by Ericsson on one of the HP laptops.  I say “driver” because there isn’t the usual collection of files including a .sys and .inf.  Instead, it’s a setup.exe.  Extract that and you get more files and another setup.exe.  Crap!  Maybe it has a silent install.  Maybe if it does we can package it up, and distribute it to a collection based on the model name of the laptop in question.  I’m even wondering if we can make it a conditional step in the task sequence where the condition is based on a ZTIGather model discovery.   It’s the only 3G modem we’ve had like this in about 8 or so laptop models so it sucks that it stands out like a sore thumb.

Error 0x800705AA: Insufficient System Resources During SCCM OSD Task Sequence

I had an interesting week this past week, doing my first production installation of System Center 2012 Configuration Manager in a production environment, with the focus of the project being on operating system and software deployment.  On Friday I had an interesting issue start to flare up while testing on some VMs.  The task sequence was failing during the installation of the operating system image.

The key log to analyse during a task sequence execution is SMSTS.LOG which can be found in WindowsTempSMSTS on the X drive.  You can get access to this log by enabling the command prompt for diagnostics in your boot image (remember to redistribute to your distribution points) and pressing F8 while the boot image is running.  In here I found:

Error 0x800705AA: Insufficient system resources

Damn!  I had to think for a few moments about this one.  Then it hit me.  I develop my reference image using a VM (snapshot right before the sysprep so I can rollback [apply snapshot], tweak and recapture) and I test on VMs before moving onto driver testing on reference hardware.  How were the VMs configured?  Dynamic memory with 512 MB startup memory.  The boot image doesn’t appear to have integration components for DM so  the 512 MB never burst up to the potential maximum memory of 4096 MB.  The boot image requires a minimum of 512 MB.  I guess the boot image needed more RAM than the startup, couldn’t avail of the maximum amount, and failed the task sequence.

The quick fix: I bumped the startup memory to 1024 MB, tested, and everything’s sorted.

Application Catalog Is The Killer Feature In System Center 2012 Configuration Manager

I deliberately picked the Application Catalog as the focal point of my demo/presentation at the System Center 2012 launch events in Dublin and Belfast because it shows how System Center 2012 recognises that IT services must change to empower the user and embrace IT controlled/secured/audited automation.

The Past

SMS 2003 was the first “System Center” product that I worked with.  We wanted something that was more powerful than Group Policy for software deployment.  The company I was working for also just signed a Microsoft enterprise agreement and we needed a software auditing solution to live up to our requirements.  So I asked one of my team, who previously did consulting on SMS 2.0, to deploy it, and I learned the product from him.

The software deployment feature was powerful.  We’d import or create a package containing the files.  Maybe we’d have to teak or create a program to install/uninstall the package.  We’d distribute the files to distribution points/secondary sites.  And then we’d advertise the required program to a collection of machines.  We never targeted users because they could roam and needlessly drag expensive software, such as Visio or Project, around with them, driving up our licensing costs.

It was easy to push out standard software like Adobe Reader.  It would go out to all Windows XP (as it was at the time) machines.  But Visio or Project?  We basically had to wait on a request.  A user would call the helpdesk asking for Visio and then a low priority ticket was created.  That ticket could wait until the higher priority tickets were dealt with.  Our Helpdesk had a 4 hour SLA so maybe 4 hours later (usually much less) they’d drop the user’s computer account into a security group for machines that should get Visio. 

And here’s why I told people that you need patience with Configuration Manager.  The process has gone unchanged … it’s just now we have a different way to tackle it.  In the past we had to push that software.  ConfigMgr/SMS would update collection memberships on a schedule, every 24 hours by default.  We had a “small” network (by Microsoft or ConfigMgr standards) so we scheduled the collection to update every hour.  Then it would query the new group membership and update its own membership. 

On the client machine, the ConfigMgr/SMS client would automatically connect to the Management Point every hour to get new policy.  At that point it would, thanks to the new Visio collection membership, realise it should install Visio.  It would then download the files and install.

Think about how long this took:

  • Helpdesk to respond – up to 4 hours (let’s go worst case scenario) – 4 hours
  • The collection to update – we’ll say 1 hour but it could have been 24 hours – 1 hour
  • The client to connect to the management point – up to 1 hour but we’ll say 1 hour

That’s a 6 hour wait for the end user to get a new application.  No wonder the business thinks that IT holds them back!  They can avail of cloud computing or a personal device (app on a tablet) in minutes, to deal with whatever business opportunity/challenge/threat is before them.  But with our push solution, IT takes 6 hours … and that could have easily been 29 hours!  That’s some “service”.

The Present

System Center 2012 is user centric.  That means the user is empowered to consume IT services in an on demand basis.  Those services are provided via System Center 2012, allowing IT to automate more, enable the user to consume as and when they need it, but IT can control, secure, and audit it.

Let’s take the Visio example.  I can create a Visio package with the automated installation.  I then create an application in System Center 2012 Configuration Manager.  I can two 2 types of deployment.  The first is a push, which is similar to what I discussed above.  That’s for when you’ want to push out software by policy.  And being a policy, the software will automatically get re-installed if it is uninstalled while the policy still applies.  There is a delay in the push, but we don’t mind.  That’s because we’re pushing out a policy to a large number of machines, and that’s probably something we do outside normal hours, and not to some “we want it now” demand.  Adobe Reader, Office, and so on are the sorts of app that you would deploy like this.

The second approach we can use is to publish the application in the Application Catalog.  Here you can list all elective software, the stuff you don’t include in your OS images or deploy on a widespread basis via policy.  Visio is a perfect example of this kind of app; it’s too expensive to deploy everywhere, and a few people will have a business case to require it.  When you create the application, you can add all sorts of text and keywords to describe the app and to make it searchable.

You can publish the URL to the Application Catalog to everyone’s browser via GPO.  And there’s a link to it in the new utility on the managed PC called Software Center.  Now a user wants Visio to open a VSD file.  The click the link to open the Application Catalog.  They can search, e.g. for .VSD file, and Visio appears in the results.  The click the Install button, and Visio installs … just like that.  It’s actually ConfigMgr doing the install, using the unattended config that you set up in the package.

Now Visio is expensive, so you don’t want everyone lashing it onto their PCs.  Not a problem!  With a mouse click, you configure the installation to require approval.  Instead of an Install button, the user is given a Request button.  They are asked to give a reason for the install and the request goes off into ConfigMgr where an administrator can review it and approve/reject it.  If it’s approved, the user will get an Install button.

The Future

We’d like that request process to be more auditable and to include non-IT staff, such as a faculty or department IT budget owner.  That’s where the Application Approval Workflow (AAW) comes in.  This combines the deployment functionality of Configuration Manager with the process and control functionality of System Center 2012 Service Manager.  Now the user can go into either the ConfigMgr Application Catalog or the portal of Service Manager, where they’d normally go to request IT services.  Requesting an approval-required application will create a service ticket in Service Manager and kick off an approval workflow. 

The engineering possibilities of workflow allow you to bring in alternative approvers based on your business or customer processes.  In other words, a budget owner can be notified of the request, read the business case, and reject/approve the install of the application.  And now IT just manages the system, instead of slowing down the business.  If there is slowness with this solution, the business can only look inwards to find a cause.

Configuration Manager 2012 Error, Past Due – Will Be Retired

I just had a bit of a head scratcher while building my ConfigMgr 2012 lab.  I had created an application to deploy Lync 2010 by policy to a collection of devices.  The “mandatory assignment” (this is old terminology for legacy packages/advertisements) was to install the Lync 2010 client as soon as possible.

I refreshed policy on my test machine and got this error in Software Center:

Past Due – Will Be Retired

Huh?!?!  I didn’t set an expiration on the deployment.  I could not figure this out.  The AppEnforce log in C:WindowsCCMLogs held the clue to this mysterious error:

Command Line: setup.exe /install /silent

The installer is called LyncSetup.exe, not Setup.exe.  I corrected the Deployment Type in my application for Lync 2010 and reran machine policy on the client machine.  The install now worked.  Then the real test: I manually uninstalled Lync, and ran the Application Deployment Evaluation Cycle on the client.  The reinstall (by policy) worked perfectly.

Managing the iPad In The Enterprise

All that kerfuffle last year about Microsoft being late to market appears to have been valid.  iPads are turning up in the business.  And I don’t mean the MD bringing one in, or one hear and another there.  I mean BIG numbers of them are turning up.  A well publicised example is SAP where they’ve deployed 12,000 iPads.  An interesting comment in the story is that the iPad is encouraging people to explore data and information, and probably empowering them to make better decisions.  A touch UI is more natural; maybe that’s part of it.  And tablets are small and light, meaning a person is more likely to bring it to and use it at coffee or lunch or home.

The interesting thing is that people aren’t talking about the entry of the iPad into this sort of market.  People Let’s face it; we’ve been expecting this. 

The conversation isn’t “Oh Microsoft are screwed and this is the death of the PC”.  We’re still early days in the “tablet at work” era, and if Microsoft don’t screw it up, Windows 8 with Office wave 15 could be a very powerful combination because of their possible integration with the normal PC and the LOB app.  I personally think 2013 will be an exciting time to be a .NET business applications architect.

But back on topic … what are people talking about?  Management.  How in the hell are businesses managing and securing these devices?  A recent survey said “Among 520 CIOs polled, 77% said they worry that further consumerization of IT will lead to greatly increased business risks”.

Right now, if you’re using iPad then you’re either trusting employees (I’m a techie meglomaniac with mixed a little [a lot] Roy from the IT Crowd so that doesn’t work for me) or they are using point solutions.

The point solutions will fall into one of two groups.  A Blackberry house, for example, will probably use a dedicated tool for controlling and configuring their RIM devices.  But along comes an iPhone or an iPad and they suddenly need another dedicated management system or something more generic. 

I think the best solution right now is to adopt a more generic mobile device management solution.  In a true consumerisation adoption, you have no idea what’s going to come in the door: Android, RIM, Apple, Microsoft, etc.  For the IT guys, the challenge is that each platform is completely different, so they’ll have to learn the strengths and weaknesses, develop a common denominator policy (PIN codes, remote wipe, etc), and then figure out how to secure each specific platform according to its unique needs.

But think about this.  That’s another management system for IT to deploy and look after.  What if you could have 1 integrated system that can manage PCs and mobile devices, configure and secure them.  We don’t have an RTM yet, but it’s coming: Configuration Manager 2012 from Microsoft System Center has mobile device management.  Information is still light on the ground on this feature, I guess all will be revealed when the products are launched.

Technorati Tags: ,

Deploy Office 2010 via ConfigMgr 2007

Yesterday I wrapped up the deployment and proof-of-concept of deploying Office 2010 with SP1 via System Center Configuration Manager 2007 R3.  It was a nice one: branch distribution points, client deployment in a mature XP network, etc.

Here’s a rough idea of what I did:

  • Install a site server in the central site.  Local SQL installation to make backup/recovery more manageable via the ConfigMgr backup task.  Boundaries were defined (the IP subnets in the ConfigMgr site).  Enable auto discovery from AD every hour.  Small network (by ConfigMgr standards) and it’s good to get changes frequently if using groups for collections.
  • Deployed branch distribution point in the local site.  I set the sample one up as a protected BDP.  This associates the subnets of the branch office with the BDP, restricting access to clients in that site.
  • Deployed some ConfigMgr clients to test machines by hand.  I did not enable client push installation (proof of concept).
  • Packaged Office 2010 using setup /admin.  Note I used SETUP_REBOOT in the setup properties (Office Customization Tool) and set it to Never.  This prevents Office 2010 setup from rebooting the machine if previous versions of Office are running during setup.  If this situation occurs, Office 2010 setup would reboot the PC with no notice to the user – bad!  Instead, I’ configured the package program to let ConfigMgr reboot the PC (no matter what – probably not a bad thing anyway).
  • Slipstreamed Office 2010 Service Pack 1 into the package.
  • Distributed the package to the Site Server’s distribution point and to the BDP.  Force the BDP to download the package by running the BDP maintenance task in the BDP server’s Configuration Manager client (Control Panel).
  • Setup up a proof of concept collection. 
  • Advertised the package setup program to the collection.  Forced policy refresh on the test machines by running the machine policy refresh in the ConfigMgr client (Control Panel).
  • Sat back and watched the goodness.

For production deployment:

  • We wanted to restrict client deployment impact on the network.  I copied the client setup files into SYSVOL and created a .bat script to run CCMSETUP with the flag to define the site name.  That would copy the ConfigMgr client setup files to DCs in every site.  I setup a GPO to run a startup script that would execute this .bat file.  That GPO could be linked to appropriate objects in AD to force setup of the client on machines.  They’d install from the local SYSVOL and eliminate any WAN impact.  Eventually, the GPO can be removed/unlinked, and client push installation can be enabled, thus hitting those last few machines that haven’t rebooted (to get the startup script to run) or any new machines that are added to the domain.  I also find that this scripted solution tends to get me better results in a mature XP network.
  • Office 2010 is to be deployed 1 site at a time.  The AD sites/OUs don’t match the physical sites (not all that unusual) so I setup a collection definition where: (system role = workstation AND (network configuration IP address = 192.168.1.% OR network configuration IP address = 192.168.2.%).  This will include all XP (or later) PCs on the site’s subnets in the collection, and exclude server machines.

From there, a new advertisement can be created to run the Office 2010 SP1 install at a pre-scheduled time.  ConfigMgr reports can be monitored to see which exceptions (problems) need to be dealt with.  The clients in the site will install from the local BDP.

For following sites, one at a time:

  • Add the branch office subnets to the ConfigMgr site boundaries.
  • Install a BDP and protect it with the site’s subnets from the boundaries list.
  • Distribute the Office 2010 package to the BDP.
  • Create a new collection specifying the subnets with the % wildcard.
  • Advertise the Office 2010 package program.

For something like this, you need to test, test, test.  You cannot test enough.  Sounds like a lot of work, but your up front time investment saves a bunch of time and money on the back end, versus a manual install to hundreds or thousands of PCs.  This works out being not so bad if you license intelligently too: ConfigMgr + SQL combined with a (desktop) Core CAL Suite (includes a bunch of CALs and a ConfigMgr management license).  And after that, you have a fine solution in ConfigMgr to manage the entire life cycle of the PCs you manage:

  • Zero touch OS image deployment
  • Software deployment
  • Patching (MSFT and third party)
  • Desired configuration management (2012 adds auto rectify)
  • Software/hardware auditing
  • License auditing/usage measurement
  • Power monitoring/policy enforcement (saving money!)
  • 2012 also adds “user centric computing” and Android/iOS device management
  • Reporting on more than you could dream of … all the way to identifying those machines that you need to replace.
  • And Dell/HP are fully invested in it as a solution, recognising the power it adds for their customers.

Jeez, I’ve totally gone over to the dark side of sales Smile Despite that, I love ConfigMgr; it allows me to play out my megalomania fantasies, even if they are limited to absolutely everything in the AD forest that I can get a ConfigMgr client onto.

Technorati Tags: ,,

ConfigMgr 2007 Management Point Won’t Install – Failed to Create the CCM_Incoming Virtual Directory

I’ve been working on a customer site for the last few days in my old stomping ground: System Center Configuration Manager (SCCM) 2007.  It’s a new deployment in a mature Windows XP network.  Today started out as a nightmare.  I had all the prereqs done but the install of the primary site server was not going well.  The management point just would not install.  The SMS_MP_CONTROL_MANAGER was reporting that:

“MP Control Manager detected MPsetup has failed to create the CCM_Incoming Virtual Directory.

Possible cause: The IIS IWAM account has expired, been disabled, or has invalid or too restrictive logon hours. You may verify this information by running the net user command line for the IWAM account. (i.e.: "net user IWAMMachineName)

Solution: Use the output to verify that the account is enabled, and logon is possible during the time of installation. Note: You can use "net user" to modify the account properties.
Possible cause: The IIS IUSR account has expired, been disabled, or has invalid or too restrictive logon hours. You may verify this information by running the net user command line for the IUSR account. (i.e.: "net user IWAMMachineName)

Solution: Use the output to verify that the account is enabled, and logon is possible during the time of installation. Note: You can use "net user" to modify the account properties.
Possible cause: The designated Web Site is disabled in IIS.

Solution: Verify that the designated Web Site is enabled, and functioning properly”.

I knew that all IIS components were installed and configured correctly: I use my Zero Touch chapter from Mastering Windows 7 Deployment as my ConfigMgr prereqs check list!  Using that, I can normally get an all green install.  But something here was wrong.  I suspected a security issue … who knows what’ll impact you in a mature network.  I googled and a number of people reported corrupt IIS metabases caused issues.  A reinstall of IIS and ConfigMgr ensued but no result.

Now I was sure an external factor was at fault.  I’d heard that some security feature had screwed up the XP machines in the past.  Something to do with Conficker.  I had GPO, antivirus, and a 3rd party management product in my sights.  We started deploying a new VM that would be dropped into an OU with blocked inheritance to prevent anything from screwing with the clean OS.  Meanwhile, I returned to the already deployed (and new) VM and Google. 

Then I found this thread on MS TechNet Forums.  The user, tymque, had found that a hack to prevent Conficker had changed some permissions to the SVCHOST registry key and the WindowsTasks folder and this broke the management point installation.  I found the default permissions on MS Support (on a Conficker subject page).  I compared the default permissions with what was in place.  They were different!  I made the required changes manually and then the management point installation (manually running mp.msi) worked.  To be safe, I ended up doing a clean reinstall of the entire site server … and got an all green as expected.

I never did find out what hacked those permissions: a bit of time pressure on this project.