Microsoft News Summary-23 May 2014

1,000,000 IOPS from Hyper-V VMs using a SOFS? Talk about nerd-vana!!! Here are the links I found interesting over the last 48 hours:

KB976424–Important Update For W2008 Or W2008 R2 DCs If You Have WS2012 Clusters

Microsoft has published an elective hotfix that they want you to know about if you haveWindows Server 2008 or Windows Server 2008 R2 domain controllers and you are running Windows Server 2012 clusters.

Symptoms

You perform an authoritative restore on the krbtgt account in a Windows Server 2008-based or in a Windows Server 2008 R2-based domain. After you perform this operation, the kpasswd protocol fails and generates a KDC_ERROR_S_PRINCIPAL_UNKNOWN error code. Additionally, you may be unable to set the password of a user by using the kpasswd protocol. Also, this issue blocks kpasswd protocol interoperability between the domain and a Massachusetts Institute of Technology (MIT) realm. For example, you cannot set the user password by using the Microsoft Identity Lifecycle Manager during user provisioning.

Note The krbtgt account is used for Kerberos authentication. The account cannot be used to log on to a domain.

You may experience additional symptoms in a Windows Server 2012-based server cluster. Assume that you try to set the password for the cluster computer object in a Windows Server 2012-based server cluster. Additionally, assume that there are Windows Server 2008-based or Windows Server 2008 R2-based domain controllers in the environment. In this situation, you receive the following error message:

CreateClusterNameCOIfNotExists (6783): Unable to set password on <ClusterName$>

To resolve this issue, apply this hotfix on the Windows Server 2008-based or Windows Server 2008 R2-based domain controllers, and then create the Windows Server 2012-based server cluster.

Note You do not need to apply this hotfix if you have Windows Server 2008 R2 Service Pack 1 installed.

Cause

When a user requests a ticket for the Kpasswd service, a flag is incorrectly set in the Kerberos ticket-granting service (TGS) request for the Kpasswd service. This behavior causes the Key Distribution Center (KDC) to incorrectly build a new service name. Therefore, an incorrect service name is used, and the KPasswd service fails.

Note The expected behavior is that the Key Distribution Center (KDC) directly copies the correct service name from the Kerberos ticket-granting tickets (TGTs).

A supported hotfix is available from Microsoft.

My TechCamp 2013 Presentation – Windows Server 2012 R2

Below you will find the slides from my presentation on “what’s new in WS2012 R2” that I did at the TechCamp 2013 community launch last week.  I focused on the OS rather than the big picture; Dave Northey (Microsoft) did the “cloud OS, etc” keynote before my session, and Damian Flynn (System Center MVP) did the “hybrid cloud & System Center” presentation later in the day.

Topics I covered:

  • Highlights of the Server perspective of the Microsoft BYOD solution
  • Networking
  • Storage
  • Virtualization
  • Cloud

A Very Important Article About Health Of Virtual DCs On Hyper-V

I strongly urge you (in other words, do it or else) to head over to Hyper-V.nu to read an article that Hans Vredevoort (Virtual Machine MVP) wrote on the July 2013 Update Rollup and what it does to prevent total corruption of Active Directory domain controllers that are running as virtual machines on Hyper-V.

Nuff said … go read it.

KB2855336 – WS2012 Update Rollup For July 2013

A number of people have reported here and on the TechNet forum that there is a problem with this update.  It appears to cause bluescreens of death on hosts that have NIC teams supporting VLAN tagging.  Please do not approve/install this update until further notice.

EDIT:

An alleged fix was included with an updated version of the July 2013 update rollup.

Microsoft has released another update rollup for July 2013 for Windows Server 2012.  There are a number of fixes included that are relevant to Hyper-V.  This update rollup is available through Windows Update: make sure you have WS2012 and Update Rollups enabled and deployed, and please use Cluster Aware Updating to update your clusters!

KB2847176: File system is recognized incorrectly after you extend a CSVFS volume on a Windows Server 2012-based cluster.

Assume that you run the Diskpart.exe command prompt utility to extend a Cluster Shared Volumes File System (CSVFS) volume on a Windows Server 2012-based cluster. The volume extends beyond the size limit of the file system. In this scenario, Disk Manager and Diskpart.exe recognize the file system as a RAW file system instead of as a CSVFS file system.

For example, assume that a file system that was formatted by using an 8 KB cluster size has a size limit of 32 TB. When you extend a CSVFS volume beyond 32 TB, Disk Manager and Diskpart.exe recognize the file system as a RAW file system.

KB2854362: Vmms.exe processes freeze on certain nodes in a Windows Server 2012-based Hyper-V failover cluster.

Assume that you deploy a Windows Server 2012-based Hyper-V failover cluster that is based on a scale-out file server cluster. When a failover starts, the Vmms.exe process freezes on some of the failover cluster nodes. Therefore, several virtual machines are in the offline state.

KB2845653: “0x0000000A” Stop error when you perform a live migration of a virtual machine on a Windows Server 2012-based cluster

Assume that you have a Windows Server 2012-based multiple-node cluster. You try to perform a live migration of a virtual machine that has a duplicated (cloned) MAC address. In this situation, the host computer crashes. Additionally, you receive the following Stop error message:

STOP: 0x0000000A (parameter1, parameter2, parameter3, parameter4)

Notes

  • This Stop error describes an IRQL_NOT_LESS_OR_EQUAL issue.
  • The parameters in this Stop error message vary, depending on the configuration of the computer.
  • Not all “Stop 0x0000000A” errors are caused by this issue

KB727972: Cluster node freezes when you restart a computer in a Windows Server 2008 R2 or Windows Server 2012 environment.

Consider the following scenario:

  • You deploy a failover cluster in a Windows Server 2008 R2 or Windows Server 2012 environment.
  • You enable the Volume Shadow Copy Service on a cluster disk.
  • You replace a host bus adapter (HBA) on a cluster node.
  • You restart the cluster node.

In this scenario, the cluster node freezes.

KB853952: Active Directory database becomes corrupted when a Windows Server 2012-based Hyper-V host server crashes.

Assume that you have a Windows Server 2012-based virtualized domain controller on a Windows Server 2012-based Hyper-V host server. When the Hyper-V host server crashes or encounters a power outage, the Active Directory database may become corrupted.

This issue occurs because the guest system requests the Hyper-V server to turn off disk caching on a disk. However, the Hyper-V server misinterprets the request and keeps disk caching enabled.

Exploring Windows Server 2012 Hyper-V Worker Process Security

In this article I want to talk a little about the security of the Hyper-V worker process in WS2012. This might give you a little more knowledge behind a potential problem that I blogged about before about KB2779204.

What is the Worker Process?

The virtual machine worker process reside in user mode (as opposed to kernel mode) in the management OS (also referred to incorrectly as the host OS, running in the root partition you can see in this diagram). There is one VMWP.EXE for every running virtual machine. It’s a small process but it plays an important role, helping Hyper-V to manage the VM.  It is responsible for coordinating all actions performed on a given virtual machine (start, stop, save, snapshot, Live Migration, etc) and is also where any device emulation happens (accessing the legacy network adapter, for instance).

The Security Changes

Let’s define something first. A VM breakout attack is where a hacker gets into the app/OS of a VM and then tries to break out from that security boundary to get onto the host and/or other VMs. This has not happened to Hyper-V but it has happened to certain other hypervisors but Microsoft wants to take no chances.

In Windows Server 2012, each worker process runs under a dedicated user account. There’s a very good preventative security reason for this. . By running the VMWP.EXE under a single restricted user account that has no rights over another other VM or to anything in the management OS (or host). A potential breakout to the VMWP.EXE would be limited to affecting just the compromised virtual machine’s files. It has no rights over anything else and therefore it can do no more damage.

In the following screenshot I’ve used SysInternals (free Microsoft tools) Process Explorer to view the properties of an instance of VMWP.EXE. Note the user account is called NT VIRTUAL MACHINE<some random thing>. You’ll also note Data Execution Prevention (DEP – a BIOS requirement for Hyper-V) is enabled and Address Space Load Randomization is set to High Entropy (to randomize memory against buffer overrun attacks).

clip_image001

The user account is created for you. There is no user or password management for you. This user is automatically made a member of a special system and hidden group called NT VIRTUAL MACHINEVirtual Machines. In local group policy (GPEDIT.MSC) on the Hyper-V host, you can see that this group has been granted a special right. Computer Configuration > Windows Settings > Security Settings > Local Policies > User Rights Assignment > Log On As A Service is configured as follows:

clip_image002

This permission allows the dedicated user account for each VMWP.EXE to log onto the management OS. This means the VMWP.EXE can start, and the virtual machine can run on this host.

The Gotcha!

Some security officers might want to customize this GPO at the local/domain level to be very restrictive. Maybe they only allow certain groups of managed service accounts to log on as a service. That could cause a problem. Imagine they do implement this restrictive GPO. That would result in each host’s NT VIRTUAL MACHINEVirtual Machines group being evicted from this right. And this could lead to the aforementioned issues in KB2779204.

“Starting or Live Migrating Hyper-V virtual machines may fail with error 0x80070569 on Windows Server 2012-based computers”

By design, as the KB article notes, Hyper-V should detect a GPO refresh every time it happens. This is normally every 90 minutes (with a random offset of 0 – 30 minutes) or whenever you purposely run GPUPDATE.EXE. When the refresh is detected then Hyper-V will repopulate the Log On As A Service right with the Virtual Machines group. That seems to work just fine for most people. But on occasion, there can be a problem, as the KB article states.

Sometimes that problem is a once-off glitch. If so, you can fix the issue by running GPUPDATE.EXE in the management OS of the affected host. Your VMs should start up OK to live migrate to this host with no issues now.

Sometimes the problem happens frequently. If that’s the case, then create an OU for the hosts with a custom GPO. I have said it before, and I’ll say it again: This should be normal practice. Your management OS’s are not like normal servers. Have a custom GPO for your hosts assigned to this Hyper-V hosts OU. It will be configured with special settings just for your hosts (restricted admin rights, AV scanning policies, etc) … including giving NT Virtual MachineVirtual Machines the Log On As A Service right. One GPO refresh later and you’re sorted.

Thanks to Ben Armstrong (Hyper-V Senior Program Manager Lead at Microsoft aka @VirtualPCGuy) for fact-checking this article for me. Admission: I did edit afterwards so mistakes are mine!

Windows 8 Deployment Resources

These resources apply to Windows 8 but you could also use them in a Windows Server 2012 deployment or cloud.

Windows XP Support ends on 8th April 2014 AND THERE WILL BE NO EXTENSIONS no matter what the penguin-hugging activation-fearing “genius” you know at the bar says.  This means no more security patches or hotfixes for XP after that date, and also means that the support statements of 3rd parties will become meaningless.  It’s time to start planning an upgrade or become a breeding ground for malware.

Microsoft Assessment and Planning (MAP) Toolkit

A free tool that is an important first step in discovering what it is that you have on the network.  Honestly, it’s a good tool.  But, if your XP network is as shagged as many I’ve encountered, then remote admin will be broken on half the PCs and MAP won’t work.  I have found that the push capability of the System Center Configuration Manager agent will get you the same information via hardware audits and Asset Intelligence.

Windows Assessment and Deployment Kit (Windows ADK)

This is a single free download that contains most of the tools you might use to plan, facilitate, and implement a Windows 8 deployment:

  • Application Compatibility Toolkit (ACT): The Application Compatibility Toolkit (ACT) helps IT Professionals understand potential application compatibility issues by identifying which applications are or are not compatible with the new versions of the Windows operating system. ACT helps to lower costs for application compatibility evaluation by providing an accurate inventory of the applications in your organization. ACT helps you to deploy Windows more quickly by helping to prioritize, test, and detect compatibility issues with your apps. By using ACT, you can become involved in the ACT Community and share your risk assessment with other ACT users. You can also test your web applications and web sites for compatibility with new releases of Internet Explorer.
    Deployment Tools: Deployment tools enable you to customize, manage, and deploy Windows images. Deployment tools can be used to automate Windows deployments, removing the need for user interaction during Windows setup. Tools included with this feature are Deployment Imaging Servicing and Management (DISM) command line tool, DISM PowerShell cmdlets, DISM API, Windows System Image Manager (Windows SIM), and OSCDIMG.
    User State Migration Tool (USMT): USMT is a scriptable command line tool that IT Professionals can use to migrate user data from a previous Windows installation to a new Windows installation. By using USMT, you can create a customized migration framework that copies the user data you select and excludes any data that does not need to be migrated. Tools included with the feature are ScanState, Loadstate, and USMTUtils command line tools.
  • Volume Activation Management Tool (VAMT): The Volume Activation Management Tool (VAMT) enables IT professionals to automate and centrally manage the activation of Windows, Windows Server, Windows ThinPC, Windows POSReady 7, select add-on product keys, and Office for computers in their organization. VAMT can manage volume activation using retail keys (or single activation keys), multiple activation keys (MAKs), or Windows Key Management Service (KMS) keys.
    Windows Performance Toolkit (WPT): Windows Performance Toolkit includes tools to record system events and analyze performance data in a graphical user interface. Tools available in this toolkit include Windows Performance Recorder, Windows Performance Analyzer, and Xperf.
    Windows Assessment Toolkit: Tools to discover and run assessments on a single computer. Assessments are tasks that simulate user activity and examine the state of the computer. Assessments produce metrics for various aspects of the system, and provide recommendations for making improvements.
    Windows Assessment Services: Tools to remotely manage settings, computers, images, and assessments in a lab environment where Windows Assessment Services is installed. This application can run on any computer with access to the server that is running Windows Assessment Services.
    Windows Preinstallation Environment (Windows PE): Minimal operating system designed to prepare a computer for installation and servicing of Windows.

Microsoft Deployment Toolkit (MDT)

MDT is a free download.  Think Ghost .. but with MUCH more functionality, e.g. capture user settings & files, blast the machine with a new image, install some software, patch it, and restore the user settings & files … all while you drink your coffee.  MDT is so good that it’s considered an essential extension to Microsoft’s corporate solution: System Center 2012 Configuration Manager SP1.

Free: who can argue with free!?!?!  Who needs Ghost!?!?!

Deploying Windows 8 with System Center 2012 Configuration Manager Service Pack 1

ConfigMgr + MDT gives you the best OS image deployment solution available.  It simply blows people away when they see it in action.

BTW, Microsoft has the driver thing sorted so you can do the single-image deployment that a sector-based image, such as Ghost, hasn’t a hope of matching.

Deliver and Deploy Windows 8

This is an aggregation of content from all across Microsoft.

Johan Arwidmark

The Deployment God of the North is a must-see if he’s presenting at an event near you.  This guy has forgotten more deployment wizardry than we mere mortals will ever learn.  He’s also the author of Deployment Fundamentals, Vol. 4 “Deploying Windows 8 and Office 2013 Using MDT 2012 Update 1” available on:

In fact, for just $9.99, I thought it was such good value that I’ve just bought the e-book myself Smile

Group Policy: Fundamentals, Security, and the Managed Desktop

You’ll want to manage and control those new deployments using GPO.  Jeremy Moskowitz writes the book on the subject:

That should be enough to get you started!

Server Posterpedia –Windows Server Poster App

A new app that features the feature poster apps for a number of server products, not just Hyper-V, has been released. You can download this app from the Microsoft Store for Windows 8.

image

Click on a poster, and it’s displayed for you:

image

You can zoom and scroll through the poster. Cleverly, the actions that you can run from the app will link you to additional information on TechNet. And there is even a link to download the original poster.  What a handy way to start learning the features of server products.  This is worth installing Windows 8 for!

Ben Armstrong posted about the app overnight, including a video of the app in action.

image

New AD Replication Status Tool

Microsoft has released a new Active Directory replication diagnostics tool called ADREPLSTATUS.  Features include:

  • Auto-discovery of the DCs and domains in the Active Directory forest to which the ADREPLSTATUS computer is joined
  • “Errors only” mode allows administrators to focus only on DCs reporting replication failures
  • Upon detection of replication errors, ADREPLSTATUS uses its tight integration with resolution content on Microsoft TechNet to display the resolution steps for the top AD Replication errors
  • Rich sorting and grouping of result output by clicking on any single column header (sort) or by dragging one or more column headers to the filter bar. Use one or both options to arrange output by last replication error, last replication success date, source DC naming context and last replication success date, etc.)
  • The ability to export replication status data so that it can be imported and viewed by source domain admins, destination domain admins or support professionals using either Microsoft Excel or ADREPLSTATUS
  • The ability to choose which columns you want displayed and their display order. Both settings are saved as a preference on the ADREPLSTATUS computer
  • Broad OS version support (Windows XP -> Windows Server 2012 Preview)

Check out the original blog post by Microsoft to learn much more.

Broken AD replication has proven to be a bit of a curse in the past. I’m amazed at how many sites (not small ones either) don’t monitor this stuff, relying on cheapware ping-based monitoring rather than the application-layer monitoring of something like System Center 2012 – Operations Manager.  They end up with fragmented AD, all sorts of weird crap happening, etc.  If you’re a consultant in a site and you’re deploying/configuring something with a reliance on AD, then here’s a handy warning sign: the customer “approves” security updates manually, and the last update to their PCs/Servers was the most recent Service Pack for the OS (usually for Windows XP).  Take a little time and check the AD replication status before you proceed Smile

Note that this new tool does not support Windows Server 2000 – that’s long since left extended support.

Virtual Machine Generation ID Document

Microsoft has just released a document on VMGeneration-ID (aka Generation ID):

This paper provides information about virtual machine generation ID capability for Windows Server 2012 and Windows 8. It also provides guidelines for developers to implement this capability in virtualization platforms