In this article I want to talk a little about the security of the Hyper-V worker process in WS2012. This might give you a little more knowledge behind a potential problem that I blogged about before about KB2779204.
What is the Worker Process?
The virtual machine worker process reside in user mode (as opposed to kernel mode) in the management OS (also referred to incorrectly as the host OS, running in the root partition you can see in this diagram). There is one VMWP.EXE for every running virtual machine. It’s a small process but it plays an important role, helping Hyper-V to manage the VM. It is responsible for coordinating all actions performed on a given virtual machine (start, stop, save, snapshot, Live Migration, etc) and is also where any device emulation happens (accessing the legacy network adapter, for instance).
The Security Changes
Let’s define something first. A VM breakout attack is where a hacker gets into the app/OS of a VM and then tries to break out from that security boundary to get onto the host and/or other VMs. This has not happened to Hyper-V but it has happened to certain other hypervisors but Microsoft wants to take no chances.
In Windows Server 2012, each worker process runs under a dedicated user account. There’s a very good preventative security reason for this. . By running the VMWP.EXE under a single restricted user account that has no rights over another other VM or to anything in the management OS (or host). A potential breakout to the VMWP.EXE would be limited to affecting just the compromised virtual machine’s files. It has no rights over anything else and therefore it can do no more damage.
In the following screenshot I’ve used SysInternals (free Microsoft tools) Process Explorer to view the properties of an instance of VMWP.EXE. Note the user account is called NT VIRTUAL MACHINE<some random thing>. You’ll also note Data Execution Prevention (DEP – a BIOS requirement for Hyper-V) is enabled and Address Space Load Randomization is set to High Entropy (to randomize memory against buffer overrun attacks).
The user account is created for you. There is no user or password management for you. This user is automatically made a member of a special system and hidden group called NT VIRTUAL MACHINEVirtual Machines. In local group policy (GPEDIT.MSC) on the Hyper-V host, you can see that this group has been granted a special right. Computer Configuration > Windows Settings > Security Settings > Local Policies > User Rights Assignment > Log On As A Service is configured as follows:
This permission allows the dedicated user account for each VMWP.EXE to log onto the management OS. This means the VMWP.EXE can start, and the virtual machine can run on this host.
The Gotcha!
Some security officers might want to customize this GPO at the local/domain level to be very restrictive. Maybe they only allow certain groups of managed service accounts to log on as a service. That could cause a problem. Imagine they do implement this restrictive GPO. That would result in each host’s NT VIRTUAL MACHINEVirtual Machines group being evicted from this right. And this could lead to the aforementioned issues in KB2779204.
“Starting or Live Migrating Hyper-V virtual machines may fail with error 0x80070569 on Windows Server 2012-based computers”
By design, as the KB article notes, Hyper-V should detect a GPO refresh every time it happens. This is normally every 90 minutes (with a random offset of 0 – 30 minutes) or whenever you purposely run GPUPDATE.EXE. When the refresh is detected then Hyper-V will repopulate the Log On As A Service right with the Virtual Machines group. That seems to work just fine for most people. But on occasion, there can be a problem, as the KB article states.
Sometimes that problem is a once-off glitch. If so, you can fix the issue by running GPUPDATE.EXE in the management OS of the affected host. Your VMs should start up OK to live migrate to this host with no issues now.
Sometimes the problem happens frequently. If that’s the case, then create an OU for the hosts with a custom GPO. I have said it before, and I’ll say it again: This should be normal practice. Your management OS’s are not like normal servers. Have a custom GPO for your hosts assigned to this Hyper-V hosts OU. It will be configured with special settings just for your hosts (restricted admin rights, AV scanning policies, etc) … including giving NT Virtual MachineVirtual Machines the Log On As A Service right. One GPO refresh later and you’re sorted.
Thanks to Ben Armstrong (Hyper-V Senior Program Manager Lead at Microsoft aka @VirtualPCGuy) for fact-checking this article for me. Admission: I did edit afterwards so mistakes are mine!
Ooooooo-k then! No need for defensive comments on that topic